ai — Engineering Performance

feat(harness-acp): introduce ACP harness adapter as a meta adapter to connect to any ACP compatible harness (#18483) ## Background AI SDK harness integrations currently require a dedicated adapter for each coding-agent runtime. ACP provides a common protocol that can support multiple runtimes through one adapter while keeping runtime-specific installation and authentication details in small profiles. Some harnesses only offer an ACP adapter as the sole way for programmatic control over their harness, so for those harnesses having an ACP adapter unblocks supporting them by our harness layer. ## Summary This adds `harness-acp`, a meta adapter for ACP v1 implementations. The adapter owns the generic sandbox bridge, protocol translation, host-tool relay, approvals, and lifecycle behavior, while callers configure the underlying ACP runtime declaratively. - Support simple and lockfile-pinned NPM acquisition for arbitrary ACP implementations. - Support direct and AI Gateway authentication with runtime-resolved environment variables and adapter-specific Gateway routes. - Translate ACP streams, native tools, host tools, approvals, skills, session modes, and resumable lifecycle state into Harness APIs. - Add complete Claude Code, Codex, and Grok Build profiles across documentation and interactive examples. ## End-to-End Verification - Exercised the added Claude Code ACP, Codex ACP, and Grok Build ACP through the interactive Harness example. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Related Issues See #16956.

feat: add first-party Code Mode package (#18091) ## Background works towards https://github.com/vercel/ai/issues/18071 the idea is to unify the different provider implementations of programmatic tool calling / code-mode. ## Summary add a new `@ai-sdk/code-mode` package that will export the necessary controls for executing the tools in that format ## End-to-End Verification - ran the example `examples/ai-functions/src/generate-text/code-mode/code-mode.ts` - ran the example `examples/ai-functions/src/stream-text/code-mode/code-mode.ts` - ran the example `http://localhost:3000/chat/code-mode` ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work - api syntax will change - docs need to be added - tool approval mechanism needs to work - workflow boundary concerns ## Related Issues towards #18071 fixes #18141

feat: Realtime API support for browser<->provider websocket connection (#13893) ## Background Part of https://github.com/vercel/ai/issues/13897 Support for Realtime API has been requested for a long time. We want to support different architectures 1. Websocket connection from browser directly to provider 2. Websocket connection from browser through user's server to provider 3. Websocket connection from browser through gateway to provider 4. Websocket connection from browser through user's server and gateway to provider This pull requests implements the first, see [Architecture](#architecture) below ## Summary Alternative implementation of the Realtime API support proposed in #13889, with a reworked developer-facing API: - **`openai.experimental_realtime('gpt-realtime')`** works in both server and browser (no separate `@ai-sdk/openai/realtime` import needed) - **`openai.experimental_realtime.getToken()`** static method for server-side ephemeral token creation - **`experimental_useRealtime`** hook returns `messages: UIMessage[]` (aligned with `useChat` format) instead of `transcript` - **`inputAudioTranscription`** session config enables rendered user messages for transcribed microphone input - **`addToolOutput(callId, result)`** for client-side tools that need manual result submission - **`onToolCall`** callback for auto-executed client-side tools - Framework-agnostic helper type: `Experimental_RealtimeSetupResponse` Provider implementations included: **OpenAI**, **Google**, and **xAI**. ElevenLabs support is split into a stacked follow-up: **#15747**. <a name="architecture"></a> ## Architecture ```mermaid sequenceDiagram participant Browser participant Server participant Provider as AI Provider<br/>(OpenAI / xAI / Gemini / ...) Browser->>Server: POST /api/setup Server->>Provider: Request token (with tool definitions) Provider-->>Server: Short-lived auth token Server-->>Browser: { token } Browser->>Provider: Open WebSocket (using token) Provider-->>Browser: Audio/text chunks (streaming) Provider-->>Browser: Tool call request Browser->>Server: App-specific tool request (optional) Server-->>Browser: App-specific tool result Browser->>Provider: Send tool result via WebSocket Provider-->>Browser: Continue streaming ``` ### Alternatives considered #### Generic RPC route for server-side tools One option was for the SDK to expose a generic server-side tool execution flow: the realtime session would receive a tool call, POST `{ name, inputs, callId }` to an `execute-tools` route, execute a matching server-side tool, and send the result back to the provider automatically. We decided against that for the initial implementation. A generic RPC route is convenient, but it creates a security-sensitive application boundary: the app must authenticate the user, bind the request to a realtime session, allowlist tool names, validate call IDs, rate limit access, and authorize each tool invocation. If the SDK provides the generic route shape, it is easy to copy into production without those controls. Instead, the SDK keeps tool execution client-driven through `onToolCall` and `addToolOutput`. Server-backed tools should call app-specific API endpoints from `onToolCall` (for example `/api/weather`), where the application can apply its normal auth, validation, authorization, and rate limiting rules. Documentation should cover secure server-backed tool calling patterns, but the SDK should not own the generic remote tool RPC abstraction yet. ## Manual verification The realtime voice example added in this PR lives at `examples/ai-e2e-next/app/realtime/page.tsx` (UI) and `examples/ai-e2e-next/app/api/realtime/[...path]/route.ts` (server-side token endpoint). 1. Add the provider API key(s) for whichever provider(s) you want to test to `examples/ai-e2e-next/.env.local`: ```bash OPENAI_API_KEY=... # OpenAI realtime (gpt-realtime) GOOGLE_GENERATIVE_AI_API_KEY=... # Google realtime (gemini-3.1-flash-live-preview) XAI_API_KEY=... # xAI realtime (grok-voice-latest) ``` 2. Start the example: ```bash cd examples/ai-e2e-next pnpm dev ``` 3. Open http://localhost:3000/realtime, pick a provider + voice, and click **Connect**. Use the microphone or type a message. Ask "what's the weather in Paris?" or "roll a dice" to exercise client-side tool calling (`onToolCall`). ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [ ] I have reviewed this pull request (self-review) - [ ] Security Audit - esp. securing remote tool calling - [ ] Update internal Architecture / how it works docs ### Future Work 1. **ElevenLabs** provider support — stacked follow-up in #15747 2. Websocket connection from browser through user's server to provider 3. Websocket connection from browser through gateway to provider 4. Websocket connection from browser through user's server and gateway to provider ## Related issues - #3176 - #3907 - #4082 - #5007 - #9559 - #12381 - #13706 - #13889 Co-authored-by: Cursor <cursoragent@cursor.com>

feat: agent tui (#15845) ## Background To quickly develop and test agents, it is helpful to have a TUI that requires very limited setup. ## Summary - integrate https://github.com/lgrammel/agent-tui as `@ai-sdk/tui` - add documentation ## Example ```ts await runAgentTUI({ agent }); ``` ## Manual Verification - [x] `pnpm tsx src/agent/openai/tui`

feat(video): externalize polling control and webhook support for generateVideo (#12515) - Adds optional `doStart`/`doStatus` methods to the experimental `VideoModelV4` spec, enabling a two-phase async model where the SDK core orchestrates polling or webhook-based completion instead of each provider implementing its own polling loop. - Adds `poll` and `webhook` parameters to `experimental_generateVideo` for user-controlled polling intervals/timeouts and webhook-based completion. - Adds optional `handleWebhookOption` to `VideoModelV4` so a provider can signal native webhook support. The SDK only invokes the user's webhook factory when the model implements it; otherwise it falls back to polling. - Replaces `doGenerate` with `doStart`/`doStatus` in the first-party models implementing `VideoModelV4` (FAL, Alibaba, KlingAI, Replicate, xAI, Google, and Google Vertex). The gateway video model remains a `doGenerate` passthrough. - Makes `doGenerate` optional on `VideoModelV4`. Third-party providers implementing only `doGenerate` continue to work unchanged, and the SDK falls back to it when no async flow is selected. - Constrains operation handles to `JSONValue`, making the provider-owned task reference serializable and suitable for future persistence and gateway support. Towards https://github.com/vercel/ai/issues/12381. ## Architecture ``` User calls: generateVideo({ model, prompt, poll?, webhook? }) ↓ Core function: Chooses flow based on model capabilities and user options ↓ ┌───────────┴───────────────┐ │ │ [Legacy / fallback] [Start/Status flow] │ (poll or webhook provided, │ or doGenerate missing) │ │ model.doGenerate(options) model.doStart(options) ↓ ↓ Provider owns polling returns { operation } ↓ ┌───────────┴────────────┐ │ │ [native webhook] [polling] │ │ await notification SDK core loop │ │ model.doStatus(op) model.doStatus(op) └───────────┬────────────┘ ↓ return result ``` The SDK core owns the polling lifecycle. Providers only implement single-shot `doStart` (submit) and `doStatus` (check once). The polling interval and timeout are controlled by the user. ## Review decisions - **`poll` and `webhook` are intentionally composable.** When a model supports native webhooks, `poll.timeoutMs` limits the webhook wait. When it does not, the same polling configuration controls the automatic fallback. This behavior is documented and covered by tests. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2821429102)) - **The first iteration uses fixed polling intervals and timeouts.** Exponential backoff and `onAttempt` were removed to keep the API small; aligning polling with shared retry/backoff primitives can be revisited separately. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2821435402)) - **Operation handles are JSON-serializable.** `operation` is typed as `JSONValue` instead of `unknown`, while remaining opaque to core. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843221880)) - **Shared cross-modality operation types are deferred.** The current types remain video-specific until another modality implements the same lifecycle, avoiding a premature provider-spec abstraction. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843151286)) - **Replicate's server-side `wait` remains provider-specific.** It affects how long Replicate holds the submission request open, while top-level `poll` controls the SDK's provider-neutral status lifecycle. It can be added through Replicate provider options independently if needed. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843203139)) ## Verification - [x] Tests added and updated - [x] Documentation added and updated - [x] Patch changeset added - [x] TypeScript, lint/format, and code consistency checks pass locally ## Future work - Expose async operations as a first-class API: start an operation, persist its handle, and resume or query it later. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843215786)) - Generalize operation types after another model modality implements the same lifecycle. - Add AI Gateway transport for the JSON-serializable start/status operation flow. - Explore a multi-notification webhook receiver if providers need intermediate status events rather than a single completion notification. ## Related issue https://github.com/vercel/ai/issues/12381 --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: josh <144584931+dancer@users.noreply.github.com> Co-authored-by: vercel-ai-sdk[bot] <225926702+vercel-ai-sdk[bot]@users.noreply.github.com> Co-authored-by: mat lenhard <mclenhard@gmail.com> Co-authored-by: Rohan Taneja <47066511+R-Taneja@users.noreply.github.com> Co-authored-by: Felix Arntz <felix.arntz@vercel.com> Co-authored-by: Walter Korman <shaper@vercel.com> Co-authored-by: Nico Albanese <49612682+nicoalbanese@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

feat: add provider versions to user-agent header (#8703) ## Background This change is added to include the provider versions in the user-agent header that we now pass. Previously, the string only appended `ai` version and runtime env ## Summary - add a `version.ts` file - export version during build by changing `tsup.config` - export verison during tests by changing `vitest.config` - mock a dummy version for tests ## Manual Verification Verified by making changes to the test file ## Tasks - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Future Work Version in user-agent added for: - [x] `gateway` - [x] `openai` - [x] `anthropic` - [x] `google` - [x] `google-vertex` - [x] `azure` - [x] `amazon-bedrock` - [x] `cohere` - [x] `mistral` - [x] `groq` - [x] `cerebras` - [x] `deepinfra` - [x] `deepseek` - [x] `fireworks` - [x] `perplexity` - [x] `replicate` - [x] `togetherai` - [x] `xai` - [x] `vercel` - [x] `openai-compatible` - [x] `elevenlabs` - [x] `assemblyai` - [x] `deepgram` - [x] `gladia` - [x] `revai` - [x] `luma` - [x] `fal` - [x] `hume` - [x] `lmnt` - [x] `langchain` - [x] `llamaindex` - [x] `valibot` ## Related Issues fix in progress for #8699 --------- Co-authored-by: Gregor Martynus <39992+gr2m@users.noreply.github.com>

more v3 -> v4 updates (#13602) - Update remaining 24 test files in `packages/ai/src` to use V4 mock models and interfaces instead of V3 counterparts - Continuation of #13567

feat(provider): add support for provider references and uploading files as supported per provider (#13816) ## Background The AI SDK supports passing media files inline or via URL, but has no way to upload files directly to a provider or reference previously uploaded files across providers. Some providers return internal file IDs (not URLs) from their upload APIs, and switching providers mid-conversation requires a way to map the same logical file to different provider-specific identifiers. ## Summary Introduces `uploadFile` as a top-level function and `ProviderReference` (`Record<string, string>`) as the provider-independent way to reference uploaded files. - **New spec types**: `SharedV4ProviderReference` (provider package), `FilesV4` interface with `uploadFile` method, `UploadFileResult`; `mediaType` and `filename`** are top-level parameters as they're widely supported and used - **New top-level API**: `uploadFile({ files, data, mediaType?, filename?, providerOptions? })` in the `ai` package, with auto-detection of media type from file bytes when not provided - **Provider implementations**: `files()` interface on Anthropic, Google, OpenAI, and xAI providers, each implementing `FilesV4.uploadFile` - Other providers don't support uploading files, or they only support uploading files for batch inference (`*.jsonl`), which we don't support anyway. - **Provider reference support in messages**: `LanguageModelV4FilePart.data` now accepts `SharedV4ProviderReference` in addition to `DataContent`; providers that support file references (Anthropic, Google, OpenAI, xAI) resolve them via `resolveProviderReference`; all other providers throw `UnsupportedFunctionalityError` - **Spec cleanup**: `file-id` and `image-file-id` tool result output types replaced with `file-reference` and `image-file-reference` using `SharedV4ProviderReference` instead of `string | Record<string, string>` - **Uploading from URL is not supported** — no provider supports this, and auto-downloading is questionable; callers should fetch first - **`reasoning-file` was not touched** — it is model-generated as part of reasoning output, so provider references are not applicable - **Docs included** — Docs about `uploadFile` and `ProviderReference`, and a new architecture guide are included ### Design decisions - `ProviderReference` is a plain `Record<string, string>` rather than a wrapper class, keeping it simple to create and merge - The `isLikelyText` heuristic for media type detection and the `documentMediaTypeSignatures` are kept internal (not exported) — they work well enough for `uploadFile` but are not general-purpose utilities - `resolveProviderReference` (provider-utils) does the lookup by provider name and throws with a clear error listing available providers when the reference doesn't contain an entry for the current provider ### Open questions 1. Should we include a `type` property in `ProviderReference` to distinguish different kinds of provider references (e.g. file vs skill, see #12855)? 2. Should `mediaType` and `filename` be top-level fields in the `uploadFile` result object? - They're currently top-level request parameters, but in the response they're in `providerMetadata`. 3. `mergeProviderReferences` is currently inlined in an example (`multi-provider.ts`) — should we offer this as a utility, or leave it for later? 4. `file-id` and `image-file-id` were removed in `toModelOutput` return value — should we deprecate them instead and/or offer auto-migration via codemod? 5. Out of scope: supporting providers that allow uploading files solely for batch inference (e.g. Cohere, Groq, Mistral), which we don't support at a provider level yet anyway - probably leave for later? ## Manual Verification Upload file examples were added for all 4 supported providers (Anthropic, Google, OpenAI, xAI), each with image, PDF, and text variants. ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work Reuse the new `ProviderReference` approach for #12855. ## Related Issues Fixes #12995

feat: Add codemod suite for AI SDK v5 migration (#7264) ## background AI SDK v5 introduces breaking changes across providers, streaming APIs, message types, and package structure. Users need automated migration tools to upgrade large codebases efficiently without manual refactoring. The existing codemod system needed better organization and version-specific commands. ## summary - reorganize 62 codemods into v4/ and v5/ directories for better structure - add version-specific CLI commands: `v4` and `v5` alongside existing `upgrade` - update documentation with proper v4/ and v5/ prefixes in migration guides - fix test fixtures and ensure all tests pass - auto-generate README with version-categorized codemod listings ## tasks - [x] reorganize codemods into v4/ and v5/ subdirectories - [x] update import paths in moved codemod files - [x] add v4 and v5 CLI commands with separate upgrade functions - [x] update README generation script for version categorization - [x] fix migration guide documentation with correct codemod names - [x] update test fixtures and resolve failing tests - [x] ensure all 62 codemods work with new structure ## future work * consider updating codemod categories

feat(harness): support request transformations in network sandbox abstraction and use it to apply credential brokering when available (#18859) ## Background Bridge-based harnesses currently forward provider credentials into sandboxed agent processes, which is far from ideal from a security perspective. Credential brokering allows us to keep those credentials only in the host environment and inject them into the relevant outgoing requests from the sandbox at the host boundary. ## Summary - Add optional `setRequestTransformations()` and `addRequestTransformations()` to the network sandbox abstraction. - Update the Vercel Sandbox implementation to support it, via its `networkPolicy` layer. - Preserve authoritative allow/deny policy and `forwardURL` rules while transformations are added, replaced, deferred, or restored across session resume. - Broker direct-provider and AI Gateway credentials for all bridge based harnesses: Claude Code, Codex, Deep Agents, Grok Build, OpenCode, and ACP-based harnesses. - Warn and retain legacy credential forwarding when a sandbox implementation does not expose additive request transformations. - Add shared credential-brokering utilities, structural conventions, tests, documentation, and an end-to-end Vercel Sandbox example. - For Codex specifically, disable websockets mode conditionally to be able to apply credential brokering. ## End-to-End Verification Ran many static and interactive examples across all bridge harnesses to verify continued correct execution with the new path of brokered credentials applied. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review)