github.com-vercel-ai

feat: Realtime API support for browser<->provider websocket connection (#13893) ## Background Part of https://github.com/vercel/ai/issues/13897 Support for Realtime API has been requested for a long time. We want to support different architectures 1. Websocket connection from browser directly to provider 2. Websocket connection from browser through user's server to provider 3. Websocket connection from browser through gateway to provider 4. Websocket connection from browser through user's server and gateway to provider This pull requests implements the first, see [Architecture](#architecture) below ## Summary Alternative implementation of the Realtime API support proposed in #13889, with a reworked developer-facing API: - **`openai.experimental_realtime('gpt-realtime')`** works in both server and browser (no separate `@ai-sdk/openai/realtime` import needed) - **`openai.experimental_realtime.getToken()`** static method for server-side ephemeral token creation - **`experimental_useRealtime`** hook returns `messages: UIMessage[]` (aligned with `useChat` format) instead of `transcript` - **`inputAudioTranscription`** session config enables rendered user messages for transcribed microphone input - **`addToolOutput(callId, result)`** for client-side tools that need manual result submission - **`onToolCall`** callback for auto-executed client-side tools - Framework-agnostic helper type: `Experimental_RealtimeSetupResponse` Provider implementations included: **OpenAI**, **Google**, and **xAI**. ElevenLabs support is split into a stacked follow-up: **#15747**. <a name="architecture"></a> ## Architecture ```mermaid sequenceDiagram participant Browser participant Server participant Provider as AI Provider<br/>(OpenAI / xAI / Gemini / ...) Browser->>Server: POST /api/setup Server->>Provider: Request token (with tool definitions) Provider-->>Server: Short-lived auth token Server-->>Browser: { token } Browser->>Provider: Open WebSocket (using token) Provider-->>Browser: Audio/text chunks (streaming) Provider-->>Browser: Tool call request Browser->>Server: App-specific tool request (optional) Server-->>Browser: App-specific tool result Browser->>Provider: Send tool result via WebSocket Provider-->>Browser: Continue streaming ``` ### Alternatives considered #### Generic RPC route for server-side tools One option was for the SDK to expose a generic server-side tool execution flow: the realtime session would receive a tool call, POST `{ name, inputs, callId }` to an `execute-tools` route, execute a matching server-side tool, and send the result back to the provider automatically. We decided against that for the initial implementation. A generic RPC route is convenient, but it creates a security-sensitive application boundary: the app must authenticate the user, bind the request to a realtime session, allowlist tool names, validate call IDs, rate limit access, and authorize each tool invocation. If the SDK provides the generic route shape, it is easy to copy into production without those controls. Instead, the SDK keeps tool execution client-driven through `onToolCall` and `addToolOutput`. Server-backed tools should call app-specific API endpoints from `onToolCall` (for example `/api/weather`), where the application can apply its normal auth, validation, authorization, and rate limiting rules. Documentation should cover secure server-backed tool calling patterns, but the SDK should not own the generic remote tool RPC abstraction yet. ## Manual verification The realtime voice example added in this PR lives at `examples/ai-e2e-next/app/realtime/page.tsx` (UI) and `examples/ai-e2e-next/app/api/realtime/[...path]/route.ts` (server-side token endpoint). 1. Add the provider API key(s) for whichever provider(s) you want to test to `examples/ai-e2e-next/.env.local`: ```bash OPENAI_API_KEY=... # OpenAI realtime (gpt-realtime) GOOGLE_GENERATIVE_AI_API_KEY=... # Google realtime (gemini-3.1-flash-live-preview) XAI_API_KEY=... # xAI realtime (grok-voice-latest) ``` 2. Start the example: ```bash cd examples/ai-e2e-next pnpm dev ``` 3. Open http://localhost:3000/realtime, pick a provider + voice, and click **Connect**. Use the microphone or type a message. Ask "what's the weather in Paris?" or "roll a dice" to exercise client-side tool calling (`onToolCall`). ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [ ] I have reviewed this pull request (self-review) - [ ] Security Audit - esp. securing remote tool calling - [ ] Update internal Architecture / how it works docs ### Future Work 1. **ElevenLabs** provider support — stacked follow-up in #15747 2. Websocket connection from browser through user's server to provider 3. Websocket connection from browser through gateway to provider 4. Websocket connection from browser through user's server and gateway to provider ## Related issues - #3176 - #3907 - #4082 - #5007 - #9559 - #12381 - #13706 - #13889 Co-authored-by: Cursor <cursoragent@cursor.com>

feat: agent tui (#15845) ## Background To quickly develop and test agents, it is helpful to have a TUI that requires very limited setup. ## Summary - integrate https://github.com/lgrammel/agent-tui as `@ai-sdk/tui` - add documentation ## Example ```ts await runAgentTUI({ agent }); ``` ## Manual Verification - [x] `pnpm tsx src/agent/openai/tui`

feat: add provider versions to user-agent header (#8703) ## Background This change is added to include the provider versions in the user-agent header that we now pass. Previously, the string only appended `ai` version and runtime env ## Summary - add a `version.ts` file - export version during build by changing `tsup.config` - export verison during tests by changing `vitest.config` - mock a dummy version for tests ## Manual Verification Verified by making changes to the test file ## Tasks - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Future Work Version in user-agent added for: - [x] `gateway` - [x] `openai` - [x] `anthropic` - [x] `google` - [x] `google-vertex` - [x] `azure` - [x] `amazon-bedrock` - [x] `cohere` - [x] `mistral` - [x] `groq` - [x] `cerebras` - [x] `deepinfra` - [x] `deepseek` - [x] `fireworks` - [x] `perplexity` - [x] `replicate` - [x] `togetherai` - [x] `xai` - [x] `vercel` - [x] `openai-compatible` - [x] `elevenlabs` - [x] `assemblyai` - [x] `deepgram` - [x] `gladia` - [x] `revai` - [x] `luma` - [x] `fal` - [x] `hume` - [x] `lmnt` - [x] `langchain` - [x] `llamaindex` - [x] `valibot` ## Related Issues fix in progress for #8699 --------- Co-authored-by: Gregor Martynus <39992+gr2m@users.noreply.github.com>

more v3 -> v4 updates (#13602) - Update remaining 24 test files in `packages/ai/src` to use V4 mock models and interfaces instead of V3 counterparts - Continuation of #13567

feat(provider): add support for provider references and uploading files as supported per provider (#13816) ## Background The AI SDK supports passing media files inline or via URL, but has no way to upload files directly to a provider or reference previously uploaded files across providers. Some providers return internal file IDs (not URLs) from their upload APIs, and switching providers mid-conversation requires a way to map the same logical file to different provider-specific identifiers. ## Summary Introduces `uploadFile` as a top-level function and `ProviderReference` (`Record<string, string>`) as the provider-independent way to reference uploaded files. - **New spec types**: `SharedV4ProviderReference` (provider package), `FilesV4` interface with `uploadFile` method, `UploadFileResult`; `mediaType` and `filename`** are top-level parameters as they're widely supported and used - **New top-level API**: `uploadFile({ files, data, mediaType?, filename?, providerOptions? })` in the `ai` package, with auto-detection of media type from file bytes when not provided - **Provider implementations**: `files()` interface on Anthropic, Google, OpenAI, and xAI providers, each implementing `FilesV4.uploadFile` - Other providers don't support uploading files, or they only support uploading files for batch inference (`*.jsonl`), which we don't support anyway. - **Provider reference support in messages**: `LanguageModelV4FilePart.data` now accepts `SharedV4ProviderReference` in addition to `DataContent`; providers that support file references (Anthropic, Google, OpenAI, xAI) resolve them via `resolveProviderReference`; all other providers throw `UnsupportedFunctionalityError` - **Spec cleanup**: `file-id` and `image-file-id` tool result output types replaced with `file-reference` and `image-file-reference` using `SharedV4ProviderReference` instead of `string | Record<string, string>` - **Uploading from URL is not supported** — no provider supports this, and auto-downloading is questionable; callers should fetch first - **`reasoning-file` was not touched** — it is model-generated as part of reasoning output, so provider references are not applicable - **Docs included** — Docs about `uploadFile` and `ProviderReference`, and a new architecture guide are included ### Design decisions - `ProviderReference` is a plain `Record<string, string>` rather than a wrapper class, keeping it simple to create and merge - The `isLikelyText` heuristic for media type detection and the `documentMediaTypeSignatures` are kept internal (not exported) — they work well enough for `uploadFile` but are not general-purpose utilities - `resolveProviderReference` (provider-utils) does the lookup by provider name and throws with a clear error listing available providers when the reference doesn't contain an entry for the current provider ### Open questions 1. Should we include a `type` property in `ProviderReference` to distinguish different kinds of provider references (e.g. file vs skill, see #12855)? 2. Should `mediaType` and `filename` be top-level fields in the `uploadFile` result object? - They're currently top-level request parameters, but in the response they're in `providerMetadata`. 3. `mergeProviderReferences` is currently inlined in an example (`multi-provider.ts`) — should we offer this as a utility, or leave it for later? 4. `file-id` and `image-file-id` were removed in `toModelOutput` return value — should we deprecate them instead and/or offer auto-migration via codemod? 5. Out of scope: supporting providers that allow uploading files solely for batch inference (e.g. Cohere, Groq, Mistral), which we don't support at a provider level yet anyway - probably leave for later? ## Manual Verification Upload file examples were added for all 4 supported providers (Anthropic, Google, OpenAI, xAI), each with image, PDF, and text variants. ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work Reuse the new `ProviderReference` approach for #12855. ## Related Issues Fixes #12995

feat: Add codemod suite for AI SDK v5 migration (#7264) ## background AI SDK v5 introduces breaking changes across providers, streaming APIs, message types, and package structure. Users need automated migration tools to upgrade large codebases efficiently without manual refactoring. The existing codemod system needed better organization and version-specific commands. ## summary - reorganize 62 codemods into v4/ and v5/ directories for better structure - add version-specific CLI commands: `v4` and `v5` alongside existing `upgrade` - update documentation with proper v4/ and v5/ prefixes in migration guides - fix test fixtures and ensure all tests pass - auto-generate README with version-categorized codemod listings ## tasks - [x] reorganize codemods into v4/ and v5/ subdirectories - [x] update import paths in moved codemod files - [x] add v4 and v5 CLI commands with separate upgrade functions - [x] update README generation script for version categorization - [x] fix migration guide documentation with correct codemod names - [x] update test fixtures and resolve failing tests - [x] ensure all 62 codemods work with new structure ## future work * consider updating codemod categories

chore(provider-util): integrate zod-to-json-schema (#8224) ## Background AI SDK 5 has an indirect reference to `zod` 3.x through `zod-to-json-schema`. This causes issues with users upgrading to `zod` 4.x (see #7935 ). ## Summary - Integrate `zod-to-json-schema` into the AI SDK - Only support strict JSON Schema 7 output (remove other targets, remove error messages) ## Manual Verification - [x] run zod/v3 structured output example ## Future Work * explore differences between jsonschema7 from zod-to-json-schema and original jsonschema7; standardize on standard jsonschema7 types if possible * extend integrated mapping to zod/v4 ## Related Issues Fixes #7935 --------- Co-authored-by: Gregor Martynus <39992+gr2m@users.noreply.github.com>

v3 -> v4 spec usage for ai@7 beta (#13549) ## Background follow up to #13001 ## Summary I missed some usage of the v3 spec

feat(ai): add OAuth for MCP clients + refactor to new package (#9127) ## Background Refactor mcp into it's own separate package + add OAuth for MCP clients ## Summary Created new package `@ai-sdk/mcp`, added OAuth provider ## Manual Verification Manual tests added, e2e example added to verify oauth works in `example/mcp/mcp-with-auth/client.ts` ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Future Work - Continue refactoring, add docs - log warning or throw error if wrong transport protocol is set for an mcp server ## Related Issues towards #8717, #9337 and #6717 --------- Co-authored-by: Gregor Martynus <39992+gr2m@users.noreply.github.com>

feat(ai): change type of experimental_context from unknown to generic (#13102) ## Background `experimental_context` for tools and step preparation was introduced in an earlier release without typing (type: `unknown`). Before marking context as stable, it should support smart generic typing for usage in tools, prepareStep, and telemetry. ## Summary Introduce generics on `experimental_context`. Each tool has a `contextSchema` for the tool-specific context. The context on the AI function call is a union of the tool contexts and can have additional properties for `prepareStep` and telemetry. ## Limitations The `experimental_context` setting is optional even when a context is needed, i.e. it will not throw type validation errors unless a context is specified. ## Manual Verification - [x] run and check types on `examples/ai-functions/src/generate-text/openai/tool-call-with-context.ts` - [x] run and check types on `examples/ai-functions/src/stream-text/openai/tool-call-with-context.ts` - [x] run and check types on `examples/ai-functions/src/agent/openai/generate-context.ts` - [x] run and check types on `examples/ai-functions/src/agent/openai/generate-context-call-options.ts` ## Future Work * mark `experimental_context` stable * add `sanitizeContext` function * add `tool` context wrapper for name mapping and safety * add `context` to `StopCondition` parameters * move `ToolSet` into `provider-utils` package * investigate how to fix context optionality limitation * remove `ToolCallOptions` type

feat: add experimental devtools package (#11050) ## Summary This PR adds `@ai-sdk/devtools`, a local development tool for debugging and inspecting AI SDK applications. It provides a web-based UI to view LLM requests, responses, tool calls, and multi-step interactions. **Key features:** - Middleware that intercepts `generateText` and `streamText` calls - Captures input parameters, prompts, output content, tool calls, token usage, and timing - Stores data locally in `.devtools/generations.json` - Web UI served at `http://localhost:4983` via `npx @ai-sdk/devtools` - Organizes data by **Runs** (complete multi-step interactions) and **Steps** (individual LLM calls) > **Note**: This package is experimental and intended for local development only. ## Manual Verification 1. Install the package and add the middleware to a model: ```typescript import { wrapLanguageModel } from "ai"; import { devToolsMiddleware } from "@ai-sdk/devtools"; const model = wrapLanguageModel({ middleware: devToolsMiddleware, model: yourModel, }); 2. Run npx @ai-sdk/devtools and open http://localhost:4983 3. Make AI SDK calls and verify they appear in the UI with correct request/response data Checklist - Tests have been added / updated (for bug fixes / features) - Documentation has been added / updated (for bug fixes / features) - A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root) - I have reviewed this pull request (self-review) Future Work - Docs - Move away from middleware --------- Co-authored-by: Gregor Martynus <39992+gr2m@users.noreply.github.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>

fix: revert zod import change (#9382) ## Background Changing `import { z } from 'zod/v4';` to `import * as z from 'zod/v4';` broke AI SDK zod schemas in some environments. ## Summary Revert import change. ## Related - Caused by #9301 - port of #9349 --------- Co-authored-by: Lars Grammel <lars.grammel@gmail.com>

chore (provider): refactor usage (language model v2) (#5653)

feat(ai): rename onStepFinish to onStepEnd (#15849) ## Background as part of our renaming consistency efforts, the `onStepFinish` callback had to be renamed to `onStepEnd` ## Summary `onStepFinish` -> `onStepEnd` ## Manual Verification ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review)

chore(provider,ai): unified Warning / SharedV3Warning system (#10631) ## Background The warnings from different model types are standardized in the v3 specification to enable unified warning logging and simplify the codebase. ## Summary * update `SharedV3Warning` type * use `SharedV3Warning` in v3 model specifications * model-specific warning types have changed to a generic `Warning` type on the ai package (**breaking change**) * move `LogWarningsFunction` type to stable ## Manual Verification - [x] run `examples/ai-core/src/generate-text/openai-warning.ts` with and without custom warning function ## Tasks - [x] update provider package - [x] update usages - [x] fix tests - [x] test functionality - [x] docs - [x] changeset ## Related Issues Closes #9818

feat(ai): experimental callbacks in ToolLoopAgent (#12717) ## Background as a follow up to https://github.com/vercel/ai/pull/12654, we will introduce the same callbacks for the `ToolLoopAgent` ## Summary - added callback `experimental_onStart` that exposes the data/events that happen at the very beginning - added callback `experimental_onStepStart` - added callback `experimental_onToolCallStart` - added callback `experimental_onToolCallFinish` ## Manual Verification ## Future Work - callbacks aren't currently passed through for streaming - will need to merge #12708 first ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review)

feat(devtools): add new devtools integration for telemetry (#14207) ## Background our devtools setup currently uses middleware to extract event info this needed to change since we now expose all the events that will be needed by devtools to initialise the spans replacement for https://github.com/vercel/ai/pull/12868 ## Summary - create new `devToolsIntegration()` that extends on the telemetry integration api - add a trace/span viewer that properly maps each step ## Manual Verification verified by running the example `examples/ai-functions/src/generate-text/anthropic/subagent-with-telemetry.ts` ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [ ] I have reviewed this pull request (self-review)

feat (ai): support model-generated files (#5216)

feat(harness): add sandbox specific expansion for harness abstraction, add `sandbox-just-bash` and `sandbox-vercel` (#15919) ## Background This is the first PR broken out from #15496, the harness abstraction layer. **All code in these new packages is considered experimental.** ## Summary - Implement the harness specific expansion of the sandbox abstraction, called `HarnessV1NetworkSandboxSession`. Its shape is driven by harness needs (see original draft PR). - The type has a `restricted()` method which returns an (effectively lower-capability) instance of `Experimental_SandboxSession`. This instance should be passed to tool calls, _not_ the network session instance. Otherwise tools would be able to perform actions that go widely beyond their authority, e.g. stopping the sandbox. - Eventually, this could live in `provider-utils` or `specification` as an expansion to the existing `Experimental_SandboxSession`, however we leave it in this experimental package, particularly because it's directly related to the harness abstraction. - Implement `HarnessV1SandboxProvider`, a basically stateless container for sandbox definition. It allows to define a specific sandbox definition in the file root, similarly to e.g. `ToolLoopAgent` (or the future `HarnessAgent`). The `createSession()` and `resumeSession()` methods are then used to create _stateful_ sandboxes by actually bootstrapping the resource. - Implement adapters for `just-bash` and Vercel Sandbox. These provide both an implementation of `Experimental_SandboxSession` and `HarnessV1NetworkSandboxSession`, accessible via the `HarnessV1SandboxProvider`. **Note:** Merging this PR will publish the first canary releases of the new packages: - `@ai-sdk/harness` - `@ai-sdk/sandbox-just-bash` - `@ai-sdk/sandbox-vercel` ## Manual Verification - Verification has been happening via the original PR #15496. - The examples in `examples/ai-functions` were updated to use these implementations instead of local ones. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review)

feat (ai): add anthropic reasoning (#4971)