Lars Grammel

feat: agent tui (#15845) ## Background To quickly develop and test agents, it is helpful to have a TUI that requires very limited setup. ## Summary - integrate https://github.com/lgrammel/agent-tui as `@ai-sdk/tui` - add documentation ## Example ```ts await runAgentTUI({ agent }); ``` ## Manual Verification - [x] `pnpm tsx src/agent/openai/tui`

fix: rename onFinish to onEnd (#15245) ## Background We are standardizes on `End` over `Finish`, and results should be accumulative. ## Summary * Renames the final lifecycle callback from onFinish to onEnd for generateText, streamText, Agent, ToolLoopAgent, and WorkflowAgent; onFinish remains as a deprecated alias, with onEnd taking precedence. * Updates final callback result semantics so usage, content, toolCalls, toolResults, files, sources, and warnings aggregate across all steps, while final-step-only data is available through finalStep. * Adds/deprecates the corresponding callback types (GenerateTextOnEndCallback, WorkflowAgentOnEndCallback) and updates telemetry to use the new onEnd event shape. * Updates docs, examples, references, and the AI SDK 7 migration guide to describe the onEnd rename and the new aggregated final event fields.

feat: add performance statistics (#15241) ## Background Statistics such as tokens per ms and time to first token are important to understand the performance of models and providers. In addition to telemetry, they can also be displayed in user interfaces. ## Summary * add `performance` property to `StepResult` with `tokensPerSecond`, `stepTimeMs`, `responseTimeMs`, `toolExecutionMs`, `timeToFirstTokenMs` * rename `durationMs` on tool execution events to `toolExecutionMs` ## Example Output ``` Step performance: [ { stepTimeMs: 2592.4770839999996, responseTimeMs: 2591.246209, tokensPerSecond: 61.36043709306976, toolExecutionMs: { call_zFG9UMazm5kY2r4q4Mn6LUyx: 0.0625 }, timeToFirstTokenMs: 2339.711542 }, { stepTimeMs: 1228.0632500000002, responseTimeMs: 1226.2540409999997, tokensPerSecond: 35.06614336205071, toolExecutionMs: { call_Ben1pOrejarzcE87EtPbXndv: 0.06979200000023411 }, timeToFirstTokenMs: 873.9773329999998 }, { stepTimeMs: 3674.4792079999997, responseTimeMs: 3673.7040419999994, tokensPerSecond: 100.17137902041159, toolExecutionMs: {}, timeToFirstTokenMs: 3110.819042 } ] ``` ## Manual Verification Run and verify: - [x] `examples/ai-functions/src/generate-text/openai/step-performance.ts` - [x] `examples/ai-functions/src/stream-text/openai/step-performance.ts` ## Future Work * add support in `WorkflowAgent` @gr2m

feat: add a per-step first-content timeout for streaming generations (#17561) ## Background Streaming transports can remain active through headers, metadata, keep-alives, empty deltas, or raw bytes without producing model content. Consumers need an SDK-level semantic deadline for the first content-bearing output of every streaming model-call step. ## Summary Adds optional timeout.firstChunkMs configuration and extraction. streamText arms a dedicated timeout for every model-call response stream, merges its abort signal into the provider call, and disarms it before forwarding the first non-empty text, reasoning, or tool-input delta, generated file, reasoning file, or tool call. Non-content activity does not affect firstChunkMs, and chunkMs now starts and resets only on semantic output. Step timers and abort listeners are cleaned up on completion, abort, setup failure, provider stream error, and cancellation, including through stitchable-stream lifecycle callbacks. ## Testing Adds Node and Edge runtime coverage for non-content activity, empty deltas, every qualifying output category, timeout identity, pre-forward disarming, multi-step re-arming, semantic chunkMs behavior, provider errors, cancellation, and timer cleanup. Also adds stitchable-stream lifecycle tests, timeout extraction tests, and compile-time public API coverage. ## End-to-end Validation - Ran the provider-independent firstChunkMs streamText example with delayed initial content; it completed successfully and emitted `The first content arrived before the deadline.` ## Documentation Updates timeout settings, the streamText reference, and ToolLoopAgent documentation with firstChunkMs semantics, qualifying output, ignored non-content activity, and content-based chunkMs behavior. ToolLoopAgent now explicitly identifies firstChunkMs and chunkMs as streaming-only. Adds a runnable mock-provider example and an ai patch changeset. ## Related Issues Fixes #17315 Co-authored-by: lgrammel <205036+lgrammel@users.noreply.github.com>

feat: add a `toolOrder` option to control the order in which tools are sent (#15811) ## Background Provider-side prompt/tool caching can benefit from stable tool definition ordering. This adds an explicit `toolOrder` option so callers can control the order in which tools are sent to provider APIs. ## Summary Adds `toolOrder` support to `generateText`, `streamText`, and `ToolLoopAgent`, including per-step overrides via `prepareStep`. Tools listed in `toolOrder` are sent first in the requested order, while omitted tools are appended alphabetically for stable defaults. Updates type coverage, unit tests, snapshots, OpenTelemetry test fixtures, API reference docs, provider examples, and adds a patch changeset for `ai`. ## Manual Verification - [x] `examples/ai-functions/src/generate-text/openai/tool-order.ts` - [x] `examples/ai-functions/src/generate-text/anthropic/tool-order.ts`

feat: add OpenAI Responses API computer tool support (#17290) ## Background Applications need typed access to OpenAI's GA computer-use tool so models can request UI actions and receive updated screenshots in multi-step workflows. ## Summary Added openai.tools.computer(), typed batched actions and safety checks, tool-choice serialization, streaming and non-streaming decoding, and computer_call_output round-tripping for image URLs and file IDs. ## Testing Added type and runtime tests covering every action variant, tool preparation and selection, streaming and non-streaming calls, safety checks, data URL and file ID screenshots, stored and stateless requests, and previousResponseId flows. ## End-to-end Validation - Ran the updated ai-functions example against live OpenAI GPT-5.4; it requested a screenshot action, received the screenshot output, and completed with an accurate screen description. ### Documentation Documented the computer tool API, action and safety-check types, execution loop, screenshot formats, persistence behavior, approvals, and security guidance in the OpenAI provider documentation. ## Related Issues Fixes #13730 Co-authored-by: realglyph123 <124026881+realglyph123@users.noreply.github.com>

feat: add timeBetweenOutputTokensMs stats (#15310) ## Background Advanced applications need detailed tracking of token performance to understand UX behavior of streams. ## Summary Add `timeBetweenOutputTokensMs` stats.

fix: prevent HTTP MCP mid-stream disconnects from crashing the process (#16608) ## Background Streamable HTTP MCP background SSE failures could escape as unhandled promise rejections after a mid-stream server disconnect, crashing Node processes instead of being contained by the transport error callback. ## Summary Updated HttpMCPTransport to observe background inbound SSE startup, reconnect, reader-loop, and close/cancel promise rejections and route non-abort errors through the existing onerror path. ## Testing Added regression coverage for an inbound SSE stream that errors and then rejects during reader cancellation; removed the temporary reproduction script and added a patch changeset. ## Related Issues Fixes #16541 Co-authored-by: jouve <1096799+jouve@users.noreply.github.com>

feat: add persistent accessible light and dark theme selection to the DevTools viewer (#17576) ## Background The DevTools viewer previously provided only a hardcoded dark palette, preventing users from selecting and retaining a readable light appearance. ## Summary Adds an accessible light/dark theme toggle, dark-default initialization, localStorage persistence, document theme state, coordinated semantic palettes, contrast-compliant status colors, focus indicators, and timeline metadata. No exported SDK, telemetry, server, provider, or captured-run API changes. ## Testing Adds six theme unit tests covering defaults, stored preferences, storage failures, document application, switching, persistence, and toggle markup. Two production Chromium tests cover keyboard switching, reload persistence, generate and stream statuses, selected message/tool/error content, timeline metadata, 4.5:1 text contrast, and 3:1 focus-ring contrast. All 30 DevTools unit tests pass. ## End-to-end Validation - The Gateway DevTools theme example completed a live generation successfully. - `pnpm --filter @ai-sdk/devtools test:e2e` built the production viewer and passed both Chromium accessibility and persistence tests. ## Documentation Updates the DevTools guide and package README with theme selection, dark-default behavior, and browser persistence; includes a Gateway usage example and an `@ai-sdk/devtools` patch changeset. ## Related Issues Fixes #17568 Co-authored-by: lgrammel <205036+lgrammel@users.noreply.github.com>

feat: add non-streaming Voxtral TTS to the Mistral provider (#17286) ## Background Users need first-class Mistral Voxtral text-to-speech support so they can generate audio with saved voices or one-off reference audio through generateSpeech. ## Summary Added a SpeechModelV4 implementation, speech and speechModel factories, public MistralSpeechModelId and MistralSpeechModelOptions types, supported output-format mapping, warnings, base64 response handling, and sensitive reference-audio redaction. ## Testing Added 20 focused speech-model tests covering factories, request mapping, voice cloning, formats, warnings, headers, custom configuration, abort signals, responses, errors, and reference-audio privacy in Node and Edge runtimes. ## End-to-end Validation - Ran the Mistral generateSpeech example against the live API and produced valid MP3 audio. - Live-validated one-off cloning using synthetic reference audio; generated valid MP3 audio without warnings. - Removed generated output artifacts after validation. ### Documentation Updated the Mistral provider documentation with speech factories, saved and preset voices, reference audio, formats, unsupported settings, privacy and consent guidance, and non-streaming constraints; also added Voxtral to the core speech model list. ## Related Issues Fixes #14245 Co-authored-by: sovetski <13520683+sovetski@users.noreply.github.com>

fix: Harden stream text processing and middleware against prototype pollution from stream part IDs (#16006) ## Background Provider stream part IDs come from upstream model responses and should not be trusted as plain object keys. `streamText` and the JSON/reasoning extraction middleware kept per-part state in `{}` objects, so a missing stream delta with an ID such as `__proto__` could resolve to `Object.prototype` and mutate shared prototype state while handling the chunk. ## Summary This PR hardens stream-part state tracking by using the existing null-prototype `createIdMap()` helper for provider-controlled IDs in: - `streamText` active text and reasoning content maps, including step resets - `extractJsonMiddleware` text block state - `extractReasoningMiddleware` reasoning extraction state It also adds regression tests for missing `__proto__` stream part IDs to verify that `Object.prototype` is not read or polluted, and includes a patch changeset for `ai`.

fix: transition client-denied tool approvals to output-denied (#17438) ## Background Client-denied tool approvals remained in `approval-responded`, causing persisted and rendered UI state to show a response instead of the terminal `output-denied` state. ## Root Cause `collectToolApprovals` discarded denied approvals whenever an existing tool result was present, including the synthetic `execution-denied` result produced by UI message conversion, preventing `streamText` from emitting `tool-output-denied`. ## Summary Denied approvals with existing `execution-denied` results are now collected for stream notification while retaining the existing result to prevent duplicate model-facing tool results. ## Testing Updated approval collection coverage and added generateText and streamText regressions verifying denial emission and execution-denied result deduplication. ## End-to-end Validation - `pnpm -C examples/ai-functions exec tsx src/reproduction/issue-17136-denied-tool-approval.ts` after rebuilding `ai` — emitted `tool-output-denied`, transitioned live and persisted state to `output-denied`, preserved the model-facing denial, and kept the approved control at `output-available`. ## Related Issues Fixes #17136 Closes #17427 Co-authored-by: Shanik1 <64074694+Shanik1@users.noreply.github.com>

fix: preserve Anthropic web search citations across multi-turn assistant message replay (#17398) ## Background Multi-turn Anthropic web-search conversations replayed previously cited assistant text without citations or encrypted indices, losing citation context. ## Root Cause Anthropic citation deltas were emitted only as source parts, which response history excludes, while assistant text serialization ignored citation metadata; this was confirmed by the failing replay reproduction and the converter and response-history code paths. ## Summary Stored web_search_result_location data on generated text metadata and serialized it back into Anthropic assistant text citation arrays. ## Testing Added inline-snapshot converter coverage and updated Anthropic generate and stream snapshots to verify citation metadata preservation. ## End-to-end Validation - `cd examples/ai-functions && pnpm tsx src/reproduction/anthropic-web-search-citation-replay.ts` — live two-turn Anthropic request passed and preserved `web_search_result_location` citations. ## Related Issues Fixes #17379 Closes #17392 Co-authored-by: flovouin <5155149+flovouin@users.noreply.github.com>

fix: Raw `JSON.parse` used in production provider code (prototype pollution risk) (#16579) ## Background Provider code was synchronously parsing LLM/tool/error JSON with raw JSON.parse, bypassing the SDK's prototype-pollution protections. ## Summary Exported secureJsonParse from @ai-sdk/provider-utils and replaced the six reported production JSON.parse call sites in Google, Anthropic, and Gateway code with secureJsonParse while preserving existing try/catch fallback behavior. ## Testing Kept the reproduction regression test that verifies prototype-polluting Google Interactions tool-call input is rejected/falls back instead of being parsed as arguments. ## Related Issues Fixes #15812

refactoring: change createExecuteToolsTransformation to executeToolsFromStream (#15262) ## Background Align function shape with `invokeCallbacksFromStream` ## Summary change createExecuteToolsTransformation to executeToolsFromStream

fix: Amazon Bedrock S3 image URLs are rejected instead of passed through to inference (#17681) ## Background The published Amazon Bedrock provider rejected S3-hosted images before inference, forcing users to load image bytes into memory. ## Root Cause The model declared no native URL support, causing the core downloader to reject s3:// URLs, while Bedrock message conversion also unconditionally rejected URL-backed files. Reproduction confirmed AI_DownloadError through generateText and AI_UnsupportedFunctionalityError through direct provider use. ## Summary The provider now recognizes s3:// image URLs and serializes user and tool-result images as Bedrock s3Location sources. Bedrock image source types and a patch changeset were updated. ## Testing Added regression coverage for native S3 URL declaration and conversion of user and tool-result S3 images. ## End-to-end Validation - `AWS_REGION=us-east-1 pnpm -C examples/ai-functions exec tsx src/reproduction/amazon-bedrock-s3-image-url.ts` using a temporary Nova variant reached live Bedrock and returned `Provided S3Location not found`, confirming pass-through instead of the previous pre-inference `AI_DownloadError`. ## Related Issues Fixes #15792 Closes #17676 Co-authored-by: tommyOtsai <171396937+tommyOtsai@users.noreply.github.com>

fix: server-response piping errors bypass caller catches (#17648) ## Background Awaited server-response piping calls could bypass caller error handling and emit stream-read failures as unhandled promise rejections. ## Root Cause The asynchronous read loop in writeToServerResponse was invoked without returning its promise, and every public piping wrapper discarded that operation. The reproduction confirmed the awaited helper returned before an InvalidResponseDataError surfaced as an unhandled rejection. ## Summary Returned Promise<void> through server-response writing, text/UI piping helpers, result methods, and the agent wrapper; updated reference documentation and added an ai patch changeset. ## Testing Added regression tests confirming internal and public text piping promises reject with stream-read errors while still ending the response. ## End-to-end Validation - `pnpm -C examples/ai-functions exec tsx --eval "<stream-error reproduction>"` — the InvalidResponseDataError was caught by the caller with no unhandled rejection. ## Related Issues Fixes #11023 Closes #17422 Co-authored-by: intellild <8379858+intellild@users.noreply.github.com>

fix: support tools with optional context (#15303) ## Background Tools with optional context schemas and with context schemas that only contained optional parameters were rejected with type errors on `toolsContext`. ## Summary Fix tool context type inference.

fix: LangChain user image conversion returning provider-specific image_url blocks (#17629) ## Background The published @ai-sdk/langchain adapter returned OpenAI-specific image_url blocks for URL and binary images instead of LangChain's canonical image blocks. ## Root Cause convertUserContent hard-coded OpenAI image serialization and constructed multimodal HumanMessages through content rather than contentBlocks. The reproduction confirmed inconsistent image output, and live ChatOpenAI validation showed canonical blocks require LangChain's v1 content marker for downstream translation. ## Summary Converted image inputs and image files to canonical LangChain image blocks and constructed multimodal messages through contentBlocks so LangChain marks and translates them as v1 content. Added a patch changeset. ## Testing Updated convertUserContent and convertModelMessages coverage for URL, binary, data-URL, and file-based images, including the v1 content marker. ## End-to-end Validation - `pnpm -C packages/langchain build && pnpm -C examples/ai-functions exec tsx -e <assertions>` — built the package and confirmed canonical URL/binary image blocks while preserving canonical non-image files. - `pnpm -C examples/next-langchain exec tsx -e <live ChatOpenAI validation>` — GPT-4o accepted the converted JPEG and identified it as a blue-and-yellow macaw. ## Related Issues Fixes #11943 Closes #17619 Co-authored-by: jessieibarra <58125016+jessieibarra@users.noreply.github.com>

fix: update step performance metrics with explicit effective, input, output, and total token throughput fields (#15307) ## Background Step performance metrics previously exposed a single output-token throughput value, which made it hard to distinguish full-request latency from streaming throughput after the first output arrived. This change makes the metrics more explicit so users can separately analyze effective request throughput, streaming output rate, input processing speed before first output, and total token throughput. ## Summary - Renamed `tokensPerSecond` to `effectiveOutputTokensPerSecond`. - Renamed `timeToFirstTokenMs` to `timeToFirstOutputTokenMs`. - Added `outputTokensPerSecond`, `inputTokensPerSecond`, and `effectiveTotalTokensPerSecond`. - Updated generateText, streamText, lifecycle events, and telemetry to use the new performance fields. - Updated docs and tests to describe and verify the new calculations. ## Related Issues Builds on and revises #15241