Lars Grammel

feat: agent tui (#15845) ## Background To quickly develop and test agents, it is helpful to have a TUI that requires very limited setup. ## Summary - integrate https://github.com/lgrammel/agent-tui as `@ai-sdk/tui` - add documentation ## Example ```ts await runAgentTUI({ agent }); ``` ## Manual Verification - [x] `pnpm tsx src/agent/openai/tui`

fix: rename onFinish to onEnd (#15245) ## Background We are standardizes on `End` over `Finish`, and results should be accumulative. ## Summary * Renames the final lifecycle callback from onFinish to onEnd for generateText, streamText, Agent, ToolLoopAgent, and WorkflowAgent; onFinish remains as a deprecated alias, with onEnd taking precedence. * Updates final callback result semantics so usage, content, toolCalls, toolResults, files, sources, and warnings aggregate across all steps, while final-step-only data is available through finalStep. * Adds/deprecates the corresponding callback types (GenerateTextOnEndCallback, WorkflowAgentOnEndCallback) and updates telemetry to use the new onEnd event shape. * Updates docs, examples, references, and the AI SDK 7 migration guide to describe the onEnd rename and the new aggregated final event fields.

feat: add performance statistics (#15241) ## Background Statistics such as tokens per ms and time to first token are important to understand the performance of models and providers. In addition to telemetry, they can also be displayed in user interfaces. ## Summary * add `performance` property to `StepResult` with `tokensPerSecond`, `stepTimeMs`, `responseTimeMs`, `toolExecutionMs`, `timeToFirstTokenMs` * rename `durationMs` on tool execution events to `toolExecutionMs` ## Example Output ``` Step performance: [ { stepTimeMs: 2592.4770839999996, responseTimeMs: 2591.246209, tokensPerSecond: 61.36043709306976, toolExecutionMs: { call_zFG9UMazm5kY2r4q4Mn6LUyx: 0.0625 }, timeToFirstTokenMs: 2339.711542 }, { stepTimeMs: 1228.0632500000002, responseTimeMs: 1226.2540409999997, tokensPerSecond: 35.06614336205071, toolExecutionMs: { call_Ben1pOrejarzcE87EtPbXndv: 0.06979200000023411 }, timeToFirstTokenMs: 873.9773329999998 }, { stepTimeMs: 3674.4792079999997, responseTimeMs: 3673.7040419999994, tokensPerSecond: 100.17137902041159, toolExecutionMs: {}, timeToFirstTokenMs: 3110.819042 } ] ``` ## Manual Verification Run and verify: - [x] `examples/ai-functions/src/generate-text/openai/step-performance.ts` - [x] `examples/ai-functions/src/stream-text/openai/step-performance.ts` ## Future Work * add support in `WorkflowAgent` @gr2m

fix(ai): fix auto-complete on provider registry and custom provider (#14811) ## Background Autocomplete on the provider registry stopped working. Most likely caused by the changes in #13001 ## Summary * restored auto-complete functionality * add type tests * add support of all model versions incl. string to CustomProvider ## Manual Verification Check auto-completes and errors in: - [x] `examples/ai-functions/src/registry/setup-registry.ts` ## Related Issues Fixes #14780 --------- Co-authored-by: Felix Arntz <felix.arntz@vercel.com>

feat(ai): add sensitiveRuntimeContext option (#14757) ## Background The `runtimeContext` can contain information that should not be exposed to telemetry, e.g. credentials. ## Summary Add concept of sensitive runtime context that allows users to restrict which runtime context properties are being sent to telemetry (opt-out). * introduce `SensitiveContext` type * introduce `sensitiveRuntimeContext` option on `generateText`, `streamText`, and `ToolLoopAgent` * restrict `activeTools` to strings and introduce `ActiveTools` helper type * rename `filter-active-tool.ts` to `filter-active-tools.ts` ## Manual Verification Run examples and verify that sensitive context is not logged by telemetry. - [x] generateText `examples/ai-functions/src/telemetry/console/generate-text.ts` - [x] streamText `examples/ai-functions/src/telemetry/console/stream-text.ts` - [x] ToolLoopAgent `examples/ai-functions/src/telemetry/console/tool-loop-agent.ts` ## Future Work * implement `sensitiveContext` for tools ## Related Issues Builds on #14536

feat: sandbox shell execution abstraction (#14949) ## Background Many agents are using filesystems through shell and file read/write tools, often in separate sandbox environments. These agents are so common that a first-class sandbox abstraction would be beneficial. ## Summary * add `Sandbox` type * add `sandbox` option to `generateText`, `streamText`, `ToolLoopAgent` * make sandbox available in `ToolExecutionOptions` ## Example Tool definition: ```ts import { tool } from 'ai'; import { z } from 'zod'; export function sandboxShellTool() { return tool({ description: 'Run a shell command', inputSchema: z.object({ command: z.string(), }), execute: async ({ command }, { sandbox }) => { // TODO figure out type inference to turn the runtime error into a type error if (!sandbox) { throw new Error('Sandbox is not available'); } return sandbox.executeCommand({ command }); }, }); } ``` Agent definition: ```ts import { openai } from '@ai-sdk/openai'; import { ToolLoopAgent } from 'ai'; import { sandboxShellTool } from '../../tools/sandbox-shell-tool'; export const sandboxAgent = new ToolLoopAgent({ model: openai('gpt-5.5'), tools: { shell: sandboxShellTool(), }, prepareCall: ({ sandbox, ...rest }) => ({ ...rest, instructions: `You are a helpful assistant that can run shell commands.\n` + `You are operating in the following sandbox: ${sandbox?.description}`, }), }); ``` Agent call: ```ts import { Bash } from 'just-bash'; import { JustBashSandbox } from '../../sandbox/just-bash-sandbox'; import { sandboxAgent } from './sandbox-agent'; const sandbox = new JustBashSandbox( new Bash({ cwd: '/home/user', }), ); const result = await sandboxAgent.stream({ prompt: 'Create a file named greeting.txt with a short greeting, then list the files and show the file contents.', sandbox, }); ``` ## Manual Verification - [x] agent - generate `src/agent/openai/generate-local-sandbox` - [x] agent - stream `src/agent/openai/stream-local-sandbox` - [x] test with ui `examples/ai-e2e-next/app/chat/sandbox/page.tsx` ## Future Work * read/write files in sandboxes * explore generic typing of sandbox tools * move sandbox shell tool into ai package * allow choosing a sandbox in prepareStep * add timeout to executeCommand * figure out sandbox streaming * need onFinalize callback (or something) that is always invoked (on finish, abort, error etc) --------- Co-authored-by: nicoalbanese <gcalbanese96@gmail.com>

feat: add a per-step first-content timeout for streaming generations (#17561) ## Background Streaming transports can remain active through headers, metadata, keep-alives, empty deltas, or raw bytes without producing model content. Consumers need an SDK-level semantic deadline for the first content-bearing output of every streaming model-call step. ## Summary Adds optional timeout.firstChunkMs configuration and extraction. streamText arms a dedicated timeout for every model-call response stream, merges its abort signal into the provider call, and disarms it before forwarding the first non-empty text, reasoning, or tool-input delta, generated file, reasoning file, or tool call. Non-content activity does not affect firstChunkMs, and chunkMs now starts and resets only on semantic output. Step timers and abort listeners are cleaned up on completion, abort, setup failure, provider stream error, and cancellation, including through stitchable-stream lifecycle callbacks. ## Testing Adds Node and Edge runtime coverage for non-content activity, empty deltas, every qualifying output category, timeout identity, pre-forward disarming, multi-step re-arming, semantic chunkMs behavior, provider errors, cancellation, and timer cleanup. Also adds stitchable-stream lifecycle tests, timeout extraction tests, and compile-time public API coverage. ## End-to-end Validation - Ran the provider-independent firstChunkMs streamText example with delayed initial content; it completed successfully and emitted `The first content arrived before the deadline.` ## Documentation Updates timeout settings, the streamText reference, and ToolLoopAgent documentation with firstChunkMs semantics, qualifying output, ignored non-content activity, and content-based chunkMs behavior. ToolLoopAgent now explicitly identifies firstChunkMs and chunkMs as streaming-only. Adds a runnable mock-provider example and an ai patch changeset. ## Related Issues Fixes #17315 Co-authored-by: lgrammel <205036+lgrammel@users.noreply.github.com>

feat: add a `toolOrder` option to control the order in which tools are sent (#15811) ## Background Provider-side prompt/tool caching can benefit from stable tool definition ordering. This adds an explicit `toolOrder` option so callers can control the order in which tools are sent to provider APIs. ## Summary Adds `toolOrder` support to `generateText`, `streamText`, and `ToolLoopAgent`, including per-step overrides via `prepareStep`. Tools listed in `toolOrder` are sent first in the requested order, while omitted tools are appended alphabetically for stable defaults. Updates type coverage, unit tests, snapshots, OpenTelemetry test fixtures, API reference docs, provider examples, and adds a patch changeset for `ai`. ## Manual Verification - [x] `examples/ai-functions/src/generate-text/openai/tool-order.ts` - [x] `examples/ai-functions/src/generate-text/anthropic/tool-order.ts`

feat: add OpenAI Responses API computer tool support (#17290) ## Background Applications need typed access to OpenAI's GA computer-use tool so models can request UI actions and receive updated screenshots in multi-step workflows. ## Summary Added openai.tools.computer(), typed batched actions and safety checks, tool-choice serialization, streaming and non-streaming decoding, and computer_call_output round-tripping for image URLs and file IDs. ## Testing Added type and runtime tests covering every action variant, tool preparation and selection, streaming and non-streaming calls, safety checks, data URL and file ID screenshots, stored and stateless requests, and previousResponseId flows. ## End-to-end Validation - Ran the updated ai-functions example against live OpenAI GPT-5.4; it requested a screenshot action, received the screenshot output, and completed with an accurate screen description. ### Documentation Documented the computer tool API, action and safety-check types, execution loop, screenshot formats, persistence behavior, approvals, and security guidance in the OpenAI provider documentation. ## Related Issues Fixes #13730 Co-authored-by: realglyph123 <124026881+realglyph123@users.noreply.github.com>

feat(ai): add experimental_refineToolInput option to ToolLoopAgent, generateText, streamText (#15000) ## Background Different LLM provider might generate slightly different tool inputs for the same tool input type, e.g. using empty string instead of `null`. Tools might come from 3rd party providers, so the input schemas might not support refinement. ## Summary Add an experimental `refineToolInput` option to `ToolLoopAgent`, `generateText`, `streamText` that can be used to modify the tool inputs as long as the expected type is retained. ## Manual Verification Execute examples and check tool inputs: - [x] `examples/ai-functions/src/generate-text/openai/refine-tool-input.ts` - [x] `examples/ai-functions/src/steam-text/openai/refine-tool-input.ts`

feat: add timeBetweenOutputTokensMs stats (#15310) ## Background Advanced applications need detailed tracking of token performance to understand UX behavior of streams. ## Summary Add `timeBetweenOutputTokensMs` stats.

refactoring: restructure Tool types (#14849) ## Background Different types of tools can have different available options. Currently we expose options on tool types that do not support it, e.g. execute on provider executed tools. ## Summary Refactor the `Tool` type into a union of `FunctionTool`, `DynamicTool`, `ProviderDefinedTool`, and `ProviderExecutedTool`. Expose the new types. Add type tests. The goal is to create a foundation for further refactoring and option restrictions. ## Future Work * restrict options available on specific tool types more

fix: prevent HTTP MCP mid-stream disconnects from crashing the process (#16608) ## Background Streamable HTTP MCP background SSE failures could escape as unhandled promise rejections after a mid-stream server disconnect, crashing Node processes instead of being contained by the transport error callback. ## Summary Updated HttpMCPTransport to observe background inbound SSE startup, reconnect, reader-loop, and close/cancel promise rejections and route non-abort errors through the existing onerror path. ## Testing Added regression coverage for an inbound SSE stream that errors and then rejects during reader cancellation; removed the temporary reproduction script and added a patch changeset. ## Related Issues Fixes #16541 Co-authored-by: jouve <1096799+jouve@users.noreply.github.com>

refactoring: remove real-time delays in unit tests (#14728) ## Background Unit tests should run without any delays. However, we use `delay` and `setTimeout` in our tests and production code, leading to slower running tests. ## Summary * upgrade `vitest` to `4.1.5` everywhere * add ci test setup patch for codemod testing to prevent test timeouts * use `delay` instead of `setTimeout` * use `vi.useFakeTimers()` and `vi.useRealTimers()` consistently in tests * advance the simulated clock with `vi.advanceTimersByTimeAsync`

fix(ai): reject system messages in messages or prompt by default (opt-in) (#14752) ## Background For historical and convenience reasons, system messages can be part of user messages or prompts, e.g. to allow interleaving regular messages and system messages. However, this creates a prompt injection risk where the user (e.g. by modifying the messages in a web ui) can override or set the system prompt. In most cases, it should only be possible to set the system prompt via the system (or instructions) property, and users should not be able to inject system messages. ## Summary * throw `InvalidPromptError` when there are system messages in the messages or prompt options * add `allowSystemInMessages` option for opting into allowing system messages in messages or prompt options ## Future Work * add `allowSystemInMessages` opt-in support to `WorkflowAgent` (if desired) @gr2m ## Related Issues Issue reported in #14749

feat: add persistent accessible light and dark theme selection to the DevTools viewer (#17576) ## Background The DevTools viewer previously provided only a hardcoded dark palette, preventing users from selecting and retaining a readable light appearance. ## Summary Adds an accessible light/dark theme toggle, dark-default initialization, localStorage persistence, document theme state, coordinated semantic palettes, contrast-compliant status colors, focus indicators, and timeline metadata. No exported SDK, telemetry, server, provider, or captured-run API changes. ## Testing Adds six theme unit tests covering defaults, stored preferences, storage failures, document application, switching, persistence, and toggle markup. Two production Chromium tests cover keyboard switching, reload persistence, generate and stream statuses, selected message/tool/error content, timeline metadata, 4.5:1 text contrast, and 3:1 focus-ring contrast. All 30 DevTools unit tests pass. ## End-to-end Validation - The Gateway DevTools theme example completed a live generation successfully. - `pnpm --filter @ai-sdk/devtools test:e2e` built the production viewer and passed both Chromium accessibility and persistence tests. ## Documentation Updates the DevTools guide and package README with theme selection, dark-default behavior, and browser persistence; includes a Gateway usage example and an `@ai-sdk/devtools` patch changeset. ## Related Issues Fixes #17568 Co-authored-by: lgrammel <205036+lgrammel@users.noreply.github.com>

feat: add non-streaming Voxtral TTS to the Mistral provider (#17286) ## Background Users need first-class Mistral Voxtral text-to-speech support so they can generate audio with saved voices or one-off reference audio through generateSpeech. ## Summary Added a SpeechModelV4 implementation, speech and speechModel factories, public MistralSpeechModelId and MistralSpeechModelOptions types, supported output-format mapping, warnings, base64 response handling, and sensitive reference-audio redaction. ## Testing Added 20 focused speech-model tests covering factories, request mapping, voice cloning, formats, warnings, headers, custom configuration, abort signals, responses, errors, and reference-audio privacy in Node and Edge runtimes. ## End-to-end Validation - Ran the Mistral generateSpeech example against the live API and produced valid MP3 audio. - Live-validated one-off cloning using synthetic reference audio; generated valid MP3 audio without warnings. - Removed generated output artifacts after validation. ### Documentation Updated the Mistral provider documentation with speech factories, saved and preset voices, reference audio, formats, unsupported settings, privacy and consent guidance, and non-streaming constraints; also added Voxtral to the core speech model list. ## Related Issues Fixes #14245 Co-authored-by: sovetski <13520683+sovetski@users.noreply.github.com>

feat: add request.messages to StepResult (#15052) ## Background Accessing the input messages for a particular step can help with compaction, e.g. when you want to continue compacted messages in `prepareStep`. ## Summary Add `request.messages` to `StepResult`. ## Example In `prepareStep`, you can access the input messages from the last step via `steps.at(-1).request.messages` ## Future Work * add `include.requestSteps` flag * limit `response.messages` on `StepResult` to only the messages from that step * add `inputMessages`, `responseMessages` to `prepareStep` params * change usage of `messages` response from `prepareStep` (continue using those messages) * implement support in `WorkflowAgent` @gr2m * add compaction example to examples and documentation ## Related Issues Relates to #9631 and #6615

fix: Harden stream text processing and middleware against prototype pollution from stream part IDs (#16006) ## Background Provider stream part IDs come from upstream model responses and should not be trusted as plain object keys. `streamText` and the JSON/reasoning extraction middleware kept per-part state in `{}` objects, so a missing stream delta with an ID such as `__proto__` could resolve to `Object.prototype` and mutate shared prototype state while handling the chunk. ## Summary This PR hardens stream-part state tracking by using the existing null-prototype `createIdMap()` helper for provider-controlled IDs in: - `streamText` active text and reasoning content maps, including step resets - `extractJsonMiddleware` text block state - `extractReasoningMiddleware` reasoning extraction state It also adds regression tests for missing `__proto__` stream part IDs to verify that `Object.prototype` is not read or polluted, and includes a patch changeset for `ai`.

fix: transition client-denied tool approvals to output-denied (#17438) ## Background Client-denied tool approvals remained in `approval-responded`, causing persisted and rendered UI state to show a response instead of the terminal `output-denied` state. ## Root Cause `collectToolApprovals` discarded denied approvals whenever an existing tool result was present, including the synthetic `execution-denied` result produced by UI message conversion, preventing `streamText` from emitting `tool-output-denied`. ## Summary Denied approvals with existing `execution-denied` results are now collected for stream notification while retaining the existing result to prevent duplicate model-facing tool results. ## Testing Updated approval collection coverage and added generateText and streamText regressions verifying denial emission and execution-denied result deduplication. ## End-to-end Validation - `pnpm -C examples/ai-functions exec tsx src/reproduction/issue-17136-denied-tool-approval.ts` after rebuilding `ai` — emitted `tool-output-denied`, transitioned live and persisted state to `output-denied`, preserved the model-facing denial, and kept the approved control at `output-available`. ## Related Issues Fixes #17136 Closes #17427 Co-authored-by: Shanik1 <64074694+Shanik1@users.noreply.github.com>