ai — Engineering Performance
22 engineers all time · Jan 2025 – Sep 2026 · built 2026-09-30 · GitHub
Performance snapshot
Today's rolling 90-day reading for ai, compared with the start of the series. Pick a window to move that comparison point.
Avg. perf / dev / mo
+82.0%
2.97 → 5.41 ETV
Active engineers
+100.0%
5.0 → 10.0
Features
+2.3pp
30.2% → 32.5%
vs. Vercel
1.1x
1.9x → 1.1x · +7% above
ai vs. Vercel
Per-engineer ETV for ai against Vercel as a whole. Both lines are 90-day rolling averages scaled to a 30-day month, so they share one axis and can be read against each other at any point. Pick a window to zoom the chart to it.
Performance Composition
Each month's output split by type of work: Features (new value), Maintenance (sustaining systems), Tests, Docs, and Fixes (rework). The yellow line is output per engineer, so when it rises each engineer is delivering more, whatever the team size did. Unit: Engineering Throughput Value (ETV).
Engineering capacity
Effective engineers behind ai, against its pre-AI baseline. Each subject has its own: ai's is 2.97 ETV / dev / mo, its first reading in Q1 2025. Per-engineer ETV divided by that gives a capacity multiple, and that multiple applied to the engineers active in the trailing 90 days turns it into engineer-equivalents. The line is the real headcount, so the gap between line and area is what the leverage is worth. Because each baseline is its own, every subject opens at 1.0x on its first day: multiples measure improvement and are not comparable between subjects.
Knowledge concentration
How dependent is this repo on a small number of engineers? Higher top-1 share = higher key-person risk.
Lars Grammel owns 33.5 % of commits.
Behind the numbers
Written summary of the work completed each month.
No monthly reports available yet.
Top engineers
Most impactful commits
Top 10 by ETV in the all-time window.
- 8.7ETVfeat(harness-acp): introduce ACP harness adapter as a meta adapter to connect to any ACP compatible harness (#18483) ## Background AI SDK harness integrations currently require a dedicated adapter for each coding-agent runtime. ACP provides a common protocol that can support multiple runtimes through one adapter while keeping runtime-specific installation and authentication details in small profiles. Some harnesses only offer an ACP adapter as the sole way for programmatic control over their harness, so for those harnesses having an ACP adapter unblocks supporting them by our harness layer. ## Summary This adds `harness-acp`, a meta adapter for ACP v1 implementations. The adapter owns the generic sandbox bridge, protocol translation, host-tool relay, approvals, and lifecycle behavior, while callers configure the underlying ACP runtime declaratively. - Support simple and lockfile-pinned NPM acquisition for arbitrary ACP implementations. - Support direct and AI Gateway authentication with runtime-resolved environment variables and adapter-specific Gateway routes. - Translate ACP streams, native tools, host tools, approvals, skills, session modes, and resumable lifecycle state into Harness APIs. - Add complete Claude Code, Codex, and Grok Build profiles across documentation and interactive examples. ## End-to-End Verification - Exercised the added Claude Code ACP, Codex ACP, and Grok Build ACP through the interactive Harness example. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Related Issues See #16956.Felix Arntz · ff0f708e · 2026-08-07
- 4.7ETVfeat(harness): add support for using adapter-native subscriptions where available (#20606) ## Background Harness adapters so far support direct auth via API keys and AI Gateway auth, but not using their native subscriptions. ## Summary This adds host-side native subscription resolution while retaining the existing `direct` and `ai-gateway` authentication modes. Direct authentication prefers explicit provider environment credentials, then falls back to a native subscription if no such credentials are present; AI Gateway authentication never reads native subscriptions. - Add shared OAuth access-token expiry and refresh helpers with a five-minute refresh window. - Resolve supported adapters' native credential files, keyrings, and CLI credentials at the host boundary. - Keep refresh tokens and real access tokens on the host while passing only fake credentials into the sandbox, which then get replaced at the host boundary via credential brokering. - Add native-subscription examples and adapter-specific coverage for every supported credential source and refresh flow. ## End-to-End Verification Ran the `native-subscription` harness examples added. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review)Felix Arntz · cdc12a15 · 2026-09-11
- 3.9ETVfeat: add first-party Code Mode package (#18091) ## Background works towards https://github.com/vercel/ai/issues/18071 the idea is to unify the different provider implementations of programmatic tool calling / code-mode. ## Summary add a new `@ai-sdk/code-mode` package that will export the necessary controls for executing the tools in that format ## End-to-End Verification - ran the example `examples/ai-functions/src/generate-text/code-mode/code-mode.ts` - ran the example `examples/ai-functions/src/stream-text/code-mode/code-mode.ts` - ran the example `http://localhost:3000/chat/code-mode` ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work - api syntax will change - docs need to be added - tool approval mechanism needs to work - workflow boundary concerns ## Related Issues towards #18071 fixes #18141Aayush Kapoor · 29d7b679 · 2026-07-30
- 3.7ETVfeat: agent tui (#15845) ## Background To quickly develop and test agents, it is helpful to have a TUI that requires very limited setup. ## Summary - integrate https://github.com/lgrammel/agent-tui as `@ai-sdk/tui` - add documentation ## Example ```ts await runAgentTUI({ agent }); ``` ## Manual Verification - [x] `pnpm tsx src/agent/openai/tui`Lars Grammel · e757741f · 2026-06-05
- 3.6ETVfeat: Realtime API support for browser<->provider websocket connection (#13893) ## Background Part of https://github.com/vercel/ai/issues/13897 Support for Realtime API has been requested for a long time. We want to support different architectures 1. Websocket connection from browser directly to provider 2. Websocket connection from browser through user's server to provider 3. Websocket connection from browser through gateway to provider 4. Websocket connection from browser through user's server and gateway to provider This pull requests implements the first, see [Architecture](#architecture) below ## Summary Alternative implementation of the Realtime API support proposed in #13889, with a reworked developer-facing API: - **`openai.experimental_realtime('gpt-realtime')`** works in both server and browser (no separate `@ai-sdk/openai/realtime` import needed) - **`openai.experimental_realtime.getToken()`** static method for server-side ephemeral token creation - **`experimental_useRealtime`** hook returns `messages: UIMessage[]` (aligned with `useChat` format) instead of `transcript` - **`inputAudioTranscription`** session config enables rendered user messages for transcribed microphone input - **`addToolOutput(callId, result)`** for client-side tools that need manual result submission - **`onToolCall`** callback for auto-executed client-side tools - Framework-agnostic helper type: `Experimental_RealtimeSetupResponse` Provider implementations included: **OpenAI**, **Google**, and **xAI**. ElevenLabs support is split into a stacked follow-up: **#15747**. <a name="architecture"></a> ## Architecture ```mermaid sequenceDiagram participant Browser participant Server participant Provider as AI Provider<br/>(OpenAI / xAI / Gemini / ...) Browser->>Server: POST /api/setup Server->>Provider: Request token (with tool definitions) Provider-->>Server: Short-lived auth token Server-->>Browser: { token } Browser->>Provider: Open WebSocket (using token) Provider-->>Browser: Audio/text chunks (streaming) Provider-->>Browser: Tool call request Browser->>Server: App-specific tool request (optional) Server-->>Browser: App-specific tool result Browser->>Provider: Send tool result via WebSocket Provider-->>Browser: Continue streaming ``` ### Alternatives considered #### Generic RPC route for server-side tools One option was for the SDK to expose a generic server-side tool execution flow: the realtime session would receive a tool call, POST `{ name, inputs, callId }` to an `execute-tools` route, execute a matching server-side tool, and send the result back to the provider automatically. We decided against that for the initial implementation. A generic RPC route is convenient, but it creates a security-sensitive application boundary: the app must authenticate the user, bind the request to a realtime session, allowlist tool names, validate call IDs, rate limit access, and authorize each tool invocation. If the SDK provides the generic route shape, it is easy to copy into production without those controls. Instead, the SDK keeps tool execution client-driven through `onToolCall` and `addToolOutput`. Server-backed tools should call app-specific API endpoints from `onToolCall` (for example `/api/weather`), where the application can apply its normal auth, validation, authorization, and rate limiting rules. Documentation should cover secure server-backed tool calling patterns, but the SDK should not own the generic remote tool RPC abstraction yet. ## Manual verification The realtime voice example added in this PR lives at `examples/ai-e2e-next/app/realtime/page.tsx` (UI) and `examples/ai-e2e-next/app/api/realtime/[...path]/route.ts` (server-side token endpoint). 1. Add the provider API key(s) for whichever provider(s) you want to test to `examples/ai-e2e-next/.env.local`: ```bash OPENAI_API_KEY=... # OpenAI realtime (gpt-realtime) GOOGLE_GENERATIVE_AI_API_KEY=... # Google realtime (gemini-3.1-flash-live-preview) XAI_API_KEY=... # xAI realtime (grok-voice-latest) ``` 2. Start the example: ```bash cd examples/ai-e2e-next pnpm dev ``` 3. Open http://localhost:3000/realtime, pick a provider + voice, and click **Connect**. Use the microphone or type a message. Ask "what's the weather in Paris?" or "roll a dice" to exercise client-side tool calling (`onToolCall`). ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [ ] I have reviewed this pull request (self-review) - [ ] Security Audit - esp. securing remote tool calling - [ ] Update internal Architecture / how it works docs ### Future Work 1. **ElevenLabs** provider support — stacked follow-up in #15747 2. Websocket connection from browser through user's server to provider 3. Websocket connection from browser through gateway to provider 4. Websocket connection from browser through user's server and gateway to provider ## Related issues - #3176 - #3907 - #4082 - #5007 - #9559 - #12381 - #13706 - #13889 Co-authored-by: Cursor <cursoragent@cursor.com>Gregor Martynus · ce769dd2 · 2026-06-05
- 3.5ETVfeat(video): externalize polling control and webhook support for generateVideo (#12515) - Adds optional `doStart`/`doStatus` methods to the experimental `VideoModelV4` spec, enabling a two-phase async model where the SDK core orchestrates polling or webhook-based completion instead of each provider implementing its own polling loop. - Adds `poll` and `webhook` parameters to `experimental_generateVideo` for user-controlled polling intervals/timeouts and webhook-based completion. - Adds optional `handleWebhookOption` to `VideoModelV4` so a provider can signal native webhook support. The SDK only invokes the user's webhook factory when the model implements it; otherwise it falls back to polling. - Replaces `doGenerate` with `doStart`/`doStatus` in the first-party models implementing `VideoModelV4` (FAL, Alibaba, KlingAI, Replicate, xAI, Google, and Google Vertex). The gateway video model remains a `doGenerate` passthrough. - Makes `doGenerate` optional on `VideoModelV4`. Third-party providers implementing only `doGenerate` continue to work unchanged, and the SDK falls back to it when no async flow is selected. - Constrains operation handles to `JSONValue`, making the provider-owned task reference serializable and suitable for future persistence and gateway support. Towards https://github.com/vercel/ai/issues/12381. ## Architecture ``` User calls: generateVideo({ model, prompt, poll?, webhook? }) ↓ Core function: Chooses flow based on model capabilities and user options ↓ ┌───────────┴───────────────┐ │ │ [Legacy / fallback] [Start/Status flow] │ (poll or webhook provided, │ or doGenerate missing) │ │ model.doGenerate(options) model.doStart(options) ↓ ↓ Provider owns polling returns { operation } ↓ ┌───────────┴────────────┐ │ │ [native webhook] [polling] │ │ await notification SDK core loop │ │ model.doStatus(op) model.doStatus(op) └───────────┬────────────┘ ↓ return result ``` The SDK core owns the polling lifecycle. Providers only implement single-shot `doStart` (submit) and `doStatus` (check once). The polling interval and timeout are controlled by the user. ## Review decisions - **`poll` and `webhook` are intentionally composable.** When a model supports native webhooks, `poll.timeoutMs` limits the webhook wait. When it does not, the same polling configuration controls the automatic fallback. This behavior is documented and covered by tests. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2821429102)) - **The first iteration uses fixed polling intervals and timeouts.** Exponential backoff and `onAttempt` were removed to keep the API small; aligning polling with shared retry/backoff primitives can be revisited separately. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2821435402)) - **Operation handles are JSON-serializable.** `operation` is typed as `JSONValue` instead of `unknown`, while remaining opaque to core. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843221880)) - **Shared cross-modality operation types are deferred.** The current types remain video-specific until another modality implements the same lifecycle, avoiding a premature provider-spec abstraction. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843151286)) - **Replicate's server-side `wait` remains provider-specific.** It affects how long Replicate holds the submission request open, while top-level `poll` controls the SDK's provider-neutral status lifecycle. It can be added through Replicate provider options independently if needed. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843203139)) ## Verification - [x] Tests added and updated - [x] Documentation added and updated - [x] Patch changeset added - [x] TypeScript, lint/format, and code consistency checks pass locally ## Future work - Expose async operations as a first-class API: start an operation, persist its handle, and resume or query it later. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843215786)) - Generalize operation types after another model modality implements the same lifecycle. - Add AI Gateway transport for the JSON-serializable start/status operation flow. - Explore a multi-notification webhook receiver if providers need intermediate status events rather than a single completion notification. ## Related issue https://github.com/vercel/ai/issues/12381 --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: josh <144584931+dancer@users.noreply.github.com> Co-authored-by: vercel-ai-sdk[bot] <225926702+vercel-ai-sdk[bot]@users.noreply.github.com> Co-authored-by: mat lenhard <mclenhard@gmail.com> Co-authored-by: Rohan Taneja <47066511+R-Taneja@users.noreply.github.com> Co-authored-by: Felix Arntz <felix.arntz@vercel.com> Co-authored-by: Walter Korman <shaper@vercel.com> Co-authored-by: Nico Albanese <49612682+nicoalbanese@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>Gregor Martynus · 79e133cc · 2026-08-03
- 3.4ETVfeat(harness): allow changing harness settings between turns via `prepareCall()` support on `HarnessAgent` (#19828) ## Background `HarnessAgent` settings were fixed when an agent was created, preventing callers from adapting instructions, skills, and tools between completed turns. `ToolLoopAgent` already supported `prepareCall()` and `callOptionsSchema`, allowing to dynamically modify certain settings in between turns. ## Summary This aligns `HarnessAgent` with the `ToolLoopAgent` call-options pattern while keeping settings fixed throughout an active turn and its continuations. - Add `callOptionsSchema` and `prepareCall` support for deriving per-turn instructions, skills, and tools. - Capture active turn settings in continuation state so suspended and resumed turns cannot pick up later configuration. - Update harness adapters to apply settings changes between turns, including ACP skill refreshes and instruction guidance. - Make `writeSkills` safely replace only harness-owned skills and skip unchanged content using a manifest with SHA-256 hashes. - Add documentation, unit coverage, and changing-settings examples for every harness. ## End-to-End Verification Ran the new `changing-settings` examples against all harnesses. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review)Felix Arntz · 14d4fc03 · 2026-08-27
- 3.3ETVfeat: add provider versions to user-agent header (#8703) ## Background This change is added to include the provider versions in the user-agent header that we now pass. Previously, the string only appended `ai` version and runtime env ## Summary - add a `version.ts` file - export version during build by changing `tsup.config` - export verison during tests by changing `vitest.config` - mock a dummy version for tests ## Manual Verification Verified by making changes to the test file ## Tasks - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Future Work Version in user-agent added for: - [x] `gateway` - [x] `openai` - [x] `anthropic` - [x] `google` - [x] `google-vertex` - [x] `azure` - [x] `amazon-bedrock` - [x] `cohere` - [x] `mistral` - [x] `groq` - [x] `cerebras` - [x] `deepinfra` - [x] `deepseek` - [x] `fireworks` - [x] `perplexity` - [x] `replicate` - [x] `togetherai` - [x] `xai` - [x] `vercel` - [x] `openai-compatible` - [x] `elevenlabs` - [x] `assemblyai` - [x] `deepgram` - [x] `gladia` - [x] `revai` - [x] `luma` - [x] `fal` - [x] `hume` - [x] `lmnt` - [x] `langchain` - [x] `llamaindex` - [x] `valibot` ## Related Issues fix in progress for #8699 --------- Co-authored-by: Gregor Martynus <39992+gr2m@users.noreply.github.com>Aayush Kapoor · 1cad0aba · 2025-09-23
- 2.8ETVfeat(provider): add support for provider references and uploading files as supported per provider (#13816) ## Background The AI SDK supports passing media files inline or via URL, but has no way to upload files directly to a provider or reference previously uploaded files across providers. Some providers return internal file IDs (not URLs) from their upload APIs, and switching providers mid-conversation requires a way to map the same logical file to different provider-specific identifiers. ## Summary Introduces `uploadFile` as a top-level function and `ProviderReference` (`Record<string, string>`) as the provider-independent way to reference uploaded files. - **New spec types**: `SharedV4ProviderReference` (provider package), `FilesV4` interface with `uploadFile` method, `UploadFileResult`; `mediaType` and `filename`** are top-level parameters as they're widely supported and used - **New top-level API**: `uploadFile({ files, data, mediaType?, filename?, providerOptions? })` in the `ai` package, with auto-detection of media type from file bytes when not provided - **Provider implementations**: `files()` interface on Anthropic, Google, OpenAI, and xAI providers, each implementing `FilesV4.uploadFile` - Other providers don't support uploading files, or they only support uploading files for batch inference (`*.jsonl`), which we don't support anyway. - **Provider reference support in messages**: `LanguageModelV4FilePart.data` now accepts `SharedV4ProviderReference` in addition to `DataContent`; providers that support file references (Anthropic, Google, OpenAI, xAI) resolve them via `resolveProviderReference`; all other providers throw `UnsupportedFunctionalityError` - **Spec cleanup**: `file-id` and `image-file-id` tool result output types replaced with `file-reference` and `image-file-reference` using `SharedV4ProviderReference` instead of `string | Record<string, string>` - **Uploading from URL is not supported** — no provider supports this, and auto-downloading is questionable; callers should fetch first - **`reasoning-file` was not touched** — it is model-generated as part of reasoning output, so provider references are not applicable - **Docs included** — Docs about `uploadFile` and `ProviderReference`, and a new architecture guide are included ### Design decisions - `ProviderReference` is a plain `Record<string, string>` rather than a wrapper class, keeping it simple to create and merge - The `isLikelyText` heuristic for media type detection and the `documentMediaTypeSignatures` are kept internal (not exported) — they work well enough for `uploadFile` but are not general-purpose utilities - `resolveProviderReference` (provider-utils) does the lookup by provider name and throws with a clear error listing available providers when the reference doesn't contain an entry for the current provider ### Open questions 1. Should we include a `type` property in `ProviderReference` to distinguish different kinds of provider references (e.g. file vs skill, see #12855)? 2. Should `mediaType` and `filename` be top-level fields in the `uploadFile` result object? - They're currently top-level request parameters, but in the response they're in `providerMetadata`. 3. `mergeProviderReferences` is currently inlined in an example (`multi-provider.ts`) — should we offer this as a utility, or leave it for later? 4. `file-id` and `image-file-id` were removed in `toModelOutput` return value — should we deprecate them instead and/or offer auto-migration via codemod? 5. Out of scope: supporting providers that allow uploading files solely for batch inference (e.g. Cohere, Groq, Mistral), which we don't support at a provider level yet anyway - probably leave for later? ## Manual Verification Upload file examples were added for all 4 supported providers (Anthropic, Google, OpenAI, xAI), each with image, PDF, and text variants. ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work Reuse the new `ProviderReference` approach for #12855. ## Related Issues Fixes #12995Felix Arntz · c29a26f4 · 2026-04-02
- 2.8ETVfeat(harness): support `askUserQuestions` tool including support for normalization across harness adapters (#20267) ## Background Harness question tools expose incompatible inputs and previously had no reliable client-side answer channel. This made structured question flows unusable through `HarnessAgent`. ## Summary - Add the normalized `askUserQuestions` tool contract, including multi-question, partial-answer, free-form, cancellation, and provider metadata support. - Translate native question calls and answers for all supported harness adapters (see below). - Preserve pending question state across suspension and cross-process continuation. - Add an interactive question renderer to the Next.js harness example. - Omit question tools from ACP adapters where the runtime cannot provide an answerable client request. Harness support: - Claude Code: Supported. - Cline: Supported. - Codex: Unsupported — SDK has no question tool. - Cursor: Unsupported — question tool is not exposed in ACP mode. - Deep Agents: Unsupported — SDK has no question tool. - fx: Unsupported — question interaction is disabled in ACP mode. - Grok Build: Supported. - OpenCode: Supported. - Pi: Unsupported — SDK has no question tool. - Example ACP harness adapters: - Claude Code ACP: Supported. - Codex ACP: Supported. ## End-to-End Verification Verified the shared question example against Claude Code, Cline, OpenCode, Grok Build, Codex ACP, and Claude Code ACP. Also verified the interactive flow in `examples/harness-e2e-next`. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work - Add support for additional ACP runtimes when they expose an answerable native question request. - Address plan mode from the original issue separately, tracked in #19944. ## Related Issues Fixes #19755Felix Arntz · 951c54d6 · 2026-09-03