ai — Engineering Performance
22 engineers all time · Jan 2025 – Nov 2026 · built 2026-11-22 · GitHub
Performance snapshot
Today's rolling 90-day reading for ai, compared with the start of the series. Pick a window to move that comparison point.
Avg. perf / dev / mo
+151.1%
2.97 → 7.46 ETV
Active engineers
+120.0%
5.0 → 11.0
Features
−5.7pp
30.2% → 24.5%
vs. Vercel
0.85x
1.9x → 0.85x · −15% below
ai vs. Vercel
Per-engineer ETV for ai against Vercel as a whole. Both lines are 90-day rolling averages scaled to a 30-day month, so they share one axis and can be read against each other at any point. Pick a window to zoom the chart to it.
Performance Composition
Each month's output split by type of work: Features (new value), Maintenance (sustaining systems), Tests, Docs, and Fixes (rework). The yellow line is output per engineer, so when it rises each engineer is delivering more, whatever the team size did. Unit: Engineering Throughput Value (ETV).
Engineering capacity
Effective engineers behind ai, against its pre-AI baseline. Each subject has its own: ai's is 2.97 ETV / dev / mo, its first reading in Q1 2025. Per-engineer ETV divided by that gives a capacity multiple, and that multiple applied to the engineers active in the trailing 90 days turns it into engineer-equivalents. The line is the real headcount, so the gap between line and area is what the leverage is worth. Because each baseline is its own, every subject opens at 1.0x on its first day: multiples measure improvement and are not comparable between subjects.
Knowledge concentration
How dependent is this repo on a small number of engineers? Higher top-1 share = higher key-person risk.
Lars Grammel owns 29.0 % of commits.
Behind the numbers
Written summary of the work completed each month.
No monthly reports available yet.
Top engineers
Most impactful commits
Top 10 by ETV in the all-time window.
- 8.7ETVfeat(harness-acp): introduce ACP harness adapter as a meta adapter to connect to any ACP compatible harness (#18483) ## Background AI SDK harness integrations currently require a dedicated adapter for each coding-agent runtime. ACP provides a common protocol that can support multiple runtimes through one adapter while keeping runtime-specific installation and authentication details in small profiles. Some harnesses only offer an ACP adapter as the sole way for programmatic control over their harness, so for those harnesses having an ACP adapter unblocks supporting them by our harness layer. ## Summary This adds `harness-acp`, a meta adapter for ACP v1 implementations. The adapter owns the generic sandbox bridge, protocol translation, host-tool relay, approvals, and lifecycle behavior, while callers configure the underlying ACP runtime declaratively. - Support simple and lockfile-pinned NPM acquisition for arbitrary ACP implementations. - Support direct and AI Gateway authentication with runtime-resolved environment variables and adapter-specific Gateway routes. - Translate ACP streams, native tools, host tools, approvals, skills, session modes, and resumable lifecycle state into Harness APIs. - Add complete Claude Code, Codex, and Grok Build profiles across documentation and interactive examples. ## End-to-End Verification - Exercised the added Claude Code ACP, Codex ACP, and Grok Build ACP through the interactive Harness example. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Related Issues See #16956.Felix Arntz · ff0f708e · 2026-08-07
- 5.3ETVfeat(harness): consistently support parallel and sequential tool approvals (#23260) ## Background Tool approval behavior varied across harnesses, especially when a model requested multiple tools in parallel or continued with tools sequentially. In particular, parallel tool approvals were not supported at all. ## Summary Coordinates approval handling and step boundaries across the shared harness agent and its adapters, so parallel and sequential tool calls can be approved and resumed consistently. - Defers pausing or completing a step until outstanding host tool executions and their results are settled. - Adds regression coverage for approval, tool result, and step-boundary handling across adapters. - Adds end-to-end examples for parallel tool approvals across harness integrations, including Pi cold resume. ## End-to-End Verification Verified via new `parallel-tool-approvals` example files and interactively via `harness-e2e-next`. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [ ] I have reviewed this pull request (self-review)Felix Arntz · 6c65131f · 2026-10-13
- 4.7ETVfeat(harness): add support for using adapter-native subscriptions where available (#20606) ## Background Harness adapters so far support direct auth via API keys and AI Gateway auth, but not using their native subscriptions. ## Summary This adds host-side native subscription resolution while retaining the existing `direct` and `ai-gateway` authentication modes. Direct authentication prefers explicit provider environment credentials, then falls back to a native subscription if no such credentials are present; AI Gateway authentication never reads native subscriptions. - Add shared OAuth access-token expiry and refresh helpers with a five-minute refresh window. - Resolve supported adapters' native credential files, keyrings, and CLI credentials at the host boundary. - Keep refresh tokens and real access tokens on the host while passing only fake credentials into the sandbox, which then get replaced at the host boundary via credential brokering. - Add native-subscription examples and adapter-specific coverage for every supported credential source and refresh flow. ## End-to-End Verification Ran the `native-subscription` harness examples added. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review)Felix Arntz · cdc12a15 · 2026-09-11
- 3.9ETVfeat: add first-party Code Mode package (#18091) ## Background works towards https://github.com/vercel/ai/issues/18071 the idea is to unify the different provider implementations of programmatic tool calling / code-mode. ## Summary add a new `@ai-sdk/code-mode` package that will export the necessary controls for executing the tools in that format ## End-to-End Verification - ran the example `examples/ai-functions/src/generate-text/code-mode/code-mode.ts` - ran the example `examples/ai-functions/src/stream-text/code-mode/code-mode.ts` - ran the example `http://localhost:3000/chat/code-mode` ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work - api syntax will change - docs need to be added - tool approval mechanism needs to work - workflow boundary concerns ## Related Issues towards #18071 fixes #18141Aayush Kapoor · 29d7b679 · 2026-07-30
- 3.7ETVfeat: agent tui (#15845) ## Background To quickly develop and test agents, it is helpful to have a TUI that requires very limited setup. ## Summary - integrate https://github.com/lgrammel/agent-tui as `@ai-sdk/tui` - add documentation ## Example ```ts await runAgentTUI({ agent }); ``` ## Manual Verification - [x] `pnpm tsx src/agent/openai/tui`Lars Grammel · e757741f · 2026-06-05
- 3.6ETVfeat: Realtime API support for browser<->provider websocket connection (#13893) ## Background Part of https://github.com/vercel/ai/issues/13897 Support for Realtime API has been requested for a long time. We want to support different architectures 1. Websocket connection from browser directly to provider 2. Websocket connection from browser through user's server to provider 3. Websocket connection from browser through gateway to provider 4. Websocket connection from browser through user's server and gateway to provider This pull requests implements the first, see [Architecture](#architecture) below ## Summary Alternative implementation of the Realtime API support proposed in #13889, with a reworked developer-facing API: - **`openai.experimental_realtime('gpt-realtime')`** works in both server and browser (no separate `@ai-sdk/openai/realtime` import needed) - **`openai.experimental_realtime.getToken()`** static method for server-side ephemeral token creation - **`experimental_useRealtime`** hook returns `messages: UIMessage[]` (aligned with `useChat` format) instead of `transcript` - **`inputAudioTranscription`** session config enables rendered user messages for transcribed microphone input - **`addToolOutput(callId, result)`** for client-side tools that need manual result submission - **`onToolCall`** callback for auto-executed client-side tools - Framework-agnostic helper type: `Experimental_RealtimeSetupResponse` Provider implementations included: **OpenAI**, **Google**, and **xAI**. ElevenLabs support is split into a stacked follow-up: **#15747**. <a name="architecture"></a> ## Architecture ```mermaid sequenceDiagram participant Browser participant Server participant Provider as AI Provider<br/>(OpenAI / xAI / Gemini / ...) Browser->>Server: POST /api/setup Server->>Provider: Request token (with tool definitions) Provider-->>Server: Short-lived auth token Server-->>Browser: { token } Browser->>Provider: Open WebSocket (using token) Provider-->>Browser: Audio/text chunks (streaming) Provider-->>Browser: Tool call request Browser->>Server: App-specific tool request (optional) Server-->>Browser: App-specific tool result Browser->>Provider: Send tool result via WebSocket Provider-->>Browser: Continue streaming ``` ### Alternatives considered #### Generic RPC route for server-side tools One option was for the SDK to expose a generic server-side tool execution flow: the realtime session would receive a tool call, POST `{ name, inputs, callId }` to an `execute-tools` route, execute a matching server-side tool, and send the result back to the provider automatically. We decided against that for the initial implementation. A generic RPC route is convenient, but it creates a security-sensitive application boundary: the app must authenticate the user, bind the request to a realtime session, allowlist tool names, validate call IDs, rate limit access, and authorize each tool invocation. If the SDK provides the generic route shape, it is easy to copy into production without those controls. Instead, the SDK keeps tool execution client-driven through `onToolCall` and `addToolOutput`. Server-backed tools should call app-specific API endpoints from `onToolCall` (for example `/api/weather`), where the application can apply its normal auth, validation, authorization, and rate limiting rules. Documentation should cover secure server-backed tool calling patterns, but the SDK should not own the generic remote tool RPC abstraction yet. ## Manual verification The realtime voice example added in this PR lives at `examples/ai-e2e-next/app/realtime/page.tsx` (UI) and `examples/ai-e2e-next/app/api/realtime/[...path]/route.ts` (server-side token endpoint). 1. Add the provider API key(s) for whichever provider(s) you want to test to `examples/ai-e2e-next/.env.local`: ```bash OPENAI_API_KEY=... # OpenAI realtime (gpt-realtime) GOOGLE_GENERATIVE_AI_API_KEY=... # Google realtime (gemini-3.1-flash-live-preview) XAI_API_KEY=... # xAI realtime (grok-voice-latest) ``` 2. Start the example: ```bash cd examples/ai-e2e-next pnpm dev ``` 3. Open http://localhost:3000/realtime, pick a provider + voice, and click **Connect**. Use the microphone or type a message. Ask "what's the weather in Paris?" or "roll a dice" to exercise client-side tool calling (`onToolCall`). ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [ ] I have reviewed this pull request (self-review) - [ ] Security Audit - esp. securing remote tool calling - [ ] Update internal Architecture / how it works docs ### Future Work 1. **ElevenLabs** provider support — stacked follow-up in #15747 2. Websocket connection from browser through user's server to provider 3. Websocket connection from browser through gateway to provider 4. Websocket connection from browser through user's server and gateway to provider ## Related issues - #3176 - #3907 - #4082 - #5007 - #9559 - #12381 - #13706 - #13889 Co-authored-by: Cursor <cursoragent@cursor.com>Gregor Martynus · ce769dd2 · 2026-06-05
- 3.5ETVfeat(harness-fx): reimplement adapter to use `libfx`, achieving clean host-sandbox separation and unlocking additional capabilities (#23362) ## Background The ACP-backed fx adapter ran entirely inside the sandbox and had several limitations due to ACP lacking certain primitives. This PR reimplements it using stable `libfx`, running the harness and model requests on the host while keeping built-in filesystem and shell operations exclusively in the sandbox, similar to Pi and Cline. It also makes the fx harness respond notably faster. Additional capabilities unlocked from this migration that weren't supported in the original adapter: - Built-in tool filtering - Mid-turn steering - Structured output - Tools with `toModelOutput()` results that include images Most importantly, the hands are now separate from the brain, which is the preferred operation mode for harnesses. ## Summary - Remove the sandbox bridge, runtime installation, and exposed-port requirement. - Support built-in tool filtering, structured output, image tool results, and mid-turn steering. - Collect parallel custom and built-in tool approvals together, with durable results and approvals across suspend/resume. Built-in execution remains sequential. - Update authentication, runtime settings, MCP/skills integration, docs, and examples for the host-based adapter. ## End-to-End Verification Ran all fx examples, including interactive testing via `harness-e2e-next`. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review)Felix Arntz · a0cd4421 · 2026-10-15
- 3.5ETVfeat(video): externalize polling control and webhook support for generateVideo (#12515) - Adds optional `doStart`/`doStatus` methods to the experimental `VideoModelV4` spec, enabling a two-phase async model where the SDK core orchestrates polling or webhook-based completion instead of each provider implementing its own polling loop. - Adds `poll` and `webhook` parameters to `experimental_generateVideo` for user-controlled polling intervals/timeouts and webhook-based completion. - Adds optional `handleWebhookOption` to `VideoModelV4` so a provider can signal native webhook support. The SDK only invokes the user's webhook factory when the model implements it; otherwise it falls back to polling. - Replaces `doGenerate` with `doStart`/`doStatus` in the first-party models implementing `VideoModelV4` (FAL, Alibaba, KlingAI, Replicate, xAI, Google, and Google Vertex). The gateway video model remains a `doGenerate` passthrough. - Makes `doGenerate` optional on `VideoModelV4`. Third-party providers implementing only `doGenerate` continue to work unchanged, and the SDK falls back to it when no async flow is selected. - Constrains operation handles to `JSONValue`, making the provider-owned task reference serializable and suitable for future persistence and gateway support. Towards https://github.com/vercel/ai/issues/12381. ## Architecture ``` User calls: generateVideo({ model, prompt, poll?, webhook? }) ↓ Core function: Chooses flow based on model capabilities and user options ↓ ┌───────────┴───────────────┐ │ │ [Legacy / fallback] [Start/Status flow] │ (poll or webhook provided, │ or doGenerate missing) │ │ model.doGenerate(options) model.doStart(options) ↓ ↓ Provider owns polling returns { operation } ↓ ┌───────────┴────────────┐ │ │ [native webhook] [polling] │ │ await notification SDK core loop │ │ model.doStatus(op) model.doStatus(op) └───────────┬────────────┘ ↓ return result ``` The SDK core owns the polling lifecycle. Providers only implement single-shot `doStart` (submit) and `doStatus` (check once). The polling interval and timeout are controlled by the user. ## Review decisions - **`poll` and `webhook` are intentionally composable.** When a model supports native webhooks, `poll.timeoutMs` limits the webhook wait. When it does not, the same polling configuration controls the automatic fallback. This behavior is documented and covered by tests. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2821429102)) - **The first iteration uses fixed polling intervals and timeouts.** Exponential backoff and `onAttempt` were removed to keep the API small; aligning polling with shared retry/backoff primitives can be revisited separately. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2821435402)) - **Operation handles are JSON-serializable.** `operation` is typed as `JSONValue` instead of `unknown`, while remaining opaque to core. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843221880)) - **Shared cross-modality operation types are deferred.** The current types remain video-specific until another modality implements the same lifecycle, avoiding a premature provider-spec abstraction. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843151286)) - **Replicate's server-side `wait` remains provider-specific.** It affects how long Replicate holds the submission request open, while top-level `poll` controls the SDK's provider-neutral status lifecycle. It can be added through Replicate provider options independently if needed. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843203139)) ## Verification - [x] Tests added and updated - [x] Documentation added and updated - [x] Patch changeset added - [x] TypeScript, lint/format, and code consistency checks pass locally ## Future work - Expose async operations as a first-class API: start an operation, persist its handle, and resume or query it later. ([discussion](https://github.com/vercel/ai/pull/12515#discussion_r2843215786)) - Generalize operation types after another model modality implements the same lifecycle. - Add AI Gateway transport for the JSON-serializable start/status operation flow. - Explore a multi-notification webhook receiver if providers need intermediate status events rather than a single completion notification. ## Related issue https://github.com/vercel/ai/issues/12381 --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: josh <144584931+dancer@users.noreply.github.com> Co-authored-by: vercel-ai-sdk[bot] <225926702+vercel-ai-sdk[bot]@users.noreply.github.com> Co-authored-by: mat lenhard <mclenhard@gmail.com> Co-authored-by: Rohan Taneja <47066511+R-Taneja@users.noreply.github.com> Co-authored-by: Felix Arntz <felix.arntz@vercel.com> Co-authored-by: Walter Korman <shaper@vercel.com> Co-authored-by: Nico Albanese <49612682+nicoalbanese@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>Gregor Martynus · 79e133cc · 2026-08-03
- 3.4ETVfeat(harness): allow changing harness settings between turns via `prepareCall()` support on `HarnessAgent` (#19828) ## Background `HarnessAgent` settings were fixed when an agent was created, preventing callers from adapting instructions, skills, and tools between completed turns. `ToolLoopAgent` already supported `prepareCall()` and `callOptionsSchema`, allowing to dynamically modify certain settings in between turns. ## Summary This aligns `HarnessAgent` with the `ToolLoopAgent` call-options pattern while keeping settings fixed throughout an active turn and its continuations. - Add `callOptionsSchema` and `prepareCall` support for deriving per-turn instructions, skills, and tools. - Capture active turn settings in continuation state so suspended and resumed turns cannot pick up later configuration. - Update harness adapters to apply settings changes between turns, including ACP skill refreshes and instruction guidance. - Make `writeSkills` safely replace only harness-owned skills and skip unchanged content using a manifest with SHA-256 hashes. - Add documentation, unit coverage, and changing-settings examples for every harness. ## End-to-End Verification Ran the new `changing-settings` examples against all harnesses. ## Checklist - [x] All commits are signed (PRs with unsigned commits cannot be merged) - [x] Tests have been added / updated (for bug fixes / features) - [x] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review)Felix Arntz · 14d4fc03 · 2026-08-27
- 3.3ETVfeat: add provider versions to user-agent header (#8703) ## Background This change is added to include the provider versions in the user-agent header that we now pass. Previously, the string only appended `ai` version and runtime env ## Summary - add a `version.ts` file - export version during build by changing `tsup.config` - export verison during tests by changing `vitest.config` - mock a dummy version for tests ## Manual Verification Verified by making changes to the test file ## Tasks - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] Formatting issues have been fixed (run `pnpm prettier-fix` in the project root) ## Future Work Version in user-agent added for: - [x] `gateway` - [x] `openai` - [x] `anthropic` - [x] `google` - [x] `google-vertex` - [x] `azure` - [x] `amazon-bedrock` - [x] `cohere` - [x] `mistral` - [x] `groq` - [x] `cerebras` - [x] `deepinfra` - [x] `deepseek` - [x] `fireworks` - [x] `perplexity` - [x] `replicate` - [x] `togetherai` - [x] `xai` - [x] `vercel` - [x] `openai-compatible` - [x] `elevenlabs` - [x] `assemblyai` - [x] `deepgram` - [x] `gladia` - [x] `revai` - [x] `luma` - [x] `fal` - [x] `hume` - [x] `lmnt` - [x] `langchain` - [x] `llamaindex` - [x] `valibot` ## Related Issues fix in progress for #8699 --------- Co-authored-by: Gregor Martynus <39992+gr2m@users.noreply.github.com>Aayush Kapoor · 1cad0aba · 2025-09-23