workflow — Engineering Performance
18 engineers all time · Oct 2025 – Nov 2026 · built 2026-11-22 · GitHub
Performance snapshot
Today's rolling 90-day reading for workflow, compared with the start of the series. Pick a window to move that comparison point.
Avg. perf / dev / mo
+344.0%
2.13 → 9.46 ETV
Active engineers
−22.2%
9.0 → 7.0
Features
−23.2pp
47.4% → 24.2%
vs. Vercel
1.1x
1.4x → 1.1x · +8% above
workflow vs. Vercel
Per-engineer ETV for workflow against Vercel as a whole. Both lines are 90-day rolling averages scaled to a 30-day month, so they share one axis and can be read against each other at any point. Pick a window to zoom the chart to it.
Performance Composition
Each month's output split by type of work: Features (new value), Maintenance (sustaining systems), Tests, Docs, and Fixes (rework). The yellow line is output per engineer, so when it rises each engineer is delivering more, whatever the team size did. Unit: Engineering Throughput Value (ETV).
Engineering capacity
Effective engineers behind workflow, against its pre-AI baseline. Each subject has its own: workflow's is 2.13 ETV / dev / mo, its first reading in Q4 2025. Per-engineer ETV divided by that gives a capacity multiple, and that multiple applied to the engineers active in the trailing 90 days turns it into engineer-equivalents. The line is the real headcount, so the gap between line and area is what the leverage is worth. Because each baseline is its own, every subject opens at 1.0x on its first day: multiples measure improvement and are not comparable between subjects.
Knowledge concentration
How dependent is this repo on a small number of engineers? Higher top-1 share = higher key-person risk.
Peter Wielander owns 31.6 % of commits.
Behind the numbers
Written summary of the work completed each month.
No monthly reports available yet.
Top engineers
Most impactful commits
Top 10 by ETV in the all-time window.
- 34.7ETV[core] Single-orchestrator runtime (spec 9) (#4661) **Spec 9 is opt-in.** New runs are spec 8 by default and behave as on `main`. Setting `WORKFLOW_SPEC_VERSION=9` creates spec 9 runs on a World that declares the in-band fence (world-local, world-postgres, world-vercel). Everything that reads, routes or writes to an existing run follows that run's stamped version, never the setting, so turning it off is safe for runs already created as spec 9. This PR adds a single-orchestrator model (spec version 9). One orchestrator delivery per run makes every decision, and step invocations only run step bodies and report outcomes. Step and wait events become plain appends that the runtime never relies on the World to refuse. The World fences the orchestrator as the only in-band writer. The spec 8 orchestrator stays, side by side. Each delivery is routed by the run's stored spec version, so runs created before opting in finish on spec 8, and spec 8 runs keep `main`'s behavior, including `main`'s queue topology and events API routes on world-vercel. See https://workflow-sdk.com/docs/changelog/spec-9Peter Wielander · 3bf8cd97 · 2026-10-29
- 8.6ETVdocs: apply Vercel technical writing standards (#3704) * docs: apply Vercel technical writing standards Audit the complete documentation corpus, package READMEs, skills, and source TSDoc/comments against the vercel-technical-writing skill and style-rules.md. Normalize sentence-case headings without changing published anchors, remove prose em dashes and filler wording, improve active voice and self-contained phrasing, standardize product/brand capitalization, American English, list punctuation, units, and code fence languages, and preserve exact runtime strings/table placeholders. All executable code is unchanged. Modified skills have their metadata versions bumped. * docs: extend writing audit to repository Markdown Apply the same technical-writing rules to design documents, compiler specifications, workbench guides, package changelogs, and the remaining tracked Markdown outside the deployed docs corpus. Preserve historical meaning, commands, output literals, table placeholders, and heading anchors. * docs: exclude generated package changelogs from auditNathan Rajlich · e1e64e3d · 2026-08-21
- 7.3ETVfeat(core): find and apply changes between versions of a frozen value (#5214) Base of a stack: this PR ← #5215 ← #5234 ← #5216. ## Problem Step arguments and return values are stored by value, so state that grows across steps (an agent's conversation history, say) is stored again in full by every step that receives it. Total storage grows with the square of the number of steps, and large payloads go to slower storage. ## Solution The stack stores **deeply frozen** values by reference: a step input that holds a value Workflow already stored becomes a reference to it, and a step output becomes only its change. Users make frozen values with immer's own `produce`, or `Object.freeze`. There is no new API. This PR adds the internals that the next one builds on, with no behavior change: - `diffState(base, next)`: the changes between two versions of a frozen value, found by identity (an object present in both is unchanged) and compared by value elsewhere. Growth at the end of an array is one `append`; key order changes are an `order` op; sparse arrays are replaced whole. - `applyStateOps(base, ops)`: rebuilds the next version, copying only the containers on changed paths. - The change format (`set`, `del`, `append`, `order`) is Workflow's own, so stored runs never depend on an immer version. immer is a dev dependency, used by the tests only. ## Validation `state.test.ts`: changes recorded and applied for appends, deletes, key order, sparse arrays, rebuilt copies and own properties named `__proto__`, `constructor` and `prototype`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com> Co-authored-by: VaguelySerious <mittgfu@gmail.com>Karthik Kalyan · 1bcd7ef5 · 2026-11-10
- 5.2ETV[world-local] Store the Local World in one SQLite database (#5235) Closes #5083 Closes #5176 Closes #5161 Closes #4203 Closes #4830 Supersedes #5153 (a fix for the same crash windows as #5083, inside the file-based storage this PR replaces; closing keywords don't close pull requests, so it needs closing by hand once this merges). Implements the SQLite world-local from the design in #5230. It supports both spec 8 (the default) and spec 9 (`WORKFLOW_SPEC_VERSION=9`). It also adds a spec 9 e2e lane for world-local on both workflow VMs, mirroring `e2e-vercel-spec9`. ## What changes **One database file: `<dataDir>/world.db`.** It is opened with the runtime's built-in driver (`node:sqlite` on Node ≥ 22.13, `bun:sqlite` on Bun), so there is no native dependency. It runs in WAL mode, and every World write is one short synchronous `BEGIN IMMEDIATE` transaction. **Storage** (`src/storage/events-storage.ts`, `store.ts`) - `events.create` reads what it validates against, appends the event(s) at `seq + 1`, and updates the run, step, wait, hook and token rows in the same transaction. A crash rolls everything back, so there are no claim files, markers, staging slots, repair paths or stale-lock rules. - The validation rules and errors are the same as before. They were ported from the old implementation and cover: - terminal-run guards and step ordering; - lazy and spec 9 creation-on-start; - `TooEarlyError`; - hook token uniqueness, retention and force-claim (the victim's `hook_disposed` is written into its own log in the same transaction); - `(runId, resumeId)` dedup; - start-Hook admission (an accepted candidate's run, Hook and `run_created` commit together; a refusal is recorded and committed before it is thrown); - `attr_set` dedup and `$retention: '0'` purge; - preload, `sinceCursor`, skipped-slot reports and the event limit. - **In-band fence (spec 9):** `seq_in_band` is a column on the run row, checked and advanced in the write's transaction. A failed write allocates nothing; there is no more "count it in case it landed". **Queue** (`src/queue.ts`): unchanged from main. It stays in memory and push-based: an enqueue is an HTTP call to the app, or a call to an in-process handler, from the process that enqueued it. That process holds delays, `timeoutSeconds` redeliveries and retries in memory. Nothing is stored. - **Startup recovery** is as on main. Every active run of the World's tag is re-enqueued with the recovery marker, after its orphaned inline steps are released (`releaseOrphanedSteps`, as on main: the interrupted step gets a `step_retrying`). That delivery re-arms the run's lost wait timers: a spec 8 replay suspends on its open waits again, and for spec 9 the marker makes the delivery re-send step messages and timers (chained in hops for long waits). Core's restart-recovery test of a sleeping run now covers both specs and both VMs. - **New, a backstop:** each recovered run also gets a plain wake for every open wait due within the queue's longest timer (about 24.8 days), due when the wait is. A wait due later gets none, because a plain wake would fire early and a spec 9 run would not treat it as a hop of the wait's chain. The recovery tests pass with the backstop turned off, so it is insurance, not the mechanism. A spec 9 wait therefore gets two wakes after a restart; one is a no-op. - **New, `close()`:** it no longer aborts a message the app has not received yet. A short-lived sender (`wf start`, a script calling `start()`) closes its World right after `start()` returns, and main aborted the first delivery at once, so the run it had just created could stay `pending`. Now `close()` waits for a message of a run until the app has written to that run (its `seq` moved past where it was when the message was sent), up to 10 s, enough for a dev server compiling the flow route on the first request. A message without a run gets up to 1 s. Main has the same race, but its slower storage almost always hid it. **Streams** (`src/streamer.ts`): chunk index assigned at commit; in-process signal plus a 100 ms poll for other processes. The poll and the end-of-stream check are primary-key seeks (the end marker is always a stream's last row, and an ended stream takes no more writes), so an idle reader costs the same however long the stream is (#4830). **Several processes on one data directory** (an app and a test runner, `@workflow/vitest`'s workers): a write waits for another process's write lock without blocking the thread. `db.transactAsync()` tries `BEGIN IMMEDIATE` with `busy_timeout = 0` and retries after yielding to the event loop, then runs the synchronous body and commit in one tick; opening a connection on that path doesn't wait either. **Other** - Earlier releases' JSON data (`runs/`, `events/`, `.locks/`, …) is not migrated and never deleted: it stays in the data directory, unread, next to `world.db`. When `world.db` is created next to that data (the first write after the upgrade, by an app, the CLI or a script), a one-time `console.info` says the earlier runs are no longer read and names the directories that can be deleted. - A read never creates the data directory or `world.db`: until the file exists, reads see an empty database (the CLI and the web UI pointed at an app that has not run yet write nothing). - Connections: kept open while in use and closed after a few idle seconds; on Windows (`brief`) a connection closes 25 ms after its burst of operations, so a data directory can still be removed. `WORKFLOW_LOCAL_SQLITE_KEEP_CONNECTIONS=1|brief|0` overrides. - ID validation (`UnsafeEntityIdError`) is kept so the same IDs stay valid. - `clear()` returns the freed space to the file system (`auto_vacuum = INCREMENTAL`, an incremental vacuum and a checkpoint after it), and the WAL is kept to 64 MiB after a checkpoint. - One `events.list` page ends early, with a cursor, once it holds about 8 MiB, so a page of large payloads doesn't hold the thread while it decodes. - Node.js 22.x's `ExperimentalWarning` for `node:sqlite` is dropped (only that warning). - Removes `proper-lockfile`. (`async-sema` stays: main's queue uses it.) - Non-test source goes from ~13.1k lines to ~6.8k. Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>Peter Wielander · c11fc400 · 2026-11-12
- 5.1ETVAdd dynamic workflows, backed by encrypted ref storage (#2062) * feat: dynamic workflows, backed by encrypted ref storage Rebase of the dynamic-workflow-source prototype onto main, reworked to use the real server-side storage path instead of stashing generated code in plaintext run metadata. `start()` now accepts workflow source as a string, for orchestration whose shape is only known after you deploy — a workflow-builder UI, a customer-defined automation, an AI-generated plan over a fixed catalog of registered steps: await start(source, [{ userId }], { dynamic: { steps: { fetchUser } } }); The source is validated, compiled to workflow VM code, and stored with the run through the same pipeline as the run's input: compressed, then encrypted with the run's key. On worlds with a blob-ref layer it lands behind a ref — inline for a small definition, object storage for a large one (see the matching workflow-server PR). Replay hydrates it and evaluates *it* instead of the deployment's bundle, so a run always executes the exact code it started on. What changed from the prototype: - Generated code no longer rides `executionContext.dynamicWorkflow`. Only small plaintext metadata does — version, source hash, export name, and the alias-to-step-id map — which is what lets a run be identified as dynamic, and its step allowlist audited, without decrypting anything. It also has to fit the 2 KB execution-context budget, which real generated code does not. - The code rides `run_created` (and the queue message, for resilient start and the turbo path, which synthesizes the run snapshot rather than reading it) as a serialized payload, or as a ref for definitions too large to send inline. - `start()` verifies the created run came back carrying its code. A backend without dynamic-source storage drops the field rather than rejecting it, so without this the run would look fine and fail much later as an unregistered-workflow error on a queue delivery. - Compilation moved to its own module. The workflow id is derived from the source *and* its step bindings, so the same definition always gets the same durable id, and arbitrary source cannot claim a static workflow's. Docs: new "Dynamic Workflows" page. The v5 "How it works" section is renamed "Advanced" to fit it — the section explains the build-time execution model, and dynamic source is the escape hatch from it. Permanent redirects cover every `/v5/docs/how-it-works/*` slug; v4 keeps its own section at the unversioned path, so those are deliberately untouched until v5 becomes the default. Tests: 35 unit tests over compilation, id derivation and validation; 4 type-level tests pinning the overload resolution; 7 e2e tests over the real round trip (generated code executing against registered steps, stable ids, encrypted-at-rest storage, replay across a suspension, the deferred ref path, the step allowlist failing a run, and validation refusing to create one). The e2e file is added to `test:e2e`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * fix(web-shared): render a dynamic run's workflow code in the run detail view The run detail sidebar keys its display map off `WorkflowRun`'s fields via an exhaustive `Record<AttributeKey, ...>`, so adding `dynamicWorkflowCode` to the run schema broke the build until it had a renderer — which is the right outcome: a payload field with no renderer would otherwise be silently dropped from the one view that exists to show it. It gets the same locked-by-default treatment as input and output, which is what the RFC asks for: generated orchestration names internal step ids, prompts and business rules, so reading it goes through the existing decrypt flow rather than being displayed by default. Ordered after input/output — the code a dynamic run actually executed is the most useful thing on its detail view, and absent on every other run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * fix: point @workflow/core's packed docs at the renamed Advanced section `prepack` copies the v5 docs tree into the published package, and it named the section by its old path, so `pnpm pack` — and with it the tarballs preview deployment — failed on a missing directory rather than on anything about the docs themselves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * docs: make the dynamic-workflow approval sample self-contained The docs type-checker checks each code block on its own, so a sample that leaned on the imports of the block above it did not compile — and neither would a reader who copied it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * docs: use Run.status in the dynamic-workflow example There is no readStatus(); status is a promise-returning getter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * test(e2e): skip dynamic-workflow tests on a backend without source storage Two fixes the first Vercel-prod run surfaced. The unsupported-backend path threw instead of skipping, so the whole dynamic suite came back red against production — which does not have the server-side storage deployed yet. It now issues a real vitest skip, the same mechanism the conformance gate uses: the gap is the backend's and the suite cannot close it, and these tests go live unchanged once the server ships. The validation test also called `getWorld()` without awaiting it, then read `.runs.list` off a promise. That assertion is gone rather than fixed: the run listing is served from an eventually-consistent store in e2e, so comparing it before and after would be a flake, and the ordering it was checking — that validation happens before any write — is pinned deterministically by the unit tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * test(e2e): read run records through the World, and skip the suspension case Three problems the first full e2e matrix surfaced, all in the test file. The assertions on stored state went through `getRun()`, which returns a Run handle whose fields are promise-returning getters over a small public surface — so `workflowName` compared two promises, and `executionContext` / `dynamicWorkflowCode` read as undefined. They now read the persisted record through `world.runs.get(runId, { resolveData: 'all' })`, which is what the rest of the suite does for storage assertions. The suspension test is skipped and left in place with the observation written down: it timed out waiting on `returnValue` on `world-local` while every non-suspending dynamic test passed on the same world, so dynamic execution works and something about the resume delivery does not. That delivery takes the read-back path in `resolveWorkflowCodeForRun` that no passing test covers. Not root-caused, and it should be before this ships beyond experiment — deleting the test would hide the finding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * fix(world-local): keep a dynamic run's code across status transitions world-local rebuilds the run record field by field on every status change, so anything not named in those four literals is dropped the first time the run moves — which for a dynamic run meant losing the workflow code the moment it went `running`, leaving a run nothing could replay. Same failure mode the neighbouring `encryptionPublicKey` lines were added to avoid, and the reason the suspension e2e test hung rather than failed: the resume delivery is the only path that reads the code back off the record, and it found nothing, so the delivery threw and was redelivered until the test timed out. That test is re-enabled here — it is what caught this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * test(e2e): skip the stored-code assertion when the backend persisted nothing Two of the twenty-seven Vercel apps got past `start()`'s fail-fast check and then failed on the stored bytes instead of skipping. That is accurate, not noise: the check reads the created run off `run_created`'s own response, and a resilient start has no response to read — so a short dynamic run can still finish, because turbo executes it from the queue message inside one invocation, while the backend stored nothing. This assertion is the thing that notices. It is the same "no dynamic-source storage" fact the other tests skip on, reached one step later, so it skips with that stated instead of failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * test(e2e): apply the stored-code skip to both assertion sites The deferred-ref test asserts stored bytes too, and only the first site got the skip — so it failed on the two apps whose runs happened to take the resilient-start path. Both now go through one helper, so a third site cannot drift from it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * chore: re-run CI The previous Vercel e2e matrix hit a backend-wide slowdown — every Vercel-facing job failed on 60s timeouts in pre-existing tests (promiseRaceStressTestWorkflow, closureVariableWorkflow, the abort suite) while all 84 local-world jobs passed. Empty commit to get a clean read, since re-running the failed jobs directly is not permitted for this token. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> * test(e2e): emit the dynamic runs' IDs as a CI artifact The dynamic tests track every run they start, but never wrote the sidecar that carries the IDs out — each e2e file runs in its own vitest worker with its own copy of the collector, and only `e2e.test.ts` was writing one. So the run IDs were collected in memory and thrown away, which is precisely what you want when you ask what a dynamic run looks like in production. Written under `e2e-dynamic-runs-<app>-vercel.json` rather than the shared per-app metadata name, which the last writer would otherwise clobber, and added to the four artifact upload lists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * test(e2e): generate the dynamic workflows inside the deployment The dynamic e2e tests called `start(source, …)` from the vitest runner. That runs, but it is not the shape an app uses, and it skipped the most important part of the feature: `dynamic.steps` has to be given *imported* step functions, so the `.stepId` the build-time transform stamped on them is what binds generated source to a registered step. The runner holds no handle on those functions, so it had to look step ids up in the deployed manifest and pass `{ stepId }` — the escape hatch, not the documented path. The workflows are now generated inside the deployment, by fixtures in `workflows/99_e2e.ts` (one file, symlinked into every workbench app). Each fixture assembles source at runtime and starts it from a step with `{ steps: { add } }`, which is how a builder UI or an LLM-generated plan actually reaches `start()`. The tests start the static fixture, take the dynamic child's run id back, and assert on the child — so the same tests exercise Vercel, local dev/prod and Postgres with no per-world branching. Fixtures added: a plain generated workflow, one whose source suspends (the resume delivery is the only path that reads the stored code back), and one whose source calls a step it was not given. `getStepMetadata` goes with them — it existed only for the manifest lookup this replaces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * fix: accept imported step functions in dynamic.steps `DynamicWorkflowStepReference` was `{ readonly stepId: string }`, so the call the docs advertise — `steps: { fetchUser, sendEmail }` with real imports — did not typecheck. The build-time transform stamps `.stepId` on step functions at runtime and nothing adds it to their declared type, so the only thing the type actually accepted was the `{ stepId }` escape hatch. Moving the e2e fixtures into the deployment is what surfaced this: app code is typechecked by the app's build, and the runner had only ever passed the escape hatch. The function arm is typed as any function, with the existing runtime check in `resolveStepId` as the real gate — it already reads `.stepId` off functions and fails with a message naming the alias. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * fix: name the created run when its dynamic code was not stored The error said the code was not stored but not which run it was talking about — and `start()` writes `run_created` *before* reading back what the backend kept, so on this path a real dynamic run does exist and only its replayability is missing. Naming it is what makes it inspectable. The e2e skip path now pulls that id out and tracks it, so the run-ID sidecar reports the run even when the test skips. Without it the ids of dynamic runs created against a backend that cannot store them were simply lost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * test(e2e): capture the child's run id, not the wrapping parent's The error reaches the runner wrapped in the parent fixture's failure, which names the parent, so a loose `wrun_` match took that instead — every run id in the sidecar was the parent's, recorded twice. Anchored to the exact wording of the message `start()` raises about the run it created. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * test(e2e): label the dynamic run in the run-id sidecar The sidecar lists two bare ids per test — the parent fixture's and the dynamic child's — with nothing saying which is which. The dynamic one is the whole reason the artifact exists, so it is now tagged `[dynamic run]`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude <noreply@anthropic.com> * fix: stop the module-syntax check rejecting identifiers that start with import/export `import` was allowed to be followed directly by an identifier character, so a line starting with `importData = …` was rejected as module syntax. The keyword now has to be followed by whitespace or a module-syntax token. `export{` and `export *` without a space are caught as well. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(dynamic-workflows): purge stored code, refuse Replay Run, parse source up front, hydrate code for o11y - $retention: 0 purges clear dynamicWorkflowCode on world-local and world-postgres alongside the other run payloads. - Replay Run refuses a dynamic run instead of creating one with no code behind it. - compileDynamicWorkflow parses the generated code, so TypeScript syntax, a reserved-word function name, or an unbalanced brace fail at start() rather than on every delivery; inline "use step" is rejected as the docs already state. - The stored code payload is real devalue under its devl prefix, so the generic hydrators (CLI inspect, web UI) can read it; hydration field lists in core, web-shared, and the CLI include dynamicWorkflowCode. - hydrateDynamicWorkflowCode normalizes a corrupt payload to SerializationError. - Changeset covers @workflow/cli and @workflow/web-shared; bump is minor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(dynamic-workflows): preflight runtime and backend support Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(dynamic-workflows): keep source out of local events Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(web): disable replay for dynamic runs Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * docs(dynamic-workflows): qualify source storage guarantees Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * test(core): exercise dynamic source compression Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * docs(dynamic-workflows): qualify changeset storage wording Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(web): guard Replay while run identity loads Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * docs(dynamic-workflows): clarify cross-deployment key preflight Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * feat(core): mark dynamic start options experimental Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(world): reject ambiguous dynamic code storage Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(core): isolate dynamic workflow compilation Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * docs(dynamic-workflows): explain runtime surface Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(core): validate top-level dynamic workflows Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(core): validate generated dynamic wrapper Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(core): isolate dynamic source bindings Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(core): pin dynamic syntax and preserve aliases Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(dynamic-workflows): validate queue names before start writes Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(core): gate dynamic starts on deployment opt-in and same deployment Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(core): fail dynamic deliveries that cannot safely execute stored code Require the deployment opt-in, a marker that derives the run's workflow name, and encr-only stored code when the run has key material. Each refusal records run_failed instead of redelivering, and executing stored code is recorded on the delivery span and in one log line. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(core): validate execution context size for dynamic starts only Static starts no longer serialize or reject their execution context before creation. world-vercel throws a WorkflowRuntimeError that names the dynamic step bindings as the cause. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(world-vercel): read a missing capabilities route as no capabilities A 404 from /v2/capabilities maps to an empty set so dynamic starts fail closed with the backend-support error; other failures still propagate. The dynamic E2E suite skips when the deployment has not opted in or its backend does not advertise dynamic-source storage. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(world): omit dynamic workflow code for resolveData none WorkflowRunWithoutData excludes dynamicWorkflowCode, and the local, Postgres, and Vercel Worlds strip it from get, getMany, and list. world-local list now filters through filterRunData. Replay reads the code back with resolveData all when a snapshot lacks it. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * docs(dynamic-workflows): document opt-in, same-deployment scope, and security model State that dynamic source runs with the deployment's full privileges and that steps is not a boundary, drop the customer-authored and AI-generated framing, and document the WORKFLOW_EXPERIMENTAL_DYNAMIC_WORKFLOWS opt-in, same-deployment-only starts, encr-only stored code, plaintext storage in Local and Postgres, the plaintext step map, and the 2,048-byte marker limit. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * refactor(core): drop the redundant dynamic runtime-version check from start Dynamic starts are same-deployment only and require the opt-in up front, so the start-local target dynamic version could only echo this process's own constant. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * ci: run dynamic workflow E2E on local and Postgres lanes The local dev, local prod, and Postgres servers opt in with WORKFLOW_EXPERIMENTAL_DYNAMIC_WORKFLOWS=1, and WORKFLOW_E2E_EXPECT_DYNAMIC_WORKFLOWS=1 makes the suite fail rather than skip on an opt-in refusal there. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(world-vercel): skip decompressing dynamic workflow code on resolveData none reads Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> * fix(world-vercel): classify corrupt dynamic workflow code as a contract error Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> --------- Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Alex Langenfeld <alex.langenfeld@vercel.com>Pranay Prakash · 20e8440f · 2026-09-29
- 4.0ETVWorld-side incrementing event ID (specVersion 6) (#3389)Peter Wielander · 6786db99 · 2026-08-11
- 3.8ETVfeat(hooks): `createHook({ experimental_force: true })` takes a held token over (#4193) * feat(hooks): createHook({ experimental_force: true }) takes a held token over A run can now register a hook token another active run holds. The World disposes the previous owner's hook in that run's own event log (`hook_disposed{forceClaimedBy}`), re-points the token, and returns the new hook with `claimedFrom`; the claimer's runtime then wakes the previous owner from that record. A run awaiting a hook whose token was taken over gets the new `HookForceClaimedError` naming the claimer, after draining the payloads that landed before the takeover. Co-authored-by: Peter Wielander <mittgfu@gmail.com>Pranay Prakash · 4f524386 · 2026-09-23
- 3.0ETV[core] Wait on child runs through a system hook instead of polling (#5048) ## Summary `await run.returnValue` on a run the current workflow started now suspends the parent on a **system hook** (`isSystem: true`, one per parent, shared by all of its children) that each child resumes when it finishes. Until now that await was a step that polled the child (long-polling where the World supports it), and the step held a worker slot for the whole wait. The docs told people to avoid this by hand-writing the hook handshake (`defineHook` + `withChildCompletionHook` wrapper + `startAndWait`). The SDK now does that handshake itself: ```ts export async function parent(ids: string[]) { "use workflow"; const runs = await Promise.all(ids.map((id) => start(child, [id]))); return await Promise.all(runs.map((run) => run.returnValue)); // no polling, no worker slot } ``` ## How it works 1. **`start()` from a workflow** (directly, or from one of its steps) stamps the parent's run id on the child (`executionContext.completionParentRunId`) and records it on the returned `Run`. It survives `Run` serialization, so a `Run` returned from a step of the parent works the same way. 2. **One system hook per parent.** When the parent awaits a child's `returnValue` (workflow VM only, and only in the run that started it), a per-run dispatcher registers the parent's completion hook, `wkf_child_run_completion:<parentRunId>`, created with `isSystem: true`, if it isn't open already. Once registration is committed, it reads every child awaited since its last read in **one** batched step (`Run.readFinishedRunIds`). Children that already finished are answered from that read. 3. **Every child resumes that same hook** with `{ runId, status }` **after** it writes `run_completed` / `run_failed`. This covers every terminal write site on both engines, including fatal setup errors, max-deliveries, replay timeout, and deployment mismatch. `Run.cancel()`, `cancelRun()` and `cancelRuns()` do the same for cancellations (the batch path once per cancelled run). The dispatcher hands each payload to the waiting child, which then reads its own result (`Run#readReturnValue`). 4. **Once no child is being waited on, the hook is disposed.** An open system hook makes every suspension of the parent path-changing, so leaving it open would turn off spec 9's run-ahead for the rest of the parent's run; closing it also stops children the parent never awaits from waking it. The next await registers it again under the same token. Resuming a token nobody registered is a cheap not-found and not an error. Why this ordering is race-free: the parent registers before it reads, and a child resumes after its terminal write. A resume that finds no hook therefore happened after the terminal write, and the parent's read, which happens after registration, sees it. **Per-child cost in the parent's log** for a fan-out of N: one `hook_received` plus one result-read step per child, plus a fixed one hook, one batched status read and one safety-net wait for the whole fan-out. The old polling path cost one long step per child, holding a worker slot. ### System hooks in the runtime The completion hook is a real system hook (`isSystem: true`). I fixed the spots that assumed every system hook is an `AbortController` hook: - **QuickJS `__releaseDisposedHook`** never released a disposed system hook's op. That exception exists so `abort()` can find its hook later, so it now applies to abort hooks only (a VM-local `isAbortSignal` marker). Without this, every dispose/re-register cycle of the completion hook leaked a pending op in the VM. `quickjs-system-hook-dispose.test.ts` fails on the old guard. - **Out-of-band observation**: adds `systemHookCount` (every SDK hook) next to `abortSignalHookCount` (abort hooks only). Both count as never inert, because other runs and steps resume them. The existing span attribute `workflow.out_of_band.abort_signal_boundaries` keeps its meaning, and a new `workflow.out_of_band.system_hook_boundaries` covers the completion hook too. - Terminal drain (node and QuickJS) already skips uncreated system hooks generically. I only corrected comments there and in the abort controller, which claimed `isSystem` exempts a hook from token-conflict checks. No World does that; Worlds only persist the flag. - `createHook` accepts an internal `isSystem` option (`SystemHookOptions`, kept off the public `HookOptions`) on both engines. ### Safety net for missed notifications While a parent waits on children, the dispatcher keeps **one** `sleep` open and, when it fires, re-reads every pending child in one step. The first check runs after 5 minutes, then the interval doubles up to 6 hours. There is never more than one outstanding wait per parent, reused across waits, so the most it costs is one extra wake. ## ⚠️ Cancellation gap: read this A parent is resumed **immediately** only when the child's own runtime records its outcome, or when the child is cancelled **through the SDK**: | How the child finished | Parent resumes | | --- | --- | | Child completed or failed (either engine) | Immediately | | `run.cancel()`, `cancelRun()`, `cancelRuns()` | Immediately | | `workflow cancel` CLI, `workflow web` (both call `cancelRun()` / `cancelRuns()`) | Immediately | | Any tool that writes the cancellation to the World / its backend API directly, without `@workflow/core` | **At the next safety-net check** (5 min → doubling → 6 h) | | The child's invocation dies after its terminal write but before the resume (the resume itself retries transient failures for ~10s) | **At the next safety-net check** | I couldn't verify from this repo how the Vercel dashboard cancels a run. If it calls the backend directly rather than through `@workflow/core`, it falls in the "next safety-net check" row: the parent isn't stranded, but it can sit for up to an interval before noticing. The real fix is for the World backend to resume `wkf_child_run_completion:<parentRunId>` itself whenever a run stamped with `completionParentRunId` reaches a terminal state, whoever wrote it. The token and payload are deliberately derivable from the child run alone so a backend can do exactly that; it's a follow-up outside this repo. The docs call this out in a warn callout and a table ([Cancellations that bypass the SDK](https://workflow-docs-git-auto-register-system-hook-for-child-wo-26a16d.vercel.sh/cookbook/advanced/child-workflows#cancellations-that-bypass-the-sdk)). ## Fallbacks to polling - `returnValue` awaited anywhere other than the parent: host code, steps, a workflow handed someone else's `Run`. - A child on another deployment that doesn't advertise the protocol (`childRunCompletionVersion` on the health check). A target that doesn't send it is never stamped. - The parent's token is held by another run (conflict). - `WORKFLOW_CHILD_RUN_COMPLETION_HOOK=0` stops new children from being stamped. Already-stamped children still notify. ## Tests - `child-run-completion.test.ts` drives the dispatcher through real replays on the node VM: register → batched read → park → notify → result, a fan-out sharing one hook and one status read, the safety net re-reading after its sleep fires (with backoff), dispose and re-register between waits, a shared wait for repeated awaits, and the non-parent and conflict fallbacks. A QuickJS case covers one system hook plus one batched read. - `quickjs-system-hook-dispose.test.ts`: a disposed system hook's op is released (fails on the old guard). - `start-child-run-completion.test.ts`: stamping (the parent id), the kill switch, cross-deployment with and without the advertised version, and `Run` serialization. - `run-child-completion.test.ts`: `Run.readFinishedRunIds` (order, metadata-only reads, missing runs, errors) and the cancel paths notifying. - `terminal-child-run-completion.test.ts` (node + QuickJS) and `max-deliveries-lifecycle.test.ts`: the parent is notified after `run_completed` / `run_failed`, never before. - `child-run-completion-notify.test.ts`: payload and token, not-found, retry, give-up. - E2E: `awaitChildReturnValueWorkflow` (success and failure) asserts exactly one completion hook, with `isSystem: true`, that the child resumed. The new `awaitChildrenFanOutWorkflow` asserts that 4 children produce **one** `hook_created` and 4 `hook_received`. - Locally: core unit tests and the full nextjs-turbopack e2e suite pass on the node VM (157) and on QuickJS. ## Docs The Child Workflows cookbook covers the one-hook-per-parent model and has a new **Cancellations that bypass the SDK** section (warn callout plus table). I updated `start()`, Starting Workflows, Workflow Composition, Runtime tuning (`WORKFLOW_CHILD_RUN_COMPLETION_HOOK`), the Postgres / Worlds worker sizing pages, the eager-processing changelog note, the world-postgres README, and the `workflow` skill. ### Docs Preview | Page | Link | | --- | --- | | Child Workflows (rewritten) | [/cookbook/advanced/child-workflows](https://workflow-docs-git-auto-register-system-hook-for-child-wo-26a16d.vercel.sh/cookbook/advanced/child-workflows) | | Child Workflows: Cancellations that bypass the SDK | [/cookbook/advanced/child-workflows#cancellations-that-bypass-the-sdk](https://workflow-docs-git-auto-register-system-hook-for-child-wo-26a16d.vercel.sh/cookbook/advanced/child-workflows#cancellations-that-bypass-the-sdk) | | `start()`: Inside a workflow function | [/docs/api-reference/workflow-api/start#inside-a-workflow-function](https://workflow-docs-git-auto-register-system-hook-for-child-wo-26a16d.vercel.sh/docs/api-reference/workflow-api/start#inside-a-workflow-function) | | Starting Workflows | [/docs/foundations/starting-workflows#starting-workflows-from-workflow-functions](https://workflow-docs-git-auto-register-system-hook-for-child-wo-26a16d.vercel.sh/docs/foundations/starting-workflows#starting-workflows-from-workflow-functions) | | Workflow Composition | [/cookbook/common-patterns/workflow-composition#adapting-to-your-use-case](https://workflow-docs-git-auto-register-system-hook-for-child-wo-26a16d.vercel.sh/cookbook/common-patterns/workflow-composition#adapting-to-your-use-case) | | Runtime tuning: `WORKFLOW_CHILD_RUN_COMPLETION_HOOK` | [/docs/configuration/runtime-tuning#workflow_child_run_completion_hook](https://workflow-docs-git-auto-register-system-hook-for-child-wo-26a16d.vercel.sh/docs/configuration/runtime-tuning#workflow_child_run_completion_hook) | | Worlds config: `queueConcurrency` | [/docs/configuration/worlds#queueconcurrency](https://workflow-docs-git-auto-register-system-hook-for-child-wo-26a16d.vercel.sh/docs/configuration/worlds#queueconcurrency) | | Postgres World: `WORKFLOW_POSTGRES_WORKER_CONCURRENCY` | [/worlds/postgres#workflow_postgres_worker_concurrency](https://workflow-docs-git-auto-register-system-hook-for-child-wo-26a16d.vercel.sh/worlds/postgres#workflow_postgres_worker_concurrency) | | Eager processing changelog | [/docs/changelog/eager-processing#parentchild-polling-holds-worker-slots](https://workflow-docs-git-auto-register-system-hook-for-child-wo-26a16d.vercel.sh/docs/changelog/eager-processing#parentchild-polling-holds-worker-slots) | 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com> Co-authored-by: Peter Wielander <29887157+VaguelySerious@users.noreply.github.com>Pranay Prakash · e1bce44d · 2026-11-04
- 3.0ETV[cli] Add `workflow migrate inngest` codemod; refresh migration guides and skill for v5 (#5317) ## Summary Migrating an Inngest app to the Workflow SDK is mostly one mechanical job: turning each `step.run(id, closure)` into a named `"use step"` function and working out which variables the closure captured, because they now have to be passed as arguments. This adds a codemod for that, `workflow migrate inngest`, and brings the migration docs and skill up to date for v5. ### `workflow migrate inngest` (`@workflow/cli`, minor) - **Captured variables:** each `createFunction()` becomes an exported `"use workflow"` function that takes the trigger event. Each `step.run()` closure becomes a module-level `"use step"` function. The variables a closure reads from the handler (the event, earlier step results, loop variables) are found with the TypeScript binder, become parameters, and are passed at the call site. Module-scope bindings are not passed. The command prints every lifted step with its captured variables. - **Other step tools:** `step.sleep`/`sleepUntil` become `sleep()`. `step.waitForEvent` (token from `match`, or from an `if` comparing a single field) and `step.waitForSignal` (`onConflict: 'replace'` maps to `experimental_force`) become a hook raced against `sleep()` through a `waitForHook()` helper the codemod adds. The helper disposes the hook when the wait ends, and makes a replaced signal wait return `null` at its timeout, as Inngest does. `waitForEvent` tokens include the workflow name, so two workflows waiting for one event don't compete. `step.invoke` becomes `start()` plus `run.returnValue`, and `step.fetch` becomes `fetch` from `workflow`. - **`step.ai.wrap`:** it evaluates its inputs inside the step, so the model object is never passed as an argument, and returns only the fields that are destructured. - **Config and context:** `retries` becomes `maxRetries`, `NonRetriableError`/`RetryAfterError` become `FatalError`/`RetryableError`, and `runId`/`attempt`/`logger` are rewritten. - **Imports:** Workflow APIs are imported under an alias when the file already binds the name, including as a type-only import. Imports, and module-level literal constants, that only the removed Inngest code used are dropped. - **Config:** options are read through `const` references (`retries: NO_RETRIES`, a config object declared elsewhere in the file). Options it can't read, or a `const` object or array that other code in the file could modify, become a TODO, not a silent default. - **No guessing:** anything it can't migrate without changing behavior stays in place under a `// TODO(workflow-migrate): ...` comment. That includes a closure that writes to a captured variable, a closure that captures a local function, `sendEvent`, flow-control options, cron triggers, `onFailure` (kept as a named function), global `fetch()` outside a step, and Inngest client calls in the workflow body. - **Report:** `--report` writes a Markdown report. It includes an Event routing table (for each event, the workflows it starts and the hook tokens it resumes) and every `inngest.send()` and `serve()` call in the project with its replacement. - **Types:** the closure's own parameters are typed from their inputs at the call site, as captured parameters are, with an explicit `any` where no type is available. - **Project integration:** it reads the project's `tsconfig.json` (files and compiler options) and uses the project's `typescript`, now an optional peer dependency. Types for captured parameters are inferred from the call sites, and every generated annotation is validated by the checker; one that doesn't resolve becomes `any`. - **Safety:** it refuses files with uncommitted changes (unless `--force`), never writes output that doesn't parse, and supports `--dry-run` and `--json`. ### How it's tested - **Unit tests:** `packages/cli/src/lib/migrate/inngest/transform.test.ts` (20 tests) covers capture analysis, refusal cases, each step-tool mapping, imports, JS output, typed output, and the report. Outputs are also run through the Workflow SWC transform to check that the expected workflows and steps register. - **Runtime test:** `workbench/vitest/workflows/inngest-migrated.ts` is the codemod's output for `packages/cli/tests/fixture/inngest/order/functions.ts`, and `inngest-migrated.test.ts` runs it in-process. It covers lifted steps with captured arguments, the `waitForEvent` hook token, the timeout path, and `NonRetriableError`. A CLI test keeps the committed file in sync with the codemod's output (formatting-insensitive). - **Manual runs:** - Against projects with the real `inngest@4` types installed; the migrated output type-checks clean against `workflow`. - Over the 30 example apps in Inngest's public SDK repo: no crashes and no unparseable output. - **Live parity check against Inngest:** - **Setup:** an Express app on `inngest@4.26.2` (and the same functions on `inngest@3.54.2`) ran against `inngest-cli dev` 1.51.0, and was migrated with this CLI. Only the app boundary was replaced, following the report's Event routing table. Each migrated app was typechecked, built with Nitro and run on the Local World. - **Result:** 11 scenarios driven over HTTP matched Inngest on both v3 and v4. They cover captures, loops, parallel steps, inputs, sleeps, fatal errors, `retries: 0` through a const, `RetryAfterError`, `invoke`, `fetch`, repeated and broadcast `waitForEvent`, timeouts, `waitForSignal` including `onConflict: 'replace'` from the same file and from another file, and `ai.wrap`. ### Docs and skill - **Inngest page:** - A "Run the codemod" section with real before/after output. - An expanded concept mapping: `waitForSignal`, `idempotency`/`singleton` to `start({ experimental_hook })`, `cancelOn` to `run.cancel()`/`AbortSignal`, `fetch`, `ai.wrap`, `runId`/`attempt`, and attributes. - A "What behaves differently" section covering result types, the sandbox, step arguments being copies, and the event shape. - A support matrix of what the codemod converts and what it leaves as TODOs, plus "One owner per token", "Verify the migration" and "Cut over" sections. - Mappings: `resumeHook(signal, { signal, data })`, `attempt - 1`, and separate rows for `idempotency` (24h retention), `singleton` skip (no retention) and `singleton` cancel. - v5 inline-step execution added to the topology section. - **CLI reference:** a `workflow migrate` section in `configuration/cli-and-web-ui`. - **Other comparison pages (Temporal, Trigger.dev, AWS Step Functions, Cloudflare):** v5 rows for idempotent starts, cancellation, `continueAsNew`, and inline steps. - The Step Functions execution-name row keeps the provider's 90-day window after close. - Temporal Memo is mapped separately from search attributes. - **`migrating-to-workflow-sdk` skill, bumped to 0.4.0:** - It runs the codemod first for Inngest. - Removed the v4 rule that told agents to wrap `start()`/`getRun()` in a step for child runs, and fixed the matching verification check. - New shared patterns for idempotent starts, attributes and cancellation. - Fixed the self-hosted bootstrap to `const world = await getWorld(); await world.start?.()`, since `getWorld()` is async in v5 and the old line silently never started the World in plain JS. - A new eval for closure lifting. ### Follow-up, not in this PR `BaseCommand.finally()` calls `process.exit()` before oclif prints a command's returned JSON or a thrown error, so for example `workflow validate --json` prints nothing. `migrate` works around this by printing its JSON and errors itself. Fixing the base class affects every command, so it belongs in its own PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>Pranay Prakash · f6eb4a2b · 2026-11-16
- 2.9ETV[world-cloudflare] Experimental Cloudflare Durable Objects World: publishable package and CI lane (#5263) Turns the Durable Objects proof of concept from #4682 into the experimental, publishable `@workflow/world-cloudflare` package (first release 0.1.0), adds `setPrecompiledQuickJSModules()` to `@workflow/core/runtime`, and adds the required `E2E Cloudflare (workerd)` CI lane. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>Pranay Prakash · fde24825 · 2026-11-13