Nathan Rajlich

docs: apply Vercel technical writing standards (#3704) * docs: apply Vercel technical writing standards Audit the complete documentation corpus, package READMEs, skills, and source TSDoc/comments against the vercel-technical-writing skill and style-rules.md. Normalize sentence-case headings without changing published anchors, remove prose em dashes and filler wording, improve active voice and self-contained phrasing, standardize product/brand capitalization, American English, list punctuation, units, and code fence languages, and preserve exact runtime strings/table placeholders. All executable code is unchanged. Modified skills have their metadata versions bumped. * docs: extend writing audit to repository Markdown Apply the same technical-writing rules to design documents, compiler specifications, workbench guides, package changelogs, and the remaining tracked Markdown outside the deployed docs corpus. Preserve historical meaning, commands, output literals, table placeholders, and heading anchors. * docs: exclude generated package changelogs from audit

fix(swc-plugin): register class expressions via an IIFE instead of by name (#3971) * fix(swc-plugin): register class expressions via an IIFE and reject unnameable classes Class expressions with "use step" methods or custom serialization were registered by module-level statements referencing the class by name. When no module-scope binding could be resolved the plugin fell back to a placeholder `AnonymousClass` identifier, which is a guaranteed ReferenceError at module evaluation (vercel/workflow#3929). Other shapes were silently wrong as well: `var A = class {}, B = class {}` registered A's steps under B, `X = class {}` assignments and classes nested inside functions emitted unresolvable references. Class expressions are now wrapped in a single IIFE that receives the class, performs every registration recorded for it, and returns it, so the registration no longer depends on a name being in scope. The class name is still needed for step/class IDs and is derived from the assigned variable, the class's own identifier, or the property key it is assigned to (`exports.Foo = class {}`, `{ Foo: class {} }`). When none is available, or the class is declared inside a function, the plugin emits a compile error instead of broken code. Class declarations keep their existing module-level output; the emitters were factored so both paths share the same statement builders. * fix(swc-plugin): generate names for anonymous class expressions instead of erroring With registration happening inside the IIFE, an anonymous class expression in a position that provides no name (`foo(class { ... })`, an array element, a conditional branch) only needs a name for its step/class IDs. Generate a deterministic `AnonymousClass<N>`, counting only anonymous classes that have something to register, instead of rejecting them. Classes declared inside a function remain an error. Dead-code elimination now keeps module-level declarations whose initializer contains a wrapped class expression: evaluating the initializer is what registers the class, and the binding may be otherwise unreferenced.

Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) (#3048) * Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay * QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed * QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols * QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity * Apply biome fixes to QuickJS engine files * Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard * CI: include generated QuickJS source assets in shared e2e build artifacts * Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine * CI: run both VM engines across all frameworks and worlds; label jobs with the engine * Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads * e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status) * e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely) * Sort imports in QuickJS serialization files (biome organizeImports) * QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal hook payloads to the target run's published X25519 public key. The shared start() path publishes that key regardless of engine, so QuickJS runs receive sealed payloads too — but the QuickJS entrypoint resolved only the bare symmetric key via importKey(), which cannot open encp envelopes. The first sealed hook payload wedged the run right after hook_received, timing out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the node engine resolves the full capability via memoizeEncryptionKey). Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric (encrypt() with RunPayloadKeys takes the encr path). Regression test seals a payload exactly as resumeHook does and round-trips it through the VM. * Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping - Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap, drawing from the seeded Math.random (identical sequences to the node engine's vm/index.ts implementations); all crypto.subtle methods throw with step-function guidance. process.env exposed as a frozen copy, matching node. - Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family methods (incl. localeCompare) throw when given an explicit locale so cross-engine divergence is loud instead of silently writing different values into the event log. No-argument forms keep working. - runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the ~1.3MB embedded WASM assets out of node-engine deployments. - runQuickJSWorkflow wraps the per-run phase so an exceptional exit disposes the VM instead of leaking it in a reused compute instance; corrected the misleading fail-loud comment (run_failed, not retry); warn when the event drain loop exhausts its iteration bound. - Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the file's workflow.* namespace. - Eval-string correlation-id interpolation uses JSON.stringify instead of quote-only escaping. - common-vm.test.ts pins the reducer/reviver superset invariant against common.ts so the duplicated sets can't silently drift. - Docs enumerate the remaining global-surface differences (subtle.digest, Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known precondition-guard gap. * QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup) #1834 made resumeHook() fall back to enqueueing the run with a hookInput payload when the direct hook_received write fails transiently, with the runtime materializing the missing event on delivery. Only the node:vm path implemented it — the QuickJS dispatch returned before the node block, so the resilient payload was silently dropped and the new e2e timed out on every quickjs leg. - runtime.ts threads hookInput into runWorkflowWithQuickJS; the entrypoint materializes the missing hook_received after loading the event log (resumeId-keyed dedup, occurredAt from the resumeId ULID, local eventData substitution for lazy/ref responses, EntityConflict / HookNotFound handling) — mirroring the node block. - processEvents drops duplicate hook_received rows sharing a resumeId (first-in-log wins), matching the node engine's EventsConsumer dedup; the seen-set lives in the VM heap so it is deterministic per replay. Verified against the dev server with WORKFLOW_VM=quickjs: the resilient resume e2e passes and the materialization is observable in the logs; all 27 hook e2e tests green. * Rerun CI * QuickJS engine: split VM-local class/step-function reducers off the hardened host codec The hardened host-side serialization (#3257) made the shared reducers/class.ts and reducers/step-function.ts depend on serialization/hardened.ts, which imports node:util and captures host intrinsics — unbundleable and meaningless inside the QuickJS guest, where the codec already runs in the guest realm. Point the VM codec at pre-hardening copies with identical wire format; the host/guest boundary hardening for this engine arrives with the host-side serde that retires the VM bundle. * QuickJS engine: enqueue explicit wait continuations instead of same-message redelivery Scheduling sleep wakeups by returning { timeoutSeconds } redelivers the CURRENT queue message. When that message is a hook-resume delivery (carrying hookInput), its redelivery re-runs the lazy-resume re-ensure in the handler prologue; if the workflow disposed the hook during the first delivery (dispose -> sleep), the re-ensure gets HookNotFound, the prologue acks the message as 'nothing left to resume', and the wait timer it carried is silently lost — the run wedges (caught by the hookDisposeTestWorkflow e2e). Enqueue fresh continuation messages instead, matching the node engine's suspension handler: getWaitContinuationDispatch for pending waits (gaining delay clamping/hop chaining and pending-wait dedup keys) and a plain immediate message for elapsed-wait / attr_set / getConflict requeues. A fresh message carries only runId, so its delivery always reaches replay. Also: read hook_received resumeId from the canonical top-level event field (eventData.resumeId is the deprecated legacy fallback), and stop passing hookInput into the entrypoint — the shared prologue in runtime.ts materializes the event for both engines. Adds a VM replay test for the hook -> dispose -> sleep shape. * Sort imports in quickjs-entrypoint (biome organizeImports) * Address review: dispatch inside run-level try/catch, queue namespace + run-origin trace carrier threading, configurable interrupt budget - Move the QuickJS engine dispatch inside the replay loop's try so escaping engine failures (MaxEventsExceededError, WASM OOM, bundle-eval errors) reach the catch that classifies and records run_failed, instead of nacking the message and burning all 48 queue redeliveries into MAX_DELIVERIES_EXCEEDED. Transient world errors still rethrow for redelivery. Updated the two comments that describe the propagation. - Thread the queue namespace from runtime.ts through runWorkflowWithQuickJS into every message publish (step dispatch, hook_conflict requeue, immediate requeue, wait continuation) — without it, publishes on a namespaced deployment land on __wkf_workflow_* while consumers listen on __<ns>_wkf_workflow_*. - Thread the run-origin nextTraceCarrier accessor through instead of capturing the current invocation context, so linked-mode invocations form a star around workflow.start rather than chaining; the hook_conflict requeue now carries a traceCarrier and requestedAt. - Replace the hardcoded 30s VM interrupt budget with the configurable replay budget (getReplayTimeoutMs, default 240s), matching the node engine. * Sort imports in quickjs-runtime (biome organizeImports)

feat(core): user-registerable workflow lifecycle hooks (registerLifecycleHooks) (#3678) * feat(core): user-registerable workflow lifecycle hooks Adds registerLifecycleHooks (exported from workflow/api) so apps can observe run terminal transitions from one central place — e.g. report every failed run to Sentry from instrumentation.ts — without wrapping each workflow body. - onRunCompleted/onRunFailed handlers receive the lazily-hydrated Run instance; onRunFailed additionally receives a WorkflowRunFailedError whose errorCode carries the classification and whose cause is the hydrated thrown value (round-tripped through the run-error serialization pipeline, so VM-realm throws surface as host-realm Errors with class identity preserved — same shape run.returnValue rejects with). - Registry lives on globalThis under Symbol.for so every bundled copy of @workflow/core shares one list; multiple registrations allowed, handlers run in registration order, and registration returns an unregister function. - Dispatch is fire-and-forget via safeWaitUntil: handlers can't delay or change the run outcome, failures are logged and swallowed, and serverless invocations stay alive while handlers finish. - Wired into every terminal writer that lands a run_completed or run_failed event: the happy-path completion, the terminal catch, the suspension-commit failure, recordFatalRunError, the max-deliveries gate, replay-budget exhaustion, the deployment guard, and both QuickJS entrypoint writers — and only on the invocation whose write actually succeeded (never on EntityConflict/RunExpired). - e2e coverage registers handlers in the Next.js workbenches' instrumentation.ts and reports observations by resuming a durable hook, so the channel works across serverless instances. - Docs: observability guide with a Sentry example + workflow/api reference page. Co-Authored-By: Nathan Rajlich <71256+TooTallNate@users.noreply.github.com> * Load lifecycle-hooks-e2e via guarded dynamic import; comment fixes The static top-level import in instrumentation.ts pulled workflow/api → world-init → @workflow/world-local → proper-lockfile → fs into every compile target of instrumentation.ts, and the non-node ones cannot resolve fs — 500ing every request in the nextjs-webpack lanes. The NEXT_RUNTIME guard only helps at runtime; the canonical Next.js pattern is a dynamic import inside the guard so each compile target dead-code- eliminates the branch. Verified: lifecycle + pages-router e2e slices green against local nextjs-webpack and nextjs-turbopack dev servers. Also: clarify the 'describe block' doc comment and correct the microtask/macrotask wording in the test flush helper. Co-Authored-By: Nathan Rajlich <71256+TooTallNate@users.noreply.github.com> * Remove em dashes from docs and comments added on this branch Per the Vercel technical writing guidelines: em dashes create ambiguity for agents parsing sentence boundaries. Replaced with periods, commas, parentheses, or colons across the new docs pages, the changeset, and the code comments this branch adds. Pre-existing em dashes elsewhere are left for the repo-wide docs audit. Co-Authored-By: Nathan Rajlich <71256+TooTallNate@users.noreply.github.com> * Apply Vercel technical writing rules to the lifecycle hooks docs Beyond the em-dash pass: the guide's intro now leads with what lifecycle hooks let you do (the first sentence is parseable as the page's purpose) instead of opening with the failure mode; passive constructions are active ('the runtime keeps the invocation alive with waitUntil', 'the runtime logs and swallows a throwing handler', 'you can register multiple hook sets', 'the backend writes that transition'); the one-word 'Semantics' heading is now the standalone statement 'How handlers behave'; and the key lazy-hydration sentence names the Run instance instead of leading with a pronoun so it reads correctly when extracted alone. Co-Authored-By: Nathan Rajlich <71256+TooTallNate@users.noreply.github.com> * fix(core): hydrate lifecycle failures from persisted errors Use persisted bytes and keys for lifecycle reporting, defer external reads, and expose workflow names without fetching runs. Preserve handler error diagnostics, strengthen regression coverage, and correct Next.js examples and best-effort delivery documentation. Co-Authored-By: Nathan Rajlich <71256+TooTallNate@users.noreply.github.com> * fix(core): sort QuickJS imports after rebase Co-Authored-By: Nathan Rajlich <71256+TooTallNate@users.noreply.github.com> --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Nathan Rajlich <71256+TooTallNate@users.noreply.github.com>

[core][world][world-vercel] Add `World.createRunId()` and region-aware queue routing (#1981) * [world-vercel] Add /run-id sub-export with tagged ULID encode/decode Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a ULID-shaped string used for workflow run IDs. Tagged values remain valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip through any system that accepts ULIDs. * [world-vercel] Add string-value assertions to run-id tests Add exact-string expectations for encoded outputs at known inputs, covering the default region/version pair, numeric region IDs, version overrides, boundary values (all-zero, all-max), the dirty-input overwrite case, and the lexicographic-order checks. Also adds an explicit byte-array expectation for the canonical ULID-spec example string and an additional first-char-range coverage test for isTagged. * [world-vercel] Remove internal-repo reference from regions doc comment * [world-vercel] Address PR review feedback on run-id sub-export - isTaggedString now fully validates the input as a 26-char Crockford Base32 ULID (delegating to ulidToBytes) instead of only inspecting the first character. This fixes false positives on inputs like '4UUUU...' that have a valid tag-bit position but invalid chars later in the string. - isTagged() now accepts `unknown` to match its documented behavior of safely rejecting non-string inputs without requiring callers to cast. - Introduce `RegionKey` for the full set of keys including 'unknown', and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the return type of `lookupRegion` and the `DecodedRunId.region` field accurately reflect that 'unknown' is never produced. Updates `encode` to reject 'unknown' as a region code string at runtime (callers wanting the unknown sentinel should pass numeric 0). * [world] [core] [world-vercel] Add World.createRunId() and region-aware queue routing - @workflow/world: add optional createRunId(input?) to the World interface so worlds can mint run IDs with embedded metadata, and add an optional 'region' field to QueueOptions for per-message routing hints. - @workflow/core: start() now delegates run ID generation to world.createRunId() when defined (falling back to a monotonic ULID otherwise), and accepts a new 'runIdInput' option that is forwarded verbatim to createRunId. When runIdInput.region is a string, it is also threaded onto the queue options so the initial workflow message is dispatched to the matching region. - @workflow/world-vercel: implement createRunId() to mint region-tagged ULIDs, preferring an explicit runIdInput.region and falling back to the VERCEL_REGION env var. The queue now resolves its destination region from (in order): an explicit opts.region, the region embedded in the payload's tagged run ID, the VERCEL_REGION env var, and finally a hardcoded 'iad1' default. This replaces the previous unconditional 'iad1' region passed to the @vercel/queue client. Monotonicity within a process is preserved by tracking the last emitted run ID and bumping the bit immediately above the 11-bit metadata window when a same-ms collision would otherwise occur, then re-stamping the requested region/version on top so metadata remains stable. * [core] [world] [world-vercel] Pass full StartOptions to World.createRunId Drop the dedicated 'runIdInput' field on StartOptions and forward the entire options bag to world.createRunId() instead. This keeps the public API surface smaller and lets each World pick the fields it recognises (e.g. world-vercel reads 'region'). The top-level 'region' option remains on StartOptionsBase and is also threaded onto the queue's per-call region opt when set. * Address review feedback: doc fixes and deterministic same-ms tests - Document the final iad1 fallback in QueueOptions.region (world) - Correct the World.createRunId doc: start() always passes an object - Fix the clientOptions comment: the handler client omits region and relies on SDK auto-detection + the ce-vqsregion header for acks - Fix a misleading QueueClient-construction comment in queue.test.ts - Freeze time in the same-ms monotonicity test so it deterministically exercises the intended path, and add a test covering the bump-above-metadata fallback when the region changes mid-millisecond Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: keep workflow-server override rewrite-compatible Export WORKFLOW_SERVER_URL_OVERRIDE while keeping the one-line const shape that workflow-server's cross-repo e2e test automation rewrites. Update world-vercel tests to import that exported value for mock origins and URL expectations instead of duplicating the temporary preview URL. * fix(world): clear region tag bit before ULID timestamp validation Region-tagged run IDs set the high bit of the ULID timestamp byte. The shared world timestamp validator used raw decodeTime(), so current tagged run IDs appeared thousands of years in the future and were rejected before reaching workflow-server. Clear the tag bit before decoding, matching the workflow-server behavior, and cover tagged IDs in tests. * fix(world-vercel): validate tagged runId timestamps via run-id decode Keep @workflow/world's ULID helpers generic; they should not know about world-vercel's region-tagged run ID layout. Instead, world-vercel decodes its tagged runId to the original ULID before using the shared timestamp validator for run_created events. Add a world-vercel regression test that a current sfo1-tagged runId passes validation. * fix(world-vercel): default run ID region to iad1 instead of unknown When neither an explicit region option nor VERCEL_REGION is available, createRunId minted a tagged ULID with the unknown (0) region sentinel, producing the tagged: true, region: null state. The server already resolves unknown/untagged runs to DEFAULT_VERCEL_REGION (iad1), so mint a concrete iad1 tag instead, keeping every run ID self-describing and routable. * test(e2e): use verbose reporter + per-test start heartbeat The default vitest reporter buffers per-file output, so a stalling e2e test produces no output until its timeout — making CI look like a silent 30-minute hang. Switch the e2e CI invocations to the verbose reporter (prints each test result as it completes) and emit a '[e2e] ▶ start:' heartbeat to stdout at the start of every test (bypassing vitest's console buffering) so a stuck test is immediately identifiable in the live CI log. * test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at combined 527+529 preview Temporarily target the workflow-server combined-527-529-preview deployment, which bundles platform-directed multi-region routing (vercel/workflow-server#527, incl. the iad1 hook pin) and durable stream state (vercel/workflow-server#529), so e2e can validate the full multi-region path end-to-end. Revert to empty on main. * fix(core): region-tag the health-check correlationId The health-check response is delivered over a Redis stream whose name (and synthetic run ID) embed the correlationId. Under platform-directed routing the responding endpoint and the polling reader can be served from different physical regions; Redis is physical-region-local, so the correlationId must carry the region for both sides to resolve the same backend. Generate the correlationId via world.createRunId() (a region-tagged ULID) when the world provides it, falling back to a plain ULID for worlds that don't tag IDs (e.g. local, single-region). The synthetic wrun_hc_<id> run ID then carries the region; workflow-server's region middleware decodes it. * Address review feedback: validate region overrides, reset server override - queue: validate opts.region and VERCEL_REGION against the known region table before routing, ignoring unrecognised codes so a bad override can't clobber the payload-derived region (Copilot) - add isKnownRegionCode() runtime guard to run-id/regions - reset WORKFLOW_SERVER_URL_OVERRIDE to '' (must be empty on main) - fold the within-PR iad1-default changeset into the main world-vercel changeset and delete it (review) - start.test: declare specVersion on createRunId mock worlds now that the merged world-compatibility check requires it - cover the new region-validation fall-through paths in queue.test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at wave-1 multi-region preview BRANCH-ONLY — revert the override to '' before merge (lint enforces). Points this PR's e2e/benchmark runs at the wave-1 multi-region workflow-server preview (vercel/workflow-server#590: iad1+sfo1+fra1 serving, staging data backends) so region-tagged runs are validated against real multi-region serving end-to-end. Also makes the unit-test mock origins in events-v4.test.ts and trace-propagation.test.ts override-aware (same pattern the rest of the file and utils.test.ts already use), so the suite passes whether or not the override is set — these two files were the only spots hardcoding https://vercel-workflow.com. * test(e2e): Vercel multi-region suite for start()'s region option Adds a dedicated e2e suite validating @workflow/world-vercel region routing end to end, run as its own CI job (e2e-vercel-multi-region) against the nextjs-turbopack workbench only — deliberately separate from e2e.test.ts, which runs as a matrix across all worlds/frameworks where Vercel-specific multi-region behavior doesn't apply. - workbench/nextjs-turbopack/vercel.json: deploy to iad1+sfo1+fra1 so region-routed flow messages have a function to land on in each region. - workflows/99_e2e.ts: regionProbeWorkflow returns the VERCEL_REGION observed by both the workflow and a step, so tests can assert the run EXECUTED in the intended region (not just that it was tagged). - packages/core/e2e/e2e-region.test.ts: per-region cases assert 1) start(..., { region }) mints a region-tagged run ID (decoded via @workflow/world-vercel/run-id), 2) the workflow + step both observed VERCEL_REGION === region, 3) the server reports the run completed; plus a concurrent all-regions case guarding against cross-region misrouting under simultaneous multi-region traffic. Skips on local deployments. - .github/workflows/tests.yml: new e2e-vercel-multi-region job mirroring e2e-vercel-prod's env/deployment-wait, running only the new suite. * test(e2e): start region probes in-function; fix getWorld await The first multi-region CI run surfaced two issues: 1. sfo1/fra1-tagged runs executed in iad1. The suite started runs from the external test process, which uses the api.vercel.com token proxy — and the proxy's queues path forwards every send to the region-less VQS host (the world's proxy-mode resolveBaseUrl ignores the region argument, and the proxy's x-vercel-vqs-api-url escape hatch only allowlists vqs-server-*.vercel.sh preview hosts). Production traffic publishes IN-FUNCTION (direct regional queue routing), so the suite now triggers start() through a new workbench route (/api/e2e-region-start) and rehydrates the run with getRun() — testing the path production actually takes. Proxy-mode regional queue routing is a known gap to address separately in api-workflow. 2. TypeError on world.runs.get: getWorld() is async and was called without await. * test(e2e): cover explicit and implicit region starts in the multi-region suite With regional VQS routing now working through the api.vercel.com proxy (vercel/api#79056 + #2789 + this branch's per-send region resolution), the suite covers both start configurations, asserting the same three properties for each (region-tagged run ID, execution in the intended region via VERCEL_REGION echoed in the return value, server-side completion): 1. EXPLICIT: start(..., { region }) called directly in the vitest runner — publishes through the token proxy, per-send region carried by x-vercel-queue-region. Restores the direct-start shape the suite had originally, plus the concurrent all-regions case. 2. IMPLICIT: dedicated per-region workbench routes (/api/e2e-region-implicit/{iad1,sfo1,fra1}), each pinned to a single region via a per-function 'regions' entry in the workbench vercel.json, calling start() with NO region option — createRunId derives the tag from the minting function's VERCEL_REGION. The test also asserts the route reported executing in its pinned region, so the implicit-tagging assertion can't pass vacuously. Replaces the interim /api/e2e-region-start route (explicit region via request body), which existed to work around the pre-#79056 proxy gap. * Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — wave-1 multi-region serving is in production workflow-server#590 (iad1+sfo1+fra1 serving) merged and deployed to production and the e2e backend, so this branch's e2e/benchmark runs no longer need to target the wave-1 preview. Restores the empty override the No Test Overrides lint job enforces for merge. The override-aware unit-test origins (events-v4/trace-propagation) stay — they are correct under any override value. * test(e2e): cross-region stream visibility (iad1 writer, sfo1 reader) Regression coverage for a backend bug that made cross-region stream reads report zero chunks on IN-PROGRESS streams (completed streams were unaffected), which forced the multi-region serving rollback. The new case exercises exactly that geometry: - crossRegionStreamWorkflow (99_e2e.ts) writes N chunks to the default output stream, then holds the stream OPEN for 45s before closing — the in-progress window is the point, since completed streams are the easy case. - The e2e starts it with region iad1, waits (same-region, via the api.vercel.com proxy) until all chunks are written, asserts the run is still 'running', then reads through a new sfo1-pinned workbench route (/api/e2e-stream-read/sfo1) that returns getTailIndex() plus its VERCEL_REGION. The reader's region served none of the stream's writes, so the reported chunk count must come from the backend's cross-region stream metadata. The test fails loudly if the route isn't actually executing in sfo1. Also bumps the explicit-region test timeout to 120s: the first case in the file absorbs every cold start at once (fresh workbench instances in up to three regions plus a cold backend preview) and was observed just over the 60s default. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE points at a multi-region backend preview that includes the fix, so this validates cross-region stream visibility end-to-end before multi-region serving is re-enabled. * test(e2e): extend multi-region suite to all 19 provisioned regions Points the suite at an all-regions backend preview and widens coverage from the wave-1 trio to every provisioned region: - Explicit path: a single concurrent all-regions case starts one tagged run per region (one shared cold-start window instead of 19 sequential ones) and aggregates per-region failures so a single region's breakage reports alongside the full picture. The trio keeps its detailed per-region cases and the 9-way concurrent-isolation case. - Implicit path: workbench gains a region-pinned /api/e2e-region-implicit/<region> route per provisioned region (19 total, shared handler), the workbench itself now deploys to all of them, and the test.each covers the full set with per-case timeouts for regional cold starts. - Multi-region CI job timeout 20m -> 35m for the sequential implicit cases. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE now targets the all-regions backend preview instead of the previous (stale, since-merged) fix preview. * test(e2e): tolerate geo-adjacent execution of queue callbacks The first all-regions run surfaced a subtle execution-locality behavior: queue delivery is guaranteed to the tagged region's dataplane and the delivery callback egresses from that region, but the consumer invocation's execution region is chosen by where that callback enters Vercel's edge — and adjacent regions can geo-resolve to each other's functions. Observed live: kix1-tagged runs (callback egressing from Osaka) deterministically executing in hnd1/Tokyo on both the explicit and implicit paths, with tagging, data placement, and completion all still strictly kix1. expectRunInRegion now asserts execution lands in the tagged region OR one of its geographic neighbors (EXECUTION_ADJACENCY), while run-ID tagging and server-side completion remain strictly the requested region. Gross misrouting (e.g. kix1 -> iad1) still fails. * Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — all-regions serving is in production The all-regions workflow-server rollout is deployed and serving production traffic from every Vercel region, so this branch's e2e no longer needs to target a branch preview. Restores the empty override the No Test Overrides lint enforces for merge. With this the PR is complete: region-tagged run IDs, region-aware queue routing, and the multi-region e2e suite (explicit + implicit + all-regions + cross-region streams) all validate against the production-default backends. * docs: fix three stale comments flagged in review - start.ts: StartOptionsBase.region fallback is iad1, not the unknown sentinel (createRunId always mints a concrete routable region) - queue.ts: example used a nonexistent start({ runIdInput }) API; the real option is start({ region }) - events.ts: decode() clears only the tag bit (top bit of the 48-bit timestamp field) — it does not restore the original untagged ULID; reword to say what actually matters for timestamp validation * test(e2e): cover hook resolve/resume for runs owned by non-iad1 regions Hooks are resolved by opaque token, which carries no region hint, so lookup and resume must work regardless of which region owns the run's data. Exercises the full follow-up-message path on sfo1- and fra1-tagged runs: create inside the workflow, resolve by token from the test process, resume twice sequentially, and assert payload order and completion. Regression coverage for the failure mode where the first message to a hook-driven app on a non-iad1 run worked but every follow-up failed with 'Hook not found'. --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

feat(core): side-effect-free serialization of workflow VM values (#3257) * feat(core): side-effect-free serialization of workflow VM values Serialization runs on the host but inspects values constructed inside the node:vm sandbox, so ordinary dynamic operations dispatch into the sandbox realm and execute workflow code: `value.toISOString()`, `Array.from(map)`, `Object.prototype.toString` (via Symbol.toStringTag), `.source`/`.flags`, `.href`, view `.buffer`/`.byteOffset`/`.byteLength`, and error `.message`/`.stack`/`.cause` reads. That is a determinism hazard. A payload is serialized exactly once and is never re-serialized on replay, so any workflow-visible side effect it triggers exists only on the live path — a patched `Date.prototype.toISOString` that consumes a seeded `Math.random()` draw, for example, shifts every subsequent draw and diverges from replay. This makes serialization side-effect free where the data allows it, and observable where it does not: - Classification uses engine brand checks (node:util types, internal-slot probes) instead of `instanceof global.X` and Object.prototype.toString, so it is immune to Symbol.hasInstance, reassigned sandbox globals, and Symbol.toStringTag spoofs. An unbranded value claiming a brand-decided tag is now classified as a plain object instead of being routed into an extractor that requires the real internal slot (unhardened devalue crashes on that input). - Extraction goes through intrinsics captured at module load — host boot, before any workflow bundle runs — invoked with explicit receivers. Internal slots are realm-agnostic, so host intrinsics read VM-realm objects without touching the sandbox's patchable prototypes. - Property access reads through descriptors, so plain data never invokes anything. Where workflow code must run because the data lives behind it — getters, proxies, custom [WORKFLOW_SERIALIZE] methods, toString() on toStringTag-branded objects like Temporal polyfills — the execution is preserved for compatibility and recorded in a new `CodecOptions.guestCodeStats` sink, surfaced as workflow.serialization.guest_code_{executions,details} span attributes. Consumers that retain a VM across steps can treat a non-empty report as "serialization may have perturbed VM state". Engine-provided accessors are deliberately not reported: V8 defines `stack` as an own accessor on every Error instance, so reporting it would flag every serialized error. Nativeness is decided with the captured host Function.prototype.toString; the bound-function caveat is documented in hardened.ts. Requires devalue 5.9.0 for the pluggable `operations` option. * chore: shorten changeset * fix(core): close review gaps in hardened serialization Five correctness fixes, all with repros: - Callable proxies were treated as engine accessors. V8 returns `function () { [native code] }` from Function.prototype.toString for a proxy around a function rather than throwing, so a proxy-wrapped getter was cached as engine-provided and invoked unreported. Gate on types.isProxy first. - Host builtins implemented in JavaScript were reported as workflow code. Node's DOMException.prototype.message/name are ordinary functions, so the nativeness test failed and every serialized DOMException reported two getter executions. They belong to the *host* realm, though, and workflow code cannot author a host-realm function — so provenance is now decided by nativeness OR host-realm `Function.prototype`, which are disjoint and together cover both cases (V8 installs `stack` per realm, so a VM error's getter is native but VM-realm). - The extraReducers at the two VM call sites were still unhardened, and they run on every value the earlier reducers do not claim — which is exactly where the report has to be complete. `instanceof global.ReadableStream/WritableStream/Request/Response` consulted Symbol.hasInstance on the sandbox class (14 invocations for an ordinary payload once the classes are patched), and AbortController's guard did a bare `value.signal` read, so a non-enumerable `signal` getter ran with an empty report. All five now walk the prototype chain and read through descriptors. - `__closureVarsFn` was invoked unreported on a purity argument that nothing checked: the property is reachable from workflow code, which can replace the compiler-generated function. step.ts now registers the generated function as trusted when it builds the proxy, so provenance is verified rather than assumed, and an unrecognized function is reported. - The URL/URLSearchParams test patched prototypes of *host* classes injected into the sandbox, mutating them for the rest of the worker process. Restored in a finally. Also, per review: - `dehydrateStepArguments` / `dehydrateWorkflowReturnValue` take an optional GuestCodeStats out-param, so a retained-VM gate can consume the report instead of it being spent on span attributes. The report-completeness tests use it to exercise the real dehydrate path. - Every intrinsic capture is now optional. The table is built at module scope, so a missing member was an import-time crash of @workflow/core rather than a degraded path; only SharedArrayBuffer was guarded, while URLSearchParams.prototype.size (Node 19.8+) and the WHATWG classes were assumed. Absent captures now make the corresponding reducer decline to match. - Documented that recording is not prevention (a recorded getter calling Math.random() still advances the run's seeded PRNG), and that a `{ kind: 'proxy' }` report implies a silent shape change (a proxied Map serializes as a plain object). - Parity coverage extended to DataView, boxed primitives, null-prototype objects, setter-only properties, DOMException, AggregateError, an accessor-valued Symbol.toStringTag, both RetryableError retryAfter paths, and a WORKFLOW_SERIALIZE class instance. * fix(core): keep identifying proxied host classes Every Next.js e2e job failed on the two webhook tests: the hook POST returned 404 because `resumeWebhook` could not serialize its step return value ("Cannot stringify arbitrary non-POJOs"), so no hook was ever registered. The value was a `NextRequest`, which Next.js hands over as a **Proxy**. `isInstanceOfPrototype` rejected proxies outright, so the Request reducer answered "not a Request" and devalue fell through to the POJO check. The reasoning behind rejecting them — that proxied built-ins were never serializable, because internal-slot reads throw on a proxy receiver — is true for `Map`/`Date`/`URL`, whose reducers read internal slots, but not for `Request`/`Response`/streams, whose reducers read ordinary properties. Next's proxy forwards those with the target as receiver, so they serialized fine before this PR. Identification now walks through proxies, matching `instanceof`, and records the traps rather than suppressing the answer. The three reducers that do read internal slots (URL, URLSearchParams, Headers) fall back to the dynamic read when the value is a proxy, so their behavior is exactly what it was before — including throwing for a bare proxy over a built-in, which threw before too. Verified against the real thing: the full nextjs-turbopack e2e suite (135 tests) passes locally, having reproduced the failure first and confirmed a reverted `serialization.ts` fixed it. The regression test uses a receiver-correcting proxy, which is what makes NextRequest work in practice; a comment records that a bare `new Proxy(request, {})` throws on undici's private slots with or without this change. * fix(core): state what the closure-fn mark proves, and correct stale docs - `isInstanceOfPrototype`'s JSDoc still described the behavior removed in 8bc462fb5 (proxies rejected without firing traps), which is the opposite of what it now does. - The `__closureVarsFn` provenance check proves the function was passed to `useStep`, not that this package generated it: `useStep` is published on the sandbox global, so workflow code can call it with a function of its own and have it marked. Renamed `registerTrustedFunction` / `isTrustedFunction` to `markUseStepClosureFn` / `isUseStepClosureFn` so the name states the boundary, and documented the laundering caveat alongside the existing ones. Marking still earns its keep — reporting every step that captures a variable would bury the signal — and closing the gap properly needs a compiler-emitted marker, which is a compiler change. - Added the missing coverage for both sides of that check: an unmarked `__closureVarsFn` is invoked and reported, a marked one is invoked and not. - `guestCodeStats` was documented as something a retained-VM gate consumes, but no runtime caller passes a sink; the executions reach telemetry from every dehydrate path regardless. Reworded both docs to say that, so the out-param is not mistaken for wiring that already exists.

fix(core): make step-argument serialization failures catchable in workflow code (#3675) * fix(core): make step-argument serialization failures catchable in workflow code A step whose arguments fail to serialize is now finalized by the suspension handler as step_created + step_failed (mirroring a step-body failure) instead of rejecting the whole suspension. The next replay — forced in-process, since no step message is dispatched for the failed step — rejects the step's promise with the SerializationError, so a try/catch around the step call observes it. Uncaught, the error propagates out of the workflow body and fails the run as a fatal USER_ERROR immediately, instead of redelivering the orchestrator message until max deliveries (49/48) as reported in production on v4. * Serialize the step_failed error with the VM global; one-sentence changeset Addresses review feedback: dehydrateStepError in finalizeUnserializableStep now receives suspension.globalThis like every other dehydration in this file. Error detection is realm-independent, so the host-created SerializationError serializes identically, but VM-realm values guest code threw into the cause chain are now detected by the realm-sensitive reducers. * Address review: QuickJS engine support, deferred-batch join, drain gate, placeholder marker, telemetry, docs - QuickJS: dumpPendingOps now catches a step input's serialization failure per-op, reframes it as a SerializationError with the same framed message as dehydrateStepArguments, and surfaces it on the pending op instead of failing the whole collection. The entrypoint's dispatchPendingOps finalizes such steps as step_created (placeholder input) + step_failed, excludes them from inline claims and queue publishes, marks them handled, and raises the requeue signal so the failure is observed even when the feed lags — mirroring the node:vm engine, so both engines agree: catchable in workflow code, USER_ERROR with the framed message when uncaught. Both step-argument e2e tests now pass on WORKFLOW_VM=quickjs. - runtime.ts: the failed-step replay path now joins suspensionResult.deferredBatchWork before continuing, so a trailing chunk commit or step-message publish rejection propagates instead of being swallowed after ack; committed inline claims are documented as deliberately handed to owned recovery. - Terminal drain: finalization is gated on a stepDispatch target. The drain caller has no replay to observe a finalization, so a completed run no longer gains failed-step rows for an unawaited unserializable step — the rethrown error is swallowed by the drain's catch, preserving its pre-existing behavior. - The placeholder input now carries a marker string ('[input unavailable: step argument serialization failed]', shared via runtime/unserializable-step.ts) so inspect/o11y don't render the failed step as a genuine zero-argument call. - New workflow.steps.failed_serialization span attribute on the suspension span, so occurrence is measurable without log search. - Docs: v5 serialization-failed error page documents where each boundary's failure surfaces (catchable step failure vs run failure) and the no-retry USER_ERROR semantics; foundations/errors-and-retries gains a Serialization Failures section with the try/catch shape. * Guard the finalization crash window; self-contained docs samples - A crash or transient failure between finalization's two durable writes leaves a lone placeholder step_created, and redelivery then dispatches the step through normal crash recovery — previously running user code with the placeholder arguments. The placeholder now carries a structural flag on the input triple's top level (which user code never controls, so no false positives), and the step executor checks it after hydration: instead of running the body, it throws the intended fatal SerializationError, completing the interrupted finalization as step_failed. Applies to both engines (they share the placeholder and the executor). - Regression tests: executor fails a placeholder-input step without running the body (and doesn't trip on a genuine argument equal to the display marker); handleSuspension rejects for redelivery when step_failed can't be written after step_created landed, leaving the recoverable placeholder behind; mixed bad-step + large fan-out returns the failure set alongside still-pending deferredBatchWork whose rejection surfaces — the contract the runtime's failed-step join (added previously) relies on. - Docs: the two new code samples are now self-contained so the docs code-sample typecheck passes.

feat(core): route sealed envelopes through the serialization layer (#3094) * feat(core): route sealed envelopes through the serialization layer Adds the plumbing that lets a cross-run writer emit `encp` payloads. The serialization layer previously had no way to express "seal to this public key": every consumer expected a symmetric `CryptoKey`, and the encrypt primitive was unconditionally AES-GCM. That gap was also a live footgun. A 32-byte X25519 public key is a structurally valid AES-256 key, so `importKey(pubkey, 'AES-GCM')` succeeds and silently produces ciphertext nobody can ever open — no compile error, no runtime error, just unreadable data. Making public keys reachable only through `sealTo()` turns that mistake from "avoided by convention" into "unrepresentable". `PayloadKey` is a widening rather than a replacement, so no existing call site changes: | Variant | Writes | Reads | Held by | | ---------------- | ------ | ------------ | ----------------------- | | `CryptoKey` | encr | encr | same-run (legacy shape) | | `RunPayloadKeys` | encr | encr, encp | the owning run, o11y | | `SealTarget` | encp | — | cross-run writers | A run's own payloads deliberately stay symmetric even when the holder could seal: sealing costs a fresh ECDH and 32 bytes per envelope and buys nothing when the writer already holds the decryption key. The variants are branded with `Symbol.for` (not `Symbol()`) because these values cross the host ↔ workflow VM realm boundary, where only the global symbol registry is shared. Both stream directions handle sealed frames, keeping the length header in the clear so frame boundaries stay findable without a key. Frames keep per-frame random nonces rather than a counter — a reconnect or a durable replay restarts the writer, and a counter would repeat `(key, nonce)` and break AES-GCM catastrophically. Regression tests assert that 100 frames of identical plaintext produce 100 distinct ciphertexts, and that ten writer incarnations never share a content key. Capability shortfalls fail loudly and specifically: opening a sealed payload with a symmetric key (or a write-only seal target, or nothing) reports "no run keypair is available" rather than surfacing as a mysterious auth-tag failure further down. * review: amortize the sealed-box KEM across stream frames Sealing each frame independently meant a fresh X25519 keygen + ECDH + HKDF per chunk — O(frames) crypto for a long stream. As review pointed out, amortizing the KEM is safe here so long as nonces stay random, which they already are; the hazard I had been guarding against was counter nonces specifically, not a long-lived content key. `createSealSession` performs one encapsulation per writer instance and reuses the content key, with a fresh random nonce per frame. `createOpenSession` mirrors it on the read side, caching decapsulation by ephemeral public key — since every frame from one writer carries the same key, that is an ECDH per writer rather than per frame. Both safety properties are preserved and now asserted directly: - nonces stay random, so sharing a content key cannot repeat `(key, nonce)` (100 identical-plaintext frames -> 100 distinct frames) - a session is scoped to one stream instance, so a reconnect or durable replay never inherits a previous content key (10 writer incarnations -> 10 distinct ephemeral keys) The envelope layout is byte-identical to one-shot `seal`, so readers cannot tell which path produced a frame and need no matching session. Two edge cases the read-side cache introduces, both covered: frames from two writers interleaved on one stream (eviction on every frame, so correctness cannot depend on hit rate), and a failed decapsulation clearing rather than poisoning the entry.

fix(core): isolate lifecycle reporting and drain stream work (#4216) Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Nathan Rajlich <71256+TooTallNate@users.noreply.github.com>

Carry run identity on step-dispatch messages; drop the blocking runs.get from the queued-step prologue (#3457) * Carry immutable run identity on step-dispatch messages; drop the blocking runs.get from the consumer prologue Closes #3456. Every queued step execution paid a runs.get round trip before its step_started claim — one RTT per branch on the TTLS-critical path, and under a 256-branch fan-out burst the read amplification drove that read to p90 ~5.1s (durabench parallel sweeps), smearing branch starts. The dispatch sites (node dispatch loop, delayed retries, the suspension handler's resilient publish, and the quickjs engine's queueStepMessage) now stamp WorkflowInvokePayload.runContext with the fields the consumer actually needs — deploymentId, specVersion, startedAt, rootRunId — all immutable for the life of a run and known from the run row the producer already holds. A consumer that receives it skips the run fetch: the run-status early exit is enforced by the step_started claim itself (RunExpired → gone, terminal step → skipped), guardDeployment takes the carried identity, and only the fan-out's LAST completer fetches the full run row, lazily, for its inline replay — once per fan-out instead of once per branch. The deployment-mismatch re-route now also preserves stepInput/runContext on the re-enqueued payload. Messages without runContext (older producers) keep the legacy prologue; messages are deployment-pinned, so mixed handling within one run cannot occur. * Address review: terminal-only lazy status gate, terminal-run start fence in local worlds, prologue telemetry, last-completer coverage - The last completer's lazy runs.get result is now gated on isTerminalWorkflowRunStatus (with a debug log): a stale 'pending' read — a run with completed steps has necessarily started — no longer silently abandons the fan-out's continuation; it falls through to the inline replay, whose next entity write is fenced server-side if the run truly ended meanwhile. - world-local / world-postgres now reject step_started on terminal runs even when the step row still reads 'running' (a redelivered start a previous delivery claimed): starting work on a finished run is never valid, and previously the body re-ran with its outcome unconsumable. In-flight steps still write their terminal events unchanged. This closes the adapter gap behind the fetch-free prologue's reliance on the step_started claim as the run-liveness check, and the prologue comment now states the contract precisely. - workflow.step.dispatch_prologue span attribute ('run_context' | 'runs_get') makes fetch-free adoption and the saved round trip observable during version-skew windows. - Restated why the eager redelivery re-ensure survives on the fetch-free path (no run fetch to overlap; still cheaper than the in-band recovery's failed-start round trip). - New two-phase fan-out coverage: a real replay emits the queued step message (asserting the stamped runContext), then its redelivery runs as the LAST completer — zero reads before the step, exactly one lazy runs.get, run completed; plus the stale-'pending' fall-through and the genuinely-terminal skip.