Nathan Rajlich

fix(world-local,world-postgres): make duplicate hook_created idempotent (#2295) * fix(world-local): make duplicate hook_created idempotent Duplicate processing of the same hook_created — same runId, hookId, and token, e.g. cross-process replay or queue redelivery — was being recorded as a hook_conflict in the event log, which then replayed as a self- conflict HookConflictError. The fix mirrors the existing step_created duplicate-correlation path: when the exclusive token claim fails and the existing claim has the same (runId, hookId), throw EntityConflictError so the runtime's existing concurrent-replay catch path swallows it. Different runId or hookId reusing the same token still produces a real hook_conflict. The persisted token claim already carried hookId; only the read schema was dropping it. The schema now preserves hookId (marked optional for backward compatibility with older claim files). Fixes #2283 * fix(world-postgres): make duplicate hook_created idempotent world-postgres has the same gap as world-local was just fixed for: the duplicate-token check in events.create unconditionally writes a hook_conflict event when an existing hook with the same token is found, even when the existing hook has the same (runId, hookId) as the incoming event. The unique partial index on workflow_events does not catch this because the duplicate path inserts hook_conflict, not hook_created. Mirror the world-local fix: when the existing hook's (runId, hookId) matches the incoming event, throw EntityConflictError so the runtime's existing concurrent-replay catch path swallows it. Different runId or hookId reusing the same token still produces a real hook_conflict. Refs #2283 * test(e2e): add regression test for hook_conflict from same-tick replay race Regression test for #1665 / #2283. A parent workflow awaits 6 child workflows with Promise.all; each child does a tiny step and creates one webhook. Awaited children flatten into the parent run, so all webhook creations land on the same workflow body. When their step resolutions align in the same tick the workflow body is re-walked and each pass submits hook_created with the same deterministic (correlationId, token). Before the world-side idempotency fix, the world wrote hook_conflict events for the duplicates and the workflow failed with HookConflictError. With the fix, duplicates throw EntityConflictError (swallowed by the suspension handler), no hook_conflict events appear in the log, and the webhooks resolve normally. Verified locally against world-local: the test fails reliably (3/3) on the unfixed code and passes reliably (5/5) on the fixed code. * test(e2e): rewrite parallelStepsThenWebhookWorkflow to match the actual #1665 repro The earlier version invoked another 'use workflow' function directly from inside the parent workflow, which is not a valid child-workflow invocation (child workflows must be spawned via start()) and didn't mirror the bug shape on #1665 anyway. Rewrite the workflow as a single 'use workflow' function that exactly mirrors Paolo's minimal repro: await Promise.all([stepA(), stepB()]); using webhook = createWebhook(); await webhook; The for-loop runs N independent iterations of that sequence in series, each disposing its webhook via 'using' before the next, to give the timing-sensitive race multiple chances to fire. The race is hard to force deterministically on fast local dev — but the same (runId, hookId) idempotency invariant is covered deterministically by the new unit tests in world-local and world-postgres. This e2e test serves as a higher-level regression net: its assertions (no hook_conflict event in the log, no HookConflictError-failed run) are correct whether the race fires or not, and will catch any future regression on a run that does hit it. * fix(world-local,world-postgres): recover crash-orphaned hook claims/rows instead of suppressing the retry Addresses review feedback on PR #2295. The original idempotency fix made duplicate same-(runId, hookId) hook_created submissions throw EntityConflictError so the suspension handler's concurrent-replay catch path swallows them. But the claim file (world-local) and hook row (world-postgres) are written before the durable hook_created event, and the writes are not atomic. A process / DB interruption between the claim/hook write and the event write leaves an orphaned claim/hook row; the retry then matched the same (runId, hookId), threw EntityConflictError, got swallowed, and the run was permanently left with no hook_created event in the log. world-local: - Add a per-(runId, hookId) in-process mutex (withHookLock) mirroring the existing withStepLock, so two same-tick concurrent calls serialize on the entity write and the dedup branch never observes an in-flight winner mid-write. - In the dedup branch, when the existing claim is for the same (runId, hookId) we are trying to create, check whether the durable hook entity actually exists on disk: - exists → real duplicate: throw EntityConflictError as before. - missing → orphaned claim from a prior crash: fall through and complete the partial write (write the hook entity with overwrite, then emit hook_created via the outer code path). world-postgres: - In the dedup branch, when the existing hook row matches the incoming (runId, hookId), check whether a hook_created event for this (runId, correlationId) already exists in the event log: - exists → real duplicate: throw EntityConflictError as before. - missing → orphaned hook row from a prior crash between hook INSERT and events INSERT: skip the hook insert (the row is already there) and let the outer code path emit hook_created, completing the partial write. Tests: - world-local: pre-seed an orphaned token claim with no matching hook entity, retry hook_created, assert hook entity and hook_created event both land (no hook_conflict, no EntityConflictError). - world-postgres: pre-seed an orphaned hook row with no matching hook_created event, retry, assert hook_created event lands (no hook_conflict, no EntityConflictError). Both tests fail on the prior implementation (EntityConflictError thrown on retry, exact symptom from the review). * fix(world-local): probe the event log (not the hook entity) to detect duplicate hook_created Addresses follow-up review on PR #2295. The previous dedup branch checked whether the durable hook entity existed on disk. But the hook entity is written before the `hook_created` event, and the two writes are not atomic, so a crash between them leaves both the claim file and the hook entity on disk with no event in the log. The dedup branch then matched on `(runId, hookId)`, found the hook entity, threw EntityConflictError, and the suspension handler swallowed the retry — permanently losing `hook_created` from the event log. The fix mirrors what the world-postgres branch already does: probe the run's event log for an existing `hook_created` event for the same `(runId, correlationId)`. The event is the durable record of a successful hook creation; the claim file and hook entity are partial- write artifacts that may exist without the event. - exists → real duplicate: throw EntityConflictError so the runtime's concurrent-replay catch path swallows it. - missing → orphaned partial write (crash at any point before the event landed): re-write the hook entity (with overwrite: true, in case a stale partial copy exists) and let the outer code path emit the hook_created event. Added a new helper findHookCreatedEvent that runs a filtered paginatedFileSystemQuery with limit:1 over the run's events. Regression test "should recover an orphaned hook entity with no matching hook_created event" added — pre-creates a hook, deletes just the hook_created event from disk to simulate a crash between the entity write and the event write, asserts the retry emits a fresh hook_created event (no hook_conflict, no swallowed EntityConflictError). I verified this test fails on the prior fix (throws `EntityConflictError: Hook "hook_orphan_entity_1" already created`, exactly as pranaygp reported) and passes on this commit. The previous test ("should recover an orphaned hook token claim with no matching hook entity") continues to pass — the event-log probe is a strict superset of the entity probe, since a missing entity always also implies a missing event. * fix(world-local): converge same-hook creation across workers via canonical eventId Addresses follow-up review on PR #2295. The previous fix made the dedup branch probe the event log to decide real-duplicate vs orphan-recovery, but the probe and the recovery write are not a single atomic operation. Two workers sharing a data directory (or two retries that lose `writeExclusive(constraintPath)` back to back) could both pass the probe (each observing no hook_created event yet), both fall through to the recovery write, and both append a hook_created event with a different eventId — producing two events in the log for the same (runId, hookId). The in-process `withHookLock` mutex does not help here because it is process-local and tag-specific. The fix persists `eventId` in the durable token claim file (written by the original `writeExclusive(constraintPath)`). On a same-(runId, hookId) dedup match, retries adopt that canonical eventId and rebuild the event with a deterministic createdAt derived from the eventId (a ULID). The outer event write switches from `writeJSON` (check-then-write, TOCTOU) to `writeExclusive` (O_CREAT|O_EXCL via temp-file + hard-link, atomic across processes). Either worker may win the publish; the other throws EntityConflictError which the runtime's existing concurrent-replay catch path swallows. Net result: exactly one hook_created event per logical creation. Backward compatibility: a claim file written before this commit lacks `eventId`. Retries that read such a claim fall back to the event-log probe + fresh-eventId recovery — the legacy behavior that does not converge across workers but cannot regress for freshly- written claims after upgrade. world-postgres already converges across workers via the partial unique index on workflow_events_entity_creation_unique (runId+correlationId+eventType for hook/step/wait_created): the loser's INSERT raises 23505 which is already translated to EntityConflictError. Regression tests: - world-local: `converges same-hook creation across workers to one event` uses two tagged storage instances sharing one data directory and fires 25 paired Promise.allSettled hook_created calls. Expected 25 hook_created events total; before this fix yielded 50. - world-postgres: `converges same-hook creation across concurrent calls to one event` exercises the same shape against the real Postgres unique index. Already converges; the test is a guard against future regressions to the catch path. Verified the world-local test fails on c7b23e1b5 with exactly the shape pranaygp reported (50 events for 25 logical creations) and passes on this commit. The earlier orphaned-claim and orphaned- entity recovery tests also continue to pass. * fix(world-local): converge legacy hook claims via recovery-marker sidecar; replace tag-proxy test with real subprocess workers Addresses follow-up review on PR #2295. Two distinct issues, both flagged by pranaygp as P1: 1. The fallback path for token claims written by versions before eventId was persisted inline (legacy claims after upgrade) still permitted the same cross-process corruption the inline fast path was fixed to prevent. Two processes both reading a legacy claim each generated their own eventId, landed their writeExclusive(eventPath) calls at different paths, and appended two hook_created events for the same (runId, hookId). Existing persisted claims after a real upgrade are exactly the state the crash-recovery branch needs to repair, so leaving the legacy path non-convergent is silent corruption, not backward compatibility. 2. The committed cross-worker convergence test used two tagged storage instances sharing one directory as a proxy for separate processes. But tags change the destination filename (events/wrun_X-evnt_Y.worker-a.json vs ...worker-b.json), so two tagged workers can each writeExclusive their own event at different paths and both fulfill. The Map-by-eventId deduplication in the assertion then masked the duplicate publication, so the test passed for the wrong reason. Implementation: - New HookRecoveryMarkerSchema (`{ eventId, hookId, runId }`) and HookRecoveryMarkerPath helper. The marker is a sidecar at hooks/tokens/<hash>.recovery.json, written via writeExclusive so the first cross-process retry pins its candidate eventId as canonical; subsequent retries read the marker and adopt that eventId. Together with the existing writeExclusive(eventPath) in the outer publish, this gives the legacy-fallback path the same single-event convergence guarantee as the inline-eventId fast path. - pinCanonicalEventIdForLegacyClaim() encapsulates the marker write-or-read. A stale marker for a different (runId, hookId) (token-reuse with leaked state) is overwritten best-effort — the common cross-worker race for the same hook still converges; only the narrow stale-token-reuse case loses convergence. - hook_disposed now also deletes the recovery marker when it deletes the token constraint file, preventing a future legacy recovery for a recycled token from latching onto a stale eventId. - The dedup branch unified: existingClaim.eventId for new claims, pinCanonicalEventIdForLegacyClaim() for legacy ones. Removed the now-redundant findHookCreatedEvent helper — the writeExclusive(eventPath) in the outer publish is the authoritative duplicate-vs-orphan detector. Tests: - New test fixture test-fixtures/hook-race-worker.ts (TypeScript, run via child_process.fork with tsx as execPath — tsx is a transitive dev dep via vitest). Each subprocess gets its own createStorage(testDir) so the in-process hookLocks Map cannot serialize across workers. - Replaced the tag-proxy test with "converges same-hook creation across separate OS processes to one event". Spawns workerCount subprocesses, releases them from a barrier into the same hook_created, asserts exactly one fulfilled + (N-1) rejected with EntityConflictError, and asserts directly on the raw events.list() result (no Map dedup) that the number of hook_created entries equals the number of logical creations. - Added "converges same-hook creation across processes when only a legacy token claim exists". Same shape, but pre-seeds the legacy claim format (`{ token, hookId, runId }` with no eventId) before each race. Verified to FAIL on 7ce66551b (both subprocesses fulfill, no convergence) and pass on this commit. - Also verified the new-eventId subprocess test FAILS when the event write is reverted to writeJSON (TOCTOU), confirming it exercises the writeExclusive-based cross-process arbitration. Both prior orphaned-claim / orphaned-entity recovery tests also continue to pass. * fix(world-local): per-lifetime recovery markers, restore event-log probe, fix CI tsx resolution Addresses three P1 review comments on PR #2295. 1. Stale recovery marker leaking across token-reuse lifetimes (pranaygp): The previous marker path used `hashToken(token)` so a stale marker for run A could leak into run B's recovery when the same token was reused after run A terminated through normal lifecycle. `deleteAllHooksForRun()` and tagged `world.clear()` deleted the token constraint and hook entity but NOT the marker sidecar, so the next legacy claim on the same token entered the stale-marker overwrite branch and the workers overwrote it non-atomically, yielding divergent publication. Fix: - Marker path now hashes `(token, runId, hookId)` together (`hookRecoveryMarkerPath` in storage/helpers.ts). Different lifetimes can never share a marker, so the stale-marker overwrite branch is removed entirely. - `hookRecoveryMarkerPath` is moved to helpers.ts and shared across events-storage.ts, hooks-storage.ts, and index.ts. - `deleteAllHooksForRun()` and tagged `world.clear()` now also delete the recovery marker for each hook (disk hygiene; per- lifetime identity makes leaks no longer corrupting). - `hook_disposed` now uses the new per-lifetime marker path too. 2. Duplicate `hook_created` event when a legacy claim's event was already published (VADE bot, also implied by pranaygp's analysis): Removing the event-log probe from the legacy fallback let a post- upgrade retry pin a new canonical eventId via the marker and publish a duplicate event at that path, even when the original pre-upgrade writer had already successfully published the event with its own (different) eventId. Fix: - Restore `findExistingHookCreatedEventId()` (renamed and made to return the eventId for clearer semantics). - Legacy fallback now probes the event log BEFORE pinning the marker; if a matching `hook_created` event already exists, throw `EntityConflictError` so the runtime's concurrent-replay catch path swallows the retry. - Inline-`eventId` fast path does NOT need the probe — the claim itself is the durable convergence key. 3. CI failure: tsx not resolvable under pnpm isolated linking (pranaygp; confirmed by ubuntu/windows unit test 60s timeouts): The previous test hard-coded `node_modules/.bin/tsx` assuming tsx would be hoisted there. But tsx was only a transitive peer dep via vitest, and pnpm's isolated linking does NOT link transitive peer deps into the workspace bin after a fresh install — so neither root nor package-local `.bin/tsx` existed in CI, the subprocess fork never started, and the barrier hung until vitest killed the test. Fix: - Add `tsx` as a direct `devDependency` of `@workflow/world- local` (pinned to 4.20.6 to match the existing transitive resolution). - Resolve via `import.meta.resolve('tsx/package.json')` and read the `bin` field dynamically, so we adapt to wherever pnpm links tsx for this package — not a hard-coded layout. - Lazy-init the resolver (no module-load IIFE) so an absent tsx fails only the convergence tests, not all 376 tests in the file. - Surface a clear error message if resolution fails, calling out the cause (transitive vs direct deps) for future readers. Also: harden the barrier helper so `error` events and pre-ready exits resolve BOTH `readyPromises` and `donePromises`, then `SIGKILL` siblings. Previously a broken child only resolved `donePromises`, leaving `Promise.all(readyPromises)` pending until the per-test timeout (60s in CI). Regression tests added: - `legacy claim whose hook_created event was already published does not append a duplicate event` — pre-seeds a legacy claim AND a pre-existing `hook_created` event with a different eventId, asserts the retry throws EntityConflictError and the log still has exactly the original event. - `converges legacy claim recovery across run lifetimes after token reuse` — runs pranaygp's full lifecycle path: race subprocess workers on run A's legacy claim, terminate run A via `run_completed` (triggers `deleteAllHooksForRun`), reuse the token in a legacy claim for run B, race subprocess workers again, asserts exactly one fulfillment + one `EntityConflictError` per race and exactly one `hook_created` event per run. Both new tests verified to fail on 2c673e436 (after rebuilding): the published-event test throws via duplicate publish instead of EntityConflictError, the token-reuse test sees both run B workers fulfill (2 events instead of 1). The existing orphaned-claim and orphaned-entity recovery tests also continue to pass. CI loop confirmed to be repaired locally by spawning subprocesses via the new resolver and intentionally breaking the worker fixture to verify the helper fails fast (~500ms) instead of hanging at the barrier. * fix(world-local): defer hook entity write until event publish commits Addresses karthikscale3's P1 review comment on PR #2295. The dedup-recovery path used to write the hook entity BEFORE the outer event publish proved whether the attempt was repairing a missing event or just colliding with an already-published `hook_created`. For already-committed duplicates, the event write then throws `EntityConflictError`, but the hook entity had already been overwritten with the retry's payload — leaving the durable hook entity and the event log inconsistent (e.g. the entity reflects the retry's metadata while the event still carries the original). karthikscale3 reproduced this on the prior head by creating `hook_created` with metadata `{ v: "a" }`, then retrying the same `(runId, hookId, token)` with metadata `{ v: "b" }` and `isWebhook: false`: the retry threw `EntityConflictError` but `hooks.get()` returned the retry's payload. Fix: defer the hook entity write until AFTER the outer `writeExclusive(eventPath)` commits. The branch now only captures the entity-to-write and its overwrite options; the actual write happens immediately after the event publish in the shared trailing block. A retry that ends in `EntityConflictError` (the event was already published) now leaves the entity untouched. The first-writer happy path and all recovery paths (orphaned- claim, orphaned-entity, cross-worker convergence, legacy claim, token-reuse across lifetimes) are unaffected — they all reach the event publish successfully, then the entity write runs as before. Regression test `does not mutate an already-committed hook entity when a duplicate hook_created retry collides` added to world-local: runs karthikscale3's exact scenario and asserts the persisted entity still carries the original metadata and isWebhook. Verified to fail on the prior commit (persisted metadata = 0xbb instead of 0xaa) and pass on this commit after rebuilding. Parallel guard test `does not mutate an already-committed hook entity when a duplicate hook_created retry collides` added to world-postgres. Postgres already protected this via `onConflictDoNothing()` on the hook INSERT, but the test guards against a future regression that adds an UPDATE/UPSERT to the dedup path. * refactor(world-local): per-instance in-process locks; drop tsx subprocess test plumbing You were right that the tsx subprocess machinery was overkill for a storage-level convergence test. Replaced with a simple two-instance in-process test that exercises the same cross-process semantics without spawning anything. The trick: `stepLocks` and `hookLocks` were module-level Maps shared by all `createEventsStorage` calls in the same process. Move them inside the function so each `createStorage(dir)` call gets its own lock map. Two storage instances sharing one data directory then behave exactly like two separate OS processes: - independent in-process `hookLocks` Maps (no in-process serialization between them), and - a shared filesystem (so the on-disk `writeExclusive` claim / marker / event publish primitives are the only thing arbitrating convergence). This is also a real architectural improvement — the global lock map was always a leaky abstraction that made unit-test simulation of the cross-process path awkward. Changes: - `stepLocks` and `hookLocks` moved from module scope into `createEventsStorage`. `withStepLock` and `withHookLock` wrappers collapsed into direct `withInProcessLock(map, key, fn)` calls at the two call sites that need them. - The three convergence regression tests in `storage.test.ts` now use `const workerA = createStorage(testDir); const workerB = createStorage(testDir);` and race `Promise.allSettled` of `events.create` from both — no subprocess, no IPC, no barrier helper, no `raceHookCreatedAcrossProcesses`. Same assertions (exactly one fulfillment + N-1 `EntityConflictError` per race, raw `events.list()` shows exactly one `hook_created` per logical creation — no Map dedup) so the regression catches are identical. - Removed: `tsx` devDep, `test-fixtures/hook-race-worker.ts`, `HOOK_RACE_WORKER` / `resolveTsxLoaderUrl` / `TSX_BIN` / `raceHookCreatedAcrossProcesses` and the `fork`/`fileURLToPath` imports they pulled in. Verified (after rebuilding world-local): - All 379 tests pass on macOS in ~1s (was ~6.7s with subprocesses). - Convergence tests confirmed to still catch the bugs: temporarily reverted the `eventId = canonicalEventId` adoption → both workers fulfilled (2 events instead of 1). Temporarily reverted the legacy-claim marker pin → same: both workers fulfilled. - No subprocess machinery means no Windows-specific quirks (cli.mjs shebang, .cmd wrappers, .bin hoisting under pnpm isolated linking, etc.) that produced the Windows CI 60s timeouts. - World-postgres still has its own parallel guard test for the karthikscale3 "no-mutate-on-duplicate" regression; that one exercises real DB concurrency and is unaffected by this change. Full repo `pnpm test` (43 packages) and the `parallelStepsThenWebhookWorkflow` e2e test against world-local both green. * fix(world-local): repair event-first hook orphans from the persisted event; skip #1665 e2e on world-postgres - A crash between the hook_created event publish and the deferred hook entity write left the event committed with the entity missing and unrepairable (retries threw EntityConflictError without materializing the entity). Retries now rebuild the entity from the PERSISTED event's payload — never the retry's eventData — via a race-safe writeExclusive, on both the canonical-eventId collision path and the legacy-claim probe path. - Skip parallelStepsThenWebhookWorkflow e2e on world-postgres: the same-tick replay pattern surfaces a separate pre-existing step_started ordering bug there (#2331). --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com>

[core][world][world-vercel] Add `World.createRunId()` and region-aware queue routing (#1981) * [world-vercel] Add /run-id sub-export with tagged ULID encode/decode Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a ULID-shaped string used for workflow run IDs. Tagged values remain valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip through any system that accepts ULIDs. * [world-vercel] Add string-value assertions to run-id tests Add exact-string expectations for encoded outputs at known inputs, covering the default region/version pair, numeric region IDs, version overrides, boundary values (all-zero, all-max), the dirty-input overwrite case, and the lexicographic-order checks. Also adds an explicit byte-array expectation for the canonical ULID-spec example string and an additional first-char-range coverage test for isTagged. * [world-vercel] Remove internal-repo reference from regions doc comment * [world-vercel] Address PR review feedback on run-id sub-export - isTaggedString now fully validates the input as a 26-char Crockford Base32 ULID (delegating to ulidToBytes) instead of only inspecting the first character. This fixes false positives on inputs like '4UUUU...' that have a valid tag-bit position but invalid chars later in the string. - isTagged() now accepts `unknown` to match its documented behavior of safely rejecting non-string inputs without requiring callers to cast. - Introduce `RegionKey` for the full set of keys including 'unknown', and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the return type of `lookupRegion` and the `DecodedRunId.region` field accurately reflect that 'unknown' is never produced. Updates `encode` to reject 'unknown' as a region code string at runtime (callers wanting the unknown sentinel should pass numeric 0). * [world] [core] [world-vercel] Add World.createRunId() and region-aware queue routing - @workflow/world: add optional createRunId(input?) to the World interface so worlds can mint run IDs with embedded metadata, and add an optional 'region' field to QueueOptions for per-message routing hints. - @workflow/core: start() now delegates run ID generation to world.createRunId() when defined (falling back to a monotonic ULID otherwise), and accepts a new 'runIdInput' option that is forwarded verbatim to createRunId. When runIdInput.region is a string, it is also threaded onto the queue options so the initial workflow message is dispatched to the matching region. - @workflow/world-vercel: implement createRunId() to mint region-tagged ULIDs, preferring an explicit runIdInput.region and falling back to the VERCEL_REGION env var. The queue now resolves its destination region from (in order): an explicit opts.region, the region embedded in the payload's tagged run ID, the VERCEL_REGION env var, and finally a hardcoded 'iad1' default. This replaces the previous unconditional 'iad1' region passed to the @vercel/queue client. Monotonicity within a process is preserved by tracking the last emitted run ID and bumping the bit immediately above the 11-bit metadata window when a same-ms collision would otherwise occur, then re-stamping the requested region/version on top so metadata remains stable. * [core] [world] [world-vercel] Pass full StartOptions to World.createRunId Drop the dedicated 'runIdInput' field on StartOptions and forward the entire options bag to world.createRunId() instead. This keeps the public API surface smaller and lets each World pick the fields it recognises (e.g. world-vercel reads 'region'). The top-level 'region' option remains on StartOptionsBase and is also threaded onto the queue's per-call region opt when set. * Address review feedback: doc fixes and deterministic same-ms tests - Document the final iad1 fallback in QueueOptions.region (world) - Correct the World.createRunId doc: start() always passes an object - Fix the clientOptions comment: the handler client omits region and relies on SDK auto-detection + the ce-vqsregion header for acks - Fix a misleading QueueClient-construction comment in queue.test.ts - Freeze time in the same-ms monotonicity test so it deterministically exercises the intended path, and add a test covering the bump-above-metadata fallback when the region changes mid-millisecond Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: keep workflow-server override rewrite-compatible Export WORKFLOW_SERVER_URL_OVERRIDE while keeping the one-line const shape that workflow-server's cross-repo e2e test automation rewrites. Update world-vercel tests to import that exported value for mock origins and URL expectations instead of duplicating the temporary preview URL. * fix(world): clear region tag bit before ULID timestamp validation Region-tagged run IDs set the high bit of the ULID timestamp byte. The shared world timestamp validator used raw decodeTime(), so current tagged run IDs appeared thousands of years in the future and were rejected before reaching workflow-server. Clear the tag bit before decoding, matching the workflow-server behavior, and cover tagged IDs in tests. * fix(world-vercel): validate tagged runId timestamps via run-id decode Keep @workflow/world's ULID helpers generic; they should not know about world-vercel's region-tagged run ID layout. Instead, world-vercel decodes its tagged runId to the original ULID before using the shared timestamp validator for run_created events. Add a world-vercel regression test that a current sfo1-tagged runId passes validation. * fix(world-vercel): default run ID region to iad1 instead of unknown When neither an explicit region option nor VERCEL_REGION is available, createRunId minted a tagged ULID with the unknown (0) region sentinel, producing the tagged: true, region: null state. The server already resolves unknown/untagged runs to DEFAULT_VERCEL_REGION (iad1), so mint a concrete iad1 tag instead, keeping every run ID self-describing and routable. * test(e2e): use verbose reporter + per-test start heartbeat The default vitest reporter buffers per-file output, so a stalling e2e test produces no output until its timeout — making CI look like a silent 30-minute hang. Switch the e2e CI invocations to the verbose reporter (prints each test result as it completes) and emit a '[e2e] ▶ start:' heartbeat to stdout at the start of every test (bypassing vitest's console buffering) so a stuck test is immediately identifiable in the live CI log. * test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at combined 527+529 preview Temporarily target the workflow-server combined-527-529-preview deployment, which bundles platform-directed multi-region routing (vercel/workflow-server#527, incl. the iad1 hook pin) and durable stream state (vercel/workflow-server#529), so e2e can validate the full multi-region path end-to-end. Revert to empty on main. * fix(core): region-tag the health-check correlationId The health-check response is delivered over a Redis stream whose name (and synthetic run ID) embed the correlationId. Under platform-directed routing the responding endpoint and the polling reader can be served from different physical regions; Redis is physical-region-local, so the correlationId must carry the region for both sides to resolve the same backend. Generate the correlationId via world.createRunId() (a region-tagged ULID) when the world provides it, falling back to a plain ULID for worlds that don't tag IDs (e.g. local, single-region). The synthetic wrun_hc_<id> run ID then carries the region; workflow-server's region middleware decodes it. * Address review feedback: validate region overrides, reset server override - queue: validate opts.region and VERCEL_REGION against the known region table before routing, ignoring unrecognised codes so a bad override can't clobber the payload-derived region (Copilot) - add isKnownRegionCode() runtime guard to run-id/regions - reset WORKFLOW_SERVER_URL_OVERRIDE to '' (must be empty on main) - fold the within-PR iad1-default changeset into the main world-vercel changeset and delete it (review) - start.test: declare specVersion on createRunId mock worlds now that the merged world-compatibility check requires it - cover the new region-validation fall-through paths in queue.test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at wave-1 multi-region preview BRANCH-ONLY — revert the override to '' before merge (lint enforces). Points this PR's e2e/benchmark runs at the wave-1 multi-region workflow-server preview (vercel/workflow-server#590: iad1+sfo1+fra1 serving, staging data backends) so region-tagged runs are validated against real multi-region serving end-to-end. Also makes the unit-test mock origins in events-v4.test.ts and trace-propagation.test.ts override-aware (same pattern the rest of the file and utils.test.ts already use), so the suite passes whether or not the override is set — these two files were the only spots hardcoding https://vercel-workflow.com. * test(e2e): Vercel multi-region suite for start()'s region option Adds a dedicated e2e suite validating @workflow/world-vercel region routing end to end, run as its own CI job (e2e-vercel-multi-region) against the nextjs-turbopack workbench only — deliberately separate from e2e.test.ts, which runs as a matrix across all worlds/frameworks where Vercel-specific multi-region behavior doesn't apply. - workbench/nextjs-turbopack/vercel.json: deploy to iad1+sfo1+fra1 so region-routed flow messages have a function to land on in each region. - workflows/99_e2e.ts: regionProbeWorkflow returns the VERCEL_REGION observed by both the workflow and a step, so tests can assert the run EXECUTED in the intended region (not just that it was tagged). - packages/core/e2e/e2e-region.test.ts: per-region cases assert 1) start(..., { region }) mints a region-tagged run ID (decoded via @workflow/world-vercel/run-id), 2) the workflow + step both observed VERCEL_REGION === region, 3) the server reports the run completed; plus a concurrent all-regions case guarding against cross-region misrouting under simultaneous multi-region traffic. Skips on local deployments. - .github/workflows/tests.yml: new e2e-vercel-multi-region job mirroring e2e-vercel-prod's env/deployment-wait, running only the new suite. * test(e2e): start region probes in-function; fix getWorld await The first multi-region CI run surfaced two issues: 1. sfo1/fra1-tagged runs executed in iad1. The suite started runs from the external test process, which uses the api.vercel.com token proxy — and the proxy's queues path forwards every send to the region-less VQS host (the world's proxy-mode resolveBaseUrl ignores the region argument, and the proxy's x-vercel-vqs-api-url escape hatch only allowlists vqs-server-*.vercel.sh preview hosts). Production traffic publishes IN-FUNCTION (direct regional queue routing), so the suite now triggers start() through a new workbench route (/api/e2e-region-start) and rehydrates the run with getRun() — testing the path production actually takes. Proxy-mode regional queue routing is a known gap to address separately in api-workflow. 2. TypeError on world.runs.get: getWorld() is async and was called without await. * test(e2e): cover explicit and implicit region starts in the multi-region suite With regional VQS routing now working through the api.vercel.com proxy (vercel/api#79056 + #2789 + this branch's per-send region resolution), the suite covers both start configurations, asserting the same three properties for each (region-tagged run ID, execution in the intended region via VERCEL_REGION echoed in the return value, server-side completion): 1. EXPLICIT: start(..., { region }) called directly in the vitest runner — publishes through the token proxy, per-send region carried by x-vercel-queue-region. Restores the direct-start shape the suite had originally, plus the concurrent all-regions case. 2. IMPLICIT: dedicated per-region workbench routes (/api/e2e-region-implicit/{iad1,sfo1,fra1}), each pinned to a single region via a per-function 'regions' entry in the workbench vercel.json, calling start() with NO region option — createRunId derives the tag from the minting function's VERCEL_REGION. The test also asserts the route reported executing in its pinned region, so the implicit-tagging assertion can't pass vacuously. Replaces the interim /api/e2e-region-start route (explicit region via request body), which existed to work around the pre-#79056 proxy gap. * Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — wave-1 multi-region serving is in production workflow-server#590 (iad1+sfo1+fra1 serving) merged and deployed to production and the e2e backend, so this branch's e2e/benchmark runs no longer need to target the wave-1 preview. Restores the empty override the No Test Overrides lint job enforces for merge. The override-aware unit-test origins (events-v4/trace-propagation) stay — they are correct under any override value. * test(e2e): cross-region stream visibility (iad1 writer, sfo1 reader) Regression coverage for a backend bug that made cross-region stream reads report zero chunks on IN-PROGRESS streams (completed streams were unaffected), which forced the multi-region serving rollback. The new case exercises exactly that geometry: - crossRegionStreamWorkflow (99_e2e.ts) writes N chunks to the default output stream, then holds the stream OPEN for 45s before closing — the in-progress window is the point, since completed streams are the easy case. - The e2e starts it with region iad1, waits (same-region, via the api.vercel.com proxy) until all chunks are written, asserts the run is still 'running', then reads through a new sfo1-pinned workbench route (/api/e2e-stream-read/sfo1) that returns getTailIndex() plus its VERCEL_REGION. The reader's region served none of the stream's writes, so the reported chunk count must come from the backend's cross-region stream metadata. The test fails loudly if the route isn't actually executing in sfo1. Also bumps the explicit-region test timeout to 120s: the first case in the file absorbs every cold start at once (fresh workbench instances in up to three regions plus a cold backend preview) and was observed just over the 60s default. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE points at a multi-region backend preview that includes the fix, so this validates cross-region stream visibility end-to-end before multi-region serving is re-enabled. * test(e2e): extend multi-region suite to all 19 provisioned regions Points the suite at an all-regions backend preview and widens coverage from the wave-1 trio to every provisioned region: - Explicit path: a single concurrent all-regions case starts one tagged run per region (one shared cold-start window instead of 19 sequential ones) and aggregates per-region failures so a single region's breakage reports alongside the full picture. The trio keeps its detailed per-region cases and the 9-way concurrent-isolation case. - Implicit path: workbench gains a region-pinned /api/e2e-region-implicit/<region> route per provisioned region (19 total, shared handler), the workbench itself now deploys to all of them, and the test.each covers the full set with per-case timeouts for regional cold starts. - Multi-region CI job timeout 20m -> 35m for the sequential implicit cases. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE now targets the all-regions backend preview instead of the previous (stale, since-merged) fix preview. * test(e2e): tolerate geo-adjacent execution of queue callbacks The first all-regions run surfaced a subtle execution-locality behavior: queue delivery is guaranteed to the tagged region's dataplane and the delivery callback egresses from that region, but the consumer invocation's execution region is chosen by where that callback enters Vercel's edge — and adjacent regions can geo-resolve to each other's functions. Observed live: kix1-tagged runs (callback egressing from Osaka) deterministically executing in hnd1/Tokyo on both the explicit and implicit paths, with tagging, data placement, and completion all still strictly kix1. expectRunInRegion now asserts execution lands in the tagged region OR one of its geographic neighbors (EXECUTION_ADJACENCY), while run-ID tagging and server-side completion remain strictly the requested region. Gross misrouting (e.g. kix1 -> iad1) still fails. * Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — all-regions serving is in production The all-regions workflow-server rollout is deployed and serving production traffic from every Vercel region, so this branch's e2e no longer needs to target a branch preview. Restores the empty override the No Test Overrides lint enforces for merge. With this the PR is complete: region-tagged run IDs, region-aware queue routing, and the multi-region e2e suite (explicit + implicit + all-regions + cross-region streams) all validate against the production-default backends. * docs: fix three stale comments flagged in review - start.ts: StartOptionsBase.region fallback is iad1, not the unknown sentinel (createRunId always mints a concrete routable region) - queue.ts: example used a nonexistent start({ runIdInput }) API; the real option is start({ region }) - events.ts: decode() clears only the tag bit (top bit of the 48-bit timestamp field) — it does not restore the original untagged ULID; reword to say what actually matters for timestamp validation * test(e2e): cover hook resolve/resume for runs owned by non-iad1 regions Hooks are resolved by opaque token, which carries no region hint, so lookup and resume must work regardless of which region owns the run's data. Exercises the full follow-up-message path on sfo1- and fra1-tagged runs: create inside the workflow, resolve by token from the test process, resume twice sequentially, and assert payload order and completion. Regression coverage for the failure mode where the first message to a hook-driven app on a non-iad1 run worked but every follow-up failed with 'Hook not found'. --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Serialize `run_failed`/`step_failed` errors through serialization pipeline (#1851) * Serialize run_failed/step_failed errors through serialization pipeline Switch run_failed, step_failed, and step_retrying events to persist the full thrown value via the workflow serialization pipeline (as SerializedData / Uint8Array) instead of a lossy { message, stack, code } StructuredError shape. Consumers hydrate via hydrateRunError / hydrateStepError to reconstruct the original thrown value, preserving Error subclass identity, cause chains, and custom properties. - WorkflowRun.error and Step.error are now SerializedData - WorkflowRun gains a top-level errorCode plaintext field - WorkflowRunFailedError.cause is now the hydrated thrown value - Adds world-postgres migration 0010_add_error_code.sql - Legacy pre-pipeline errorJson records surface as undefined on read * Update Next.js workbenches for new WorkflowRunFailedError.cause type cause is now `unknown` (the hydrated thrown value) rather than `Error & { code }`. Defensively extract Error-shaped fields when the hydrated value is an Error, otherwise round-trip the raw value, and expose the new `errorCode` classification field. * Update docs for WorkflowRunFailedError.cause: unknown The hydrated `cause` is now `unknown` (the original thrown value through the serialization pipeline) and the error classification has moved to the top-level `errorCode` property. Update the two affected docs pages and the `TSDoc` interface to reflect the new shape, and narrow `cause` with `instanceof Error` before accessing fields. * Expand test coverage for the run/step error serialization pipeline Unit tests: - 19 new dehydrate/hydrate{Step,Run}Error round-trip tests covering FatalError, plain Error, built-in Error subclasses, non-Error thrown values (string, plain object), cause chains, encryption round-trip, the binary format prefix contract, and the unserializable / unknown- format error paths. - 5 new tests for Run.returnValue when the run is failed: hydrated FatalError + cause as cause, plain Error preservation, non-Error thrown values surfaced verbatim, cross-class cause chains, and the hydration-failure fallback that still surfaces errorCode. E2E tests (new, in 99_e2e.ts + e2e.test.ts): - Step throw → workflow catch round-trips a FatalError with a TypeError cause chain, asserting class identity, fatal marker, and cause name + message all survive the step_failed event pipeline. - Workflow throw → run_failed reaches status with the new top-level errorCode metadata exposed (cause-shape coverage lives at the unit level, since the SWC plugin's class registration is not invoked in the plain-Node e2e runner). - Workflow throw of a non-Error value round-trips that value verbatim as WorkflowRunFailedError.cause. Adjustments to existing assertions: - error.cause is now ; tests narrow with and use the new top-level field instead of . - step.error / run.error from CLI --withData are now hydrated payloads: unregistered class instances surface as Instance refs whose carries the original message + stack. Observability hydration: - hydrateStepIO / hydrateWorkflowIO in serialization-format.ts now hydrate the field via hydrateData, so the CLI and web UI continue to surface readable run/step error messages and stacks. * Tighten error serialization changeset description * Trim error serialization changeset to a single sentence * Resolve FatalError/RetryableError revivers via cross-realm registry When a workflow runs in a Node `vm` context, its bundled `@workflow/errors` is a different module instance than the host's import (separate prototype chains, separate class identity). Calling `new FatalError(...)` from the host-side reviver produces a host-realm instance that fails `err instanceof FatalError` checks in the workflow code — even when the serialized payload was correctly tagged via the dedicated `FatalError` reducer. Surfaced by the local-prod e2e "step throw round-trips FatalError" test on Next.js Turbopack: each route gets its own bundled chunk, so the flow handler's `@workflow/errors` and the workflow VM bundle's `@workflow/errors` are two distinct copies of the same module. Fix: - Each bundled copy of `@workflow/errors` self-registers its `FatalError` and `RetryableError` classes on `globalThis` via `Symbol.for("@workflow/errors//FatalError")` / `Symbol.for("@workflow/errors//RetryableError")`. First load wins per realm; the descriptor is non-writable / non-configurable to make accidental clobbering loud. - The revivers in `@workflow/core`'s common reducers module read the consumer's `globalThis` (passed in as `global`) to pick up the realm-local class, falling back to the host-imported class when no registration is present (e.g. in the CLI / test runner). * Use `types.isNativeError` to remap workflow stacks across VM realms The runtime's run-failure path computes a source-map-remapped stack and then assigns it back onto the thrown value via `if (err instanceof Error) err.stack = errorStack`. Workflows run inside a Node `vm` context, so a workflow-thrown error is an instance of the VM realm's `Error` — `instanceof` against the host realm's `Error` returns `false`, the assignment is skipped, and the serialized `run_failed` event carries the un-remapped (bundled-line- number) stack instead of the source-mapped one. Switch the gate to `types.isNativeError`, which uses V8's internal type tag and works across realms — same approach already in place for the serialization reducers. Caught by the local-prod e2e "nested function calls preserve message and stack trace" and "cross-file imports preserve message and stack trace" tests, which assert that the persisted run-error stack contains `99_e2e.ts` / `helpers.ts`. * Sync CLI revivers with core + add toJSON shim for Error subclasses Two issues with the CLI's hand-rolled reviver list: 1. It hadn't been updated for the new first-class Error subclass reducers (`TypeError`, `RangeError`, `FatalError`, `RetryableError`, etc.). devalue throws "Unknown type X" when it encounters a reduced value with no matching reviver, and `hydrateResourceIO` swallows that error and surfaces the raw `Uint8Array` payload — so `step.error` / `run.error` showed up as raw byte dumps in `workflow inspect` output. 2. Even with all the right revivers, `Error.prototype`'s `message` / `stack` / `cause` are non-enumerable, so `JSON.stringify` (used by `workflow inspect --json`) drops them — leaving the subclass-specific enumerable fields (e.g. `FatalError.fatal`) visible but the actual error data missing. Fix: - Build the CLI reviver set on top of `getCommonRevivers()` from `@workflow/core` so the CLI stays in sync with the runtime's reducer set automatically. New core reducers/revivers will Just Work without any CLI-side change. - Wrap each Error reviver from the common set with a thin shim that attaches a non-enumerable `toJSON` method to the produced `Error` instance. `JSON.stringify` calls `toJSON` and gets a full object (`name` + `message` + `stack` + `cause` + any enumerable subclass fields like `fatal` / `retryAfter` / `errors`); `util.inspect` ignores `toJSON` and renders the canonical `Error: msg\\n at ...` format. Best of both worlds for CLI output without compromising the runtime hydration path. Caught by the local-prod e2e "basic step error preserves" and "cross-file step error preserves" tests, which read `failedStep.error.message` / `.stack` from the CLI's JSON output. * Clarify parseErrorJson JSDoc to match its always-null return The previous JSDoc described preserving legacy values "for best-effort hydration" which contradicted the implementation, where legacy errors are intentionally surfaced as absent (the pre-pipeline shapes can't be hydrated by the new error revivers). Rewrite the comment so the contract matches behavior. Also rename the now-unused parameter to `_errorJson` to reflect that the function ignores it. Caught by a code review on #1851. * Refine error-handler ergonomics on the step / run hot paths Three review-driven adjustments that all touch the queue handlers and their interaction with the error serialization pipeline: 1. Memoize the per-run encryption key fetch. The step handler used to eagerly fetch + import the key at the top of every step delivery so the value would be in scope for every potential dehydrateStepError path. That pessimized step-started early-return cases (the fetch happens unconditionally even when the step never reaches user code) and required duplicating the same boilerplate at four call sites in runtime.ts. Introduce `memoizeEncryptionKey(world, run)` in runtime/helpers.ts that returns a lazy, single-fetch accessor; step-handler / runtime call sites use `await getEncryptionKey()` instead. The first caller pays the fetch cost, subsequent callers await the cached promise, and steps that fail before any encryption-aware work happens skip the fetch entirely. 2. Preserve the prior attempt's serialized error as the cause on the defensive max-retries-exceeded `step_failed` re-invocation guard. The existing comment explicitly opted out of cause attachment, but the symmetric post-failure path below already does this and the reviewer is right that consumers shouldn't have to walk the step_retrying event history to recover the underlying error. Best- effort: if hydration of the prior `step.error` throws, fall back to a FatalError without cause rather than letting the event write itself fail. 3. Document the intentional `unflatten` throw in `hydrateStepError` / `hydrateRunError` for non-Uint8Array input. SDK version is pinned per workflow run via skew protection so the non-binary branch is dead in production; if a misshapen value reaches it, surfacing the throw via the surrounding o11y try/catch is more debuggable than masking it. Add a comment so future reviewers don't reach for a defensive fallback. A standalone `falls back to plaintext` suggestion on the run_failed key fetch was rejected: when encryption is configured we should fail loudly rather than silently emit plaintext error data. The queue's redelivery semantics will retry the key fetch; persistent KMS outages get logged with the existing "persistent error preventing the run from being terminated" message rather than a security regression. * Hydrate `event.eventData.error` in event listings `hydrateEventData` enumerated the per-event fields that need hydration (`result`, `input`, `output`, `metadata`, `payload`) but omitted the new `error` field on `step_failed`, `step_retrying`, and `run_failed` events. Without this branch, o11y tools that list events (e.g. `workflow inspect events`) surface the raw `Uint8Array` payload instead of a hydrated `{ name, message, stack, … }` object even though the entity-level `Run.error` / `Step.error` paths already hydrate. Mirrors the existing per-field branches; the `try/catch` leaves the field un-hydrated on parse failure rather than failing the whole event view. Adds a unit test. * Use `.is()` static checks in `classifyRunError` for cross-realm safety Workflows execute inside a separate `vm` realm: the `WorkflowRuntimeError` class bundled into the workflow code and the host-imported one are distinct constructors, so an `err instanceof WorkflowRuntimeError` check on a VM-thrown error returns `false` and we'd misclassify genuine runtime errors (corrupted event log, missing timestamps, workflow/step not registered) as user errors. Switch to each subclass's `.is()` static (a name-based duck check that works across realms). Since `WorkflowRuntimeError.is` only matches its own concrete name, enumerate every concrete subclass we want to recognize (`StepNotRegisteredError`, `WorkflowNotRegisteredError`) in a `RUNTIME_ERROR_CHECKS` table; keep that table in sync with the class hierarchy in `@workflow/errors`. Existing `classify-error.test.ts` already covers `WorkflowRuntimeError` and `WorkflowNotRegisteredError` cases — both still pass. * Add e2e coverage for step throws of non-Error values We had `errorWorkflowThrowNonErrorValue` (workflow body throws a plain object — round-trips verbatim as `WorkflowRunFailedError.cause`) but no symmetric coverage for the step-throw side. Step-throw goes through a different code path: non-Error values aren't recognized as `FatalError` (no `name === 'FatalError'`) nor `RetryableError`, so they take the transient retry path. After max retries the runtime wraps the original thrown value as `cause` on a fresh `FatalError` which the workflow's catch block then sees. Add a workflow that throws a recognizable plain object from a step with `maxRetries = 0` (so we exhaust on first attempt and avoid a long test wait) and a workflow that asserts the wrapped FatalError shape: `isFatal`, `instanceof FatalError`, message includes the original object's serialized form, `cause` is the original non-Error object verbatim with structure preserved. Documents the current retry-then-wrap behavior so any future change to "non-Error throws skip retries" semantics has to update the test. * Note legacy postgres error-data loss in the run/step error changeset Pre-upgrade failed runs that wrote into world-postgres's deprecated `error` text column can't be hydrated through the new pipeline (the shape is incompatible with the new revivers). The new runtime intentionally surfaces them as `error: undefined` on read; the original payload is still readable directly from the `errorJson` column for manual inspection. Add a one-sentence note to the changeset's migration text so consumers upgrading don't get blindsided by suddenly-empty error fields on historical runs.

Refactor: Extract serialization into modular architecture and wire into existing pipeline (#1299) * Add serialization module foundation: types, codec interface, format prefix Start of the serialization refactor (separate from snapshot-runtime). New files: - serialization/types.ts — SerializationFormat enum, SerializableSpecial interface, Reducers/Revivers types - serialization/codec.ts — Codec interface with formatPrefix, serialize, deserialize, and optional deserializeLegacy - serialization/format.ts — Format prefix encode/decode/peek, moved from the monolithic serialization.ts The Codec interface enables future alternative formats (CBOR, JSON) while keeping the devalue implementation as the current default. * Add reducers, devalue codec, encryption, and mode-specific modules Serialization refactor Phase 1: create the new module structure alongside the existing monolithic serialization.ts (which continues to work). New files: - serialization/reducers/common.ts — Date, Error, Map, Set, URL, BigInt, typed arrays, Headers, Request, Response, RegExp, URLSearchParams - serialization/reducers/class.ts — Class/Instance with WORKFLOW_SERIALIZE/ DESERIALIZE support - serialization/reducers/step-function.ts — StepFunction with closure vars - serialization/codec-devalue.ts — devalue Codec implementation - serialization/encryption.ts — composable encrypt/decrypt layer - serialization/workflow.ts — synchronous, no encryption, for VM use - serialization/step.ts — async with encryption, for step handler - serialization/client.ts — async with encryption, for start() API - serialization/index.ts — re-exports all public API - serialization/serialization.test.ts — 25 focused tests All modes compose their reducer/reviver sets from the shared building blocks. Cross-mode compatibility verified: data serialized in any mode can be deserialized in any other mode (for common types). Existing 108 serialization tests continue to pass unchanged. * Add sub-path exports for workflow serialization module - Add ./serialization/workflow export to @workflow/core package.json - Add ./internal/serialization re-export to workflow meta-package - The workflow bundle can now import serialize/deserialize via: import { serialize, deserialize } from 'workflow/internal/serialization' Full test suite passes: 493 tests across 22 files (including 25 new serialization module tests). * Address code review feedback 1. Fix reducer composition order: Class/Instance reducers now come BEFORE common reducers in all three modes (workflow, step, client). This ensures custom Error subclasses with WORKFLOW_SERIALIZE are handled by the Instance reducer before the generic Error reducer (devalue uses first-match-wins semantics). 2. Fix encryption decrypt() to fail fast when encrypted data is encountered without a decryption key, instead of silently returning encrypted bytes that would fail later with an unhelpful format error. 3. Remove Request/Response from common reducers — they don't have matching common revivers, so including them caused asymmetric behavior (serialize as Request, deserialize as plain object). Request/Response handling belongs in mode-specific modules that can provide proper revivers. 4. Document Node.js dependency in the workflow serialization re-export. The current implementation uses node:util and Buffer. For the QuickJS VM (snapshot runtime), these will need polyfills — tracked separately. * Move reducer/reviver composition into the devalue codec The Codec interface now takes a SerializationMode ('workflow', 'step', 'client') instead of raw reducers/revivers. The reducer/reviver composition is internal to the devalue codec implementation. This is the right abstraction because reducers/revivers are devalue- specific concepts. A future CBOR codec would handle Date, typed arrays, Map, Set natively via the CBOR type system — it wouldn't use reducers at all. A JSON codec would only support standard JSON types. The mode-specific modules (workflow.ts, step.ts, client.ts) are now simpler — they just pass the mode string to the codec. * Replace SerializationFormatType enum with open-ended FormatPrefix type The format prefix is now a branded string type validated by isFormatPrefix() — any 4-character [a-z0-9] string is valid. This removes the hard-coded enum of known formats, making the system truly open for extension: type FormatPrefix = string & { __brand: 'FormatPrefix' }; function isFormatPrefix(value: string): value is FormatPrefix; The SerializationFormat object still provides well-known constants ('devl', 'encr') but they're now just typed constants, not an exhaustive enum. peekFormatPrefix() and decodeFormatPrefix() use isFormatPrefix() for validation instead of checking against a known list. Unknown but valid prefixes (e.g. 'cbor', 'json', 'v2b1') are accepted — the caller decides whether they can handle the format. 6 new isFormatPrefix tests covering: valid strings, too short, too long, uppercase, special characters. 1 new test for unknown-but-valid prefixes. * Wire modular serialization modules into serialization.ts, add 138 unit tests Replace duplicate format prefix, reducer/reviver, and encryption helper code in the monolithic serialization.ts with imports from the modular serialization/ directory. This completes the refactoring started in the earlier additive-only commits. Key changes: - serialization.ts now imports types, format prefix, common/class/step-function reducers and revivers, and encryption helpers from ./serialization/ modules - Removed ~450 lines of duplicate code from serialization.ts - Made encryption error messages consistent between old and new modules - Added 138 comprehensive unit tests covering types, format prefix, encryption, codec, all three reducer modules, all three mode modules, cross-mode compatibility, and edge cases - Updated one existing test assertion for new error message wording * Address code review feedback - encryption.ts: throw WorkflowRuntimeError instead of plain Error in decrypt() to preserve the error contract from legacy maybeDecrypt() - format.ts: document that open-ended prefix validation ([a-z0-9]{4}) is intentional for forward compatibility — callers check support - errors.ts: extract duplicated formatSerializationError into shared utility, remove 4 copies from workflow.ts, step.ts, client.ts - codec-devalue.ts: document that globalThis default is a known limitation; legacy dehydrate/hydrate path still supports custom global * Fix codec-devalue.ts comment: clarify modular modules are not used in current runtime The globalThis default is not a limitation for the current runtime — all serialization goes through dehydrate*/hydrate* in serialization.ts which passes the correct global. The modular modules are infrastructure for the future snapshot runtime where serialization runs inside the VM. * Wire dehydrate/hydrate functions through modular serialize/deserialize The dehydrate*/hydrate* functions in serialization.ts now delegate to the modular mode modules (workflowModule, stepModule, clientModule) instead of directly calling devalue stringify/parse/unflatten. Key changes: - Extended Codec interface with CodecOptions (global, extraReducers, extraRevivers) so the codec can receive VM globals and mode-specific stream/Request/Response handlers - devalueCodec threads global through to all reducer/reviver factories so instanceof checks work across VM boundaries - Mode modules (workflow.ts, step.ts, client.ts) accept CodecOptions and pass them through to the codec - dehydrate*/hydrate* functions now call module serialize/deserialize with stream and Request/Response reducers/revivers passed as extras - v1Compat path remains inline (pre-codec, uses stringify + revive) - Error context strings preserved via try/catch re-wrapping * Bump changeset from patch to minor for serialization refactor Return types of public get*Reducers/get*Revivers functions narrowed from Reducers/Revivers to Partial<Reducers>/Partial<Revivers>, which is a TypeScript-level breaking change. Also adds new sub-path exports (@workflow/core/serialization/workflow, workflow/internal/serialization) which is additive. Minor bump is the appropriate semver for both. * Remove unused workflow/internal/serialization re-export and @workflow/core/serialization/workflow sub-path Both exports had zero consumers in the repo. The workflow/internal/serialization export was previously removed on main in #1082 for the same reason. The modular workflow.serialize/deserialize is still reachable via @workflow/core/serialization when needed. These exports can be reintroduced by the snapshot runtime branch if/when it actually needs them. Also updates the changeset to drop the 'new sub-path exports' bullet. * Downgrade changeset from minor to patch After auditing actual consumers of the narrowed return types (getExternalReducers/getWorkflowReducers/getExternalRevivers/getWorkflowRevivers now return Partial<Reducers>/Partial<Revivers>), no in-repo or external consumer indexes specific keys on the returned object in a way that would break. The only internal caller that did (runtime/run.ts) was updated in this same PR. The narrowing is type-safer but effectively invisible at runtime and for idiomatic callers that spread or forward the object. Since the refactor is internally restructuring only, patch is the appropriate semver bump. * Trim serialization-refactor changeset * Dedup formatSerializationError: import from serialization/errors.ts The legacy serialization.ts had its own inlined copy of formatSerializationError. Now that the helper is exported from serialization/errors.ts (already consumed by workflow.ts/step.ts/client.ts), import it here too to keep the single source of truth.

[swc-plugin] Capture lexical `this` for nested arrow step functions (#1935) * [swc-plugin] Capture lexical `this` for nested arrow step functions When a nested arrow `"use step"` references the enclosing function/method's `this`, plumb that `this` through the workflow runtime so the step body sees the correct receiver. - Workflow mode wraps the step proxy with `.bind(this)`, so invoking the proxy captures the caller's `this` as `thisVal` on the queue item. - Step mode hoists the body as a regular `function` (not an arrow) so the runtime's `stepFn.apply(thisVal, args)` rebinds `this` inside the hoisted body. Detection only fires for arrows, since arrows inherit `this` lexically. Nested non-arrow functions/methods/getters/setters introduce their own `this`, so the detector stops at those boundaries. The runtime already supported `thisVal` for instance-method steps; this PR is purely a compiler change to feed the existing pipeline. Caveat: capture works at runtime only when the captured value is serializable across the workflow->step boundary (i.e. the enclosing class implements `WORKFLOW_SERIALIZE`/`WORKFLOW_DESERIALIZE`). Refs vercel/workflow#1865 * Address PR review: preserve step proxy metadata + tighter `this` detection - core: Override `.bind` on step proxies so the bound function retains `stepId` and `__closureVarsFn`. Without this, a bound proxy that flows through workflow serialization (e.g. as a step argument) would be treated as a non-serializable plain function by `getStepFunctionReducer`. - swc-plugin: Detector now also walks `arrow.params` so `this` references in default values / destructuring initializers (e.g. `(x = this.foo) => ...`) trigger the `.bind(this)` path. - swc-plugin: Class bodies inside the arrow body are now treated as `this`-binding boundaries — `this` inside class field initializers, methods, etc. is bound to the class instance, not the outer arrow. The detector still walks `extends` clauses and computed property keys because those are evaluated in the surrounding scope. - spec.md: Sharpen the note about `this` in step bodies — it's syntactically allowed but only meaningful for instance-method steps and lexical-`this` arrow steps; other shapes compile but `this` will be whatever the caller of the step proxy passes. - Add `lexical-this-detector-edge-cases` fixture covering both the default-param positive case and the inner-class false-positive guard. - Strengthen the runtime test to assert `stepId` / `__closureVarsFn` survive `.bind(...)`. * [swc-plugin] Fix `arguments` closure-var capture; drop dead `this`/`arguments` checks - Add `arguments` to `is_global_identifier` so it's not captured as a closure variable. Previously a nested `function`-form step like function step() { 'use step'; return arguments[0]; } was hoisted with `const { arguments } = ...` (a strict-mode syntax error) and the body's `arguments[0]` resolved against the destructured binding instead of the function's intrinsic `arguments` object. - Remove dead `ForbiddenExpression` checks for `this` and `arguments` in `visit_mut_this_expr` / `visit_mut_ident`. The `'use step'` / `'use workflow'` directives are stripped during the module-level traversal before children are visited, so `in_step_function` / `in_workflow_function` are never observed as true here in practice. The existing `step-with-this-arguments-super` fixture explicitly documents that all three identifiers are allowed in step bodies. - Tighten the spec note about `arguments` accordingly: it works in `function`-form steps (reflecting positional args) but is not captured for arrow-form steps; use `...args` for that case. - Add `nested-step-arguments` fixture pinning down the new behavior.

fix(core,errors): classify SDK encryption failures as RUNTIME_ERROR (#2145) * fix(core,errors): classify SDK encryption failures as RUNTIME_ERROR SDK-level AES-GCM encrypt/decrypt failures are never the user's fault, but the run-failure classifier was tagging them as USER_ERROR because the native Web Crypto OperationError (most commonly raised by AESCipherJob.onDone on GCM auth-tag mismatch) does not match any RUNTIME_ERROR_CHECKS entry. Introduce a new RuntimeDecryptionError (subclass of WorkflowRuntimeError) that the encryption module throws when subtle.encrypt/subtle.decrypt fails, with the original DOMException as cause plus diagnostic context (operation, byteLength, printable/hex format prefix of the input header). classifyRunError now picks it up via RUNTIME_ERROR_CHECKS, so these failures surface as RUNTIME_ERROR with a proper named class for dashboards and triage. * Trim changeset description to one sentence * Trim historical-context comments * docs: add runtime-decryption-failed troubleshooting page (v4 + v5) * fix(core): round-trip RuntimeDecryptionError context, fix formatPrefix, propagate through serialization wrappers Addresses review feedback on #2145: - Add a RuntimeDecryptionError reducer/reviver (+ SerializableSpecial entry + globalThis registration) so its `context` (operation, byteLength, formatPrefix) survives the dehydrate/hydrate run-error round trip instead of being dropped by the generic Error reducer. - Stop capturing `formatPrefix` in the low-level encryption layer, which only sees the stripped AES payload (nonce bytes), not the outer `encr` marker. The serialization layer now attaches the real envelope prefix. - Rethrow RuntimeDecryptionError unchanged from the serialize/dehydrate catch blocks instead of reframing it as a SerializationError, so an encryption failure during dehydration stays a RUNTIME_ERROR rather than being misclassified as USER_ERROR. * fix(core): enrich stream decrypt errors with envelope prefix + fix lint - Mirror the catch/enrich/rethrow block from serialization/encryption.ts around the stream-path aesGcmDecrypt() call so auth-tag failures on encrypted stream frames also carry context.formatPrefix = 'encr' (addresses review feedback). Add a tampered-frame test. - Fix all auto-fixable Biome lint findings in the touched files (template literals, useless try/catch wrappers, optional chaining, non-null assertions).

[core] Add wire-level framing for byte streams (#1853) Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com>

fix(world-local): prevent path traversal via request-supplied IDs (#1829) * fix(world-local): prevent path traversal via request-supplied IDs Request-supplied identifiers (runId, eventId, stepId, hookId, correlationId, stream names, and tags) flowed directly into path.join() calls, allowing a client to send values like '../../../package' and cause the backend to read or write files outside the workflow data directory. Add a centralized validator (assertSafeEntityId) that rejects IDs which are empty, start with '.', or contain path separators or NUL bytes. Apply it at each storage-layer entry point that composes IDs into filesystem paths: fs.taggedPath / readJSONWithFallback / paginatedFileSystemQuery, the runs / steps / events / hooks storage methods, and the streamer. * address review feedback - UnsafeEntityIdError now extends WorkflowWorldError for consistency with other storage-layer errors and the platform error-to-HTTP mapping. - Add resolveWithinBase(basedir, ...segments) containment helper and apply it at every taggedPath / readJSONWithFallback / .locks path construction site in events-storage and legacy, so a forgotten assertSafeEntityId at a future call site can't silently regress. - Truncate attacker-controlled values in the error message. - Drop unused assertSafeEntityIds helper and the unreachable typeof check under the TS signature. - Fix docstrings on assertSafeEntityId / taggedPath JSDoc example / filePrefix validation comment to match what the code actually does. - handleLegacyEvent now re-asserts runId locally so the invariant is documented at the call site instead of implicitly inherited from events.create. --------- Co-authored-by: JJ Kasper <jj@jjsweb.site>

[core] Fix process crash from rejected waitUntil promises (#2336) Co-authored-by: Peter Wielander <mittgfu@gmail.com>

[world-vercel] Add /run-id sub-export with tagged ULID encode/decode (#1978) * [world-vercel] Add /run-id sub-export with tagged ULID encode/decode Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a ULID-shaped string used for workflow run IDs. Tagged values remain valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip through any system that accepts ULIDs. * [world-vercel] Add string-value assertions to run-id tests Add exact-string expectations for encoded outputs at known inputs, covering the default region/version pair, numeric region IDs, version overrides, boundary values (all-zero, all-max), the dirty-input overwrite case, and the lexicographic-order checks. Also adds an explicit byte-array expectation for the canonical ULID-spec example string and an additional first-char-range coverage test for isTagged. * [world-vercel] Remove internal-repo reference from regions doc comment * [world-vercel] Address PR review feedback on run-id sub-export - isTaggedString now fully validates the input as a 26-char Crockford Base32 ULID (delegating to ulidToBytes) instead of only inspecting the first character. This fixes false positives on inputs like '4UUUU...' that have a valid tag-bit position but invalid chars later in the string. - isTagged() now accepts `unknown` to match its documented behavior of safely rejecting non-string inputs without requiring callers to cast. - Introduce `RegionKey` for the full set of keys including 'unknown', and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the return type of `lookupRegion` and the `DecodedRunId.region` field accurately reflect that 'unknown' is never produced. Updates `encode` to reject 'unknown' as a region code string at runtime (callers wanting the unknown sentinel should pass numeric 0). * [world-vercel] Move tagged-ULID metadata to the top of randomness Address review feedback on #1978: 1. **Metadata at top of randomness, not bottom.** Place `regionId` (6 bits) in the high bits of byte[6] and `version` (5 bits) straddling bytes 6 and 7, leaving the bottom 69 bits of randomness untouched by `encode`. This means a `monotonicFactory()`-style ULID generator's intra-millisecond bottom-bit increments survive encoding intact, so consecutive `encode(ulid(), region, { version })` calls with the same metadata produce strictly increasing strings. Previously the metadata sat in the bottom 11 bits — exactly the bits the monotonic factory uses — causing same-ms collisions/inversions. 2. **DecodedRunId is now a discriminated union.** When `tagged: false`, the `regionId`, `version`, and `region` fields are typed as `null` instead of being populated with garbage bits from arbitrary ULIDs. This forces callers to discriminate on `tagged` before reading metadata. 3. **regionIdFor: keep runtime backstop, mark as ignored for coverage.** The unreachable-in-TS branch stays as a defensive runtime check for callers crossing a JS/TS boundary; an istanbul/c8 ignore comment keeps coverage tools quiet. Doc strings and tests updated accordingly. The new layout adds a test verifying that a sequence of incrementing-bottom-bit ULIDs (simulating `monotonicFactory()`) round-trips through `encode` as a strictly increasing sequence. 108/108 world-vercel tests pass; typecheck clean.

Auto-remove workflow packages from serverExternalPackages (#1481) * Warn when serverExternalPackages hides workflow-enabled packages Add a build-time warning when packages in serverExternalPackages contain workflow code ('use step', 'use workflow', or serialization classes). These packages are completely invisible to the workflow compiler when externalized, causing silent runtime failures. The warning detects workflow patterns via two methods: - Fast path: check package.json dependencies for @workflow/serde - Thorough path: read the package entry file and run pattern detection Also adds documentation in the serialization guide about the externalization footgun for 3rd-party packages. * Auto-remove workflow packages from serverExternalPackages When workflow-enabled dependencies are externalized in Next.js, compiler transforms are skipped and runtime failures follow. Detect those packages in withWorkflow, remove them from serverExternalPackages for the current build, and keep a generalized externalPackages warning fallback for non-Next builders. * Address review feedback: add entry-point limitation comment and missing test case

[core] Exclude inline step execution from replay timeout (#2013) * [core] Exclude inline step execution from replay timeout The v5 combined workflow+step handler wraps inline step bodies in the same setTimeout(..., REPLAY_TIMEOUT_MS) guard that previously only bounded the v4 'workflows' function's fast deterministic replay. As a result, any workflow with a single step exceeding 240s hard-fails with FatalError: Workflow replay exceeded maximum duration (240s) after 4 attempts — even though the step could legitimately run for the full function maxDuration (up to 800s on Pro Fluid). Replace the setTimeout guard with a per-invocation budget that only accumulates non-step time. pauseReplayBudget() / resumeReplayBudget() bracket each executeStep() call, and the loop checks the budget at iteration boundaries. The retry-then-fail semantics from #1567 are preserved verbatim for the pure-replay case. Also adds a WORKFLOW_REPLAY_TIMEOUT_MS env var override (clamped to 30s..780s) so operators can adjust the bare-replay ceiling without patching @workflow/core. Fixes #2009. * Address PR review feedback - Extract budget bookkeeping into ReplayBudget class (replay-budget.ts) with sentinel-protected idempotent pause()/resume() to avoid double-counting in future refactors that nest step execution - Restore VERCEL_URL gate around process.exit(1) so a long pure-replay in local dev/non-Vercel runtimes can't hard-kill the host process - Warn (once per distinct raw value) when WORKFLOW_REPLAY_TIMEOUT_MS is clamped or rejected, so misconfiguration is observable - Correct Hobby maxDuration comment (60s standard / 300s Fluid) - Document budget-check responsiveness trade-off vs. old setTimeout - Tighten describe-error test assertions to match the full new hint - Shorten changeset description - Add ReplayBudget unit tests (9) including 8-minute step regression - Add warn-once tests for getReplayTimeoutMs (extended) * Replace VERCEL_URL gate with World capability Per review feedback, gating runtime behavior on process.env.VERCEL_URL leaks deployment-environment concerns into @workflow/core. Replace the check with a new optional capability on the World interface: processExitTriggersQueueRedelivery (default false). - @workflow/world: declare the new optional capability on World - @workflow/world-vercel: set it to true (Vercel fails the function invocation on non-zero exit and VQS redelivers via fresh invocation) - @workflow/core: handleReplayBudgetExhausted reads world.processExitTriggersQueueRedelivery instead of process.env.VERCEL_URL; behavior is otherwise unchanged - Add 4 unit tests for handleReplayBudgetExhausted covering both branches (exit-for-redelivery and best-effort run_failed) --------- Co-authored-by: Peter Wielander <mittgfu@gmail.com>

cli: show world-specific run fields in inspect output via World.describeRun (#2896) * cli: show run region in inspect output via World.regionForRunId Adds an optional reverse-lookup hook to the World interface — regionForRunId(runId): string | null — so tooling can display a run's region generically. Worlds without a regional dimension simply omit the hook and no region output appears. - @workflow/world: new optional interface member (documented: must not throw; null = undeterminable) - @workflow/world-vercel: implements it from the run-ID region tag (tagged -> embedded region, untagged legacy -> default region, malformed -> null) - @workflow/cli: 'workflow inspect runs' gains a region column (between workflowName and status) and 'workflow inspect run <id>' a region property, in both table and JSON output — only when the world defines the hook * Generalize the inspect hook: World.describeRun display fields Replaces regionForRunId on the World interface with describeRun, per review: worlds may want to expose more than a region, and the information need not be encoded in the run ID — describeRun receives the run entity itself (loosely typed, mirroring createRunId), so a world can derive fields from executionContext or any other property. Each returned key becomes an inspect column/property; null values are preserved in structured output ('applicable but undeterminable' vs. the hook being absent entirely). - world-vercel: describeRun returns { region } decoded from the run ID tag (regionForRunId stays exported as a utility); entities without a usable runId contribute nothing - CLI listing: columns come from the union of keys the world returns for the page, inserted before status; both analytics and storage paths; detached call site binds this - CLI showRun: merges the world fields into detail/JSON output via a method-style call (preserves this), keeping nulls - tests: field merging (multi-key), null preservation in JSON, hook absent, and world-vercel describeRun coverage incl. no-runId entities * cli: evaluate describeRun defensively Per review: the World interface says describeRun is pure and must not throw, but it is an external extension point and the CLI should not trust that. New safeWorldFields helper, used by both the listing and showRun paths: - a throwing implementation contributes no fields instead of crashing the inspect command - keys that already exist on the run row are dropped, so a world can never overwrite canonical fields (status, runId, ...) in output Tests: canonical fields survive a clobbering describeRun (extra keys still merged); a throwing describeRun leaves rows untouched and the command succeeds. * Allow async describeRun implementations Per review: widening a sync signature to async later would break every consumer, while accepting sync-or-async from day one is free — sync implementations (like world-vercel's) remain valid, and consumers simply await, which handles both. The performance intent lives on as documented guidance: the hook is called once per displayed run, so implementations should stay cheap and avoid I/O; the CLI evaluates a page's rows concurrently so an async world costs one await per page, not per row. Promise rejections get the same treatment as throws: no fields, never a crash. * Update packages/world/src/interfaces.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Nathan Rajlich <n@n8.io> --------- Signed-off-by: Nathan Rajlich <n@n8.io> Co-authored-by: Peter Wielander <mittgfu@gmail.com>

Fix Next workflow module specifier root (#2455)

feat(core): support passing parent WritableStream to child workflow via start() (#2059) * test(e2e): cover WritableStream passed as start() argument Adds an e2e workflow + test where a parent workflow gets a WritableStream via getWritable(), forwards it through start() to a child workflow, and the child step writes raw bytes to it. Asserts the external reader on the parent's stream observes the exact bytes the child wrote. * fix(core): avoid double-framing when WritableStream is forwarded via start() When a workflow's getWritable() handle is passed across start() to a child workflow, the parent step's reviver wraps it in a serialize transform that pipes into a workflow server stream. Until now, getExternalReducers.WritableStream then installed a second serialize transform on top of that — so every chunk the child step wrote got devalue-framed twice but only deframed once on the reader side, and external consumers saw the inner frame instead of the original bytes. Fix: tag every user-visible writable that's already backed by a workflow server stream with its (runId, name). When the external reducer recognizes those tags during dehydration, it bridges bytes straight from the new child-side server stream to the original server stream instead of piping through the user's writable. That leaves the producer-side serialize transform (installed once by the child's step reviver) as the only framing layer in the chain. * fix(core): forward (runId, name) when a tagged WritableStream crosses start() Replaces the previous in-process bridge with first-class writable forwarding at the descriptor level. When a parent workflow's getWritable() handle is passed as an argument to a child workflow, the dehydrated descriptor now carries the original (runId, name). The child run's step-side reviver opens the writable against the parent's server stream directly and resolves the parent run's encryption key (encrypt-only) via getEncryptionKeyForRun. This removes the architectural limitation that the bridge could only stay alive for the duration of the parent step process — on Vercel that capped forwarding at ~15 minutes regardless of the child run's lifetime, dropping any writes the child made after the parent step process exited. importKey() now accepts a usages parameter, defaulting to ['encrypt', 'decrypt']. The cross-run forwarding path imports with ['encrypt'] only so a compromised child run cannot decrypt any existing data on the parent's stream — only contribute new writes. * test: rename writable-forwarded workflows and cover step-context getWritable() Addresses PR review: - Rename writableForwardedToChildChildWorkflow → writableForwardedChildWorkflow (drops the duplicated 'Child' segment). - Split writableForwardedToChildWorkflow into two variants covered by a test.each: writableForwardedFromWorkflowWorkflow (workflow-context getWritable, the original test) and writableForwardedFromStepWorkflow (step-context getWritable passed directly into start() from the same step that called getWritable()). - Terser changeset description.

fix(web-shared): hydrate FatalError/RetryableError and Error subclasses in o11y (#1942) * fix(web-shared): hydrate FatalError/RetryableError and Error subclasses in o11y The web o11y reviver set was missing entries for the recently-added serialization types (FatalError, RetryableError, the built-in Error subclasses, AggregateError, DOMException), causing devalue.unflatten to throw "Unknown type X" and the UI to surface "Failed to load resource details" whenever a step or run failed with one of these error types. Adds the missing revivers to getWebRevivers() and a regression test that round-trips real values through the runtime's dehydrateStepError back through the web reviver set. * fix(web-shared): address review feedback on error revivers - Pass `cause` through ErrorOptions to the subclass constructor instead of assigning afterwards, matching `getCommonRevivers` in core. This gives the resulting `cause` property the same engine-set, non-enumerable semantics as a freshly thrown Error in the consumer realm. - Guard `RetryableError.retryAfter` against missing/undefined values from older runtime payloads — without it, `new Date(undefined)` produces an Invalid Date rather than the property being absent. Add a defensive test that drives the reviver directly with a payload missing the field.

[swc-plugin] Preserve imports referenced by hoisted nested steps (#1944) * [swc-plugin] Preserve imports referenced by hoisted nested steps Dead-code elimination ran before nested step functions were hoisted out of workflow bodies, so imports referenced only by hoisted step bodies were incorrectly stripped from the step bundle, causing a ReferenceError at runtime. Move DCE to run after hoisting in visit_mut_program. * [swc-plugin] Namespace nested step IDs under non-exported workflow functions Anonymous steps nested inside callback properties of a non-exported workflow function were registered with an unnamespaced step ID in step mode while the workflow-mode proxy looked them up under the workflow function name, causing a runtime 'step not found' failure. Set current_workflow_function_name in visit_mut_fn_decl for non-exported workflow functions to match the behavior in visit_mut_export_decl. Also clarify the fixture comment to distinguish step-mode and workflow-mode behavior per reviewer feedback. * [swc-plugin] Namespace nested step IDs across all workflow declaration shapes Extends the previous fix to cover all three non-exported workflow declaration forms (async function decl, const arrow, const fn-expr) by visiting the workflow body with workflow context before replacing it, and corrects the __internal_workflows manifest comment to report the same prefixed step IDs that are registered at runtime and looked up by the workflow-mode WORKFLOW_USE_STEP proxy. Adds a dedicated regression fixture covering all three shapes.

Atomically dedupe duplicate step_created/wait_created events in world-local (#1877) Concurrent invocations producing identical correlationIds (as the snapshot runtime does by design across replays) previously both succeeded and persisted duplicate events. step_created had no guard at all; wait_created used a TOCTOU read-then-check that allowed both writers through under concurrency. Both now claim a per-(runId, correlationId) constraint file with O_CREAT|O_EXCL before writing, so the loser surfaces as EntityConflictError — which the runtime's dedup catch path already handles.

Enforce per-(run, correlation) uniqueness for entity-creating events in world-postgres (#1878) Adds a unique partial index on workflow_events(run_id, correlation_id, type) filtered to step_created/hook_created/wait_created, and translates the resulting unique-violation (pg code 23505, surfaced via DrizzleQueryError.cause) into EntityConflictError. The steps table already deduped via onConflictDoNothing, but the event row still inserted, leaving duplicate events in the log. Now both rows are kept consistent and the runtime's existing dedup catch path handles concurrent writers cleanly.

[world-vercel] Validate ref resolve responses before use (#2035) * [world-vercel] Validate ref resolve responses before use When workflow-server returns a ref body to the SDK, the bytes are fed into the workflow runtime's event log and deserialized via `decodeFormatPrefix`. The SDK always writes ref payloads with at least a 4-byte format prefix (see `encodeWithFormatPrefix` in `@workflow/core`), so a zero-byte response — or one whose length disagrees with `Content-Length` — is never a valid stored value. Before this change, `resolveRefDescriptor` had no validation: a 200 with an empty body would be passed downstream as a zero-length Uint8Array, which then failed deep inside replay with: Data too short to contain format prefix: expected at least 4 bytes, got 0 By that point the workflow's in-memory event snapshot is already poisoned with the empty payload, so every subsequent replay deterministically reproduces the same failure, downstream `resumeHook()` calls surface as `Hook not found`, and the run only unsticks when stale-run cleanup terminates the sandbox. This catches the failure at the transport boundary instead, where it can be retried as a `WorkflowWorldError`. Both an empty body and a length mismatch (truncated streaming response) are rejected. This is the SDK-side companion to vercel/workflow-server#432, which adds the same validation on the server side. * Address review: reject <4-byte bodies, handle malformed Content-Length Three review changes: 1. Reject any body shorter than the 4-byte format-prefix length, not just zero-byte bodies. The SDK guarantees every stored ref payload starts with a 4-byte format prefix (FORMAT_PREFIX_LENGTH in @workflow/core), so a 1-3 byte body would also fail downstream replay with the same 'Data too short to contain format prefix' error this PR exists to prevent. 2. Parse Content-Length safely with parseInt + Number.isFinite + non-negative checks instead of bare Number(). A non-numeric value like 'abc' would otherwise produce NaN and silently surface as a 'truncated' error, masking the real cause. Malformed values are treated as absent; the minimum-length check still defends against actual truncation in that case. 3. Add tests for the truncated-body-without-Content-Length case (chunked transfer where Content-Length validation can't see the truncation), and for a malformed Content-Length header that should be ignored rather than misreported as truncation. The validation logic also moves into a small assertValidRefBody helper to keep the inner trace function under the noExcessiveCognitiveComplexity limit. * Address review: scope 4-byte minimum to binary refs, strict Content-Length parsing - Only apply the 4-byte format-prefix minimum to application/octet-stream payloads; CBOR refs can legitimately be 1-byte primitives (true/0/null). - Require Content-Length to be a plain run of digits before comparing; parseInt would otherwise accept numeric-prefixed garbage ('12junk' -> 12). - Make the changeset succinct. * Address review: skip Content-Length check for compressed responses fetch/undici transparently decompresses gzip/br bodies but leaves Content-Length describing the encoded (compressed) size, so comparing it against the decompressed byteLength would reject valid compressed refs as a phantom 'ref-body-length-mismatch'. Skip the comparison when a non-identity Content-Encoding is present; an absent or 'identity' encoding is still validated. Adds regression tests for both cases.