Nathan Rajlich

fix(world-local,world-postgres): make duplicate hook_created idempotent (#2295) * fix(world-local): make duplicate hook_created idempotent Duplicate processing of the same hook_created — same runId, hookId, and token, e.g. cross-process replay or queue redelivery — was being recorded as a hook_conflict in the event log, which then replayed as a self- conflict HookConflictError. The fix mirrors the existing step_created duplicate-correlation path: when the exclusive token claim fails and the existing claim has the same (runId, hookId), throw EntityConflictError so the runtime's existing concurrent-replay catch path swallows it. Different runId or hookId reusing the same token still produces a real hook_conflict. The persisted token claim already carried hookId; only the read schema was dropping it. The schema now preserves hookId (marked optional for backward compatibility with older claim files). Fixes #2283 * fix(world-postgres): make duplicate hook_created idempotent world-postgres has the same gap as world-local was just fixed for: the duplicate-token check in events.create unconditionally writes a hook_conflict event when an existing hook with the same token is found, even when the existing hook has the same (runId, hookId) as the incoming event. The unique partial index on workflow_events does not catch this because the duplicate path inserts hook_conflict, not hook_created. Mirror the world-local fix: when the existing hook's (runId, hookId) matches the incoming event, throw EntityConflictError so the runtime's existing concurrent-replay catch path swallows it. Different runId or hookId reusing the same token still produces a real hook_conflict. Refs #2283 * test(e2e): add regression test for hook_conflict from same-tick replay race Regression test for #1665 / #2283. A parent workflow awaits 6 child workflows with Promise.all; each child does a tiny step and creates one webhook. Awaited children flatten into the parent run, so all webhook creations land on the same workflow body. When their step resolutions align in the same tick the workflow body is re-walked and each pass submits hook_created with the same deterministic (correlationId, token). Before the world-side idempotency fix, the world wrote hook_conflict events for the duplicates and the workflow failed with HookConflictError. With the fix, duplicates throw EntityConflictError (swallowed by the suspension handler), no hook_conflict events appear in the log, and the webhooks resolve normally. Verified locally against world-local: the test fails reliably (3/3) on the unfixed code and passes reliably (5/5) on the fixed code. * test(e2e): rewrite parallelStepsThenWebhookWorkflow to match the actual #1665 repro The earlier version invoked another 'use workflow' function directly from inside the parent workflow, which is not a valid child-workflow invocation (child workflows must be spawned via start()) and didn't mirror the bug shape on #1665 anyway. Rewrite the workflow as a single 'use workflow' function that exactly mirrors Paolo's minimal repro: await Promise.all([stepA(), stepB()]); using webhook = createWebhook(); await webhook; The for-loop runs N independent iterations of that sequence in series, each disposing its webhook via 'using' before the next, to give the timing-sensitive race multiple chances to fire. The race is hard to force deterministically on fast local dev — but the same (runId, hookId) idempotency invariant is covered deterministically by the new unit tests in world-local and world-postgres. This e2e test serves as a higher-level regression net: its assertions (no hook_conflict event in the log, no HookConflictError-failed run) are correct whether the race fires or not, and will catch any future regression on a run that does hit it. * fix(world-local,world-postgres): recover crash-orphaned hook claims/rows instead of suppressing the retry Addresses review feedback on PR #2295. The original idempotency fix made duplicate same-(runId, hookId) hook_created submissions throw EntityConflictError so the suspension handler's concurrent-replay catch path swallows them. But the claim file (world-local) and hook row (world-postgres) are written before the durable hook_created event, and the writes are not atomic. A process / DB interruption between the claim/hook write and the event write leaves an orphaned claim/hook row; the retry then matched the same (runId, hookId), threw EntityConflictError, got swallowed, and the run was permanently left with no hook_created event in the log. world-local: - Add a per-(runId, hookId) in-process mutex (withHookLock) mirroring the existing withStepLock, so two same-tick concurrent calls serialize on the entity write and the dedup branch never observes an in-flight winner mid-write. - In the dedup branch, when the existing claim is for the same (runId, hookId) we are trying to create, check whether the durable hook entity actually exists on disk: - exists → real duplicate: throw EntityConflictError as before. - missing → orphaned claim from a prior crash: fall through and complete the partial write (write the hook entity with overwrite, then emit hook_created via the outer code path). world-postgres: - In the dedup branch, when the existing hook row matches the incoming (runId, hookId), check whether a hook_created event for this (runId, correlationId) already exists in the event log: - exists → real duplicate: throw EntityConflictError as before. - missing → orphaned hook row from a prior crash between hook INSERT and events INSERT: skip the hook insert (the row is already there) and let the outer code path emit hook_created, completing the partial write. Tests: - world-local: pre-seed an orphaned token claim with no matching hook entity, retry hook_created, assert hook entity and hook_created event both land (no hook_conflict, no EntityConflictError). - world-postgres: pre-seed an orphaned hook row with no matching hook_created event, retry, assert hook_created event lands (no hook_conflict, no EntityConflictError). Both tests fail on the prior implementation (EntityConflictError thrown on retry, exact symptom from the review). * fix(world-local): probe the event log (not the hook entity) to detect duplicate hook_created Addresses follow-up review on PR #2295. The previous dedup branch checked whether the durable hook entity existed on disk. But the hook entity is written before the `hook_created` event, and the two writes are not atomic, so a crash between them leaves both the claim file and the hook entity on disk with no event in the log. The dedup branch then matched on `(runId, hookId)`, found the hook entity, threw EntityConflictError, and the suspension handler swallowed the retry — permanently losing `hook_created` from the event log. The fix mirrors what the world-postgres branch already does: probe the run's event log for an existing `hook_created` event for the same `(runId, correlationId)`. The event is the durable record of a successful hook creation; the claim file and hook entity are partial- write artifacts that may exist without the event. - exists → real duplicate: throw EntityConflictError so the runtime's concurrent-replay catch path swallows it. - missing → orphaned partial write (crash at any point before the event landed): re-write the hook entity (with overwrite: true, in case a stale partial copy exists) and let the outer code path emit the hook_created event. Added a new helper findHookCreatedEvent that runs a filtered paginatedFileSystemQuery with limit:1 over the run's events. Regression test "should recover an orphaned hook entity with no matching hook_created event" added — pre-creates a hook, deletes just the hook_created event from disk to simulate a crash between the entity write and the event write, asserts the retry emits a fresh hook_created event (no hook_conflict, no swallowed EntityConflictError). I verified this test fails on the prior fix (throws `EntityConflictError: Hook "hook_orphan_entity_1" already created`, exactly as pranaygp reported) and passes on this commit. The previous test ("should recover an orphaned hook token claim with no matching hook entity") continues to pass — the event-log probe is a strict superset of the entity probe, since a missing entity always also implies a missing event. * fix(world-local): converge same-hook creation across workers via canonical eventId Addresses follow-up review on PR #2295. The previous fix made the dedup branch probe the event log to decide real-duplicate vs orphan-recovery, but the probe and the recovery write are not a single atomic operation. Two workers sharing a data directory (or two retries that lose `writeExclusive(constraintPath)` back to back) could both pass the probe (each observing no hook_created event yet), both fall through to the recovery write, and both append a hook_created event with a different eventId — producing two events in the log for the same (runId, hookId). The in-process `withHookLock` mutex does not help here because it is process-local and tag-specific. The fix persists `eventId` in the durable token claim file (written by the original `writeExclusive(constraintPath)`). On a same-(runId, hookId) dedup match, retries adopt that canonical eventId and rebuild the event with a deterministic createdAt derived from the eventId (a ULID). The outer event write switches from `writeJSON` (check-then-write, TOCTOU) to `writeExclusive` (O_CREAT|O_EXCL via temp-file + hard-link, atomic across processes). Either worker may win the publish; the other throws EntityConflictError which the runtime's existing concurrent-replay catch path swallows. Net result: exactly one hook_created event per logical creation. Backward compatibility: a claim file written before this commit lacks `eventId`. Retries that read such a claim fall back to the event-log probe + fresh-eventId recovery — the legacy behavior that does not converge across workers but cannot regress for freshly- written claims after upgrade. world-postgres already converges across workers via the partial unique index on workflow_events_entity_creation_unique (runId+correlationId+eventType for hook/step/wait_created): the loser's INSERT raises 23505 which is already translated to EntityConflictError. Regression tests: - world-local: `converges same-hook creation across workers to one event` uses two tagged storage instances sharing one data directory and fires 25 paired Promise.allSettled hook_created calls. Expected 25 hook_created events total; before this fix yielded 50. - world-postgres: `converges same-hook creation across concurrent calls to one event` exercises the same shape against the real Postgres unique index. Already converges; the test is a guard against future regressions to the catch path. Verified the world-local test fails on c7b23e1b5 with exactly the shape pranaygp reported (50 events for 25 logical creations) and passes on this commit. The earlier orphaned-claim and orphaned- entity recovery tests also continue to pass. * fix(world-local): converge legacy hook claims via recovery-marker sidecar; replace tag-proxy test with real subprocess workers Addresses follow-up review on PR #2295. Two distinct issues, both flagged by pranaygp as P1: 1. The fallback path for token claims written by versions before eventId was persisted inline (legacy claims after upgrade) still permitted the same cross-process corruption the inline fast path was fixed to prevent. Two processes both reading a legacy claim each generated their own eventId, landed their writeExclusive(eventPath) calls at different paths, and appended two hook_created events for the same (runId, hookId). Existing persisted claims after a real upgrade are exactly the state the crash-recovery branch needs to repair, so leaving the legacy path non-convergent is silent corruption, not backward compatibility. 2. The committed cross-worker convergence test used two tagged storage instances sharing one directory as a proxy for separate processes. But tags change the destination filename (events/wrun_X-evnt_Y.worker-a.json vs ...worker-b.json), so two tagged workers can each writeExclusive their own event at different paths and both fulfill. The Map-by-eventId deduplication in the assertion then masked the duplicate publication, so the test passed for the wrong reason. Implementation: - New HookRecoveryMarkerSchema (`{ eventId, hookId, runId }`) and HookRecoveryMarkerPath helper. The marker is a sidecar at hooks/tokens/<hash>.recovery.json, written via writeExclusive so the first cross-process retry pins its candidate eventId as canonical; subsequent retries read the marker and adopt that eventId. Together with the existing writeExclusive(eventPath) in the outer publish, this gives the legacy-fallback path the same single-event convergence guarantee as the inline-eventId fast path. - pinCanonicalEventIdForLegacyClaim() encapsulates the marker write-or-read. A stale marker for a different (runId, hookId) (token-reuse with leaked state) is overwritten best-effort — the common cross-worker race for the same hook still converges; only the narrow stale-token-reuse case loses convergence. - hook_disposed now also deletes the recovery marker when it deletes the token constraint file, preventing a future legacy recovery for a recycled token from latching onto a stale eventId. - The dedup branch unified: existingClaim.eventId for new claims, pinCanonicalEventIdForLegacyClaim() for legacy ones. Removed the now-redundant findHookCreatedEvent helper — the writeExclusive(eventPath) in the outer publish is the authoritative duplicate-vs-orphan detector. Tests: - New test fixture test-fixtures/hook-race-worker.ts (TypeScript, run via child_process.fork with tsx as execPath — tsx is a transitive dev dep via vitest). Each subprocess gets its own createStorage(testDir) so the in-process hookLocks Map cannot serialize across workers. - Replaced the tag-proxy test with "converges same-hook creation across separate OS processes to one event". Spawns workerCount subprocesses, releases them from a barrier into the same hook_created, asserts exactly one fulfilled + (N-1) rejected with EntityConflictError, and asserts directly on the raw events.list() result (no Map dedup) that the number of hook_created entries equals the number of logical creations. - Added "converges same-hook creation across processes when only a legacy token claim exists". Same shape, but pre-seeds the legacy claim format (`{ token, hookId, runId }` with no eventId) before each race. Verified to FAIL on 7ce66551b (both subprocesses fulfill, no convergence) and pass on this commit. - Also verified the new-eventId subprocess test FAILS when the event write is reverted to writeJSON (TOCTOU), confirming it exercises the writeExclusive-based cross-process arbitration. Both prior orphaned-claim / orphaned-entity recovery tests also continue to pass. * fix(world-local): per-lifetime recovery markers, restore event-log probe, fix CI tsx resolution Addresses three P1 review comments on PR #2295. 1. Stale recovery marker leaking across token-reuse lifetimes (pranaygp): The previous marker path used `hashToken(token)` so a stale marker for run A could leak into run B's recovery when the same token was reused after run A terminated through normal lifecycle. `deleteAllHooksForRun()` and tagged `world.clear()` deleted the token constraint and hook entity but NOT the marker sidecar, so the next legacy claim on the same token entered the stale-marker overwrite branch and the workers overwrote it non-atomically, yielding divergent publication. Fix: - Marker path now hashes `(token, runId, hookId)` together (`hookRecoveryMarkerPath` in storage/helpers.ts). Different lifetimes can never share a marker, so the stale-marker overwrite branch is removed entirely. - `hookRecoveryMarkerPath` is moved to helpers.ts and shared across events-storage.ts, hooks-storage.ts, and index.ts. - `deleteAllHooksForRun()` and tagged `world.clear()` now also delete the recovery marker for each hook (disk hygiene; per- lifetime identity makes leaks no longer corrupting). - `hook_disposed` now uses the new per-lifetime marker path too. 2. Duplicate `hook_created` event when a legacy claim's event was already published (VADE bot, also implied by pranaygp's analysis): Removing the event-log probe from the legacy fallback let a post- upgrade retry pin a new canonical eventId via the marker and publish a duplicate event at that path, even when the original pre-upgrade writer had already successfully published the event with its own (different) eventId. Fix: - Restore `findExistingHookCreatedEventId()` (renamed and made to return the eventId for clearer semantics). - Legacy fallback now probes the event log BEFORE pinning the marker; if a matching `hook_created` event already exists, throw `EntityConflictError` so the runtime's concurrent-replay catch path swallows the retry. - Inline-`eventId` fast path does NOT need the probe — the claim itself is the durable convergence key. 3. CI failure: tsx not resolvable under pnpm isolated linking (pranaygp; confirmed by ubuntu/windows unit test 60s timeouts): The previous test hard-coded `node_modules/.bin/tsx` assuming tsx would be hoisted there. But tsx was only a transitive peer dep via vitest, and pnpm's isolated linking does NOT link transitive peer deps into the workspace bin after a fresh install — so neither root nor package-local `.bin/tsx` existed in CI, the subprocess fork never started, and the barrier hung until vitest killed the test. Fix: - Add `tsx` as a direct `devDependency` of `@workflow/world- local` (pinned to 4.20.6 to match the existing transitive resolution). - Resolve via `import.meta.resolve('tsx/package.json')` and read the `bin` field dynamically, so we adapt to wherever pnpm links tsx for this package — not a hard-coded layout. - Lazy-init the resolver (no module-load IIFE) so an absent tsx fails only the convergence tests, not all 376 tests in the file. - Surface a clear error message if resolution fails, calling out the cause (transitive vs direct deps) for future readers. Also: harden the barrier helper so `error` events and pre-ready exits resolve BOTH `readyPromises` and `donePromises`, then `SIGKILL` siblings. Previously a broken child only resolved `donePromises`, leaving `Promise.all(readyPromises)` pending until the per-test timeout (60s in CI). Regression tests added: - `legacy claim whose hook_created event was already published does not append a duplicate event` — pre-seeds a legacy claim AND a pre-existing `hook_created` event with a different eventId, asserts the retry throws EntityConflictError and the log still has exactly the original event. - `converges legacy claim recovery across run lifetimes after token reuse` — runs pranaygp's full lifecycle path: race subprocess workers on run A's legacy claim, terminate run A via `run_completed` (triggers `deleteAllHooksForRun`), reuse the token in a legacy claim for run B, race subprocess workers again, asserts exactly one fulfillment + one `EntityConflictError` per race and exactly one `hook_created` event per run. Both new tests verified to fail on 2c673e436 (after rebuilding): the published-event test throws via duplicate publish instead of EntityConflictError, the token-reuse test sees both run B workers fulfill (2 events instead of 1). The existing orphaned-claim and orphaned-entity recovery tests also continue to pass. CI loop confirmed to be repaired locally by spawning subprocesses via the new resolver and intentionally breaking the worker fixture to verify the helper fails fast (~500ms) instead of hanging at the barrier. * fix(world-local): defer hook entity write until event publish commits Addresses karthikscale3's P1 review comment on PR #2295. The dedup-recovery path used to write the hook entity BEFORE the outer event publish proved whether the attempt was repairing a missing event or just colliding with an already-published `hook_created`. For already-committed duplicates, the event write then throws `EntityConflictError`, but the hook entity had already been overwritten with the retry's payload — leaving the durable hook entity and the event log inconsistent (e.g. the entity reflects the retry's metadata while the event still carries the original). karthikscale3 reproduced this on the prior head by creating `hook_created` with metadata `{ v: "a" }`, then retrying the same `(runId, hookId, token)` with metadata `{ v: "b" }` and `isWebhook: false`: the retry threw `EntityConflictError` but `hooks.get()` returned the retry's payload. Fix: defer the hook entity write until AFTER the outer `writeExclusive(eventPath)` commits. The branch now only captures the entity-to-write and its overwrite options; the actual write happens immediately after the event publish in the shared trailing block. A retry that ends in `EntityConflictError` (the event was already published) now leaves the entity untouched. The first-writer happy path and all recovery paths (orphaned- claim, orphaned-entity, cross-worker convergence, legacy claim, token-reuse across lifetimes) are unaffected — they all reach the event publish successfully, then the entity write runs as before. Regression test `does not mutate an already-committed hook entity when a duplicate hook_created retry collides` added to world-local: runs karthikscale3's exact scenario and asserts the persisted entity still carries the original metadata and isWebhook. Verified to fail on the prior commit (persisted metadata = 0xbb instead of 0xaa) and pass on this commit after rebuilding. Parallel guard test `does not mutate an already-committed hook entity when a duplicate hook_created retry collides` added to world-postgres. Postgres already protected this via `onConflictDoNothing()` on the hook INSERT, but the test guards against a future regression that adds an UPDATE/UPSERT to the dedup path. * refactor(world-local): per-instance in-process locks; drop tsx subprocess test plumbing You were right that the tsx subprocess machinery was overkill for a storage-level convergence test. Replaced with a simple two-instance in-process test that exercises the same cross-process semantics without spawning anything. The trick: `stepLocks` and `hookLocks` were module-level Maps shared by all `createEventsStorage` calls in the same process. Move them inside the function so each `createStorage(dir)` call gets its own lock map. Two storage instances sharing one data directory then behave exactly like two separate OS processes: - independent in-process `hookLocks` Maps (no in-process serialization between them), and - a shared filesystem (so the on-disk `writeExclusive` claim / marker / event publish primitives are the only thing arbitrating convergence). This is also a real architectural improvement — the global lock map was always a leaky abstraction that made unit-test simulation of the cross-process path awkward. Changes: - `stepLocks` and `hookLocks` moved from module scope into `createEventsStorage`. `withStepLock` and `withHookLock` wrappers collapsed into direct `withInProcessLock(map, key, fn)` calls at the two call sites that need them. - The three convergence regression tests in `storage.test.ts` now use `const workerA = createStorage(testDir); const workerB = createStorage(testDir);` and race `Promise.allSettled` of `events.create` from both — no subprocess, no IPC, no barrier helper, no `raceHookCreatedAcrossProcesses`. Same assertions (exactly one fulfillment + N-1 `EntityConflictError` per race, raw `events.list()` shows exactly one `hook_created` per logical creation — no Map dedup) so the regression catches are identical. - Removed: `tsx` devDep, `test-fixtures/hook-race-worker.ts`, `HOOK_RACE_WORKER` / `resolveTsxLoaderUrl` / `TSX_BIN` / `raceHookCreatedAcrossProcesses` and the `fork`/`fileURLToPath` imports they pulled in. Verified (after rebuilding world-local): - All 379 tests pass on macOS in ~1s (was ~6.7s with subprocesses). - Convergence tests confirmed to still catch the bugs: temporarily reverted the `eventId = canonicalEventId` adoption → both workers fulfilled (2 events instead of 1). Temporarily reverted the legacy-claim marker pin → same: both workers fulfilled. - No subprocess machinery means no Windows-specific quirks (cli.mjs shebang, .cmd wrappers, .bin hoisting under pnpm isolated linking, etc.) that produced the Windows CI 60s timeouts. - World-postgres still has its own parallel guard test for the karthikscale3 "no-mutate-on-duplicate" regression; that one exercises real DB concurrency and is unaffected by this change. Full repo `pnpm test` (43 packages) and the `parallelStepsThenWebhookWorkflow` e2e test against world-local both green. * fix(world-local): repair event-first hook orphans from the persisted event; skip #1665 e2e on world-postgres - A crash between the hook_created event publish and the deferred hook entity write left the event committed with the entity missing and unrepairable (retries threw EntityConflictError without materializing the entity). Retries now rebuild the entity from the PERSISTED event's payload — never the retry's eventData — via a race-safe writeExclusive, on both the canonical-eventId collision path and the legacy-claim probe path. - Skip parallelStepsThenWebhookWorkflow e2e on world-postgres: the same-tick replay pattern surfaces a separate pre-existing step_started ordering bug there (#2331). --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com>

Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) (#3048) * Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay * QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed * QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols * QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity * Apply biome fixes to QuickJS engine files * Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard * CI: include generated QuickJS source assets in shared e2e build artifacts * Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine * CI: run both VM engines across all frameworks and worlds; label jobs with the engine * Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads * e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status) * e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely) * Sort imports in QuickJS serialization files (biome organizeImports) * QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal hook payloads to the target run's published X25519 public key. The shared start() path publishes that key regardless of engine, so QuickJS runs receive sealed payloads too — but the QuickJS entrypoint resolved only the bare symmetric key via importKey(), which cannot open encp envelopes. The first sealed hook payload wedged the run right after hook_received, timing out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the node engine resolves the full capability via memoizeEncryptionKey). Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric (encrypt() with RunPayloadKeys takes the encr path). Regression test seals a payload exactly as resumeHook does and round-trips it through the VM. * Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping - Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap, drawing from the seeded Math.random (identical sequences to the node engine's vm/index.ts implementations); all crypto.subtle methods throw with step-function guidance. process.env exposed as a frozen copy, matching node. - Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family methods (incl. localeCompare) throw when given an explicit locale so cross-engine divergence is loud instead of silently writing different values into the event log. No-argument forms keep working. - runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the ~1.3MB embedded WASM assets out of node-engine deployments. - runQuickJSWorkflow wraps the per-run phase so an exceptional exit disposes the VM instead of leaking it in a reused compute instance; corrected the misleading fail-loud comment (run_failed, not retry); warn when the event drain loop exhausts its iteration bound. - Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the file's workflow.* namespace. - Eval-string correlation-id interpolation uses JSON.stringify instead of quote-only escaping. - common-vm.test.ts pins the reducer/reviver superset invariant against common.ts so the duplicated sets can't silently drift. - Docs enumerate the remaining global-surface differences (subtle.digest, Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known precondition-guard gap. * QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup) #1834 made resumeHook() fall back to enqueueing the run with a hookInput payload when the direct hook_received write fails transiently, with the runtime materializing the missing event on delivery. Only the node:vm path implemented it — the QuickJS dispatch returned before the node block, so the resilient payload was silently dropped and the new e2e timed out on every quickjs leg. - runtime.ts threads hookInput into runWorkflowWithQuickJS; the entrypoint materializes the missing hook_received after loading the event log (resumeId-keyed dedup, occurredAt from the resumeId ULID, local eventData substitution for lazy/ref responses, EntityConflict / HookNotFound handling) — mirroring the node block. - processEvents drops duplicate hook_received rows sharing a resumeId (first-in-log wins), matching the node engine's EventsConsumer dedup; the seen-set lives in the VM heap so it is deterministic per replay. Verified against the dev server with WORKFLOW_VM=quickjs: the resilient resume e2e passes and the materialization is observable in the logs; all 27 hook e2e tests green. * Rerun CI * QuickJS engine: split VM-local class/step-function reducers off the hardened host codec The hardened host-side serialization (#3257) made the shared reducers/class.ts and reducers/step-function.ts depend on serialization/hardened.ts, which imports node:util and captures host intrinsics — unbundleable and meaningless inside the QuickJS guest, where the codec already runs in the guest realm. Point the VM codec at pre-hardening copies with identical wire format; the host/guest boundary hardening for this engine arrives with the host-side serde that retires the VM bundle. * QuickJS engine: enqueue explicit wait continuations instead of same-message redelivery Scheduling sleep wakeups by returning { timeoutSeconds } redelivers the CURRENT queue message. When that message is a hook-resume delivery (carrying hookInput), its redelivery re-runs the lazy-resume re-ensure in the handler prologue; if the workflow disposed the hook during the first delivery (dispose -> sleep), the re-ensure gets HookNotFound, the prologue acks the message as 'nothing left to resume', and the wait timer it carried is silently lost — the run wedges (caught by the hookDisposeTestWorkflow e2e). Enqueue fresh continuation messages instead, matching the node engine's suspension handler: getWaitContinuationDispatch for pending waits (gaining delay clamping/hop chaining and pending-wait dedup keys) and a plain immediate message for elapsed-wait / attr_set / getConflict requeues. A fresh message carries only runId, so its delivery always reaches replay. Also: read hook_received resumeId from the canonical top-level event field (eventData.resumeId is the deprecated legacy fallback), and stop passing hookInput into the entrypoint — the shared prologue in runtime.ts materializes the event for both engines. Adds a VM replay test for the hook -> dispose -> sleep shape. * Sort imports in quickjs-entrypoint (biome organizeImports) * Address review: dispatch inside run-level try/catch, queue namespace + run-origin trace carrier threading, configurable interrupt budget - Move the QuickJS engine dispatch inside the replay loop's try so escaping engine failures (MaxEventsExceededError, WASM OOM, bundle-eval errors) reach the catch that classifies and records run_failed, instead of nacking the message and burning all 48 queue redeliveries into MAX_DELIVERIES_EXCEEDED. Transient world errors still rethrow for redelivery. Updated the two comments that describe the propagation. - Thread the queue namespace from runtime.ts through runWorkflowWithQuickJS into every message publish (step dispatch, hook_conflict requeue, immediate requeue, wait continuation) — without it, publishes on a namespaced deployment land on __wkf_workflow_* while consumers listen on __<ns>_wkf_workflow_*. - Thread the run-origin nextTraceCarrier accessor through instead of capturing the current invocation context, so linked-mode invocations form a star around workflow.start rather than chaining; the hook_conflict requeue now carries a traceCarrier and requestedAt. - Replace the hardcoded 30s VM interrupt budget with the configurable replay budget (getReplayTimeoutMs, default 240s), matching the node engine. * Sort imports in quickjs-runtime (biome organizeImports)

[core][world][world-vercel] Add `World.createRunId()` and region-aware queue routing (#1981) * [world-vercel] Add /run-id sub-export with tagged ULID encode/decode Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a ULID-shaped string used for workflow run IDs. Tagged values remain valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip through any system that accepts ULIDs. * [world-vercel] Add string-value assertions to run-id tests Add exact-string expectations for encoded outputs at known inputs, covering the default region/version pair, numeric region IDs, version overrides, boundary values (all-zero, all-max), the dirty-input overwrite case, and the lexicographic-order checks. Also adds an explicit byte-array expectation for the canonical ULID-spec example string and an additional first-char-range coverage test for isTagged. * [world-vercel] Remove internal-repo reference from regions doc comment * [world-vercel] Address PR review feedback on run-id sub-export - isTaggedString now fully validates the input as a 26-char Crockford Base32 ULID (delegating to ulidToBytes) instead of only inspecting the first character. This fixes false positives on inputs like '4UUUU...' that have a valid tag-bit position but invalid chars later in the string. - isTagged() now accepts `unknown` to match its documented behavior of safely rejecting non-string inputs without requiring callers to cast. - Introduce `RegionKey` for the full set of keys including 'unknown', and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the return type of `lookupRegion` and the `DecodedRunId.region` field accurately reflect that 'unknown' is never produced. Updates `encode` to reject 'unknown' as a region code string at runtime (callers wanting the unknown sentinel should pass numeric 0). * [world] [core] [world-vercel] Add World.createRunId() and region-aware queue routing - @workflow/world: add optional createRunId(input?) to the World interface so worlds can mint run IDs with embedded metadata, and add an optional 'region' field to QueueOptions for per-message routing hints. - @workflow/core: start() now delegates run ID generation to world.createRunId() when defined (falling back to a monotonic ULID otherwise), and accepts a new 'runIdInput' option that is forwarded verbatim to createRunId. When runIdInput.region is a string, it is also threaded onto the queue options so the initial workflow message is dispatched to the matching region. - @workflow/world-vercel: implement createRunId() to mint region-tagged ULIDs, preferring an explicit runIdInput.region and falling back to the VERCEL_REGION env var. The queue now resolves its destination region from (in order): an explicit opts.region, the region embedded in the payload's tagged run ID, the VERCEL_REGION env var, and finally a hardcoded 'iad1' default. This replaces the previous unconditional 'iad1' region passed to the @vercel/queue client. Monotonicity within a process is preserved by tracking the last emitted run ID and bumping the bit immediately above the 11-bit metadata window when a same-ms collision would otherwise occur, then re-stamping the requested region/version on top so metadata remains stable. * [core] [world] [world-vercel] Pass full StartOptions to World.createRunId Drop the dedicated 'runIdInput' field on StartOptions and forward the entire options bag to world.createRunId() instead. This keeps the public API surface smaller and lets each World pick the fields it recognises (e.g. world-vercel reads 'region'). The top-level 'region' option remains on StartOptionsBase and is also threaded onto the queue's per-call region opt when set. * Address review feedback: doc fixes and deterministic same-ms tests - Document the final iad1 fallback in QueueOptions.region (world) - Correct the World.createRunId doc: start() always passes an object - Fix the clientOptions comment: the handler client omits region and relies on SDK auto-detection + the ce-vqsregion header for acks - Fix a misleading QueueClient-construction comment in queue.test.ts - Freeze time in the same-ms monotonicity test so it deterministically exercises the intended path, and add a test covering the bump-above-metadata fallback when the region changes mid-millisecond Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: keep workflow-server override rewrite-compatible Export WORKFLOW_SERVER_URL_OVERRIDE while keeping the one-line const shape that workflow-server's cross-repo e2e test automation rewrites. Update world-vercel tests to import that exported value for mock origins and URL expectations instead of duplicating the temporary preview URL. * fix(world): clear region tag bit before ULID timestamp validation Region-tagged run IDs set the high bit of the ULID timestamp byte. The shared world timestamp validator used raw decodeTime(), so current tagged run IDs appeared thousands of years in the future and were rejected before reaching workflow-server. Clear the tag bit before decoding, matching the workflow-server behavior, and cover tagged IDs in tests. * fix(world-vercel): validate tagged runId timestamps via run-id decode Keep @workflow/world's ULID helpers generic; they should not know about world-vercel's region-tagged run ID layout. Instead, world-vercel decodes its tagged runId to the original ULID before using the shared timestamp validator for run_created events. Add a world-vercel regression test that a current sfo1-tagged runId passes validation. * fix(world-vercel): default run ID region to iad1 instead of unknown When neither an explicit region option nor VERCEL_REGION is available, createRunId minted a tagged ULID with the unknown (0) region sentinel, producing the tagged: true, region: null state. The server already resolves unknown/untagged runs to DEFAULT_VERCEL_REGION (iad1), so mint a concrete iad1 tag instead, keeping every run ID self-describing and routable. * test(e2e): use verbose reporter + per-test start heartbeat The default vitest reporter buffers per-file output, so a stalling e2e test produces no output until its timeout — making CI look like a silent 30-minute hang. Switch the e2e CI invocations to the verbose reporter (prints each test result as it completes) and emit a '[e2e] ▶ start:' heartbeat to stdout at the start of every test (bypassing vitest's console buffering) so a stuck test is immediately identifiable in the live CI log. * test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at combined 527+529 preview Temporarily target the workflow-server combined-527-529-preview deployment, which bundles platform-directed multi-region routing (vercel/workflow-server#527, incl. the iad1 hook pin) and durable stream state (vercel/workflow-server#529), so e2e can validate the full multi-region path end-to-end. Revert to empty on main. * fix(core): region-tag the health-check correlationId The health-check response is delivered over a Redis stream whose name (and synthetic run ID) embed the correlationId. Under platform-directed routing the responding endpoint and the polling reader can be served from different physical regions; Redis is physical-region-local, so the correlationId must carry the region for both sides to resolve the same backend. Generate the correlationId via world.createRunId() (a region-tagged ULID) when the world provides it, falling back to a plain ULID for worlds that don't tag IDs (e.g. local, single-region). The synthetic wrun_hc_<id> run ID then carries the region; workflow-server's region middleware decodes it. * Address review feedback: validate region overrides, reset server override - queue: validate opts.region and VERCEL_REGION against the known region table before routing, ignoring unrecognised codes so a bad override can't clobber the payload-derived region (Copilot) - add isKnownRegionCode() runtime guard to run-id/regions - reset WORKFLOW_SERVER_URL_OVERRIDE to '' (must be empty on main) - fold the within-PR iad1-default changeset into the main world-vercel changeset and delete it (review) - start.test: declare specVersion on createRunId mock worlds now that the merged world-compatibility check requires it - cover the new region-validation fall-through paths in queue.test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(world-vercel): point WORKFLOW_SERVER_URL_OVERRIDE at wave-1 multi-region preview BRANCH-ONLY — revert the override to '' before merge (lint enforces). Points this PR's e2e/benchmark runs at the wave-1 multi-region workflow-server preview (vercel/workflow-server#590: iad1+sfo1+fra1 serving, staging data backends) so region-tagged runs are validated against real multi-region serving end-to-end. Also makes the unit-test mock origins in events-v4.test.ts and trace-propagation.test.ts override-aware (same pattern the rest of the file and utils.test.ts already use), so the suite passes whether or not the override is set — these two files were the only spots hardcoding https://vercel-workflow.com. * test(e2e): Vercel multi-region suite for start()'s region option Adds a dedicated e2e suite validating @workflow/world-vercel region routing end to end, run as its own CI job (e2e-vercel-multi-region) against the nextjs-turbopack workbench only — deliberately separate from e2e.test.ts, which runs as a matrix across all worlds/frameworks where Vercel-specific multi-region behavior doesn't apply. - workbench/nextjs-turbopack/vercel.json: deploy to iad1+sfo1+fra1 so region-routed flow messages have a function to land on in each region. - workflows/99_e2e.ts: regionProbeWorkflow returns the VERCEL_REGION observed by both the workflow and a step, so tests can assert the run EXECUTED in the intended region (not just that it was tagged). - packages/core/e2e/e2e-region.test.ts: per-region cases assert 1) start(..., { region }) mints a region-tagged run ID (decoded via @workflow/world-vercel/run-id), 2) the workflow + step both observed VERCEL_REGION === region, 3) the server reports the run completed; plus a concurrent all-regions case guarding against cross-region misrouting under simultaneous multi-region traffic. Skips on local deployments. - .github/workflows/tests.yml: new e2e-vercel-multi-region job mirroring e2e-vercel-prod's env/deployment-wait, running only the new suite. * test(e2e): start region probes in-function; fix getWorld await The first multi-region CI run surfaced two issues: 1. sfo1/fra1-tagged runs executed in iad1. The suite started runs from the external test process, which uses the api.vercel.com token proxy — and the proxy's queues path forwards every send to the region-less VQS host (the world's proxy-mode resolveBaseUrl ignores the region argument, and the proxy's x-vercel-vqs-api-url escape hatch only allowlists vqs-server-*.vercel.sh preview hosts). Production traffic publishes IN-FUNCTION (direct regional queue routing), so the suite now triggers start() through a new workbench route (/api/e2e-region-start) and rehydrates the run with getRun() — testing the path production actually takes. Proxy-mode regional queue routing is a known gap to address separately in api-workflow. 2. TypeError on world.runs.get: getWorld() is async and was called without await. * test(e2e): cover explicit and implicit region starts in the multi-region suite With regional VQS routing now working through the api.vercel.com proxy (vercel/api#79056 + #2789 + this branch's per-send region resolution), the suite covers both start configurations, asserting the same three properties for each (region-tagged run ID, execution in the intended region via VERCEL_REGION echoed in the return value, server-side completion): 1. EXPLICIT: start(..., { region }) called directly in the vitest runner — publishes through the token proxy, per-send region carried by x-vercel-queue-region. Restores the direct-start shape the suite had originally, plus the concurrent all-regions case. 2. IMPLICIT: dedicated per-region workbench routes (/api/e2e-region-implicit/{iad1,sfo1,fra1}), each pinned to a single region via a per-function 'regions' entry in the workbench vercel.json, calling start() with NO region option — createRunId derives the tag from the minting function's VERCEL_REGION. The test also asserts the route reported executing in its pinned region, so the implicit-tagging assertion can't pass vacuously. Replaces the interim /api/e2e-region-start route (explicit region via request body), which existed to work around the pre-#79056 proxy gap. * Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — wave-1 multi-region serving is in production workflow-server#590 (iad1+sfo1+fra1 serving) merged and deployed to production and the e2e backend, so this branch's e2e/benchmark runs no longer need to target the wave-1 preview. Restores the empty override the No Test Overrides lint job enforces for merge. The override-aware unit-test origins (events-v4/trace-propagation) stay — they are correct under any override value. * test(e2e): cross-region stream visibility (iad1 writer, sfo1 reader) Regression coverage for a backend bug that made cross-region stream reads report zero chunks on IN-PROGRESS streams (completed streams were unaffected), which forced the multi-region serving rollback. The new case exercises exactly that geometry: - crossRegionStreamWorkflow (99_e2e.ts) writes N chunks to the default output stream, then holds the stream OPEN for 45s before closing — the in-progress window is the point, since completed streams are the easy case. - The e2e starts it with region iad1, waits (same-region, via the api.vercel.com proxy) until all chunks are written, asserts the run is still 'running', then reads through a new sfo1-pinned workbench route (/api/e2e-stream-read/sfo1) that returns getTailIndex() plus its VERCEL_REGION. The reader's region served none of the stream's writes, so the reported chunk count must come from the backend's cross-region stream metadata. The test fails loudly if the route isn't actually executing in sfo1. Also bumps the explicit-region test timeout to 120s: the first case in the file absorbs every cold start at once (fresh workbench instances in up to three regions plus a cold backend preview) and was observed just over the 60s default. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE points at a multi-region backend preview that includes the fix, so this validates cross-region stream visibility end-to-end before multi-region serving is re-enabled. * test(e2e): extend multi-region suite to all 19 provisioned regions Points the suite at an all-regions backend preview and widens coverage from the wave-1 trio to every provisioned region: - Explicit path: a single concurrent all-regions case starts one tagged run per region (one shared cold-start window instead of 19 sequential ones) and aggregates per-region failures so a single region's breakage reports alongside the full picture. The trio keeps its detailed per-region cases and the 9-way concurrent-isolation case. - Implicit path: workbench gains a region-pinned /api/e2e-region-implicit/<region> route per provisioned region (19 total, shared handler), the workbench itself now deploys to all of them, and the test.each covers the full set with per-case timeouts for regional cold starts. - Multi-region CI job timeout 20m -> 35m for the sequential implicit cases. BRANCH-ONLY (revert before merge, lint enforces): WORKFLOW_SERVER_URL_OVERRIDE now targets the all-regions backend preview instead of the previous (stale, since-merged) fix preview. * test(e2e): tolerate geo-adjacent execution of queue callbacks The first all-regions run surfaced a subtle execution-locality behavior: queue delivery is guaranteed to the tagged region's dataplane and the delivery callback egresses from that region, but the consumer invocation's execution region is chosen by where that callback enters Vercel's edge — and adjacent regions can geo-resolve to each other's functions. Observed live: kix1-tagged runs (callback egressing from Osaka) deterministically executing in hnd1/Tokyo on both the explicit and implicit paths, with tagging, data placement, and completion all still strictly kix1. expectRunInRegion now asserts execution lands in the tagged region OR one of its geographic neighbors (EXECUTION_ADJACENCY), while run-ID tagging and server-side completion remain strictly the requested region. Gross misrouting (e.g. kix1 -> iad1) still fails. * Revert WORKFLOW_SERVER_URL_OVERRIDE to '' — all-regions serving is in production The all-regions workflow-server rollout is deployed and serving production traffic from every Vercel region, so this branch's e2e no longer needs to target a branch preview. Restores the empty override the No Test Overrides lint enforces for merge. With this the PR is complete: region-tagged run IDs, region-aware queue routing, and the multi-region e2e suite (explicit + implicit + all-regions + cross-region streams) all validate against the production-default backends. * docs: fix three stale comments flagged in review - start.ts: StartOptionsBase.region fallback is iad1, not the unknown sentinel (createRunId always mints a concrete routable region) - queue.ts: example used a nonexistent start({ runIdInput }) API; the real option is start({ region }) - events.ts: decode() clears only the tag bit (top bit of the 48-bit timestamp field) — it does not restore the original untagged ULID; reword to say what actually matters for timestamp validation * test(e2e): cover hook resolve/resume for runs owned by non-iad1 regions Hooks are resolved by opaque token, which carries no region hint, so lookup and resume must work regardless of which region owns the run's data. Exercises the full follow-up-message path on sfo1- and fra1-tagged runs: create inside the workflow, resolve by token from the test process, resume twice sequentially, and assert payload order and completion. Regression coverage for the failure mode where the first message to a hook-driven app on a non-iad1 run worked but every follow-up failed with 'Hook not found'. --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

feat(core): side-effect-free serialization of workflow VM values (#3257) * feat(core): side-effect-free serialization of workflow VM values Serialization runs on the host but inspects values constructed inside the node:vm sandbox, so ordinary dynamic operations dispatch into the sandbox realm and execute workflow code: `value.toISOString()`, `Array.from(map)`, `Object.prototype.toString` (via Symbol.toStringTag), `.source`/`.flags`, `.href`, view `.buffer`/`.byteOffset`/`.byteLength`, and error `.message`/`.stack`/`.cause` reads. That is a determinism hazard. A payload is serialized exactly once and is never re-serialized on replay, so any workflow-visible side effect it triggers exists only on the live path — a patched `Date.prototype.toISOString` that consumes a seeded `Math.random()` draw, for example, shifts every subsequent draw and diverges from replay. This makes serialization side-effect free where the data allows it, and observable where it does not: - Classification uses engine brand checks (node:util types, internal-slot probes) instead of `instanceof global.X` and Object.prototype.toString, so it is immune to Symbol.hasInstance, reassigned sandbox globals, and Symbol.toStringTag spoofs. An unbranded value claiming a brand-decided tag is now classified as a plain object instead of being routed into an extractor that requires the real internal slot (unhardened devalue crashes on that input). - Extraction goes through intrinsics captured at module load — host boot, before any workflow bundle runs — invoked with explicit receivers. Internal slots are realm-agnostic, so host intrinsics read VM-realm objects without touching the sandbox's patchable prototypes. - Property access reads through descriptors, so plain data never invokes anything. Where workflow code must run because the data lives behind it — getters, proxies, custom [WORKFLOW_SERIALIZE] methods, toString() on toStringTag-branded objects like Temporal polyfills — the execution is preserved for compatibility and recorded in a new `CodecOptions.guestCodeStats` sink, surfaced as workflow.serialization.guest_code_{executions,details} span attributes. Consumers that retain a VM across steps can treat a non-empty report as "serialization may have perturbed VM state". Engine-provided accessors are deliberately not reported: V8 defines `stack` as an own accessor on every Error instance, so reporting it would flag every serialized error. Nativeness is decided with the captured host Function.prototype.toString; the bound-function caveat is documented in hardened.ts. Requires devalue 5.9.0 for the pluggable `operations` option. * chore: shorten changeset * fix(core): close review gaps in hardened serialization Five correctness fixes, all with repros: - Callable proxies were treated as engine accessors. V8 returns `function () { [native code] }` from Function.prototype.toString for a proxy around a function rather than throwing, so a proxy-wrapped getter was cached as engine-provided and invoked unreported. Gate on types.isProxy first. - Host builtins implemented in JavaScript were reported as workflow code. Node's DOMException.prototype.message/name are ordinary functions, so the nativeness test failed and every serialized DOMException reported two getter executions. They belong to the *host* realm, though, and workflow code cannot author a host-realm function — so provenance is now decided by nativeness OR host-realm `Function.prototype`, which are disjoint and together cover both cases (V8 installs `stack` per realm, so a VM error's getter is native but VM-realm). - The extraReducers at the two VM call sites were still unhardened, and they run on every value the earlier reducers do not claim — which is exactly where the report has to be complete. `instanceof global.ReadableStream/WritableStream/Request/Response` consulted Symbol.hasInstance on the sandbox class (14 invocations for an ordinary payload once the classes are patched), and AbortController's guard did a bare `value.signal` read, so a non-enumerable `signal` getter ran with an empty report. All five now walk the prototype chain and read through descriptors. - `__closureVarsFn` was invoked unreported on a purity argument that nothing checked: the property is reachable from workflow code, which can replace the compiler-generated function. step.ts now registers the generated function as trusted when it builds the proxy, so provenance is verified rather than assumed, and an unrecognized function is reported. - The URL/URLSearchParams test patched prototypes of *host* classes injected into the sandbox, mutating them for the rest of the worker process. Restored in a finally. Also, per review: - `dehydrateStepArguments` / `dehydrateWorkflowReturnValue` take an optional GuestCodeStats out-param, so a retained-VM gate can consume the report instead of it being spent on span attributes. The report-completeness tests use it to exercise the real dehydrate path. - Every intrinsic capture is now optional. The table is built at module scope, so a missing member was an import-time crash of @workflow/core rather than a degraded path; only SharedArrayBuffer was guarded, while URLSearchParams.prototype.size (Node 19.8+) and the WHATWG classes were assumed. Absent captures now make the corresponding reducer decline to match. - Documented that recording is not prevention (a recorded getter calling Math.random() still advances the run's seeded PRNG), and that a `{ kind: 'proxy' }` report implies a silent shape change (a proxied Map serializes as a plain object). - Parity coverage extended to DataView, boxed primitives, null-prototype objects, setter-only properties, DOMException, AggregateError, an accessor-valued Symbol.toStringTag, both RetryableError retryAfter paths, and a WORKFLOW_SERIALIZE class instance. * fix(core): keep identifying proxied host classes Every Next.js e2e job failed on the two webhook tests: the hook POST returned 404 because `resumeWebhook` could not serialize its step return value ("Cannot stringify arbitrary non-POJOs"), so no hook was ever registered. The value was a `NextRequest`, which Next.js hands over as a **Proxy**. `isInstanceOfPrototype` rejected proxies outright, so the Request reducer answered "not a Request" and devalue fell through to the POJO check. The reasoning behind rejecting them — that proxied built-ins were never serializable, because internal-slot reads throw on a proxy receiver — is true for `Map`/`Date`/`URL`, whose reducers read internal slots, but not for `Request`/`Response`/streams, whose reducers read ordinary properties. Next's proxy forwards those with the target as receiver, so they serialized fine before this PR. Identification now walks through proxies, matching `instanceof`, and records the traps rather than suppressing the answer. The three reducers that do read internal slots (URL, URLSearchParams, Headers) fall back to the dynamic read when the value is a proxy, so their behavior is exactly what it was before — including throwing for a bare proxy over a built-in, which threw before too. Verified against the real thing: the full nextjs-turbopack e2e suite (135 tests) passes locally, having reproduced the failure first and confirmed a reverted `serialization.ts` fixed it. The regression test uses a receiver-correcting proxy, which is what makes NextRequest work in practice; a comment records that a bare `new Proxy(request, {})` throws on undici's private slots with or without this change. * fix(core): state what the closure-fn mark proves, and correct stale docs - `isInstanceOfPrototype`'s JSDoc still described the behavior removed in 8bc462fb5 (proxies rejected without firing traps), which is the opposite of what it now does. - The `__closureVarsFn` provenance check proves the function was passed to `useStep`, not that this package generated it: `useStep` is published on the sandbox global, so workflow code can call it with a function of its own and have it marked. Renamed `registerTrustedFunction` / `isTrustedFunction` to `markUseStepClosureFn` / `isUseStepClosureFn` so the name states the boundary, and documented the laundering caveat alongside the existing ones. Marking still earns its keep — reporting every step that captures a variable would bury the signal — and closing the gap properly needs a compiler-emitted marker, which is a compiler change. - Added the missing coverage for both sides of that check: an unmarked `__closureVarsFn` is invoked and reported, a marked one is invoked and not. - `guestCodeStats` was documented as something a retained-VM gate consumes, but no runtime caller passes a sink; the executions reach telemetry from every dehydrate path regardless. Reworded both docs to say that, so the out-param is not mistaken for wiring that already exists.

fix(core,errors): classify SDK encryption failures as RUNTIME_ERROR (#2145) * fix(core,errors): classify SDK encryption failures as RUNTIME_ERROR SDK-level AES-GCM encrypt/decrypt failures are never the user's fault, but the run-failure classifier was tagging them as USER_ERROR because the native Web Crypto OperationError (most commonly raised by AESCipherJob.onDone on GCM auth-tag mismatch) does not match any RUNTIME_ERROR_CHECKS entry. Introduce a new RuntimeDecryptionError (subclass of WorkflowRuntimeError) that the encryption module throws when subtle.encrypt/subtle.decrypt fails, with the original DOMException as cause plus diagnostic context (operation, byteLength, printable/hex format prefix of the input header). classifyRunError now picks it up via RUNTIME_ERROR_CHECKS, so these failures surface as RUNTIME_ERROR with a proper named class for dashboards and triage. * Trim changeset description to one sentence * Trim historical-context comments * docs: add runtime-decryption-failed troubleshooting page (v4 + v5) * fix(core): round-trip RuntimeDecryptionError context, fix formatPrefix, propagate through serialization wrappers Addresses review feedback on #2145: - Add a RuntimeDecryptionError reducer/reviver (+ SerializableSpecial entry + globalThis registration) so its `context` (operation, byteLength, formatPrefix) survives the dehydrate/hydrate run-error round trip instead of being dropped by the generic Error reducer. - Stop capturing `formatPrefix` in the low-level encryption layer, which only sees the stripped AES payload (nonce bytes), not the outer `encr` marker. The serialization layer now attaches the real envelope prefix. - Rethrow RuntimeDecryptionError unchanged from the serialize/dehydrate catch blocks instead of reframing it as a SerializationError, so an encryption failure during dehydration stays a RUNTIME_ERROR rather than being misclassified as USER_ERROR. * fix(core): enrich stream decrypt errors with envelope prefix + fix lint - Mirror the catch/enrich/rethrow block from serialization/encryption.ts around the stream-path aesGcmDecrypt() call so auth-tag failures on encrypted stream frames also carry context.formatPrefix = 'encr' (addresses review feedback). Add a tampered-frame test. - Fix all auto-fixable Biome lint findings in the touched files (template literals, useless try/catch wrappers, optional chaining, non-null assertions).

feat(core): route sealed envelopes through the serialization layer (#3094) * feat(core): route sealed envelopes through the serialization layer Adds the plumbing that lets a cross-run writer emit `encp` payloads. The serialization layer previously had no way to express "seal to this public key": every consumer expected a symmetric `CryptoKey`, and the encrypt primitive was unconditionally AES-GCM. That gap was also a live footgun. A 32-byte X25519 public key is a structurally valid AES-256 key, so `importKey(pubkey, 'AES-GCM')` succeeds and silently produces ciphertext nobody can ever open — no compile error, no runtime error, just unreadable data. Making public keys reachable only through `sealTo()` turns that mistake from "avoided by convention" into "unrepresentable". `PayloadKey` is a widening rather than a replacement, so no existing call site changes: | Variant | Writes | Reads | Held by | | ---------------- | ------ | ------------ | ----------------------- | | `CryptoKey` | encr | encr | same-run (legacy shape) | | `RunPayloadKeys` | encr | encr, encp | the owning run, o11y | | `SealTarget` | encp | — | cross-run writers | A run's own payloads deliberately stay symmetric even when the holder could seal: sealing costs a fresh ECDH and 32 bytes per envelope and buys nothing when the writer already holds the decryption key. The variants are branded with `Symbol.for` (not `Symbol()`) because these values cross the host ↔ workflow VM realm boundary, where only the global symbol registry is shared. Both stream directions handle sealed frames, keeping the length header in the clear so frame boundaries stay findable without a key. Frames keep per-frame random nonces rather than a counter — a reconnect or a durable replay restarts the writer, and a counter would repeat `(key, nonce)` and break AES-GCM catastrophically. Regression tests assert that 100 frames of identical plaintext produce 100 distinct ciphertexts, and that ten writer incarnations never share a content key. Capability shortfalls fail loudly and specifically: opening a sealed payload with a symmetric key (or a write-only seal target, or nothing) reports "no run keypair is available" rather than surfacing as a mysterious auth-tag failure further down. * review: amortize the sealed-box KEM across stream frames Sealing each frame independently meant a fresh X25519 keygen + ECDH + HKDF per chunk — O(frames) crypto for a long stream. As review pointed out, amortizing the KEM is safe here so long as nonces stay random, which they already are; the hazard I had been guarding against was counter nonces specifically, not a long-lived content key. `createSealSession` performs one encapsulation per writer instance and reuses the content key, with a fresh random nonce per frame. `createOpenSession` mirrors it on the read side, caching decapsulation by ephemeral public key — since every frame from one writer carries the same key, that is an ECDH per writer rather than per frame. Both safety properties are preserved and now asserted directly: - nonces stay random, so sharing a content key cannot repeat `(key, nonce)` (100 identical-plaintext frames -> 100 distinct frames) - a session is scoped to one stream instance, so a reconnect or durable replay never inherits a previous content key (10 writer incarnations -> 10 distinct ephemeral keys) The envelope layout is byte-identical to one-shot `seal`, so readers cannot tell which path produced a frame and need no matching session. Two edge cases the read-side cache introduces, both covered: frames from two writers interleaved on one stream (eviction on every frame, so correctness cannot depend on hit rate), and a failed decapsulation clearing rather than poisoning the entry.

[core] Add wire-level framing for byte streams (#1853) Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com>

feat(core): add `encp` sealed-box encryption primitive (#3093) * feat(core): add `encp` sealed-box encryption primitive Cross-run writes (hook resumptions targeting another run, forwarded writable stream frames) currently require the writer to hold the recipient run's symmetric key, which also grants decrypt capability and costs a ~350ms `run-key` API round trip across a deployment boundary. Add the crypto foundation for sealing those writes to a public key instead. Both keys descend from the per-run key material `K` that `World.getEncryptionKeyForRun()` already returns, so key acquisition, the World interface, and the Vercel API are all untouched: K ├── AES-256 key = K used directly → 'encr' (unchanged) └── X25519 scalar = HKDF(K, label) → 'encp' └── public key (published, not secret) `sealed-box.ts` implements an ECIES-style construction over the same primitives as HPKE base mode (DHKEM(X25519, HKDF-SHA256), AES-256-GCM), binding both public keys into the KDF `info` as HPKE's `kem_context` does to prevent key-substitution attacks. The deviation from strict RFC 9180 framing is documented, and the HKDF labels are versioned so a conformant profile can be added later without touching existing payloads. Nothing produces `encp` payloads yet — this is the primitive only. The o11y layer is hardened defensively so sealed payloads render as ciphertext rather than throwing `Unsupported serialization format`, and `hydrateDataWithKey` skips the AES path for them since opening a sealed payload needs the private scalar rather than the symmetric key. - optional AAD on the AES helpers, used to bind `projectId|runId` - `encapsulate`/`decapsulate` split so stream writers can amortize the KEM across frames; documented that they must keep random per-frame nonces and re-encapsulate per connection attempt, since a long-lived content key plus counter nonces would repeat `(key, nonce)` after a reconnect or a durable replay - public key derivation is cross-validated against node:crypto's native X25519 in tests, since it reads the public half out of a JWK export * review: tighten sealed-box docs, key-length checks, and constants Addresses review feedback on the sealed-box primitive: - The module doc pointed at `getSerializeStream` as enforcing the re-encapsulate-per-writer rule, but nothing in-tree uses `encapsulate` yet, and the stream path added later seals per frame instead. Reworded to state the two rules as the caller's contract, since this module enforces neither. - `derivePublicKeyFromScalar` now asserts the JWK-derived public key is 32 bytes. That decode is the one place this module trusts an external encoding; a short value would otherwise fail much later inside key agreement with a far less obvious message. - `open()` used bare 12/16 for the nonce and tag sizes. Those now come from exported `NONCE_LENGTH`/`TAG_BYTES` in the AES layer, so the wire format check cannot drift from the implementation.

Make resumeHook() resilient to transient hook_received event write failures (#1834) * Make resumeHook() resilient to transient hook_received event write failures When events.create('hook_received') fails with a retryable error (429/5xx), resumeHook() now dispatches the queue message with a `hookInput` payload carrying the dehydrated hook payload. The workflow runtime materializes the missing hook_received event from that payload on its next delivery, mirroring the existing resilient-start behavior of start() / run_created / run_started. Returned Hook carries a new `resilientResume: true` flag when the fallback path was taken. Both write paths share a client-minted `resumeId` as an idempotency key so the runtime can dedup if the direct write actually committed but the client saw a transient error. Uses a sequential write-then-queue flow (not parallel) to avoid a dedup race on the happy path: hook_received events have no entity-level conflict guard (unlike run_created), so a duplicate written before the direct write commits would double-deliver the payload to the workflow. * Fix resilient resume: use local payload in materialized hook_received event The server returns a 'lazy' response for hook_received event creation, where eventData.payload may be a RefDescriptor (when the payload exceeded the inline size and was offloaded to blob storage) rather than the raw bytes. Pushing this directly to the in-memory events array caused the workflow VM to fail with 'Invalid input' when trying to deserialize the RefDescriptor as a Uint8Array. Substitute the eventData we already have locally so the in-memory event matches what getWorkflowRunEvents would return after client-side ref hydration. * Gate resilient resume on target runtime capability; carry hook token; export ResumedHook; docs - Only take the resilient path when the target run's recorded @workflow/core version understands hookInput on the queue payload. Runs keep executing on the deployment they were created on (skew protection), and older runtimes parse the queue message with a schema that silently strips unknown fields - the resume payload would be lost while resumeHook() reported success. Fail fast (propagate the original event-write error) for such runs instead, preserving the caller's ability to retry. - Carry the hook token on hookInput and write it into the materialized hook_received event so it gets the same replay-divergence guard as a directly written event (#2030 parity). - Export ResumedHook from @workflow/core/runtime and workflow/api. - Add changelog page and update resumeHook() API reference docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: correct capability cutoff, drop own-version escape hatch, replay-side resumeId dedup Review fixes for the resilient-resume capability gate and dedup: - Bump the supportsQueueHookInput cutoff to 5.0.0-beta.39: 5.0.0-beta.38 is published WITHOUT this feature (its queue-payload schema strips hookInput), so classifying it as capable would silently lose resume payloads. The cutoff is now a single exported constant (QUEUE_HOOK_INPUT_MIN_VERSION) with a TODO(release) requiring re-verification at merge time. - Remove the own-version exact-match escape hatch entirely: version strings do not identify builds (a published beta.38 and a main-built tarball can share a version string while differing in content), so the check could declare a featureless published deployment capable. Pre-release builds now fall back to fail-fast until the version is bumped past the cutoff — the safe direction. Tests simulate a capable target explicitly. - Make duplicate suppression authoritative at the replay boundary: replay now dedups hook_received events sharing a resumeId (same resume attempt), so even when concurrent redelivery of the same queue message double-materializes the event (no World enforces uniqueness on hook_received), the payload reaches workflow code exactly once. This is a pure function of the persisted log, keeping replay deterministic. The runtime's snapshot check remains as best-effort write suppression, with its comment corrected to say so; the EntityConflictError catch is kept as the forward-compatible signal for planned server-side (runId, resumeId) uniqueness, with its comment corrected to say it is defensive today. - Stamp materialized hook_received events with occurredAt decoded from the resumeId ULID so resiliently-resumed hooks are timestamped at resume time rather than after the queue round-trip. - Pin the cross-version compat contract in a test: the direct write is resumeId-only (no digest or negotiation fields), which later server-side idempotency work must keep accepting. - Exercise the published boundary (5.0.0-beta.38) in fail-fast tests, and make the capability tests self-check against the exported cutoff constant instead of restating literals. - Docs: changelog date June -> July 2026, dash consistency, and document the replay-side dedup guarantee. * Encode release-gate and successor-rebase contracts into code comments Comment-only changes capturing the review agreements so they survive the parallel-resume successor rebase (no behavior change): - capabilities.ts: the QUEUE_HOOK_INPUT_MIN_VERSION re-verification point is the actual combined SDK release (after the successor lands and its server-side dedup is deployed), not source-merge time — this PR merges source-only and no SDK is published from it alone. Every Version Packages merge in between moves the earliest possible carrier. - workflow/hook.ts + runtime.ts: scope the replay-side resumeId dedup honestly as defense-in-depth over the persisted log, not a cross-invocation exactly-once guarantee — concurrent invocations replaying pre-duplicate snapshots each see only their own row; the storage-level (runId, resumeId) constraint in the successor work is the correctness boundary. The set stays useful post-constraint for logs written before it deployed. - runtime.ts: document the EntityConflictError swallow's known gap while the branch is defensive (this invocation's local log lacks the payload; progress relies on the other writer's delivery or redelivery) and pin the rebase contract for when the constraint makes it live: a matching claim must append the canonical event locally and succeed; a real conflict must rethrow for redelivery. - resume-hook-resilient.test.ts: reframe the wire-shape pin as a tripwire rather than a permanent contract — the successor deliberately widens it (ID/digest pair + attestation) before any SDK release, so the resumeId-only shape never ships as a published server contract. --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

[core] Fix process crash from rejected waitUntil promises (#2336) Co-authored-by: Peter Wielander <mittgfu@gmail.com>

QuickJS engine: host-side, side-effect-free serialization via handles (#3263) * Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay * QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed * QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols * QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity * Apply biome fixes to QuickJS engine files * Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard * CI: include generated QuickJS source assets in shared e2e build artifacts * Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine * CI: run both VM engines across all frameworks and worlds; label jobs with the engine * Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads * e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status) * e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely) * Sort imports in QuickJS serialization files (biome organizeImports) * QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal hook payloads to the target run's published X25519 public key. The shared start() path publishes that key regardless of engine, so QuickJS runs receive sealed payloads too — but the QuickJS entrypoint resolved only the bare symmetric key via importKey(), which cannot open encp envelopes. The first sealed hook payload wedged the run right after hook_received, timing out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the node engine resolves the full capability via memoizeEncryptionKey). Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric (encrypt() with RunPayloadKeys takes the encr path). Regression test seals a payload exactly as resumeHook does and round-trips it through the VM. * Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping - Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap, drawing from the seeded Math.random (identical sequences to the node engine's vm/index.ts implementations); all crypto.subtle methods throw with step-function guidance. process.env exposed as a frozen copy, matching node. - Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family methods (incl. localeCompare) throw when given an explicit locale so cross-engine divergence is loud instead of silently writing different values into the event log. No-argument forms keep working. - runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the ~1.3MB embedded WASM assets out of node-engine deployments. - runQuickJSWorkflow wraps the per-run phase so an exceptional exit disposes the VM instead of leaking it in a reused compute instance; corrected the misleading fail-loud comment (run_failed, not retry); warn when the event drain loop exhausts its iteration bound. - Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the file's workflow.* namespace. - Eval-string correlation-id interpolation uses JSON.stringify instead of quote-only escaping. - common-vm.test.ts pins the reducer/reviver superset invariant against common.ts so the duplicated sets can't silently drift. - Docs enumerate the remaining global-surface differences (subtle.digest, Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known precondition-guard gap. * QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup) #1834 made resumeHook() fall back to enqueueing the run with a hookInput payload when the direct hook_received write fails transiently, with the runtime materializing the missing event on delivery. Only the node:vm path implemented it — the QuickJS dispatch returned before the node block, so the resilient payload was silently dropped and the new e2e timed out on every quickjs leg. - runtime.ts threads hookInput into runWorkflowWithQuickJS; the entrypoint materializes the missing hook_received after loading the event log (resumeId-keyed dedup, occurredAt from the resumeId ULID, local eventData substitution for lazy/ref responses, EntityConflict / HookNotFound handling) — mirroring the node block. - processEvents drops duplicate hook_received rows sharing a resumeId (first-in-log wins), matching the node engine's EventsConsumer dedup; the seen-set lives in the VM heap so it is deterministic per replay. Verified against the dev server with WORKFLOW_VM=quickjs: the resilient resume e2e passes and the materialization is observable in the logs; all 27 hook e2e tests green. * QuickJS engine: inline step execution via live-VM continuation loop + WASM module caching * Address review: exclusive inline step claims, self-write requeue, in-loop event ceiling - Inline steps now claim via a lazy step_started carrying the input (step_created deferred, atomic create-claim in the world), with ownerMessageId stamped and authoritativeAttempt=1 — a concurrent invocation racing on the same fresh step loses with EntityConflictError and skips instead of both bare-starting the step and double-running the body. This also removes the stepsCreatedByUs set, whose 'created by us' invariant didn't survive the swallowed create-race conflict; redelivery backstops now key on hasCreatedEvent. - dispatchPendingOps' createdAttributeEvent/createdGetConflictHook signals are consumed again: when the loop exits suspended without ever reading back a self-written attr_set / getConflict hook_created (eventually-consistent listing lag), the entrypoint requeues immediately instead of parking the run awaiting_external with its unblocking event already written. - The server-supplied event ceiling is re-checked at the top of every continuation-loop turn (seenEventIds.size), so a single invocation fanning out inline can no longer grow the log arbitrarily past the operator's limit. The quickjs dispatch in runtime.ts converts MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the guard's throw previously nacked forever, parking runaway runs in 'running'. - Documented the deliberate decision that the platform function timeout is the only bound on inline chaining (budget parked per batch), matching the node engine. * QuickJS engine: host-side, side-effect-free serialization via handles (Re-applied onto the review-fixed base; original commits da2723016 + 9814ed9ac squashed.) Replace the in-VM serde bundle with a host-side codec (runtime/quickjs-serde.ts) built on quickjs-wasi 3.3's introspection primitives and devalue 5.9's pluggable stringify/parse operations — mirroring the node:vm engine's architecture. Review fixes incorporated: - reducer/reviver key sets are pinned against codec-devalue-vm's workflow mode by exhaustiveness tests (exact order for reducers — first match wins), so the handle-space codec can't silently drift from the shared value-space sets. - the devalue entry in minimumReleaseAgeExclude is removed: the exact version is pinned via the workspace catalog + lockfile, so the cooldown waiver was unnecessary (verified with both frozen and regular installs). - eval-string interpolation inherits the JSON.stringify(cid) hardening from the base branch. * Address review: NUL-safe string extraction, deterministic retryAfter, pass-scoped handle disposal, byte-cache lifecycle - NUL (U+0000) safety across the WASM boundary: handle.toString() routes through JS_ToCString and silently truncates at the first NUL, and the C-string key APIs mangle NUL-bearing property keys (drop or collide). guestString() detects truncation by comparing against the handle's true guest length and recovers via in-VM JSON.stringify escaping; shapeOf verifies its fast host-string key list against a guest Object.keys count (+ duplicate check) and re-extracts through key handles on mismatch; get/hasOwn route NUL-bearing keys through length-aware guest string handles. All string funnels (primitives, symbol descriptions, error fields via chained/own reads, Headers entries, RegExp source/flags, URL href) go through guestString. Regression-tested down to the truncate-vs-collide enumeration shapes; fixes nullByteWorkflow on the quickjs e2e legs. - RetryableError's absent/invalid retryAfter fallback now reads the GUEST clock (the deterministic replay clock at the WASI layer) via a captured Date.now instead of the host wall clock — the in-VM reducer was replay-stable by construction and the host port silently lost that. - Pass-scoped handle disposal: serialize/deserialize sweep every intermediate handle their pass creates (call/invoke results, descriptor reads, dups, parse-op constructions), closing the ~one-leaked-handle-per-value-node growth across long-lived inline sessions. Implemented with module-owned tracking rather than vm.withScope: the library scope also captures the handles the host-callback trampoline wraps around C-owned argv pointers, and disposing those (Map/Set/Headers forEach visitors run mid-pass) double-frees guest values — observed as WASM memory corruption. identities is cleared per pass so freed-pointer reuse cannot alias entries across passes. - Byte-cache lifecycle: terminal drain now shares the per-VM cache with the suspension path (re-serializing an op at drain could re-invoke getters and produce different bytes for what the log treats as one value), and entries for settled ops — which neither collection filter can match again — are evicted, bounding the cache by the live pending set. * Adopt quickjs-wasi 3.3.1: withScope handle sweeping, real memoryLimit accounting, loud unregistered-callback failures 3.3.1 ships the three fixes this branch surfaced upstream: - Borrowed host-callback handles (vercel-labs/quickjs-wasi#31): the trampoline's this/argv handles are scope-exempt, making vm.withScope safe around host callbacks. The serde's module-owned pass-disposal apparatus (passDisposal/track/runWithPassDisposal and ~18 track() wraps) is replaced by withScope in serialize/deserialize — simpler, and strictly more complete: every handle constructed during the pass is swept, not just the ones our creation funnels saw. Bench parity confirmed (within ~10% on the 50k-node extreme case, unchanged elsewhere; still 2.6-100x over the in-VM codec). - Real memoryLimit accounting (vercel-labs/quickjs-wasi#33): the engine's 256 MB VM ceiling now actually bounds retained guest allocations (usable-size was 0 on wasm32-wasi before, so the limit never accumulated). - Unregistered host callbacks throw (vercel-labs/quickjs-wasi#34): guest calls into missing callbacks fail loud instead of silently returning undefined — protection this engine wants for snapshot-restore re-registration bugs. Also merges origin/main (undici 7.29.0). * Address review: lossless lone-surrogate string extraction, portable base64 P1 — the guestString length check was insufficient: JS_ToCString has TWO corruptions (NUL truncation, lone-surrogate -> U+FFFD replacement) and they can cancel — the replacement expansion offsets the truncation so the extracted length matches the true guest length. A bare lone surrogate can also replace 1:1 with no length change at all. Worse, the JSON.stringify slow path was itself lossy for lone surrogates: QuickJS passes them through raw, and the C-string extraction of ITS output corrupts them. - guestString accepts the fast value only when length matches AND it contains no U+FFFD (legitimate U+FFFD strings take the loss-free slow path); the slow path now escapes INSIDE the VM to printable ASCII via a new captured escapeString intrinsic (WTF-16-safe per-code-unit \uXXXX escaping), then JSON-parses host-side. - shapeOf's fast-key acceptance adds a U+FFFD scan alongside the count/duplicate checks (lone-surrogate keys corrupt with count and uniqueness intact). - get()/hasOwn() route keys through guest string handles when they carry a NUL or an UNPAIRED surrogate (paired surrogates - emoji keys - encode fine through the C-string APIs; vm.newString is verified WTF-16-preserving for the handle path). - Tests: the reviewer's exact length-canceling case, bare lone surrogates, legit-U+FFFD passthrough, byte parity with the reference codec, and lone-surrogate/mixed keys. P2 — the codec's base64 helpers no longer carry an unconditional Node Buffer dependency: feature-detected Uint8Array.fromBase64/toBase64 when available, Buffer when present, btoa/atob loop otherwise — keeping WASM-only/non-Node hosts (Cloudflare Workers) viable. * Adopt quickjs-wasi 3.4.0: delete the NUL/surrogate string machinery 3.4.0 ships lossless string transport (vercel-labs/quickjs-wasi#35 — found by this PR's review cycle), so the SDK-side detection and escape machinery is deleted wholesale: - guestString (length + U+FFFD detection, in-VM escape fallback) — plain toString() is lossless now - the escapeString / hasOwnCall / jsonStringify / objectKeys captured intrinsics - keyNeedsHandleLookup and the handle-keyed get/hasOwn routing — the library routes inexpressible keys itself - shapeOf's guest-key verification pass (count/duplicate/U+FFFD scans) — enumeration is lossless Net ~130 lines and four captured intrinsics removed; the serde now uses the plain quickjs-wasi surface everywhere. Test honesty fix that 3.4.0 forced: the earlier lone-surrogate round-trip tests passed only via mutual corruption — the pre-3.4.0 lossy host→guest transport corrupted the guest comparison literals identically to the wire. With an honest transport they exposed that the WIRE itself (devalue emits lone surrogates raw; the wire is UTF-8) degrades lone surrogates to U+FFFD — in the node engine's reference codec exactly as here, verified. Bug-compatible parity is the load-bearing property (event logs replay across engines), so those tests now assert byte parity with the reference codec plus guest-observed equality with the reference codec's own round trip; NULs are devalue-escaped and asserted to survive exactly. Wire-level surrogate preservation is a product-wide devalue/UTF-8 question, tracked separately from this engine.

fix(core): don't observe idle while a committed delivery is parked behind its deferral (#3198)

feat: publish each run's X25519 public key on the run entity (#3095) * feat: publish each run's X25519 public key on the run entity A cross-run writer needs the recipient run's public key to seal a payload to it. Derive that key at `start()` and stamp it on the run, so a hook resumption or a forwarded-stream writer can find it on a run fetch it was already making instead of spending ~350ms on `run-key`. The key is derived from the per-run key material `getEncryptionKeyForRun()` already returns, so nothing about key acquisition changes. It is not secret: the matching private scalar is never stored anywhere, only re-derived on demand from the deployment's own env seed. Storing it beside run metadata therefore does not weaken the run's confidentiality. **Presence is the writer-side gate for sealed envelopes.** A run only carries a public key if the runtime that created it could also open one — which holds by construction, since derivation and `encp` dispatch both live in `@workflow/core`, so any core that can stamp can also open. Runs are pinned to their creating deployment, so the capability this attests to is still accurate at resume time. Writers seal iff the field is set and otherwise fall back to the symmetric path, which makes version skew degrade gracefully instead of wedging a run. The field rides on `run_created`, and is mirrored onto the queued `runInput` so the resilient-start path (server recreates the run from the queue message when the `run_created` write failed) doesn't silently produce a run that can't receive sealed writes. world-vercel's compile-time wire-contract guard caught the new field before it could be silently dropped on the v4 path, exactly as designed — routed into the frame meta block as plaintext metadata. Also adds browser- and VM-safe base64 helpers to `sealed-box.ts`, since neither `Buffer` nor `btoa` can be assumed in every context that module runs in. `base64ToBytes` returns undefined on malformed input rather than throwing, so a corrupt stored key degrades to "no usable public key" and falls back to the symmetric path instead of crashing a resumption. Both are cross-validated against `Buffer` in tests. * review: fix public-key loss on resilient start and lifecycle updates Two real bugs found in review, both in the local worlds. Neither surfaces as an error — a run just silently stops accepting sealed cross-run writes and falls back to the slow symmetric path forever. **Resilient start dropped the key.** When a `run_started` arrives for a run that was never created, world-local and world-postgres rebuild the run from the queued message. Neither copied `encryptionPublicKey` onto the run row or the synthetic `run_created` event they write. That is precisely the scenario this field exists to survive. (The equivalent server-side path was already handled.) **world-local also wiped the key on every lifecycle transition.** Its run_started / run_completed / run_failed / run_cancelled handlers rewrite the whole run document field-by-field, so any field not explicitly listed is dropped — meaning the key was lost on the *first* `run_started`, not just on the resilient path. All four rebuild sites now carry it. world-postgres is safe here by construction because it issues column-scoped SQL UPDATEs rather than rewriting the row. **base64 decoding is now strict.** The decoder accepted shapes that cannot describe a whole number of bytes (`length % 4 === 1`) and ignored anything after a mid-string `=`, returning a short array instead of `undefined`. That is worse than throwing: a corrupt stored key looked *present*, so callers sealed to garbage rather than taking the symmetric fallback. Now rejects out-of-alphabet characters, bad lengths, misplaced padding, and non-zero trailing bits — with a round-trip test over every length 0–48 to make sure the strictness does not overshoot. * fix: send encryptionPublicKey in the v4 POST frame meta `splitEventDataForV4` lifted the run's public key into the frame meta and `events.ts` spread that meta into `CreateEventV4Input`, but `buildPostFrameMeta` — which copies meta onto the wire field by field — never forwarded `encryptionPublicKey`, and the field was missing from `CreateEventV4Input` entirely. Because the meta is applied with a spread, TypeScript's excess-property check doesn't fire, so the key was computed, put in the meta, and then silently dropped before the request was sent. The server therefore never received the key, never stored it on the run entity, and every cross-run writer fell back to the symmetric envelope. Every symptom pointed away from the SDK: a deliberately oversized key was accepted rather than rejected (the field never arrived), the key was absent from the run row, and `resumeHook()` always chose `encr`. Add the field to `CreateEventV4Input`, forward it in `buildPostFrameMeta`, and cover it for both `run_created` and resilient-start `run_started`. Also add a generic guard asserting that every field the splitter puts in the meta reaches the wire, so the next omission in this hand-maintained mapping fails a test instead of silently degrading encryption.

QuickJS engine: baseline-snapshot startup optimization (~25× faster VM boot) (#3342) * Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay * QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed * QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols * QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity * Apply biome fixes to QuickJS engine files * Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard * CI: include generated QuickJS source assets in shared e2e build artifacts * Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine * CI: run both VM engines across all frameworks and worlds; label jobs with the engine * Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads * e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status) * e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely) * Sort imports in QuickJS serialization files (biome organizeImports) * QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal hook payloads to the target run's published X25519 public key. The shared start() path publishes that key regardless of engine, so QuickJS runs receive sealed payloads too — but the QuickJS entrypoint resolved only the bare symmetric key via importKey(), which cannot open encp envelopes. The first sealed hook payload wedged the run right after hook_received, timing out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the node engine resolves the full capability via memoizeEncryptionKey). Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric (encrypt() with RunPayloadKeys takes the encr path). Regression test seals a payload exactly as resumeHook does and round-trips it through the VM. * Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping - Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap, drawing from the seeded Math.random (identical sequences to the node engine's vm/index.ts implementations); all crypto.subtle methods throw with step-function guidance. process.env exposed as a frozen copy, matching node. - Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family methods (incl. localeCompare) throw when given an explicit locale so cross-engine divergence is loud instead of silently writing different values into the event log. No-argument forms keep working. - runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the ~1.3MB embedded WASM assets out of node-engine deployments. - runQuickJSWorkflow wraps the per-run phase so an exceptional exit disposes the VM instead of leaking it in a reused compute instance; corrected the misleading fail-loud comment (run_failed, not retry); warn when the event drain loop exhausts its iteration bound. - Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the file's workflow.* namespace. - Eval-string correlation-id interpolation uses JSON.stringify instead of quote-only escaping. - common-vm.test.ts pins the reducer/reviver superset invariant against common.ts so the duplicated sets can't silently drift. - Docs enumerate the remaining global-surface differences (subtle.digest, Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known precondition-guard gap. * QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup) #1834 made resumeHook() fall back to enqueueing the run with a hookInput payload when the direct hook_received write fails transiently, with the runtime materializing the missing event on delivery. Only the node:vm path implemented it — the QuickJS dispatch returned before the node block, so the resilient payload was silently dropped and the new e2e timed out on every quickjs leg. - runtime.ts threads hookInput into runWorkflowWithQuickJS; the entrypoint materializes the missing hook_received after loading the event log (resumeId-keyed dedup, occurredAt from the resumeId ULID, local eventData substitution for lazy/ref responses, EntityConflict / HookNotFound handling) — mirroring the node block. - processEvents drops duplicate hook_received rows sharing a resumeId (first-in-log wins), matching the node engine's EventsConsumer dedup; the seen-set lives in the VM heap so it is deterministic per replay. Verified against the dev server with WORKFLOW_VM=quickjs: the resilient resume e2e passes and the materialization is observable in the logs; all 27 hook e2e tests green. * QuickJS engine: inline step execution via live-VM continuation loop + WASM module caching * Address review: exclusive inline step claims, self-write requeue, in-loop event ceiling - Inline steps now claim via a lazy step_started carrying the input (step_created deferred, atomic create-claim in the world), with ownerMessageId stamped and authoritativeAttempt=1 — a concurrent invocation racing on the same fresh step loses with EntityConflictError and skips instead of both bare-starting the step and double-running the body. This also removes the stepsCreatedByUs set, whose 'created by us' invariant didn't survive the swallowed create-race conflict; redelivery backstops now key on hasCreatedEvent. - dispatchPendingOps' createdAttributeEvent/createdGetConflictHook signals are consumed again: when the loop exits suspended without ever reading back a self-written attr_set / getConflict hook_created (eventually-consistent listing lag), the entrypoint requeues immediately instead of parking the run awaiting_external with its unblocking event already written. - The server-supplied event ceiling is re-checked at the top of every continuation-loop turn (seenEventIds.size), so a single invocation fanning out inline can no longer grow the log arbitrarily past the operator's limit. The quickjs dispatch in runtime.ts converts MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the guard's throw previously nacked forever, parking runaway runs in 'running'. - Documented the deliberate decision that the platform function timeout is the only bound on inline chaining (budget parked per batch), matching the node engine. * QuickJS engine: host-side, side-effect-free serialization via handles (Re-applied onto the review-fixed base; original commits da2723016 + 9814ed9ac squashed.) Replace the in-VM serde bundle with a host-side codec (runtime/quickjs-serde.ts) built on quickjs-wasi 3.3's introspection primitives and devalue 5.9's pluggable stringify/parse operations — mirroring the node:vm engine's architecture. Review fixes incorporated: - reducer/reviver key sets are pinned against codec-devalue-vm's workflow mode by exhaustiveness tests (exact order for reducers — first match wins), so the handle-space codec can't silently drift from the shared value-space sets. - the devalue entry in minimumReleaseAgeExclude is removed: the exact version is pinned via the workspace catalog + lockfile, so the cooldown waiver was unnecessary (verified with both frozen and regular installs). - eval-string interpolation inherits the JSON.stringify(cid) hardening from the base branch. * Address review: NUL-safe string extraction, deterministic retryAfter, pass-scoped handle disposal, byte-cache lifecycle - NUL (U+0000) safety across the WASM boundary: handle.toString() routes through JS_ToCString and silently truncates at the first NUL, and the C-string key APIs mangle NUL-bearing property keys (drop or collide). guestString() detects truncation by comparing against the handle's true guest length and recovers via in-VM JSON.stringify escaping; shapeOf verifies its fast host-string key list against a guest Object.keys count (+ duplicate check) and re-extracts through key handles on mismatch; get/hasOwn route NUL-bearing keys through length-aware guest string handles. All string funnels (primitives, symbol descriptions, error fields via chained/own reads, Headers entries, RegExp source/flags, URL href) go through guestString. Regression-tested down to the truncate-vs-collide enumeration shapes; fixes nullByteWorkflow on the quickjs e2e legs. - RetryableError's absent/invalid retryAfter fallback now reads the GUEST clock (the deterministic replay clock at the WASI layer) via a captured Date.now instead of the host wall clock — the in-VM reducer was replay-stable by construction and the host port silently lost that. - Pass-scoped handle disposal: serialize/deserialize sweep every intermediate handle their pass creates (call/invoke results, descriptor reads, dups, parse-op constructions), closing the ~one-leaked-handle-per-value-node growth across long-lived inline sessions. Implemented with module-owned tracking rather than vm.withScope: the library scope also captures the handles the host-callback trampoline wraps around C-owned argv pointers, and disposing those (Map/Set/Headers forEach visitors run mid-pass) double-frees guest values — observed as WASM memory corruption. identities is cleared per pass so freed-pointer reuse cannot alias entries across passes. - Byte-cache lifecycle: terminal drain now shares the per-VM cache with the suspension path (re-serializing an op at drain could re-invoke getters and produce different bytes for what the log treats as one value), and entries for settled ops — which neither collection filter can match again — are evicted, bounding the cache by the live pending set. * Adopt quickjs-wasi 3.3.1: withScope handle sweeping, real memoryLimit accounting, loud unregistered-callback failures 3.3.1 ships the three fixes this branch surfaced upstream: - Borrowed host-callback handles (vercel-labs/quickjs-wasi#31): the trampoline's this/argv handles are scope-exempt, making vm.withScope safe around host callbacks. The serde's module-owned pass-disposal apparatus (passDisposal/track/runWithPassDisposal and ~18 track() wraps) is replaced by withScope in serialize/deserialize — simpler, and strictly more complete: every handle constructed during the pass is swept, not just the ones our creation funnels saw. Bench parity confirmed (within ~10% on the 50k-node extreme case, unchanged elsewhere; still 2.6-100x over the in-VM codec). - Real memoryLimit accounting (vercel-labs/quickjs-wasi#33): the engine's 256 MB VM ceiling now actually bounds retained guest allocations (usable-size was 0 on wasm32-wasi before, so the limit never accumulated). - Unregistered host callbacks throw (vercel-labs/quickjs-wasi#34): guest calls into missing callbacks fail loud instead of silently returning undefined — protection this engine wants for snapshot-restore re-registration bugs. Also merges origin/main (undici 7.29.0). * QuickJS engine: baseline-snapshot startup optimization Evaluating the workflow bundle dominates VM startup (~74ms of a ~77ms boot for the 1.3MB e2e bundle) and full event replay pays it on EVERY invocation — a large share of the quickjs engine's TTFS gap vs node:vm, where V8 compiles the same script in single-digit ms. The bundle is identical across all runs of a deployment, so the engine now hydrates one VM per function instance (bootstrap + bundle eval), snapshots its memory, and starts every invocation with QuickJS.restore (~3ms) instead of re-evaluating. Measured on the real generated e2e flow bundle (154 workflows), boot to first suspension: fresh 79.4ms -> restored 3.2ms (24.8x). First invocation pays hydrate+restore (85.8ms, ~= one fresh boot); every subsequent invocation — including every replay wake — gets the discount. Determinism: replay requires module-scope user code to observe the run-seeded PRNG and deterministic clock, and a restored heap carries whatever module scope computed at hydrate time. The hydrate therefore runs with draw-counting placeholder host fns and a read-counting clock; a bundle that consumed either is marked ineligible and every invocation falls back to fresh evaluation (node:vm-parity semantics preserved exactly). When the gate passes, restore is byte-equivalent to fresh eval: the per-run host fns (random / __generateNanoid / __generateUlid) re-register by NAME on the restored VM before the workflow body runs, so the seeded draw sequence — and every correlationId — is identical. Pinned by a parity test that feeds Math.random() into a step input and byte-compares the serialized ops across fresh, first-restore and cached-restore invocations. Cache: per function instance, keyed on the bundle string (reference-stable in generated flow routes), promise-deduped for concurrent first invocations, capped at 4 entries; hydrate rejections evict for retry while eval failures cache as ineligible (the fresh path re-evaluates and surfaces the real, source-mapped error). Kill switch: WORKFLOW_QUICKJS_BASELINE_SNAPSHOT=0. * Address review: intrinsics-replacement gate, hydrate-failure fallback, shared clock helper, review nits - Serialization-intrinsics gate (the substantive finding): the restore path's serde captures intrinsics from the restored heap — AFTER module scope ran — while the fresh path captures before user code. A bundle that replaced a captured intrinsic at module scope (e.g. a Date.prototype.toISOString polyfill) without touching PRNG/clock passed the eligibility gate yet would serialize differently on the two paths. captureIntrinsicsSignature (exported from quickjs-serde) identity-fingerprints every to-be-captured value; the hydrate compares it before and after bundle eval and marks any replacement ineligible. Expression-created entries (makeSparseArray, makeThunk, hasOwnCall) are excluded — they get fresh identities per eval and cannot be replaced by user code. Gate test added with a toISOString polyfill. - Hydrate-failure fallback: a getBaselineEntry rejection (infrastructure — vm.snapshot() under memory pressure, QuickJS.create failing) no longer fails the invocation; it logs a warning and falls back to fresh evaluation, with the cached promise already evicted for retry. - initWorkflowVM now uses the shared makeDeterministicClockWasi helper its doc claimed it shared, so the two clock implementations cannot drift. - getCompiledAssets() awaited once per call site (hydrate + restore). - WORKFLOW_TURBO JSDoc reattached to isTurboEnabled (the baseline constant had been inserted between doc and function). - Parity test saves/restores any pre-existing WORKFLOW_QUICKJS_BASELINE_SNAPSHOT env value instead of deleting it. * Fix source-map remapping for workflows sharing a baseline snapshot The baseline cache is keyed on the bundle, which every workflow in a deployment shares — but the hydrate evaluated the bundle with the FIRST caller's workflowId as the eval filename. That name is baked into the snapshot's compiled code, so on the restore path every OTHER workflow's stack frames referenced the first hydrator's id, and remapErrorStack (which matches frames by the failing run's module specifier) never matched them — raw bundle line numbers leaked into user-visible stacks for any workflow outside the first hydrator's module. Hydrate now evaluates under a workflow-independent constant (BASELINE_BUNDLE_FILENAME), and the entrypoint's three remap sites (failed-branch stack, hydrated error, cause chain) remap against BOTH filename spaces — the run's module specifier covers fresh-path frames, the constant covers snapshot-path frames; remapErrorStack early-exits on a cheap includes() for whichever space has no frames. Regression test: a two-module bundle hydrated under workflow A, with workflow B failing through the restored snapshot — B's stack must reference the constant filename and not A's id. * Address review: lossless lone-surrogate string extraction, portable base64 P1 — the guestString length check was insufficient: JS_ToCString has TWO corruptions (NUL truncation, lone-surrogate -> U+FFFD replacement) and they can cancel — the replacement expansion offsets the truncation so the extracted length matches the true guest length. A bare lone surrogate can also replace 1:1 with no length change at all. Worse, the JSON.stringify slow path was itself lossy for lone surrogates: QuickJS passes them through raw, and the C-string extraction of ITS output corrupts them. - guestString accepts the fast value only when length matches AND it contains no U+FFFD (legitimate U+FFFD strings take the loss-free slow path); the slow path now escapes INSIDE the VM to printable ASCII via a new captured escapeString intrinsic (WTF-16-safe per-code-unit \uXXXX escaping), then JSON-parses host-side. - shapeOf's fast-key acceptance adds a U+FFFD scan alongside the count/duplicate checks (lone-surrogate keys corrupt with count and uniqueness intact). - get()/hasOwn() route keys through guest string handles when they carry a NUL or an UNPAIRED surrogate (paired surrogates - emoji keys - encode fine through the C-string APIs; vm.newString is verified WTF-16-preserving for the handle path). - Tests: the reviewer's exact length-canceling case, bare lone surrogates, legit-U+FFFD passthrough, byte parity with the reference codec, and lone-surrogate/mixed keys. P2 — the codec's base64 helpers no longer carry an unconditional Node Buffer dependency: feature-detected Uint8Array.fromBase64/toBase64 when available, Buffer when present, btoa/atob loop otherwise — keeping WASM-only/non-Node hosts (Cloudflare Workers) viable. * Merge quickjs-host-serde (lossless surrogate extraction, portable base64) into quickjs-baseline-snapshot The new escapeString captured intrinsic is expression-created (a fresh guest closure per capture eval), so it joins makeSparseArray/makeThunk/ hasOwnCall in captureIntrinsicsSignature's exclusion list — without this the baseline hydrate gate would classify every bundle ineligible (the byte-parity test catches exactly that, as it did when hasOwnCall was missed). * Address review: pre-eval serde capture root, adopted by pointer from the snapshot The intrinsics-replacement gate was structurally losing: its own post-eval probe executed guest-reachable code (CAPTURE_INTRINSICS calls Object.getOwnPropertyDescriptor / Object.getPrototypeOf), those dependencies were not in the identity signature, and a module-scope stateful wrapper around them both evaded detection AND had its side effects baked into the snapshot — fresh returned 0 from the reviewer's counter repro while restore returned the probe's call count. Replace detection with prevention: ALL guest-touching serde initialization (intrinsics capture, branded samples, well-known symbol lookups) is bundled into one CAPTURE_ROOT expression evaluated in the baseline VM BEFORE the bundle — the same capture-before-user-code ordering the fresh path has always had. The container handle's box lives in the snapshot's linear memory, its raw pointer rides the BaselineEntry, and every restored VM re-adopts it (adoptSerdeRoot) — serde init then performs only plain-data property reads and C-level classId reads: NO guest code executes after user code has run, on either path. Consequences: - the identity-signature gate and its expression-created skip-list are deleted (nothing to detect — module-scope intrinsic patching is now HARMLESS on the snapshot path, not merely detectable) - polyfill bundles become ELIGIBLE for the optimization and serialize through pristine intrinsics identically on both paths (test flipped from gating to byte-equality) - process.env injection converted from guest-source eval to handle-based installProcessEnv (captured Object.freeze + vm.hostToHandle): the old evalCode ran JSON.parse post-eval on the restore path only, the same observable-divergence class - the reviewer's stateful-wrapper repro is a regression test: the counter must be zero and identical across fresh and restored invocations * Adopt quickjs-wasi 3.4.0: delete the NUL/surrogate string machinery 3.4.0 ships lossless string transport (vercel-labs/quickjs-wasi#35 — found by this PR's review cycle), so the SDK-side detection and escape machinery is deleted wholesale: - guestString (length + U+FFFD detection, in-VM escape fallback) — plain toString() is lossless now - the escapeString / hasOwnCall / jsonStringify / objectKeys captured intrinsics - keyNeedsHandleLookup and the handle-keyed get/hasOwn routing — the library routes inexpressible keys itself - shapeOf's guest-key verification pass (count/duplicate/U+FFFD scans) — enumeration is lossless Net ~130 lines and four captured intrinsics removed; the serde now uses the plain quickjs-wasi surface everywhere. Test honesty fix that 3.4.0 forced: the earlier lone-surrogate round-trip tests passed only via mutual corruption — the pre-3.4.0 lossy host→guest transport corrupted the guest comparison literals identically to the wire. With an honest transport they exposed that the WIRE itself (devalue emits lone surrogates raw; the wire is UTF-8) degrades lone surrogates to U+FFFD — in the node engine's reference codec exactly as here, verified. Bug-compatible parity is the load-bearing property (event logs replay across engines), so those tests now assert byte parity with the reference codec plus guest-observed equality with the reference codec's own round trip; NULs are devalue-escaped and asserted to survive exactly. Wire-level surrogate preservation is a product-wide devalue/UTF-8 question, tracked separately from this engine. * rerun CI

[world-vercel] Add /run-id sub-export with tagged ULID encode/decode (#1978) * [world-vercel] Add /run-id sub-export with tagged ULID encode/decode Encodes a tag bit, 5-bit version, and 6-bit Vercel region ID into a ULID-shaped string used for workflow run IDs. Tagged values remain valid 26-char Crockford-Base32 ULIDs so they still sort and round-trip through any system that accepts ULIDs. * [world-vercel] Add string-value assertions to run-id tests Add exact-string expectations for encoded outputs at known inputs, covering the default region/version pair, numeric region IDs, version overrides, boundary values (all-zero, all-max), the dirty-input overwrite case, and the lexicographic-order checks. Also adds an explicit byte-array expectation for the canonical ULID-spec example string and an additional first-char-range coverage test for isTagged. * [world-vercel] Remove internal-repo reference from regions doc comment * [world-vercel] Address PR review feedback on run-id sub-export - isTaggedString now fully validates the input as a 26-char Crockford Base32 ULID (delegating to ulidToBytes) instead of only inspecting the first character. This fixes false positives on inputs like '4UUUU...' that have a valid tag-bit position but invalid chars later in the string. - isTagged() now accepts `unknown` to match its documented behavior of safely rejecting non-string inputs without requiring callers to cast. - Introduce `RegionKey` for the full set of keys including 'unknown', and narrow `RegionCode` to `Exclude<RegionKey, 'unknown'>` so the return type of `lookupRegion` and the `DecodedRunId.region` field accurately reflect that 'unknown' is never produced. Updates `encode` to reject 'unknown' as a region code string at runtime (callers wanting the unknown sentinel should pass numeric 0). * [world-vercel] Move tagged-ULID metadata to the top of randomness Address review feedback on #1978: 1. **Metadata at top of randomness, not bottom.** Place `regionId` (6 bits) in the high bits of byte[6] and `version` (5 bits) straddling bytes 6 and 7, leaving the bottom 69 bits of randomness untouched by `encode`. This means a `monotonicFactory()`-style ULID generator's intra-millisecond bottom-bit increments survive encoding intact, so consecutive `encode(ulid(), region, { version })` calls with the same metadata produce strictly increasing strings. Previously the metadata sat in the bottom 11 bits — exactly the bits the monotonic factory uses — causing same-ms collisions/inversions. 2. **DecodedRunId is now a discriminated union.** When `tagged: false`, the `regionId`, `version`, and `region` fields are typed as `null` instead of being populated with garbage bits from arbitrary ULIDs. This forces callers to discriminate on `tagged` before reading metadata. 3. **regionIdFor: keep runtime backstop, mark as ignored for coverage.** The unreachable-in-TS branch stays as a defensive runtime check for callers crossing a JS/TS boundary; an istanbul/c8 ignore comment keeps coverage tools quiet. Doc strings and tests updated accordingly. The new layout adds a test verifying that a sequence of incrementing-bottom-bit ULIDs (simulating `monotonicFactory()`) round-trips through `encode` as a strictly increasing sequence. 108/108 world-vercel tests pass; typecheck clean.

QuickJS engine: inline step execution + WASM module caching (#3049) * Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay * QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed * QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols * QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity * Apply biome fixes to QuickJS engine files * Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard * CI: include generated QuickJS source assets in shared e2e build artifacts * Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine * CI: run both VM engines across all frameworks and worlds; label jobs with the engine * Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads * e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status) * e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely) * Sort imports in QuickJS serialization files (biome organizeImports) * QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal hook payloads to the target run's published X25519 public key. The shared start() path publishes that key regardless of engine, so QuickJS runs receive sealed payloads too — but the QuickJS entrypoint resolved only the bare symmetric key via importKey(), which cannot open encp envelopes. The first sealed hook payload wedged the run right after hook_received, timing out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the node engine resolves the full capability via memoizeEncryptionKey). Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric (encrypt() with RunPayloadKeys takes the encr path). Regression test seals a payload exactly as resumeHook does and round-trips it through the VM. * Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping - Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap, drawing from the seeded Math.random (identical sequences to the node engine's vm/index.ts implementations); all crypto.subtle methods throw with step-function guidance. process.env exposed as a frozen copy, matching node. - Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family methods (incl. localeCompare) throw when given an explicit locale so cross-engine divergence is loud instead of silently writing different values into the event log. No-argument forms keep working. - runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the ~1.3MB embedded WASM assets out of node-engine deployments. - runQuickJSWorkflow wraps the per-run phase so an exceptional exit disposes the VM instead of leaking it in a reused compute instance; corrected the misleading fail-loud comment (run_failed, not retry); warn when the event drain loop exhausts its iteration bound. - Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the file's workflow.* namespace. - Eval-string correlation-id interpolation uses JSON.stringify instead of quote-only escaping. - common-vm.test.ts pins the reducer/reviver superset invariant against common.ts so the duplicated sets can't silently drift. - Docs enumerate the remaining global-surface differences (subtle.digest, Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known precondition-guard gap. * QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup) #1834 made resumeHook() fall back to enqueueing the run with a hookInput payload when the direct hook_received write fails transiently, with the runtime materializing the missing event on delivery. Only the node:vm path implemented it — the QuickJS dispatch returned before the node block, so the resilient payload was silently dropped and the new e2e timed out on every quickjs leg. - runtime.ts threads hookInput into runWorkflowWithQuickJS; the entrypoint materializes the missing hook_received after loading the event log (resumeId-keyed dedup, occurredAt from the resumeId ULID, local eventData substitution for lazy/ref responses, EntityConflict / HookNotFound handling) — mirroring the node block. - processEvents drops duplicate hook_received rows sharing a resumeId (first-in-log wins), matching the node engine's EventsConsumer dedup; the seen-set lives in the VM heap so it is deterministic per replay. Verified against the dev server with WORKFLOW_VM=quickjs: the resilient resume e2e passes and the materialization is observable in the logs; all 27 hook e2e tests green. * QuickJS engine: inline step execution via live-VM continuation loop + WASM module caching * Address review: exclusive inline step claims, self-write requeue, in-loop event ceiling - Inline steps now claim via a lazy step_started carrying the input (step_created deferred, atomic create-claim in the world), with ownerMessageId stamped and authoritativeAttempt=1 — a concurrent invocation racing on the same fresh step loses with EntityConflictError and skips instead of both bare-starting the step and double-running the body. This also removes the stepsCreatedByUs set, whose 'created by us' invariant didn't survive the swallowed create-race conflict; redelivery backstops now key on hasCreatedEvent. - dispatchPendingOps' createdAttributeEvent/createdGetConflictHook signals are consumed again: when the loop exits suspended without ever reading back a self-written attr_set / getConflict hook_created (eventually-consistent listing lag), the entrypoint requeues immediately instead of parking the run awaiting_external with its unblocking event already written. - The server-supplied event ceiling is re-checked at the top of every continuation-loop turn (seenEventIds.size), so a single invocation fanning out inline can no longer grow the log arbitrarily past the operator's limit. The quickjs dispatch in runtime.ts converts MaxEventsExceededError into run_failed / MAX_EVENTS_EXCEEDED — the guard's throw previously nacked forever, parking runaway runs in 'running'. - Documented the deliberate decision that the platform function timeout is the only bound on inline chaining (budget parked per batch), matching the node engine. * Fix lost wait continuation for waits that elapse mid-iteration (sleepWinsRace flake) The pre-inline wait-continuation sweep skipped waits with resumeMs <= 0. A wait whose deadline falls between the iteration's elapsed-wait pass (which saw it as still pending and wrote nothing) and this sweep got NEITHER a wait_completed NOR a continuation — and the inline batch then blocked the invocation for the full step duration with no wake armed anywhere. For Promise.race(step, sleep) that silently hands the race to the step: the sleep's wait_completed is never written and the run completes with the wrong winner. The vulnerable window spans the iteration's dispatch + feed network round-trips, so on world-vercel a 1s sleep landed in it roughly half the time (the ~50% sleepWinsRaceWorkflow failure rate in the Vercel quickjs e2e legs), while world-local's sub-ms round-trips masked it locally. Match the node engine (Math.max(1000, resumeAtMs - now) in suspension-handler.ts): always arm the continuation, clamping already-elapsed waits to the 1s minimum — the continuation invocation's pre-VM elapsed check completes them. Waits whose wait_completed this invocation already wrote are skipped. Diagnosed from run wrun_41KZ73HR4H0GZ6RYD1WQHZX822 (CI run 30942512953): wait_created at +0.5s for a 1s sleep, no wait_completed ever, step_completed at +10.8s wins the race.

[core] Exclude inline step execution from replay timeout (#2013) * [core] Exclude inline step execution from replay timeout The v5 combined workflow+step handler wraps inline step bodies in the same setTimeout(..., REPLAY_TIMEOUT_MS) guard that previously only bounded the v4 'workflows' function's fast deterministic replay. As a result, any workflow with a single step exceeding 240s hard-fails with FatalError: Workflow replay exceeded maximum duration (240s) after 4 attempts — even though the step could legitimately run for the full function maxDuration (up to 800s on Pro Fluid). Replace the setTimeout guard with a per-invocation budget that only accumulates non-step time. pauseReplayBudget() / resumeReplayBudget() bracket each executeStep() call, and the loop checks the budget at iteration boundaries. The retry-then-fail semantics from #1567 are preserved verbatim for the pure-replay case. Also adds a WORKFLOW_REPLAY_TIMEOUT_MS env var override (clamped to 30s..780s) so operators can adjust the bare-replay ceiling without patching @workflow/core. Fixes #2009. * Address PR review feedback - Extract budget bookkeeping into ReplayBudget class (replay-budget.ts) with sentinel-protected idempotent pause()/resume() to avoid double-counting in future refactors that nest step execution - Restore VERCEL_URL gate around process.exit(1) so a long pure-replay in local dev/non-Vercel runtimes can't hard-kill the host process - Warn (once per distinct raw value) when WORKFLOW_REPLAY_TIMEOUT_MS is clamped or rejected, so misconfiguration is observable - Correct Hobby maxDuration comment (60s standard / 300s Fluid) - Document budget-check responsiveness trade-off vs. old setTimeout - Tighten describe-error test assertions to match the full new hint - Shorten changeset description - Add ReplayBudget unit tests (9) including 8-minute step regression - Add warn-once tests for getReplayTimeoutMs (extended) * Replace VERCEL_URL gate with World capability Per review feedback, gating runtime behavior on process.env.VERCEL_URL leaks deployment-environment concerns into @workflow/core. Replace the check with a new optional capability on the World interface: processExitTriggersQueueRedelivery (default false). - @workflow/world: declare the new optional capability on World - @workflow/world-vercel: set it to true (Vercel fails the function invocation on non-zero exit and VQS redelivers via fresh invocation) - @workflow/core: handleReplayBudgetExhausted reads world.processExitTriggersQueueRedelivery instead of process.env.VERCEL_URL; behavior is otherwise unchanged - Add 4 unit tests for handleReplayBudgetExhausted covering both branches (exit-for-redelivery and best-effort run_failed) --------- Co-authored-by: Peter Wielander <mittgfu@gmail.com>

cli: show world-specific run fields in inspect output via World.describeRun (#2896) * cli: show run region in inspect output via World.regionForRunId Adds an optional reverse-lookup hook to the World interface — regionForRunId(runId): string | null — so tooling can display a run's region generically. Worlds without a regional dimension simply omit the hook and no region output appears. - @workflow/world: new optional interface member (documented: must not throw; null = undeterminable) - @workflow/world-vercel: implements it from the run-ID region tag (tagged -> embedded region, untagged legacy -> default region, malformed -> null) - @workflow/cli: 'workflow inspect runs' gains a region column (between workflowName and status) and 'workflow inspect run <id>' a region property, in both table and JSON output — only when the world defines the hook * Generalize the inspect hook: World.describeRun display fields Replaces regionForRunId on the World interface with describeRun, per review: worlds may want to expose more than a region, and the information need not be encoded in the run ID — describeRun receives the run entity itself (loosely typed, mirroring createRunId), so a world can derive fields from executionContext or any other property. Each returned key becomes an inspect column/property; null values are preserved in structured output ('applicable but undeterminable' vs. the hook being absent entirely). - world-vercel: describeRun returns { region } decoded from the run ID tag (regionForRunId stays exported as a utility); entities without a usable runId contribute nothing - CLI listing: columns come from the union of keys the world returns for the page, inserted before status; both analytics and storage paths; detached call site binds this - CLI showRun: merges the world fields into detail/JSON output via a method-style call (preserves this), keeping nulls - tests: field merging (multi-key), null preservation in JSON, hook absent, and world-vercel describeRun coverage incl. no-runId entities * cli: evaluate describeRun defensively Per review: the World interface says describeRun is pure and must not throw, but it is an external extension point and the CLI should not trust that. New safeWorldFields helper, used by both the listing and showRun paths: - a throwing implementation contributes no fields instead of crashing the inspect command - keys that already exist on the run row are dropped, so a world can never overwrite canonical fields (status, runId, ...) in output Tests: canonical fields survive a clobbering describeRun (extra keys still merged); a throwing describeRun leaves rows untouched and the command succeeds. * Allow async describeRun implementations Per review: widening a sync signature to async later would break every consumer, while accepting sync-or-async from day one is free — sync implementations (like world-vercel's) remain valid, and consumers simply await, which handles both. The performance intent lives on as documented guidance: the hook is called once per displayed run, so implementations should stay cheap and avoid I/O; the CLI evaluates a page's rows concurrently so an async world costs one await per page, not per row. Promise rejections get the same treatment as throws: no fields, never a crash. * Update packages/world/src/interfaces.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Nathan Rajlich <n@n8.io> --------- Signed-off-by: Nathan Rajlich <n@n8.io> Co-authored-by: Peter Wielander <mittgfu@gmail.com>

feat: return the run public key from the capability probe (#3099) * feat: return the run public key from the capability probe This removes the last key-lookup request from the cross-deployment hot paths. `start()` already blocks on a capability probe for every cross-deployment call, and the probe responder executes *inside the target deployment*, where the run's key material is available locally. So the public key can ride back on a response the caller is already awaiting, at no additional latency, and the `run-key` API request disappears. Three properties make this better than keeping the request: - The wait is already being paid. Folding the key into the existing response removes the request outright rather than relocating it. - A public key is exactly what this channel can carry. The probe response stream is deliberately unauthenticated, which would disqualify shipping the symmetric key over it — but a public key is not secret. - It reduces privilege. The caller ends up able to seal the workflow arguments but not read them back; fetching the symmetric key granted full read access to a run it merely launched. `runId` is now minted before the probe rather than just after it. `createRunId()` reads only `opts`, which is fully resolved by that point, so the move has no other dependency — and a test asserts the id sent to the probe is the one actually created. Everything is best-effort. The probe is already failure-tolerant (2s timeout, errors swallowed) and is skipped entirely for same-deployment starts and for worlds without a streams API. When no key comes back — old target, timeout, encryption disabled, or a malformed value — `start()` falls back to the existing lookup plus symmetric encryption. Key derivation failures inside the responder are caught and logged so the probe still reports health and capabilities, which callers depend on for reasons unrelated to encryption. * fix: keep the health-check discriminator on runId-bearing probes `QueuePayloadSchema` is an ordered union and `z.object` strips keys the matching member doesn't declare. Adding an optional `runId` to `HealthCheckPayloadSchema` made a probe payload also satisfy `WorkflowInvokePayloadSchema`, whose only required field is `runId`. Because the invoke member came first, world-vercel's queue handler parsed a runId-bearing probe down to `{ runId }`, dropping `__healthCheck` and `correlationId`. The runtime dispatches on `__healthCheck` before falling through to the invoke schema, so the probe was reinterpreted as "replay this run": it POSTed `run_started` for a run that does not exist yet, 404'd, failed the handler, and retried indefinitely. The probe never answered and the cross-deployment `start()` timed out — which also regressed the pre-existing capability detection, not just the new key lookup. Order the health-check member first; it requires `__healthCheck: true`, which no invoke or step payload carries, so invoke and step payloads still resolve to their own members. Also reorder `getPhysicalQueueName` to match health checks before the runId branch, so under `WORKFLOW_SEQUENTIAL_REPLAYS=1` a probe keeps its per-probe topic instead of queueing behind the run it is preparing.

Fix Next workflow module specifier root (#2455)