Karthik Kalyan

Lazy hook resumption: parallel event write + queue publish (#3230) * feat(core): lazy hook resumption via parallel event write + queue publish (rebased onto #1834 + #3145) Rebase of #3230 onto current main (267765375 + #1834 resilient resumeHook + #3145 event-count-gated replay restart). Reconstructed as a single commit since `git rebase -i` is unavailable in this environment. Reconciliation vs the pre-rebase branch: - Replaces #1834's version-prediction (`supportsQueueHookInput`, `QUEUE_HOOK_INPUT_MIN_VERSION`) with #3230's capability protocol (persisted `hookResumeInputVersion` + static `hookResumeDedup`). - One idempotency protocol: a single `resumeId` + SHA-256 payload digest per resume, sent to both the direct event write and the queue `hookInput`. - Two execution tiers: backend+consumer attest dedup -> parallel `Promise.allSettled(event write, queue publish)`; otherwise plain sequential (no hookInput/resumeId, event-write errors propagate). - Consumer re-ensures the `hook_received` event (keyed by resumeId/digest) after event loading, before replay; skips when already preloaded. - Preserves #3145: event-count guard, `preconditionReinvocations`, in-process replay restart, `insertEventByEventId`. - Removes #1834's resumeId-only test (never released); adds parallel + consumer-preload + world-local dedup/producer-consumer suites. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(core): read top-level event.resumeId in replay dedup; reconcile unreleased #1834 docs/changeset - hook.ts: dedup hook_received replay on top-level event.resumeId (the backend now hoists it to a first-class column), with the legacy nested eventData.resumeId retained as a deprecated parse-only fallback. - workflow.test.ts: cover dedup across both top-level and legacy nested forms. - resume-hook.ts: emit producer recovery telemetry when a transient event-write failure is swallowed on the parallel path. - resume-hook.consumer-preload.test.ts: add terminal-run (consume) and transient-conflict (rethrow/redeliver) re-ensure cases. - Consolidate the two overlapping changesets into resilient-resume-hook.md and delete the redundant lazy-hook-resumption.md. - Docs: return type back to Promise<Hook> (resume-hook.mdx), rewrite the resilience changelog to the final parallel/deduplicated design, and correct the WORKFLOW_DISABLE_LAZY_HOOK_RESUME resilience wording. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs,core: rename "Resilient hook resume" → "Lazy hook resume" for consistency - changelog/index.mdx: update the changelog entry title. - hook.ts: update the dedup comment label to "Lazy-resume dedup". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: give #3230 its own changeset instead of repurposing #1834's The lazy-hook-resume work had been folded into #1834's pre-existing `resilient-resume-hook.md` changeset. Give this PR its own changeset and delete the superseded #1834 one, whose `resilientResume: true` flag promise no longer holds (resumeHook() returns plain Promise<Hook>). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: add #3230's own lazy-hook-resumption changeset Follow-up to 63d877178, which deleted #1834's superseded changeset but did not stage the replacement. Adds this PR's own changeset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: tighten lazy-hook-resumption changeset Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: leave #1834's resilient-resume changeset/changelog/docs untouched Restore #1834's own artifacts that #3230 had rewritten: - .changeset/resilient-resume-hook.md (restored verbatim) - docs/.../changelog/resilient-resume.mdx (restored verbatim) - docs/.../changelog/index.mdx (restored verbatim) #3230 keeps only its own changeset plus the two docs its code/config genuinely require: the resumeHook() Promise<Hook> return type (ResumedHook is removed from the code) and the new WORKFLOW_DISABLE_LAZY_HOOK_RESUME env var. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Reconcile #1834 ResumedHook contract with #3230 parallel resume Preserve the resilient-resume contract from #1834 on the parallelized resumeHook() fast path instead of dropping it: - Restore the `ResumedHook` type (Hook + optional `resilientResume`) and its exports (`@workflow/core/runtime`, `workflow/api`); resumeHook/resumeHookImpl return `Promise<ResumedHook>`. - Set `resilientResume: true` on the swallow-recover branch (transient direct write failure + successful queue dispatch), absent on the happy/sequential paths. - Restore the producer OTEL convention `workflow.hook.resilient_resume` and the consumer `workflow.hook.resilient_resume_materialized`, wired where the consumer re-ensures the event. - Restore the consumer `occurredAt` derivation from the resume ULID so the materialized hook_received is dated to resume time, not queue-round-trip time. - Fix the #3230 changeset's contradictory "Still returns Promise<Hook>" line and update the resilient-resume changelog + resume-hook API reference to the shipped parallel/dedup behavior. - Port the #1834 failure-path coverage into resume-hook.parallel.test.ts (non-retryable event-write rethrow, both-fail prioritizes the queue error, resilientResume flag + payload delivery on the recovered path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: drop dead nested resumeId fallback, remove server PR link - Drop the legacy nested `eventData.resumeId` fallback in the hook consumer. The nested form was only ever written by unreleased preview builds and is stripped by `EventSchema` parsing (the `hook_received` eventData schema does not declare it), so the fallback was dead code. Dedup now keys solely off the top-level `event.resumeId` column. Repoint the replay dedup test to the surviving top-level path (it previously exercised the nested form only by building unparsed Event objects in memory). - Remove the internal workflow-server PR reference from world-vercel's capability note (the link 404s outside the org); the note keeps the same information without the dead link. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Fix compressed workflow error display (#2680) * Fix compressed workflow data display * Add OSS web no-key hydration regression * Scope compression normalization to read paths; tidy hydration Address review feedback on the compressed-data fix: - world-vercel: keep gzip/zstd decompression on the o11y/display read paths (getStep/getRun/getEvent/getWorkflowRunEvents/getHook) but not on the runtime event-append path (world.events.create, createStep, updateStep). That path is runtime-only and re-hydrates every payload via the decompress-aware helpers, so decompressing at the adapter was redundant work on the TTFB-sensitive run_started/inline-delta path and skewed the runtime's deserialize compression telemetry to `codec: none`. deserializeStep is now shape-only; normalizeStepData runs in the read filter. Adds a regression test pinning the write-path pass-through. - serialized-data: drop dead `errorRef`/`metadataRef` normalization (refs are descriptor objects, never compressed byte payloads). - web: in the wait-entity path, filter events by correlationId before hydrating so an encryption key doesn't decrypt the whole event page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Peter Wielander <peter.wielander@vercel.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

perf(core): path-independent stream write batching (group commit in the server writable) (#3078) * perf(core): move stream write batching into WorkflowServerWritableStream (group commit) Batching previously lived in flushablePipe's coalescing loop, so it only engaged on paths that used flushablePipe (getWritable). A raw ReadableStream crossing a workflow/step boundary is piped with native pipeTo(), which does not pull chunk N+1 until write(chunk N) resolves — and write() resolved only after the flush timer AND the server round trip, so the buffer never held more than one chunk and every token became its own server request. The sink now group-commits: - write() resolves when the chunk enters a bounded client buffer; the bound counts buffered AND in-request chunks (WORKFLOW_STREAM_MAX_INFLIGHT_CHUNKS, preserving its documented meaning) plus a byte bound (WORKFLOW_STREAM_MAX_BUFFERED_BYTES, new, default 8 MiB, documented in runtime-tuning). A full buffer applies backpressure until a group lands durably. - The flush interval is a real group-commit window; chunks arriving while a request is in flight accumulate and form the next writeMulti group. One request in flight at a time preserves chunk order. - Per-request wire limits (1,000 chunks / 1 MiB) split groups exactly as the coalescing pipe did; an oversized single chunk goes alone. - Durability moved to an explicit barrier (STREAM_DRAIN_SYMBOL): close() drains before closing; flushablePipe adopts the barrier so lock-release completion (step completion) still means 'everything written is durable'; abort() DRAINS the accepted prefix (never closing) so a producer error after acked writes cannot lose data — native pipeTo aborts the sink on source failure; and a failed pipe drains before settling so a step failure is not persisted ahead of the emitted prefix. A dispatch failure retains the group, poisons the sink, and surfaces at the next write/close/drain. flushablePipe is now a plain per-chunk pump responsible only for lock-release completion and durability tracking; its coalescing machinery and STREAM_WRITE_BATCH_SYMBOL are removed. Covered: native-pipeTo batching (the regression), awaited per-chunk loops coalescing into one writeMulti, in-flight accumulation, wire-cap splits (count/byte/oversized), in-flight-inclusive backpressure for both bounds, sequential fallback without writeMulti, source-error prefix delivery through abort, failed-pipe drain-before-reject, early-ack sticky errors, turbo run-ready barrier gating (incl. dwell telemetry), drain-barrier adoption/rejection, and group-level flush spans. 1,572 core unit tests pass; e2e tier requires a deployment and was not run here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(core): re-dispatch a chunk buffered in the request settle gap Review (bot): a write landing between the dispatch loop's empty-buffer exit and the reaction clearing the in-flight marker armed no timer (scheduleGroupCommit saw the marker set) and was never dispatched on an open stream — only a later write/close/drain would pick it up. The settle reaction now re-dispatches when the buffer is non-empty, treating the chunk as an in-request arrival; drain waiters settle with the new chain. Regression test aims a write at the settle gap and asserts both chunks flush without a close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(core): document abort-drain boundedness and terminal-run conflict handling Review note: the abort-path drain is deliberately un-timeboxed (a bound would drop acked chunks); its worst case is owned by the World transport's finite timeout/retry budget, and a teardown-driven drain into an already-terminal run rejects into the existing catch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(core): poll instead of fixed sleeps for dispatch assertions The native-pipeTo batching test flaked on a slow CI runner: a fixed 25ms wait raced the 10ms commit window plus scheduler jitter. All 'dispatch has happened' assertions now poll the expectation (bounded); intentional negatives keep their fixed windows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

perf(core): initialize lazy hook replay from hook_received stream (#3345) * perf(core): initialize lazy hook replay from hook_received stream On a lazy hook queue delivery, the consumer's idempotent hook_received re-ensure is hoisted above run_started and doubles as the invocation's setup request: it asks the World to return the current replay log with the write (new advisory CreateEventParams.preloadEvents), so one HTTP request yields the canonical event, the reconstructed run, and the complete replay log — skipping both the run_started POST and the initial events.list. - world: optional `preloadEvents?: true` on CreateEventParams, the hook_received dual of skipPreload; Worlds may ignore it - world-vercel: createHookReceivedPreloadEventV4 sends the frame Accept on eligible hook_received posts and decodes either response mode — frames via the response decoder extracted from the LIST consumer (GET behavior unchanged), CBOR via the shared materialized-response mapping. The run is reconstructed from run_created/run_started (plus attr_set folds), the canonical event found by x-wf-event-id, and resumeId now survives frame decoding so the runtime can match it - core: new fast path before the generic run-state setup, guarded on hookInput.resumeId + payloadDigest; a validated COMPLETE preload (hasMore false — this path has no cursor-continuation machinery) initializes workflowRun/preloadedEvents/maxEventsLimit directly, anything else falls back to the run_started setup without re-posting the hook; error classification matches the existing re-ensure (terminal → consume, transient → redeliver); setup source reported via workflow.resume_setup_source (never workflow.hook.resilient_resume_materialized, which stays a recovery-only signal) - producer resumeHook() is unchanged and never sets preloadEvents Based directly on main (no dependency on #3124/#3191); pairs with workflow-server's streamed hook_received replay-log response, which deploys first — the SDK negotiates per request and falls back safely against older servers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * address review: lazy fallback, retryable resume, terminal telemetry - world-vercel: the preload request keeps hook_received's lazy remoteRefBehavior — a supporting server owns frame-body resolution, while an older server now answers the CBOR fallback without resolving an S3-backed payload the runtime would discard - world-vercel: the atomic lazy-resume shape (resumeId + digest) opts into withEventPostRetry via idempotentHookResume — the (runId, resumeId) claim makes the POST idempotent-on-retry; legacy/partial hook_received shapes stay single-attempt, definitive 4xx stays non-retryable (unit + adapter + trace-propagation coverage) - core: a terminal event found in the preload records workflow.resume_setup_source=hook_received_stream and the run's actual terminal status on the span before consuming the delivery - core: document resilient_resume_materialized as the legacy/non-atomic re-ensure signal (claim ownership is not observable client-side, so the hoisted path deliberately never emits it) and resume_setup_source as a latency signal, not proof of event creation; note the Option A skip is now unreachable for atomic resumes - world: spell out the full preload usability contract on preloadEvents (complete hasMore-false log, non-null cursor, run/startedAt/maxEvents, lifecycle events, matching resumeId, list ordering, read-after-write consistency); bump @workflow/world to minor - new QuickJS sourcing tests (VM mocked): an attested complete preload is used verbatim with no events.list, a non-attested hook-containing preload is refetched, and an attested empty preload is not trusted Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

[world][web][cli] o11y: window-aware runs listing (#2812) * web: infinite scroll for the runs table Replace Previous/Next cursor paging with front-style infinite scroll: a useInfiniteList hook accumulates cursor pages with per-run dedup and generation-guarded resets, and useLoadMoreOnScroll drives loadMore from an IntersectionObserver sentinel (400px prefetch margin, guarded against double-fetch, observed against the table's scroll container). Footer now shows the loaded count and the analytics lookback window. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * web: back the runs infinite list with SWR so tab switches serve from cache Rewrite useInfiniteList on useSWRInfinite: pages are keyed by [cacheKey, cursor] in SWR's global cache, so unmount/remount (switching tabs) restores fetched pages instantly instead of refetching. Revalidation is conservative because analytics list queries are expensive: revalidateFirstPage and revalidateIfStale are off; freshness comes from the Refresh button and the visibility-change auto-reload, which map to reload() (reset to first page + revalidate). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * world: expose startTime/endTime on analytics runs listing The workflow-server /v2/analytics/runs endpoint has accepted a bounded startTime/endTime window since it shipped, and is significantly faster with one (the window prunes the ClickHouse scan: ~2s for 12h vs ~8s for the default 30-day entitlement window). The world client never exposed the params, so the CLI and web UI could only issue windowless requests. Pass them through so clients can send bounded windows (e.g. a period picker like front's workflows o11y). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * web: front-style period picker for the runs list Add a time-window picker (1h/6h/24h/3d/7d/30d, default 24h, URL-backed via ?period=) that sends an explicit startTime/endTime window through fetchRuns -> world.analytics.runs.list, keeping the ClickHouse scan bounded. The window is frozen per selection/refresh so all cursor pages share the same bounds, and it participates in the SWR cache key. Plan tiers are honored data-driven from the server's pageInfo: presets longer than the plan's observability lookback are disabled in the picker (labeled Observability Plus when an upgrade is available), and a 402 observability-upgrade-required response renders through the existing upgrade-required error handling. The footer now labels the selected window instead of the plan lookback. The runtime (local) fallback path ignores the window since the storage API has no time filter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * web: allow status filtering without a workflow name filter The status dropdown was disabled on Vercel backends until a workflow was selected — a limitation of the runtime DynamoDB API's index design. The runs list now reads via world.analytics, whose ClickHouse query filters derived status independently of workflowName, so drop the guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * cli: time-window flags for runs listing; widen name lookups past the default window The analytics backend now defaults windowless runs listings to the trailing 24h. Replicate the web's window support in the CLI: - 'workflow inspect runs' gains --since/--until (relative durations like 30m/12h/7d/2w, or timestamps) which are sent as an explicit startTime/endTime window. Out-of-plan windows surface through the existing observability-upgrade-required handling; non-analytics backends warn that the flags are ignored. - 'workflow start <name>' resolves the workflow's latest run via a windowless (default-window) listing and now retries across the plan's whole observability window on a miss, so names idle for more than a day keep resolving. - Bulk 'workflow cancel' matches across the plan window up front — a run can sleep or wait on a hook for days without recent events, so the default recent window must not bound cancellation matching. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: tighten changeset descriptions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * web: persist frozen listing windows across remounts; minimize lockfile diff Address review findings: - The frozen startTime/endTime lived in component state, but RunsTable fully remounts on tab switches, so every remount minted a new SWR cache key — the cached-pages restore never hit and cache entries grew unboundedly (one per key, including every 5s local-backend poll tick). Move the frozen windows to a module-scope store keyed by period: a remount reuses the stored window (same cache key, instant restore), and the window only advances on explicit refresh/reload. Non-analytics backends now send no window at all (the runtime APIs ignore it anyway), which also hides the period picker and window label there. - Regenerate pnpm-lock.yaml from main so the diff contains only the swr addition (plus its own use-sync-external-store dependency), dropping the unrelated docs-importer radix-ui re-resolutions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

otel: explicit traceparent injection + linked-trace mode for bounded per-invocation traces (#2363) * otel: explicit traceparent injection + linked-trace mode for bounded per-invocation traces - Add WORKFLOW_TRACE_MODE ('linked' default, 'continuous' legacy) to the workflow and step queue handlers. In linked mode, WORKFLOW_V2/STEP spans start a new trace root with span links to the incoming delivery context and the run-origin context, and re-enqueued messages forward the ORIGINAL run-origin trace carrier unchanged. - world-vercel now explicitly injects W3C traceparent/tracestate/baggage headers on outgoing workflow-server HTTP requests from inside the client span (no-op without an OTEL SDK registered). - New workflow.trace.mode span attribute; unit tests for both modes and for header injection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * changeset: call out behavioral telemetry changes of the linked default Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add v5 observability tracing page Documents OTEL spans/attributes, linked trace mode and WORKFLOW_TRACE_MODE, span links, context propagation, and the v4 behavior-change callout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * otel: human-friendly span names for workflow and step spans WORKFLOW_V2/STEP prefixes with full machine names (workflow//./src/...//fn) become workflow.execute / step.execute / workflow.start with the short function name. New workflowDisplayName/stepDisplayName helpers in @workflow/utils handle both raw and queue-sanitized name forms; full names remain in the workflow.name/step.name attributes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * changeset: merge span-name and linked-trace notes into one changeset Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update trace-shape prose to renamed span names Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: replace ascii trace diagram with mermaid Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * address review: empty carriers, shared trace helpers, mode warning, name edge cases, consumer span kind - Treat an empty ({}) trace carrier as absent everywhere the trace-mode logic branches, so linked mode falls back to a fresh origin instead of forwarding a useless {} forever; workflow.trace.propagated now reports whether a usable carrier arrived. - Extract the duplicated linked-mode logic into shared telemetry helpers getNextTraceCarrier() and buildInvocationSpanLinks(), used by both the workflow and step queue handlers; resume-hook now uses linkToTraceCarrier (gaining the isSpanContextValid guard). - Warn once per distinct unrecognized WORKFLOW_TRACE_MODE value instead of silently selecting linked. - shortNameFromSanitized: map default/__default to the module short name (mirroring parseName) and document the `$`-sanitization limitation. - Queue-delivered workflow.execute spans now use the CONSUMER span kind, matching queue-delivered step.execute spans; docs span table and changeset updated accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

cli: read list views from world.analytics when available (#2648) * Add workflow analytics world APIs * cli: read list views from world.analytics when available inspect list views (runs, steps, events, hooks, sleeps) now read from the optional world.analytics namespace when the active backend provides one, falling back to the runtime storage APIs otherwise. Payload and detail views are unchanged. Deprecate --with-data for list views; payloads are viewable per-resource via 'inspect <resource> <id>'. * cli: keep hook listing on the runtime storage API The analytics read path omits ownerId (and the secret hook token), so routing hook listing through it silently drops the ownerId column. Keep inspect hooks on the runtime APIs, consistent with the web observability UI. Runs, steps, events, and sleeps continue to use the analytics read path when available. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Handle analytics access metadata in CLI * test(cli): preserve analytics pageInfo in json output * fix(cli): paginate analytics sleeps output * fix(cli): correct deprecation message flag name to --withData The list-view deprecation warning referenced '--with-data', but the actual oclif flag is '--withData' (with '-d' alias); '--with-data' errors with "Nonexistent flag". Fix the warning text, the doc comment, and the changeset to reference the real flag name. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): preserve inspect json array output * fix(cli): fall back when analytics lists are empty --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Use and show event occurredAt (#2613)

telemetry: move client stream spans from world-vercel to core (#2901) * fix(deps): dedupe @opentelemetry/api to a single workspace instance The lockfile resolved both 1.9.0 and 1.9.1, so the copy that registers the tracer provider (via @vercel/otel in the app) and the copy a package imports could differ. The API's global-registration version check rejects a consumer newer than the registered copy and silently hands back a noop tracer — which is why world-vercel's spans (workflow.stream.write/ chunk_rtt, read.connect, its http spans) never reached Datadog from deployed apps while core's spans flowed in the same process. Root-caused via the DEBUG=workflow:* run on #2900: import succeeds, no warn, spans dropped. Pin a single version via a workspace override so every bundle shares one API instance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * debug: one-shot OTEL runtime diagnostic in core + world-vercel; DEBUG on turbopack workbench The dedupe alone did not restore world-vercel span emission (verified on this PR's own preview: stream traffic flowed, zero workflow.stream.write spans). Under DEBUG=workflow:*, both packages now log once how their module instance of @opentelemetry/api sees the world — global registration version, provider/delegate/tracer/probe constructor names, and whether a probe span is recording. Diffing the core line (spans work) against the world-vercel line (spans dropped) in one deployment's logs pinpoints the divergence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * debug: log span identity for named world-vercel spans; namespace otel probes per package Diag round 1 showed world-vercel's tracer records and instrumentedFetch handles the stream PUTs, yet the named spans are unfindable in the backend. Round 2: log traceId/spanId/isRecording for every named instrumentedFetch span under DEBUG so export can be checked for a specific span id, and split the probe span names (.core / .world_vercel) so per-package export is attributable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * telemetry: emit stream RPC latencies from core (chunk_rtt, connect_ms, close span) world-vercel's instrumentedFetch spans never export from deployed apps (root cause still open — see PR discussion), so the operationally needed client-side latency signals move one layer up to core, whose spans are proven to export: - workflow.stream.write.chunk_rtt on the workflow.stream.flush span: the World write RPC duration, network included (same attribute key as world-vercel's per-request span so queries are layer-agnostic). - workflow.stream.read.connect_ms on the workflow.stream.read span: the world.streams.get await (read dispatch -> stream handle). - new workflow.stream.close span: the close RPC round trip. Bonus: measured at the World interface, these cover world-local and world-postgres too, not just Vercel deployments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * telemetry: emit read-completion span (total duration, chunks, bytes) Completes the read-side picture: workflow.stream.read.complete is back-dated to the read dispatch so its duration is the total read, with chunk/byte counts for throughput. Cancelled reads emit nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: drop DEBUG from turbopack workbench; tighten changeset Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * telemetry: cover createReconnectingFramedStream in read telemetry Ordinary serialized streams read through createReconnectingFramedStream (which calls world.streams.get directly), so connect_ms / ttfc / read.complete never fired for that path — only WorkflowServerReadableStream was instrumented. Wire the same helpers into the framed reader: first- connect duration, first-frame TTFC, and completion totals — plus workflow.stream.read.reconnects, which only this path can know. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

web: read observability list views from world.analytics when available (#2647) * Add workflow analytics world APIs * web: read observability list views from world.analytics when available Route the runs/steps/events/hooks list server actions through the optional world.analytics namespace when the backend provides one, falling back to the runtime storage APIs otherwise. Events listing only uses the analytics path when no payload data is requested. Detail/get actions, streams, and mutations are unchanged. * web: keep events and hooks list reads on the runtime storage API The Events tab and trace viewer derive step names and wait resumeAt from resolved event payloads, and the hooks table needs the secret token and ownerId for its resume/copy-token actions. The metadata-only analytics rows do not carry these, so only the runs and steps list views use world.analytics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * web: read events list from world.analytics with a runtime-shape remap The events list/trace consumers read only top-level eventType, correlationId, and createdAt; event payloads are loaded lazily per event via fetchEvent(..., 'all') on the runtime path. Map the flat analytics event rows into the runtime Event shape (reconstructing eventData.stepName) so fetchEvents and fetchEventsByCorrelationId can use the analytics read path when available. Hooks remain on the runtime path (they need the secret token + ownerId). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Handle analytics access metadata in web --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

feat(cli): bulk-cancel runs in a single operation (#3348) * feat(cli): bulk-cancel runs in a single operation Replace the per-run cancel loop in `workflow cancel` with one `cancelRuns` call, validate `--limit` (1-500), print a compact outcome summary with per-run lines for surfaced failures, and exit nonzero only when a run fails. The bulk logic lives in a dependency-injected `performBulkCancel` helper so it is unit-testable without an oclif harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cli): address bulk cancel review feedback --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

feat(world,world-vercel,core): bulk run cancellation primitive (#3347) * feat(world,world-vercel,core): bulk run cancellation primitive Add a bulk cancellation contract to @workflow/world (schemas, types, and an optional Storage['runs'].cancelMany method), implement it in @workflow/world-vercel via a single POST /v4/runs/cancel request, and add a cancelRuns runtime helper to @workflow/core that uses the world fast path when available and otherwise falls back to bounded-concurrency (max 20) single-run cancellation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Update packages/world/src/interfaces.ts Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> --------- Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com>

feat(core): resume hooks from stored resumeContext and seal to the run key (#3125) Hooks can carry an optional `resumeContext` mirrored from the run at creation time. When present, `resumeHook`/`resumeWebhook` resume directly from it instead of fetching the full run, saving a round trip per resume. When the context also carries the run's `encryptionPublicKey`, the resume seals its payload (`encp`) directly to that key. Combined with the sealed envelope work (#3093-#3096), a default webhook resume then needs neither a run read nor a cross-deployment run-key lookup: the key is resolved only when the hook actually stores metadata that must be hydrated symmetrically. Everything falls back transparently to the full run fetch and symmetric key when the context (or the public key within it) is absent, so new clients interoperate with old servers and vice versa. - world: optional `encryptionPublicKey` on `HookResumeContext` - world-postgres: `resume_context` column migration - core: combined fast-path + seal in resume-hook; fast-path control-flow suite split from the real-serialization crypto suite - world-vercel: cover the `getEncryptionKeyForRun(runId, { deploymentId })` overload the fast path relies on - web-shared: render `resumeContext` in the attribute panel Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Stamp run IDs on world spans (#2508) * Stamp run IDs on world spans * Apply suggestions from code review Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> --------- Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com>

[core] Thread queue namespace through o11y run actions and bound healthCheck stream reads (#2874) * [core] Thread queue namespace through o11y run actions and bound healthCheck stream reads Deployments that use a queue namespace (e.g. eve's __eve_wkf_workflow_* topics) could not be targeted by cross-context callers: start(), recreateRunFromExisting(), reenqueueRun(), and wakeUpRun() always built queue names from the caller's WORKFLOW_QUEUE_NAMESPACE env, so dashboard replays published to topics the target deployment has no consumer for. Additionally, healthCheck()'s poll loop only checked its timeout between iterations while world.streams.get() itself was unbounded — against workflow-server, which holds unwritten streams open for ~2 minutes, a 2s capability probe hung until the caller's function timed out (the observed 30s 504s on dashboard replay). - Add a `namespace` option to StartOptionsBase, RecreateRunOptions, StopSleepOptions, and new ReenqueueRunOptions; thread it into getWorkflowQueueName() and the cross-deployment capability probe. - Fold the inline `namespace` param into HealthCheckOptions. - Race streams.get() against the remaining health-check budget. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Shorten changeset Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

feat(world-vercel): client-observed stream write/read e2e latency spans (#2857) * feat(world-vercel): name stream client spans + add stream attributes Stream write/read requests already share the instrumented HTTP envelope (a CLIENT span + W3C trace-context injection), but the spans were named for the bare HTTP verb (`http PUT`/`http GET`) and carried only generic HTTP attributes — so stream latency couldn't be sliced per run/stream. Name these spans for their operation (`workflow.stream.write` / `workflow.stream.read`) and tag them with `workflow.run.id`, `workflow.stream.name`, `workflow.stream.operation` (write | write_multi | close | read), and `workflow.stream.start_index` (read). Implemented via new optional `spanName`/`attributes` fields on `instrumentedFetch`, so other callers are unaffected. Additive OTEL only: no behavior change when no OpenTelemetry SDK is registered (the span is undefined and attributes are dropped). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Shorten changeset summary Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add client-observed end-to-end read TTFC span The read GET can't report a client-measured latency back to the server (the value only exists after the response starts streaming), so capture it purely in the SDK's own OTEL: watch response.body for the first non-empty chunk and emit a workflow.stream.read span back-dated to read dispatch, whose duration is the end-to-end time-to-first-chunk (incl. the network hop) via workflow.stream.read.ttfc_ms. Rename the fetch/connect span to workflow.stream.read.connect. No-op without an OTEL SDK registered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add client-observed e2e write latency attribute The write PUT is request/response and the server acks only after capturing the chunk, so the workflow.stream.write span duration already equals the client->server write latency. Expose it as a named attribute workflow.stream.write.e2e_ms (via a durationAttribute option on instrumentedFetch) for direct querying, parallel to the read ttfc_ms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: document stream spans and latency attributes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Emit read TTFC span from the core reader instead of a world-vercel transform Per review: move the client-observed time-to-first-chunk measurement out of a TransformStream wrapper in world-vercel and into WorkflowServerReadableStream in core, emitting workflow.stream.read on the first non-empty chunk reaching the consumer. Removes the passthrough, measures at the reader abstraction, and is backend-agnostic. world-vercel keeps the workflow.stream.read.connect HTTP span; the recordElapsedSpan helper now lives in @workflow/core. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Apply suggestion from @VaguelySerious Signed-off-by: Peter Wielander <mittgfu@gmail.com> * Update docs/content/docs/v5/observability/tracing.mdx Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> * Address review: rename write.e2e_ms -> write.chunk_rtt; docs + changeset wording Per review, rename the write attribute to workflow.stream.write.chunk_rtt (it's a per-chunk client<->server round-trip, not a full e2e), update the docs row wording for both write and read attributes, and shorten the changeset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Signed-off-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com>

[web-shared][web] Fix events tab search (#2107) * Add server-backed exact ID search to the Events tab. Replace client-side substring filtering with API lookups for full correlation and event IDs so searches work beyond the first loaded page. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix exact ID search dimming and support wrun_ correlation IDs. Disable group dimming for server search results and accept run IDs in the exact ID parser so run-level correlation search works. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix dimmed row when searching by event ID for run-level events. Map selectedGroupKey to __run__ for run-level search results so the matched row is treated as related instead of dimmed. Co-authored-by: Cursor <cursoragent@cursor.com> * Remove run ID search from Events tab exact ID lookup. Workflow-server only accepts step, wait, and hook correlation IDs — not wrun_. Update the search placeholder and validation toast accordingly. Co-authored-by: Cursor <cursoragent@cursor.com> * Harden exact ID search UX and correlation fetch limits. Normalize lowercase ULIDs, scope Enter toasts to ID-like input, abort stale searches, disable search when unavailable, expand parser tests, and cap correlation pagination in workflow web. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix search clear race and surface truncated correlation results. Guard successful exact-ID search against aborted requests, invalidate in-flight work when the input clears, and return truncation metadata from correlation pagination. Co-authored-by: Cursor <cursoragent@cursor.com> * Differentiate exact ID search errors from not-found results. Return a discriminated union from onExactIdSearch and show search errors in the Events tab instead of mislabeling them as missing IDs. Co-authored-by: Cursor <cursoragent@cursor.com> * Apply suggestion from @VaguelySerious Signed-off-by: Peter Wielander <mittgfu@gmail.com> --------- Signed-off-by: Peter Wielander <mittgfu@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Peter Wielander <mittgfu@gmail.com>

[web-shared] Fix message-only error rendering (#2251) * Fix message-only error rendering * Handle non-string error stacks

feat(web): bulk-cancel selected runs from the runs table (#3349) * feat(cli): bulk-cancel runs in a single operation Replace the per-run cancel loop in `workflow cancel` with one `cancelRuns` call, validate `--limit` (1-500), print a compact outcome summary with per-run lines for surfaced failures, and exit nonzero only when a run fails. The bulk logic lives in a dependency-injected `performBulkCancel` helper so it is unit-testable without an oclif harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cli): address bulk cancel review feedback * feat(web): bulk-cancel selected runs in a single request Thread a bulkCancelRuns action through the server action, RPC route, rpc-client, and client wrappers, backed by core's cancelRuns. The runs table now cancels the selected pending/running runs in one call, caps a batch at BULK_CANCEL_MAX_RUN_IDS (disabling the button with guidance above the cap), and reports a single outcome-summary toast covering only the categories that occurred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

telemetry: stream flush span + otel load diagnostics (#2891) * telemetry: emit client-observed workflow.stream.write span per flush batch Complements the existing workflow.stream.read TTFC span: each flushed batch emits a back-dated CLIENT span covering the app-perceived write latency (buffer dwell + RPC), with buffer_dwell_ms / chunks / bytes attributes so client-side batching cost (flush timer, turbo run-ready barrier) can be told apart from network/server time. Failed flushes keep the batch's original t0 so a retried batch reports its full dwell. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: tighten changeset Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * telemetry: rename flush span to workflow.stream.flush; DEBUG-log world-vercel OTEL load failure - workflow.stream.write is taken by world-vercel's per-request RPC span (#2857, chunk_rtt); the per-batch flush span gets its own name so the two stay distinguishable in trace queries. Attributes move to workflow.stream.flush.{buffer_dwell_ms,chunks,bytes}, operation=flush. - world-vercel's @opentelemetry/api load failure was silently latched as null, which also swallows bundler/resolution failures in apps that DO register a tracer (observed in production: workbench apps emit core spans but none of world-vercel's). Log the reason under DEBUG=workflow:* so the failure mode is diagnosable. - Document workflow.stream.flush in the tracing docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: tighten changeset Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>