Sunil Pai

Recovery: e2e + unit coverage, and fix runFiber recovery starvation/backoff (#1729) * test(ai-chat,think): fix racy fiber-cleanup checks and add continue-path e2e The recovery e2e tests asserted `hasFiberRows() === false` the instant recovery was detected, racing the continuation/retry turn that recovery legitimately re-runs in a fresh fiber. Poll until the fiber rows settle instead. Also fixes the ai-chat e2e worker, which emitted the legacy `0:{json}` stream framing that AIChatAgent never parses (it reads `data:` SSE frames), so no chunk was ever persisted and recovery only ever saw an empty partial. Emit proper `data:` frames and stream enough chunks to cross the ResumableStream flush threshold, enabling a new continue-path test (non-empty partial -> resume the same assistant message). Co-authored-by: Cursor <cursoragent@cursor.com> * test(ai-chat): e2e coverage for chat recovery budget exhaustion Adds a deterministic exhaustion harness: agents whose turn hangs and produces no recovery progress, so repeated SIGKILLs drive the recovery budget without racing real streamed content. Covers onExhausted firing with reason no_progress_timeout, recovery_aborted, and work_budget_exceeded, plus the persisted terminal banner (#1645). Extracts the shared wrangler/WebSocket e2e plumbing into harness.ts. max_attempts (alarm-debounce forces >30s spacing) and stable_timeout (not feasibly deterministic in-process) are left to unit coverage. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ai-chat): e2e coverage for continue:false / persist:false recovery outcomes Interrupts a turn after a non-empty partial has flushed, then asserts the two onChatRecovery branches that suppress the default behavior: - { continue: false } persists the partial as a durable assistant message but does not re-run the turn (onChatMessage invoked once). - { persist: false, continue: false } drops a plain-text partial (no settled tool results) and does not re-run. Adds an onChatMessage invocation counter + assistant-text accessor to the test agent to distinguish "persisted partial" from a continuation. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): e2e for context-overflow compaction recovery Add ThinkContextOverflowE2EAgent plus an in-process (no process kill) e2e covering the opt-in contextOverflow recovery paths: reactive compact-and-retry that recovers a turn, reactive budget exhaustion that surfaces a terminal context_overflow error, and the proactive guard that compacts pre-step when reported usage crosses the headroom budget. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ai-chat): e2e coverage for stream-buffer cleanup alarm (#1706) and recovering-status broadcast (#1620) Add two deterministic e2e tests in @cloudflare/ai-chat: - #1706 stream-buffer cleanup alarm: new ChatBufferCleanupAgent exposes @callable inspectors (buffer/chunk row counts, _cleanupStreamBuffers schedule count, forced future sweep, hasReclaimableStreams). Asserts a completed turn arms exactly one cleanup alarm, a second turn does not stack a duplicate, and a forced future-now sweep reclaims all buffers so a fully-swept DO reports no reclaimable streams. - #1620 recovering-status broadcast: drives a SIGKILL/restart recovery of a slow-stream turn and asserts the durable cf:chat:recovering flag transitions active -> cleared (via a new getRecoveringFlag @callable), plus a live WS frame collector observes the cf_agent_chat_recovering clear broadcast. The durable flag is the deterministic source of truth because the live frame is not replayed on connect. Adds a createFrameCollector helper to harness.ts and registers ChatBufferCleanupAgent under a new v5 migration tag. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): e2e for durable-submission recovery on start Add ThinkSubmissionRecoveryE2EAgent plus an e2e covering the three _recoverSubmissionsOnStart transitions: messages-not-applied re-enqueues as pending, applied-but-unrecoverable surfaces as error, and a recoverable in-flight submission (real mid-stream SIGKILL) is left running and driven to completion by the scheduled continuation. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agents): arm follow-up alarm for pending runFiber recovery `_scheduleNextAlarm()` only rescheduled for active keepAlive leases, due schedules, and facet runs — never for orphaned `cf_agents_runs` rows or interrupted/pending managed ledger fibers still awaiting recovery. Because orphaned fibers hold no keepAlive ref, a scan that yielded on `fiberRecoveryScanDeadlineMs` (or a pass that retained a repeatedly-throwing unmanaged recovery hook) never got another alarm, so the remaining fibers starved. Add `_hasPendingFiberRecovery()` and arm a follow-up alarm whenever recovery work is outstanding, so multi-pass recovery resumes and eventually drains every fiber (and ages out poison rows via `fiberRecoveryMaxAgeMs`). Co-authored-by: Cursor <cursoragent@cursor.com> * test(agents): e2e coverage for poison-row aging, scan-deadline yield, and concurrent fiber recovery Add three runFiber recovery e2e tests (real `wrangler dev` + SIGKILL/restart against `--persist-to`): - poison-row aging: an unmanaged fiber whose `onFiberRecovered` always throws is retained for retry across alarm passes, then dropped with a `max_age_exceeded` skip once it exceeds `fiberRecoveryMaxAgeMs`. - scan-deadline yield: a tiny `fiberRecoveryScanDeadlineMs` forces a single alarm pass to yield (`scan_deadline_exceeded`) partway through 20 orphaned fibers; subsequent passes drain the rest with no starvation. - concurrent fibers: N concurrent fibers (mixed managed + unmanaged) are all recovered after a kill, covering the gap that prior tests only recovered a single fiber. New DO test agents record recovery signals (hook invocations + skip reasons) into durable SQL so assertions survive DO eviction between polls. Shared spawn/kill/RPC harness lives in `recovery-helpers.ts`. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): e2e for messenger reply-fiber recovery Add ThinkMessengerRecoveryE2EAgent plus an e2e covering MESSENGER_REPLY_FIBER_NAME recovery via _handleInternalFiberRecovery: a streaming-stage interruption posts the apology (apologize mode), and an accepted-stage interruption recovers in answer mode and re-drives reply delivery. Uses an in-memory fake chat adapter that records posts into agent SQL; full streamed-answer rendering is deferred (needs a complete adapter/real transport). Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): e2e for workflow-turn recovery + notification drain replay Add ThinkWorkflowRecoveryE2EAgent (reuses STEP_PROMPT_WORKFLOW with a deterministic mock structured model). Covers the happy path (structured workflow turn completes, notification drains, workflow resumes with the validated output) and the recovery path (mid-stream SIGKILL): on restart the turn is reconciled to a terminal submission and the workflow-notification drain replays it so the workflow is unblocked. Documents the deferred gap that an interrupted structured turn is recovered as skipped rather than completed. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): rename e2e callable to avoid Think.getWorkflow collision The workflow-recovery e2e agent's @callable shadowed the inherited Think.getWorkflow(workflowId) with an incompatible signature, failing typecheck. Rename it to inspectWorkflowRun. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agents): exponential backoff for the runFiber-recovery follow-up alarm The follow-up alarm added for pending fiber recovery fired every keepAliveIntervalMs with no backoff, so a repeatedly-throwing recovery hook — or a `fiberRecoveryMaxAgeMs: 0` ("retain forever") row whose hook keeps throwing — would wake the DO on every tick indefinitely (the perpetual-heartbeat hazard #1707 guards against). Track consecutive no-progress recovery scans and back the alarm off exponentially (capped at 5 min); any scan that recovers a fiber (including a scan-deadline yield that drained part of a batch) resets it, so legitimate multi-pass draining stays prompt. Adds e2e coverage: retain-forever poison-row backoff cadence, and multi-pass recovery for a sub-agent (facet) child driven by the parent alarm. Co-authored-by: Cursor <cursoragent@cursor.com> * test(agents): fast unit coverage for runFiber recovery alarm re-arm + backoff Adds deterministic, in-process unit tests (no process kill / timers) that drive `_checkRunFibers` + `_scheduleNextAlarm` directly and inspect the physical alarm: the starvation re-arm (alarm armed while a retained recovery row is pending), exponential backoff across no-progress scans, backoff reset on forward progress, and no alarm once recovery drains. Previously this behavior was only covered by the nightly e2e suite. Adds getCurrentAlarm/getRecoveryNoProgressScans/simulateAlarmCycle test helpers to the run-fiber test agent. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(agents): note the fiberRecoveryMaxAgeMs:0 warm-DO trade-off A repeatedly-throwing recovery hook with fiberRecoveryMaxAgeMs:0 ("retain forever") is retried on the capped backoff indefinitely, so the Durable Object never idle-evicts while the un-recoverable row exists. Document this in the option JSDoc and docs/durable-execution.md, and recommend a finite age. Bounding recovery by attempts is tracked in #1728. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

feat(agents): detached (background) agent-tool runs — durable completion, progress & milestones (#1758) * design: add RFC for detached (background) agent-tool runs Adds the design record for first-class detached sub-agent runs with a durable named-method completion hook and progress/milestone signaling, in response to cloudflare/agents#1752. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(agents): core detached agent-tool runs with durable completion hook Implements the core of rfc-detached-agent-tools (cloudflare/agents#1752): - `runAgentTool(cls, { detached })` dispatches a sub-agent without awaiting, returning `{ runId, status: "running" }`. Fire-and-forget (`detached: true`) or a durable per-run callback (`detached: { onFinish: "methodName" }`). - Durable, eviction-surviving completion delivery via a single guarded funnel with two independent ledger slots (finish / give-up) using a claim+lease, so delivery is exactly-once on the happy path and at-least-once under failure — a premature give-up can never dedupe a child's real late completion (the #1752 production incident). - Warm fast path (waitUntil) + durable self-scheduling reconcile backbone (this.schedule) that self-cancels once no detached run remains. - Reconcile fork: detached runs are never sealed `interrupted` on a lost observer (the normal state for a background run); the backbone owns them and re-arms on restart. - Absolute `maxBudgetMs` give-up ceiling (default 24h, finite because a detached run has no observer to notice a leak) surfaced as `interrupted`/`budget-exceeded`. - `cancelAgentTool(runId)` by-id cancellation through the same guarded path. Schema bumped to v10 with detached + ledger columns (idempotent migrations). Co-authored-by: Cursor <cursoragent@cursor.com> * test(agents): cover detached delivery ledger (exactly-once + two-slot) Drives the detached delivery funnel directly: exactly-once on terminal, dedupe under concurrent fast-path/backbone deliveries, and the independent finish/give-up slots so a budget give-up never dedupes a child's real late completion (#1752). Also switches the ledger claim to rowsWritten() since UPDATE ... RETURNING row counts are not a reliable claim signal on Workers SQLite. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(agents): document detached runs + inspectAgentToolRun null contract Adds a changeset (minor) and a "Detached (background) runs" section to the agent-tools doc covering the detached handle, durable onFinish, budget give-up, explicit cancellation, and the inspectAgentToolRun null-means-not-yet contract from #1752. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(agents): satisfy oxlint + oxfmt for detached changes Co-authored-by: Cursor <cursoragent@cursor.com> * feat(think): detached runAgentTool notify convenience Adds `detached: { notify: true }` sugar: when a detached sub-agent run finishes, a Think agent injects the result back into the chat via submitMessages (idempotent per run + status) so the model reacts, without hand-wiring onFinish. Override formatDetachedCompletion() to customize. Wired generically in the base Agent by resolving the conventional notify hook by name so the core stays decoupled from the chat layer. Co-authored-by: Cursor <cursoragent@cursor.com> * example(agents-as-tools): demonstrate detached background runs Adds a `research_background` tool that dispatches a Researcher with `detached: { notify: true }` (returns immediately, result posted back into the chat on completion) and a `cancelBackground(runId)` callable built on cancelAgentTool. Updates the system prompt and README to cover the background flow. Co-authored-by: Cursor <cursoragent@cursor.com> * test(agents): bump expected schema version to 10 for detached columns Co-authored-by: Cursor <cursoragent@cursor.com> * feat(agents): ephemeral progress + durable milestones for agent-tool runs Adds two complementary signaling channels to long-running agent-tool runs, plus a round of detached-run hardening from the deep review pass. Progress (4a) — ephemeral: - `reportProgress({ fraction, message, phase, data? })` emits transient `data-agent-progress` parts; coalesced, latest-snapshot-only, persisted as `progress_json`. Surfaced via `AgentToolRunState.progress`, the `onProgress` hook, and `inspectAgentToolRun`. - Shared `AgentToolProgressEmitter` centralizes coalescing/persistence so Think and AIChatAgent share one code path; `forget(runId)` clears coalescing state on terminal cleanup (no per-run map leak). - No-progress budget (`detachedNoProgressBudgetMs`) gives up a detached run that reports initial activity then goes silent; resets on any progress/milestone. Milestones (4b) — durable: - `reportProgress({ milestone, data })` persists an ordered, observable record (new `cf_*_agent_tool_milestones` tables), broadcasts a `data-agent-milestone` part (never coalesced), and surfaces `AgentToolRunState.milestones`. - `detached: { onMilestones }` injects an at-most-once chat notification on both the warm path and the reconcile cold path. Two modes — the `string[]` shorthand defaults to "narrate": - "narrate" (default): synthetic assistant message, no model turn — a cheap status line for milestones the agent needn't act on. - "react": user-role turn so the model responds (steer / start dependent work). Costs a turn. Override prose via `formatDetachedMilestone()`. Client/UX: - Reducer projects milestones (deduped by sequence) and advances the progress snapshot monotonically. Synthetic notify messages render as "Agent event" with milestone/result badges instead of a raw user/assistant bubble. - agents-as-tools example emits a "sources-gathered" milestone and uses the narrate shorthand; tray renders progress + milestone chips. Detached-run hardening (review pass): - Terminal `agent-tool-event` is always broadcast on cancel (synthesize seq). - Generic `onFinish` delivery serialized against the turn queue via `_runDetachedDelivery` (Think/AIChatAgent enqueue on `_turnQueue`). - `_armDetachedBackbone` race guarded with an arming mutex. - Failed detached deliveries re-throw and emit `agent_tool:detached:delivery_failed`; accumulating live runs emit `agent_tool:detached:live_count_warning`. Schema bumped to 11. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(think): explain why submission drain uses allowNested Document at the call site that the submission drain runs fire-and-forget, so a nested submission turn (e.g. a detached-finish notify calling submitMessages mid-turn) is admitted safely without deadlock — and why the flag is applied to all submissions rather than scoped to detached notify. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(ai-chat): detached notify + onMilestones parity with Think AIChatAgent now implements the parent-side detached chat conveniences that were previously Think-only, so `detached: { notify }` and `detached: { onMilestones }` work on an AIChatAgent parent instead of being silently dropped (the docs and the base _deliverDetachedMilestone JSDoc already claimed support). - formatDetachedCompletion + _cfDetachedNotifyFinish (auto-wired by name) inject the completion as a user turn the model reacts to; idempotent per (runId, status) via a deterministic message id. - formatDetachedMilestone + _deliverDetachedMilestone override deliver milestone notifications: "narrate" (default) persists an assistant line with no model turn; "react" injects a user turn + reply. Idempotent per (runId, name). AIChatAgent has no durable-submission layer and TurnQueue has no re-entrancy bypass, so the shared _injectDetachedNotification helper avoids self-deadlock by dispatching the react turn three ways: - inside our own serialized finish-delivery slot -> run inline + awaited (the ledger only marks delivered after it resolves, so the reaction is eviction-safe); - inside a FOREIGN active turn (e.g. cancelAgentTool mid-turn) -> fire-and-forget a turn that runs once the slot frees (inline would interleave, enqueue-await would deadlock); the persisted message makes it best-effort-durable; - otherwise -> enqueue + await normally. An in-flight id set dedupes a concurrent warm-tail + backbone delivery of the same milestone within one isolate (milestones have no ledger claim), while messages.some() dedupes re-delivery against persisted history across eviction. Tests: ai-chat detached-notify covers notify react + idempotency, narrate (no model turn), and react milestones. Docs/changesets/RFC updated to reflect that notify + onMilestones are chat-host conveniences (Think + AIChatAgent). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com>

Make nightly e2e trustworthy + harden chat-recovery / agent-tool coverage (#1797) * fix(ai-chat): make nightly e2e trustworthy and deterministic The ai-chat Playwright nightly had been red for weeks, timing out at the 30-minute job limit with no actionable output. Root cause was twofold: - Specs navigated to `about:blank` (a null origin) before opening WebSockets to `localhost`, which Chromium's Private Network Access rules reject — so every test failed to connect, and with no globalTimeout the job ran until the CI ceiling. - The remote `ai` binding made `wrangler dev` slow to become ready, and Playwright's port-only readiness check raced ahead of it. Fixes: - Add a `/__health` endpoint to the e2e worker and make all specs navigate there (same-origin) before connecting WebSockets. - Split the suite into a deterministic, AI-free config (`wrangler.mock.jsonc`, no credentials) and a bounded Workers-AI config, each with globalTimeout, retries, and real `/__health` readiness polling. - Run them as two nightly jobs so the deterministic suite is a green gate independent of remote AI. Also fix the 3MB compaction e2e test: it asserted the pre-refactor notice strings ("too large to persist" / "Preview:") instead of the current structured-truncation marker. Compaction itself works correctly on the setMessages path (verified against persistMessages + the row-size-guard unit tests); updated the assertions and un-quarantined it. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ai-chat): bump Workers-AI e2e retries to 3 The real-model LLM specs are inherently nondeterministic — the small Workers AI model occasionally varies wording or step boundaries and trips a strict assertion on the first attempt, but re-runs recover cleanly. Raise CI retries from 1 to 3 for the LLM-only suite (deterministic suite stays at 1). Retries only fire on failure, so green runs are unaffected. Co-authored-by: Cursor <cursoragent@cursor.com> * ci(nightly): run deployed recovery suites on the nightly schedule The two Layer-5 DEPLOYED suites (ai-chat recovery on real edge, Think recovery probe on real edge) were gated behind a `RUN_DEPLOYED_E2E` repo variable that was never set, so they had never run in CI — only locally. That left the real-edge recovery paths unverified and free to silently rot. Enable them on the nightly `schedule` event while keeping manual `workflow_dispatch` runs opt-in (via the `run_deployed` input or the repo variable), so ad-hoc debugging dispatches don't deploy billable Workers unintentionally. Each suite still uniquely names and always deletes its throwaway Worker. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): re-apply channel policy on recovered pre-stream retry turns Adds recovery × channels coverage (previously zero) and fixes a real bug it surfaced: `continueLastTurn` re-resolved the channel from the persisted user message on recovery, but `_retryLastUserTurn` (the pre-stream retry path used by `_chatRecoveryRetry`) admitted the recovered turn without re-resolving the channel. So a turn interrupted before streaming was retried with the default policy instead of its channel's instructions / tool narrowing, even though the `metadata.channel` stamp survived. The new workers-pool test `channel-recovery.test.ts` asserts both the stamp survives and per-channel policy is re-applied across BOTH recovery paths (continue + retry); the retry case reproduced the bug before the fix. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ai-chat): sub-agent SIGKILL agent-tool recovery e2e (Think parity) ai-chat had full SIGKILL chat-recovery e2e but no sub-agent coverage, while Think tests an agent-tool child recovering across a process kill. Closes that parity gap: adds a `ChatRecoveryHelperParent` (plain Agent) that drives a `runAgentTool` run whose child is an `AIChatAgent` (`ChatRecoveryHelperChild`, slow finite stream + chatRecovery + default continue recovery). The test starts the run, confirms it is in-flight, SIGKILLs + restarts wrangler against the same persist dir, and asserts the parent re-attaches to the still-running child and collects its real terminal (`completed`) instead of abandoning it as `interrupted` (#1630). Auto-runs in the nightly e2e-ai-chat-recovery job (matches the e2e include glob); no CI change. Test-only, no changeset. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think,ai-chat): re-bind agent-tool child run_id across recovery A recovered agent-tool child turn minted a fresh request id but left the child-run row's request_id pointing at the pre-eviction turn, breaking frame attribution. A long-running recovered child then forwarded nothing to the parent's re-attach tail and was sealed `interrupted` once the no-progress budget elapsed, even while still advancing. Both recovery paths (continueLastTurn / _retryLastUserTurn) now re-bind the child-run row and the in-memory attribution map to the current turn's request id, keeping frames flowing across recovery so the parent re-attaches to the real terminal. - think: _rebindAgentToolChildRunRequestId + wiring; un-skip the reattach-budget e2e (now green, 199s) and add deterministic workers-pool unit coverage. - ai-chat: parity fix + deterministic unit coverage. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): cover nested agent-tools, concurrency cap, runTurn × recovery, action-pause × recovery Adds deterministic Workers-pool coverage for agent-tool composability and recovery interactions that were previously untested: - nested-agent-tools: 3-level chain (grandparent → middle → grandchild) completes and grandchild observation does not bridge to the grandparent - max-concurrent-agent-tools: over-cap runs are rejected, freed slots reuse, Infinity allows unbounded concurrency - run-turn-recovery: runTurn wraps execution in a recovery fiber and cleans up, and a fresh runTurn composes cleanly on a recovery-resolved transcript - action-pause-recovery: a parked durable pause is not seen as a pending interaction by recovery and stays approvable across an active incident Test-only; no public API change. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think,ai-chat): harden agent-tool recovery coverage + unify child-run predicate Production (behavior-preserving hardening, no migration): - Unify the in-flight child-run predicate across @cloudflare/think and @cloudflare/ai-chat on an explicit `status IN ('starting','running')` in both `_rebindAgentToolChildRunRequestId` and `_agentToolRunForRequest`. Both child-run tables already carry `status` AND `completed_at`, and terminal rows set them together (the lifecycle invariant), so this is equivalent to the prior `completed_at IS NULL` / `status = 'running'` checks but states intent and stays consistent. Document the one-child-run-row-per-DO invariant (child DOs are addressed by runId) that makes the rebind unambiguous. Coverage: - rebind no-op safety (think + ai-chat): no table, settled row, and defensive newest-of-many-active-rows selection — locks the "no-op on non-child recovery" contract. - maxConcurrentAgentTools: soft-terminal `interrupted` runs do NOT occupy a slot (re-issue after recovery is never cap-blocked); in-flight `running` runs do. - channel × recovery: per-channel TOOL policy (not just instructions) is re-applied on a recovered turn; a turn with no channel stamp falls back to default policy; channel re-resolution AND agent-tool request_id rebind compose on the SAME recovered turn. - nested agent-tools: each nesting level enforces its own maxConcurrentAgentTools independently of its parent. - action-pause × recovery: a parked durable pause stays approvable across a CONTINUE recovery incident (the retry path was already covered). - ai-chat agent-tool recovery e2e: document the coverage split (deterministic unit suite is the tight rebind gate; SIGKILL e2e is the integration smoke) and why we don't replicate Think's slow no-progress-budget e2e. Behavior-preserving + test-only; no changeset. Co-authored-by: Cursor <cursoragent@cursor.com> * Add test-coverage matrix and useAgent tests Introduce a living design doc `design/test-coverage-matrix.md` and link it from design/AGENTS.md and design/README.md to track feature × test-layer coverage and quarantined tests. Add a new browser React test `packages/agents/src/react-tests/useAgentToolEvents.test.tsx` to exercise live-vs-replay dedupe and terminal guards for agent-tool events. Also set a 15-minute Playwright `globalTimeout` in `packages/codemode/e2e/playwright.config.ts` so slow/flaky e2e runs produce reports instead of silent CI cancellations. * test: harden browser/hook coverage, resolve skip debt, add nightly e2e jobs Workstream A — browser/hook depth: - New real-Worker agent-tool replay-on-reconnect suite (agents react-tests) backed by a deterministic, LLM-free TestAgentToolStubChild; proves live-vs-replay dedupe + typed interrupted-cause survival across a real socket reconnect. - New deterministic Think resume-handshake suite (think react-tests) driving the real useAgentChat: single ACK per RESUMING (#1733), replay-no-dup, and idle transcript restore. Workstream B — coverage matrix: - design/test-coverage-matrix.md per-cell precision + A1/A2 rows; cross-linked from root + agents AGENTS.md; recorded idle-connect transcript-delivery as a keep-divergent row in the chat-recovery RFC convergence matrix. Workstream C — nightly hygiene (.github/workflows/nightly.yml): - Add codemode Playwright e2e job and agents browser-connector job; enable the gated Workers-AI tanstack leg. Workstream D — skip debt (all resolved): - basepath HTTP custom-path: replaced the meaningless "not 404" skip with two real positive tests (method+body forwarding; instance resolution). - workflow-error-reporting step.do: un-skipped via introspector disableRetryDelays() (real throw + retry exhaustion, no wall-clock waits). - workflow-integration x5: stale comments + one stale assertion; all un-skipped. - shell git clone: reclassified to an opt-in Group A gate (RUN_GIT_CLONE_E2E, forwarded into the pool via vitest.config.ts), repinned to octocat/Hello-World. - useAgent async query: fixed a missing `await` on the suspending render. Test-only + CI + docs; no package public API change (no changeset needed). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

AI Gateway resumable streaming: RFC + experimental harnesses + workers-ai-provider@3.2.0 bump (#1764) * design: RFC + harness for AI Gateway resumable streaming in workers-ai-provider Adds the design record and the empirical harness behind merging ai-gateway-provider's capabilities into workers-ai-provider, centered on AI Gateway native resumable streaming (cf-aig-run-id + /resume). design/rfc-workers-ai-gateway-merge.md (+ index entry in design/AGENTS.md): Capability-driven dual-transport design. The run path (env.AI.run) gives resumable streaming; the gateway path (env.AI.gateway(id).run([...])) gives server-side fallback + caching. They are disjoint, so a delegate selects the transport from the requested options and warns/errors on incompatible mixes. Covers: dispatch mechanics (run-path forwarding fetch + gateway-path capture/redispatch), slug canonicalization, routing-layer providers (OpenRouter), fallback modes, a tiered resume-expiry recovery ladder, an error taxonomy with a nested attempt tree, Agents SDK / Think chat-recovery integration, provider-specific interactions (OpenAI Responses API scoped out; model-agnostic user-message continuation instead of deprecated Anthropic prefill), and a two-tier (unit + live e2e) test strategy. experimental/gateway-resume/ — the harness that validated it against a live account with unified billing: - Resume contract: cf-aig-run-id is issued for dash-catalog models on the new run API (not @cf/* yet); resume(from=0) reproduces the full stream byte-for-byte; `from` is an SSE EVENT index, not a byte offset; replay is provider-native SSE (so a provider-matched parser is required). - Transport split: cf-aig-run-id only on env.AI.run; cf-aig-step (server fallback) + cf-aig-cache-status (caching) only on env.AI.gateway().run([]). - Request-side passthrough (/passthrough): real @ai-sdk/openai (.chat) and @ai-sdk/anthropic bodies dispatched through env.AI.run parse cleanly (text, tools, usage, finish); cf-aig-run-id surfaces on result.response.headers; anthropic-version header survives. This is the gate the whole run-path delegate design rested on. - Delegate engine (src/delegate.ts, /delegate): reference implementation of the capability matrix + transport selection + forwarding fetch, exercised end-to-end through streamText. All scenarios verified: run-path resume, server fallback (cf-aig-step increments on a real bad-primary fallthrough), caching, conflict errors, escape hatch, both providers. - Buffer TTL: ~330-360s (~5.5 min); expiry contract is a clean 404 {"error":"Request not found"} (vs 200/0-bytes past-end, vs 500 for a malformed id) — the signal the tiered recovery ladder branches on. Reproduced by ttl-sweep.sh / ttl-sweep-fine.sh. The harness is the prototype that the workers-ai-provider/gateway-delegate feature (in the cloudflare/ai repo) was ported from. Co-authored-by: Cursor <cursoragent@cursor.com> * design: validate transparent resume reconnect/replay (RFC §7.1) Prototype + live-validate the resume reconnect layer in the harness, then mark RFC §7.1 BUILT + VALIDATED. experimental/gateway-resume: - src/resumable.ts: createResumableStream — SSE event-boundary buffering (emit only complete events, buffer the trailing partial), terminator counting for the resume `from` index, reconnect via env.AI.fetch(/resume?from=N), 404 -> ResumeExpiredError. Includes a dropAfterEvents fault injector for testing. - src/index.ts: /resume-stream endpoint runs streamText through a run-path @ai-sdk model whose body is wrapped in the resumable stream, with ?dropAfter=N fault injection, reporting reconnect count + completion. Live results (proves transparent in-stream recovery, the previously unproven part): clean run -> 0 reconnects, finishReason stop; injected drop after 20 and after 80 events (openai) -> 1 reconnect each, complete parse, no error; anthropic native SSE drop after 5 events -> 1 reconnect, complete parse. Byte alignment holds because partial events are discarded and resume realigns on the boundary. RFC: §7 now carries a BUILT + VALIDATED status note, §7.1 documents the boundary-buffering correctness detail (not just a counter), and History records the implementation landing in cloudflare/ai (branch feat/workers-ai-provider-gateway-delegate). Co-authored-by: Cursor <cursoragent@cursor.com> * design: validate cross-invocation re-attach (RFC §7.1 / §9) Generalize the harness resumable stream + add a /reattach endpoint, and update the RFC to reflect the built re-attach primitive + the concrete Layer B seam. experimental/gateway-resume: - resumable.ts: optional `initial` + `fromEvent` (re-attach with no initial body, starting from resume?from=fromEvent) + onProgress(eventOffset) hook. - index.ts: /reattach simulates invocation #1 starting a run, then invocation #2 (after eviction) re-attaching with no initial body. Asserts (a) from=0 reproduces the full response through the @ai-sdk parser, and (b) from=mid is byte-exact against the known tail. Live results: openai (152 events) and anthropic (21 events) both pass — from=0 parses to finishReason stop with full text; from=mid is byte-equal to the tail (openai 22481==22481, anthropic 1814==1814). RFC: §7.1 status note now documents both modes (in-stream wrap + cross-invocation re-attach) and the onProgress capture surface; §9 plug-in steps updated to the concrete delegate hooks (onDispatch -> runId, onProgress -> eventOffset, stash in the chat fiber, re-attach via createResumableStream({ fromEvent }) on onChatRecovery). Co-authored-by: Cursor <cursoragent@cursor.com> * feat(experimental): gateway-resume-think — Layer B re-attach recipe for Think A self-contained Think agent demonstrating the missing "Layer B" recovery from RFC §9: on Durable Object eviction mid-turn, re-attach byte-exactly to the same AI Gateway run (cf-aig-run-id) instead of regenerating (which re-spends tokens). Pattern: getModel() capture cf-aig-run-id (onRunId) + live SSE offset (onProgress); this.stash({ runId, eventOffset }) — survives eviction (throttled to every 8th event). ‹DO evicted› onChatRecovery(ctx) planResume(ctx.recoveryData) — re-attach if the buffer is still live (TTL window), else fall back to the default; arms the next continuation + returns { continue: true }. continueLastTurn() getModel() returns a re-attach model that replays the exact tail from the stashed offset via createResumableStream ({ runId, fromEvent }) — zero new tokens. Files: src/plan.ts pure Layer-B decision (re-attach vs fallback; TTL + shape guards) src/resume.ts resumable stream (vendored from workers-ai-provider), re-attach mode src/gateway-model.ts AI SDK model over env.AI.run: buildCaptureModel / buildReattachModel src/server.ts GatewayResumeAgent extends Think wiring it together src/layer-b.test.ts 9 hermetic tests (planResume decisions + re-attach stream) scripts/driver.mjs live end-to-end (start → ctx.abort → assert reattach) README.md the pattern, how to run hermetic tests + live deploy Modeled on experimental/chat-recovery-probe (ctx.abort eviction, deploy/driver validation). Vendors a copy of the resume primitive so the experiment is self-contained; the shipping version lives in workers-ai-provider. Validation: tsc clean; oxlint/oxfmt clean; 9 unit tests green; wrangler deploy --dry-run bundles with AI + DO bindings. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(gateway-resume-think): deterministic live driver + validated re-attach Validated the Layer-B recipe against a real deploy (gateway "default", openai/gpt-5.4): a mid-stream ctx.abort() leaves a stashed { runId, eventOffset }; onChatRecovery re-attaches byte-exactly from the stashed offset and the turn converges with zero new tokens. ✓ captured run 47e67890a911… at event 86 → interrupt (ctx.abort, mid-stream) ✓ re-attached to run 47e67890a911… from event 88 ✓ turn converged driver.mjs: wait until the run-id is CAPTURED before interrupting (no fixed sleep racing the gateway round-trip — the earlier 2s interrupt beat the first byte and fell back), tolerant JSON reads across the abort/reboot window, and assert both the reattach decision and convergence. wrangler vars: GATEWAY "default" (the real resume-capable gateway) instead of the placeholder. README: document the live run + the buffered-tail semantics (resume replays what was buffered up to the abort; it does not regenerate). Co-authored-by: Cursor <cursoragent@cursor.com> * design: prove gateway runs are detached (keep generating after disconnect) Add a /detach probe to the gateway-resume harness: start a run, read a few SSE events, then reader.cancel() to simulate the originating request dropping mid-stream, and sample resume?from=0 over time. Finding: the run is server-driven / detached. The first resume?from=0 BLOCKS while tailing the live run (openai/gpt-5.4 6.7s -> 513 events; anthropic claude-opus-4.7 8.6s -> 24 events) and replays the COMPLETE stream including the terminal event ([DONE] / message_stop), despite only 3 events consumed before the disconnect. Upstream generation is not tied to the originating socket -> re-attach is genuinely zero-loss. Also extend /reattach to parse a mid-offset re-attach through @ai-sdk: from=0 -> full message (772 chars, stop); from=83/167 -> byte-exact tail, parses cleanly (stop), 392 chars (the tail text, not corruption). This corrects a wrong caveat in the gateway-resume-think recipe: the demo's 510 chars was the correct tail (events 88->end), to be concatenated with the prefix Layer A already delivered = full message, zero new tokens. Docs: RFC §7.1/§9 (detachment finding + prefix+tail merge model), both harness READMEs. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(gateway-resume-think): prove zero-loss reconstruction + fix offset throttle Add a ground-truth verification path and prove, end-to-end on a real DO eviction, that recovery reconstructs the FULL answer with zero regenerated tokens. - /gw/verify (+ parseReattachText): parse resume?from=0 to the full run text and compare to the recovered assistant message. Live result: recovered === full run, byte-identical (e.g. 17303 chars), zero tokens. - Recovery now re-attaches from event 0 (full replay) instead of the stashed tail offset. The run is detached (completes server-side after the originating disconnect, per experimental/gateway-resume /detach), so from=0 replays the complete buffer for zero tokens and continueLastTurn REPLACES the partial leaf with it -> provably whole. A tail re-attach risks dropping the prefix under replace semantics + the Layer-A<->SSE offset-space mismatch (RFC §9.4). - Fix the stash throttle: SSE offsets JUMP (a chunk can carry several events), so `eventOffset % N === 0` never lands and only the offset-0 stash survived. Switch to a delta-based throttle (advanced >= N since last stash). Verified: 38 stashes, offset 305 captured across eviction. - Driver: poll verify over time (distinguishes a still-streaming continuation from a real stall) and report pre-eviction stash diagnostics. - Hermetic test: full(from=0) === prefix(0..k) + tail(from=k) through the openai parser (seam regression guard). - Docs: recipe README + RFC §9 updated with the validated from=0 design and the throttle/seam learnings. Co-authored-by: Cursor <cursoragent@cursor.com> * design: reconcile RFC with as-built implementation (cloudflare/ai#573) Update the AI Gateway merge RFC to match what actually shipped: - Status → accepted/implemented (#573); Decision → accepted with the deliberate deviations from the original sketch called out. - §3: add an "As built" callout (single createWorkersAI entry, internal delegate, wire-format-keyed plugins, optional default gateway, per-call fallback, per-slug autocomplete, per-provider runWireFormat, metadata). - §3a: mark the generic openai-compatible wrapper resolved — folded into the wire-format-keyed openai plugin. - Open questions: resolve the ones #573 answered (wrapper descriptor shape, per-slug autocomplete typing, fallback ergonomics, conflict granularity, openai-compatible). TTL configurability, OpenRouter internal-routing recoverability, Responses API, continuation boundary, and the framework cf-aig-run-id capture seam remain open follow-ups. - History: add the 2026-06-16 consolidation entry (single entry, literal types, metadata, alibaba/minimax, runWireFormat, default gateway, hardened e2e). Co-authored-by: Cursor <cursoragent@cursor.com> * update workers-ai-provider --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com>

feat(think): add opt-in, read-only HTTP fetch tool (#1821) * feat(think): add opt-in, read-only HTTP fetch tool Adds a new `@cloudflare/think/tools/fetch` export and a `fetchTools` property on `Think` that lets agents read allowlisted HTTP resources (docs, APIs) directly. Disabled by default. `createFetchTools()` generates a generic, allowlisted `fetch_url` tool plus one `fetch_<name>` tool per named service-binding/`Fetcher` target, so the model gets a clear per-target surface instead of one polymorphic tool. Safety model (Workers-grounded): - GET-only; mutations are out of scope for v1. - SSRF defenses: blocks private/loopback/link-local/CGNAT/ULA hosts, `*.internal` and `*.localhost`, and IP literals in decimal/octal/hex/ shorthand forms; URL normalization; rejects embedded credentials. - Allowlist-aware redirect policy with cross-origin header stripping; bare-origin patterns auto-expand to cover all subpaths. - Separate size limits for download (`maxBytes`), model context (`maxModelChars`), and a `response: "workspace"` spill for large bodies. - Model header allowlist; fixed binding headers take precedence over model-provided ones. - Markdown-first default `Accept` (configurable / disablable) so content-negotiating endpoints return clean text instead of HTML. - `tool:fetch` observability event for an egress audit trail. Hardening: allowlist globs are precompiled to RegExp at config time; unconsumed response bodies are cancelled on non-2xx and redirect paths. Integrated into the `assistant` example (kitchen-sink) with workspace spill enabled. Includes Worker-runtime unit + Think integration tests (`fetch-tools.test.ts` + `agents/fetch-tools.ts` fixture), README/docs/ design updates, test-coverage-matrix row, and a changeset. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): block IPv4-mapped IPv6 addresses in SSRF guard The WHATWG URL parser serializes `[::ffff:127.0.0.1]` to hex form `[::ffff:7f00:1]`, which the dotted-decimal-only regex never matched, so IPv4-mapped IPv6 addresses bypassed our own private-network check (the Workers egress layer still blocked them, but the defense-in-depth layer had a gap). Decode the trailing two hextets back into dotted IPv4 and reuse the v4 rules. Adds test coverage for both mapped forms. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

fix(chat): keep AI SDK status correct when reconnect races the pre-stream window (#1784) (#1803) * fix(chat): keep AI SDK status correct when reconnect races the pre-stream window (#1784) A chat turn spends a window between "request accepted" and "first chunk produced" in neither the active-stream nor terminal-replay state: it is queued, debouncing, waiting on waitForMcpConnections, or running async setup inside onChatMessage before a stream object exists. A client that reconnected or re-mounted in that window was answered with cf_agent_stream_resume_none and gave up, so the turn the server went on to stream never drove the client's AI SDK `status` — the UI stayed stuck at "ready" until a full remount. Server (agents/chat): - New shared PreStreamTurns tracker (pre-stream-turns.ts) for accepted-but-not- yet-streamed turns and the connections parked waiting on them, mirroring ContinuationState (pure data + send-through-callback). - New server->client cf_agent_stream_pending frame (protocol + wire-types + golden builder). - ResumeHandshake now parks resume requests that arrive during the pre-stream window and emits STREAM_PENDING ("keep waiting") instead of RESUME_NONE, then flushes parked connections into the normal STREAM_RESUMING handshake on stream start. Continuation affinity is relaxed via an optional isConnectionPresent host hook so a transparent reconnect (whose connection id changed) can resume a continuation whose original owner connection is gone. Client: - ws-chat-transport: handleStreamPending() extends the resume probe from the 5s fast path to a 60s backstop so the probe stays open across the gap. - useAgentChat re-probes the stream on a transparent socket reopen (e.g. a 1006 reconnect that does not remount the component) so status recovers. Hosts: - Wired the begin/park/flush/settle lifecycle into both AIChatAgent and @cloudflare/think. - Skipped turns (supersede / queue generation change) settle WITHOUT releasing parked connections (releaseParked: false), so a client parked during the window survives onto the successor turn instead of being cut loose by a premature RESUME_NONE in the supersede/settle microtask race. Hibernation: PreStreamTurns is in-memory only and is safe because the pre-stream window cannot overlap hibernation — a turn between begin() and stream start is an unresolved message-handler promise that pins the DO in memory, so eviction only happens once a durable stream exists (resumed via ResumableStream) or the turn finished. Documented as an invariant on each _preStream field. Tests: - Unit: PreStreamTurns (incl. skip-path contract), handshake park/pending/flush/ affinity, golden STREAM_PENDING frame. - Transport: STREAM_PENDING keep-waiting timeout extension (incl. tool continuation path). - Integration (workers runtime, real AIChatAgent): onConnect/resume-request park -> pending -> resuming, and parked-client-survives-overlapping-submits. - React hook: transparent reconnect re-probe + STREAM_PENDING keepalive. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ai-chat): guarantee pre-stream turn settles if a pre-turn-body step throws (#1784) AIChatAgent.begin()'d the pre-stream turn before persistMessages, _mergeQueuedUserMessages, and the queued mcp.waitForConnections / _setRequestContext steps. A throw in any of those never reached the settle in chatTurnBody's finally, so the request id stayed stuck in _preStream (hasInFlight() true forever) and every later client was parked on STREAM_PENDING (60s) instead of getting an immediate STREAM_RESUME_NONE — until chat clear or DO eviction. Wrap the whole post-begin() scope in a top-level try/finally that always calls _settlePreStreamTurn, mirroring @cloudflare/think. Idempotent with the existing happy-path settle. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

feat(think): git init + Oxlint/Oxfmt for scaffolding, interactive template, drop unused helpers (#1817) * feat(think): git init + Oxlint/Oxfmt for scaffolding, interactive template, drop unused helpers Scaffolding (create-think + `think init`) - Initialize a git repository for newly scaffolded and augmented apps, with a guard (`git rev-parse --is-inside-work-tree`) that skips cleanly when the target is already inside a repo — avoids reinit and accidental nested repos. Cross-platform spawn (`shell` on win32); a missing/broken git binary warns and continues. - create-think now prompts for a starter template when `--template` is omitted, with a TTY guard that falls back to `basic` non-interactively and robust numeric-choice handling (out-of-range falls back / re-prompts). - Starters ship with Oxlint + Oxfmt config (.oxlintrc.json / .oxfmtrc.json) and a `check` script (format check + lint + typecheck); `think init` augmentation writes the canonical 149-line .gitignore. - Consolidated shared CLI plumbing (git/npm spawning, repo guard, fs helpers) into create-think/src/cli-utils.ts so create-think and `think init` cannot drift; trimmed lib.ts re-exports to only what crosses the package boundary. Framework cleanup - Removed the unused declarative `agent()` helper (framework/agent.ts) in favor of class-based agents. - Removed identity helpers `defineMessengers`, `defineScheduledTasks`, and `defineChannels` in favor of plain typed object returns; updated all usages across examples, starters, tests, and docs. - Inlined `__isThinkAgentExport` into `__isAgentClass` and dropped the dead `reexportDefault` branch in inferAgentExport. Repo-wide config hygiene - Normalized Tailwind `@source` declarations in styles.css to a single node_modules path (npm install no longer needs the workspace-root fallback); added the missing `@cloudflare/kumo` direct dependency to forever-fibers, which was sourcing it without declaring it. - Normalized `$schema` paths in wrangler.jsonc to the nearest node_modules. Verified: pnpm run check (sherif + exports + oxfmt + oxlint + typecheck, 113/113) clean; create-think tests 16/16. Co-authored-by: Cursor <cursoragent@cursor.com> * docs: update wrangler $schema and Tailwind @source conventions to match scaffolding The PR normalized every example's wrangler.jsonc $schema to `./node_modules/wrangler/config-schema.json` and each styles.css @source to `../node_modules/...`, but the convention guides still documented the old monorepo-root-relative paths. - examples/AGENTS.md: update the wrangler $schema bullet and the Kumo @source CSS block to the standalone-friendly paths, with a note on why they resolve both in the workspace and when copied out. - design/visuals.md: fix the @source code block and replace the now-incorrect "must point to the hoisted package at monorepo root" note. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Fix sub-agent workflow origins (#1430) * Fix sub-agent workflow origins Support workflows started from sub-agent facets by preserving the origin path for callbacks, RPC, tracking, and docs. Made-with: Cursor * Harden sub-agent workflow origins: hibernation-safe broadcast, RPC guard, version check, docs, tests - Make facet broadcastToClients() await the parent hop so a facet that hibernates right after the workflow RPC no longer silently drops the message - Match real DO-stub RPC semantics in _cf_invokeAgentPath: refuse built-in/ prototype and JS-internal method names; align error messaging - Validate AgentWorkflowOrigin.version so an SDK mismatch fails clearly - Fix stale onWorkflowCallback doc comment (_workflow_handleCallback RPC, not HTTP) - Document routing constraints: name-based resolution, facet-local workflow tracking (getWorkflows scoping), class names must survive bundling - Add tests: callback routing to a facet evicted mid-workflow, deleted-mid-flight guard + unsafe-method rejection, and HTTP reach via routeSubAgentRequest while a workflow runs * test: cover legacy workflow resume without origin payload Adds a regression test for the backward-compat path in AgentWorkflow._initAgent(): a workflow started before this change carries only __agentName/__agentBinding/ __workflowName (no __agentOrigin) and must still resolve its originating Agent by name+binding for callbacks and this.agent RPC. Starts TEST_WORKFLOW with legacy-only params (bypassing runWorkflow, which now always injects __agentOrigin) and asserts the RPC result lands and the tracking row reaches 'complete'. * docs: correct facet broadcast lifetime rationale Workerd tracks outbound actor RPC through IoContext::awaitIo/addTask and promotes actor tasks to wait-until tasks even when the JS promise is not awaited. Correct the prior claim that a facet could hibernate and drop the root broadcast. Keep the await for accurate completion, ordering, and error propagation. Also rename the explicit abort coverage from eviction/hibernation to restart coverage. * revert: keep facet workflow broadcast fire-and-forget Broadcast delivery to clients is best-effort everywhere in facet routing: facet broadcast(), the connection bridge, and connection-routing broadcasts all fire-and-forget their root hop on main. A facet does not care whether the client socket is still connected, and workerd keeps the outbound actor RPC alive after the caller returns, so the message is still delivered. Awaiting only the workflow path would impose a stronger completion/error contract than the rest of the broadcast API without any durable-delivery guarantee. Revert _workflow_broadcast() to the standard broadcast() path and drop the related changeset bullet. * chore: trim sub-agent workflow origin patch --------- Co-authored-by: Matt Carey <matt@cloudflare.com> Co-authored-by: Matt <77928207+mattzcarey@users.noreply.github.com>

Recovery-engine convergence: ai-chat recovers interrupted server tools like Think (#1794) * feat(ai-chat): recover interrupted server-tool calls like Think (recovery-engine convergence) Closes the last real behavior gap between the two AI-SDK chat hosts so ai-chat is a strict subset of Think's recovery behavior. - agents/chat: extract Think's transcript repair into a shared @internal primitive (repair-transcript.ts: repairInterruptedToolParts + toolPartHasSettledResult) and the orphan-persist core (orphan-persist.ts: persistReconstructedOrphan). Think delegates to both (behavior-neutral). - ai-chat: adopt the shared repair via an overridable repairInterruptedToolPart hook; flip a dead server-tool input-available orphan to an errored result so convertToModelMessages no longer 400s with AI_MissingToolResultsError. - ai-chat: run repair before EVERY inference chokepoint (live submit, auto-continuation, _runProgrammaticChatTurn, continueLastTurn), guarded by !hasPendingClientInteraction() so a pending client tool is never clobbered — matching Think repairing before every inference, incl. chatRecovery=false. - ai-chat: waitUntilStable gains an optional pendingInteraction predicate; the recovery paths pass the narrow client-only predicate so a dead server orphan no longer blocks stability. Hard constraint honored: repair only ever reshapes assistant tool parts; user messages and metadata.channel are untouched. Tests: shared repair-transcript unit tests; ai-chat e2e (server orphan recovers instead of exhausting) + chokepoint repair/guard tests. Full suites green (ai-chat 612, think 791). Co-authored-by: Cursor <cursoragent@cursor.com> * feat(ai-chat): server-only repair scope + cross-host recovery conformance suite Lock-in follow-up to the recovery-engine convergence. - agents/chat: repairInterruptedToolParts gains an optional shouldRepair(part) skip predicate (default: repair all, so @cloudflare/think is unchanged). - ai-chat: scope repair per-part to dead SERVER orphans via shouldRepair = !partAwaitsClientInteraction(...), replacing the coarse whole-transcript guard. A fresh dead-server orphan at the leaf is now repaired even when an unrelated abandoned client orphan sits earlier in history; a part still awaiting a client is left verbatim. - agents/chat: add recovery-conformance.test.ts — runs the shared repair + predicate primitives under both host wirings over a canonical scenario table, pinning the subset relationship (identical for server orphans; intentional divergence for client orphans) and the "ai-chat never repairs more than Think" invariant. - docs: correct stale §Cutover claim (incident shape/id/keys are shared in recovery-incident.ts) + progress-log entry. Full suites green: agents 477, ai-chat 613, think 791. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

fix(chat-recovery): bound Durable Object memory-limit (OOM) crash loops (#1825) (#1826) * fix(chat-recovery): bound Durable Object memory-limit (OOM) crash loops (#1825) A chat-recovery turn whose Durable Object isolate exceeds its 128 MB memory limit could loop forever, re-running the (billable) turn on every platform alarm retry. The isolate streams a little content before the reset, which bumps the durable progress counter; on the next wake recovery reads that as forward progress and resets both progress-keyed bounds (maxAttempts, noProgressTimeoutMs), and because each crash lands inside the alarm-debounce window the attempt counter is pinned too. With maxRecoveryWork defaulting to Infinity, no instrument could ever seal the turn, so the model ran forever. This lands a layered fix: 1. Finite maxRecoveryWork default (1000, was Infinity). The work meter is the one signal that keeps climbing across the loop, so a finite default seals a runaway with reason="work_budget_exceeded". 2. OOM-specific in-DO budget (chatRecovery.maxOomRetries, default 3). A memory reset re-OOMs on re-run (the turn's working set, not the platform, is the cause), so it is classified as a distinct deterministic failure rather than a deploy-style transient: it is NOT deferred and retried forever. Each crash bumps a durable per-incident oomAttempts counter; after a small number of tries it seals with reason="out_of_memory". Fast and attributable. 3. Alarm-boundary circuit breaker (Agent.alarm()) as the universal backstop for OOMs that bypass the in-DO budgets entirely - thrown before the budget code runs (boot-time state hydration), or whose own small writes also OOM under memory pressure. Left unhandled such an error propagates out of alarm() and the platform auto-retries forever. alarm() now intercepts ONLY Durable Object memory-limit resets at the outermost frame, where the heavy turn has unwound and GC has reclaimed its footprint, so the seal/purge writes can land where mid-turn ones OOMed. A durable strike counter (static maxAlarmMemoryLimitStrikes, default 3) tolerates a few resets - backing off the looping rows so the retry is not a hot loop - then seals the recovery (out_of_memory) and surgically purges ONLY the looping schedule rows, leaving unrelated scheduled tasks intact. Emits a new alarm:memory_limit_reset event. Everything except memory-limit resets re-throws exactly as before. Supporting changes: - Broaden + export isDurableObjectMemoryLimitReset(error): matches the shared "exceeded its memory limit" fragment so truncated/reworded surfacings observed in real #1825 logs still classify. Sibling to isDurableObjectCodeUpdateReset / isPlatformTransientError. - _executeScheduleCallback now DEFERS (re-throws) memory-limit resets for one-shot rows instead of swallowing them after in-process retries, so the error reaches the alarm-boundary breaker; track the executing row id so the breaker can purge the exact looping row. - think/ai-chat override _cf_recoveryAlarmCallbacks() and _cf_sealMemoryLimitedRecovery() to target their recovery continuation callbacks and terminalize active incidents (banner + onExhausted + seal). - Remove the redundant result-path OOM handling in continueLastTurn: those turns are already terminalized, so it only risked wasteful reschedules and duplicate terminal signals. Adds unit + integration coverage (predicate, listActiveChatRecoveryIncidents, alarm circuit breaker), an RFC follow-up section, docs, and changesets. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(chat-recovery): reset alarm OOM strike counter on a clean alarm The alarm-boundary memory-limit strike counter (maxAlarmMemoryLimitStrikes, #1825) is documented as counting CONSECUTIVE alarm OOM resets, but it was only ever deleted when the breaker sealed — never after a clean alarm — so it actually tracked LIFETIME resets. A Durable Object hitting rare, non-consecutive transient spikes (e.g. one a month) would eventually reach the strike budget and wrongly seal healthy recovery work. alarm() now best-effort clears cf_agents:oom_alarm_strikes after a clean _cf_runAlarmBody() so strikes must be consecutive to seal. The clear reads first and only writes when a strike is recorded, so the common no-strike path costs no write. Adds a regression test (strike recorded -> clean alarm resets to 0 -> next OOM starts at strike 1). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>