Sunil Pai

Recovery: e2e + unit coverage, and fix runFiber recovery starvation/backoff (#1729) * test(ai-chat,think): fix racy fiber-cleanup checks and add continue-path e2e The recovery e2e tests asserted `hasFiberRows() === false` the instant recovery was detected, racing the continuation/retry turn that recovery legitimately re-runs in a fresh fiber. Poll until the fiber rows settle instead. Also fixes the ai-chat e2e worker, which emitted the legacy `0:{json}` stream framing that AIChatAgent never parses (it reads `data:` SSE frames), so no chunk was ever persisted and recovery only ever saw an empty partial. Emit proper `data:` frames and stream enough chunks to cross the ResumableStream flush threshold, enabling a new continue-path test (non-empty partial -> resume the same assistant message). Co-authored-by: Cursor <cursoragent@cursor.com> * test(ai-chat): e2e coverage for chat recovery budget exhaustion Adds a deterministic exhaustion harness: agents whose turn hangs and produces no recovery progress, so repeated SIGKILLs drive the recovery budget without racing real streamed content. Covers onExhausted firing with reason no_progress_timeout, recovery_aborted, and work_budget_exceeded, plus the persisted terminal banner (#1645). Extracts the shared wrangler/WebSocket e2e plumbing into harness.ts. max_attempts (alarm-debounce forces >30s spacing) and stable_timeout (not feasibly deterministic in-process) are left to unit coverage. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ai-chat): e2e coverage for continue:false / persist:false recovery outcomes Interrupts a turn after a non-empty partial has flushed, then asserts the two onChatRecovery branches that suppress the default behavior: - { continue: false } persists the partial as a durable assistant message but does not re-run the turn (onChatMessage invoked once). - { persist: false, continue: false } drops a plain-text partial (no settled tool results) and does not re-run. Adds an onChatMessage invocation counter + assistant-text accessor to the test agent to distinguish "persisted partial" from a continuation. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): e2e for context-overflow compaction recovery Add ThinkContextOverflowE2EAgent plus an in-process (no process kill) e2e covering the opt-in contextOverflow recovery paths: reactive compact-and-retry that recovers a turn, reactive budget exhaustion that surfaces a terminal context_overflow error, and the proactive guard that compacts pre-step when reported usage crosses the headroom budget. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ai-chat): e2e coverage for stream-buffer cleanup alarm (#1706) and recovering-status broadcast (#1620) Add two deterministic e2e tests in @cloudflare/ai-chat: - #1706 stream-buffer cleanup alarm: new ChatBufferCleanupAgent exposes @callable inspectors (buffer/chunk row counts, _cleanupStreamBuffers schedule count, forced future sweep, hasReclaimableStreams). Asserts a completed turn arms exactly one cleanup alarm, a second turn does not stack a duplicate, and a forced future-now sweep reclaims all buffers so a fully-swept DO reports no reclaimable streams. - #1620 recovering-status broadcast: drives a SIGKILL/restart recovery of a slow-stream turn and asserts the durable cf:chat:recovering flag transitions active -> cleared (via a new getRecoveringFlag @callable), plus a live WS frame collector observes the cf_agent_chat_recovering clear broadcast. The durable flag is the deterministic source of truth because the live frame is not replayed on connect. Adds a createFrameCollector helper to harness.ts and registers ChatBufferCleanupAgent under a new v5 migration tag. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): e2e for durable-submission recovery on start Add ThinkSubmissionRecoveryE2EAgent plus an e2e covering the three _recoverSubmissionsOnStart transitions: messages-not-applied re-enqueues as pending, applied-but-unrecoverable surfaces as error, and a recoverable in-flight submission (real mid-stream SIGKILL) is left running and driven to completion by the scheduled continuation. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agents): arm follow-up alarm for pending runFiber recovery `_scheduleNextAlarm()` only rescheduled for active keepAlive leases, due schedules, and facet runs — never for orphaned `cf_agents_runs` rows or interrupted/pending managed ledger fibers still awaiting recovery. Because orphaned fibers hold no keepAlive ref, a scan that yielded on `fiberRecoveryScanDeadlineMs` (or a pass that retained a repeatedly-throwing unmanaged recovery hook) never got another alarm, so the remaining fibers starved. Add `_hasPendingFiberRecovery()` and arm a follow-up alarm whenever recovery work is outstanding, so multi-pass recovery resumes and eventually drains every fiber (and ages out poison rows via `fiberRecoveryMaxAgeMs`). Co-authored-by: Cursor <cursoragent@cursor.com> * test(agents): e2e coverage for poison-row aging, scan-deadline yield, and concurrent fiber recovery Add three runFiber recovery e2e tests (real `wrangler dev` + SIGKILL/restart against `--persist-to`): - poison-row aging: an unmanaged fiber whose `onFiberRecovered` always throws is retained for retry across alarm passes, then dropped with a `max_age_exceeded` skip once it exceeds `fiberRecoveryMaxAgeMs`. - scan-deadline yield: a tiny `fiberRecoveryScanDeadlineMs` forces a single alarm pass to yield (`scan_deadline_exceeded`) partway through 20 orphaned fibers; subsequent passes drain the rest with no starvation. - concurrent fibers: N concurrent fibers (mixed managed + unmanaged) are all recovered after a kill, covering the gap that prior tests only recovered a single fiber. New DO test agents record recovery signals (hook invocations + skip reasons) into durable SQL so assertions survive DO eviction between polls. Shared spawn/kill/RPC harness lives in `recovery-helpers.ts`. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): e2e for messenger reply-fiber recovery Add ThinkMessengerRecoveryE2EAgent plus an e2e covering MESSENGER_REPLY_FIBER_NAME recovery via _handleInternalFiberRecovery: a streaming-stage interruption posts the apology (apologize mode), and an accepted-stage interruption recovers in answer mode and re-drives reply delivery. Uses an in-memory fake chat adapter that records posts into agent SQL; full streamed-answer rendering is deferred (needs a complete adapter/real transport). Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): e2e for workflow-turn recovery + notification drain replay Add ThinkWorkflowRecoveryE2EAgent (reuses STEP_PROMPT_WORKFLOW with a deterministic mock structured model). Covers the happy path (structured workflow turn completes, notification drains, workflow resumes with the validated output) and the recovery path (mid-stream SIGKILL): on restart the turn is reconciled to a terminal submission and the workflow-notification drain replays it so the workflow is unblocked. Documents the deferred gap that an interrupted structured turn is recovered as skipped rather than completed. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): rename e2e callable to avoid Think.getWorkflow collision The workflow-recovery e2e agent's @callable shadowed the inherited Think.getWorkflow(workflowId) with an incompatible signature, failing typecheck. Rename it to inspectWorkflowRun. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agents): exponential backoff for the runFiber-recovery follow-up alarm The follow-up alarm added for pending fiber recovery fired every keepAliveIntervalMs with no backoff, so a repeatedly-throwing recovery hook — or a `fiberRecoveryMaxAgeMs: 0` ("retain forever") row whose hook keeps throwing — would wake the DO on every tick indefinitely (the perpetual-heartbeat hazard #1707 guards against). Track consecutive no-progress recovery scans and back the alarm off exponentially (capped at 5 min); any scan that recovers a fiber (including a scan-deadline yield that drained part of a batch) resets it, so legitimate multi-pass draining stays prompt. Adds e2e coverage: retain-forever poison-row backoff cadence, and multi-pass recovery for a sub-agent (facet) child driven by the parent alarm. Co-authored-by: Cursor <cursoragent@cursor.com> * test(agents): fast unit coverage for runFiber recovery alarm re-arm + backoff Adds deterministic, in-process unit tests (no process kill / timers) that drive `_checkRunFibers` + `_scheduleNextAlarm` directly and inspect the physical alarm: the starvation re-arm (alarm armed while a retained recovery row is pending), exponential backoff across no-progress scans, backoff reset on forward progress, and no alarm once recovery drains. Previously this behavior was only covered by the nightly e2e suite. Adds getCurrentAlarm/getRecoveryNoProgressScans/simulateAlarmCycle test helpers to the run-fiber test agent. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(agents): note the fiberRecoveryMaxAgeMs:0 warm-DO trade-off A repeatedly-throwing recovery hook with fiberRecoveryMaxAgeMs:0 ("retain forever") is retried on the capped backoff indefinitely, so the Durable Object never idle-evicts while the un-recoverable row exists. Document this in the option JSDoc and docs/durable-execution.md, and recommend a finite age. Bounding recovery by attempts is tracked in #1728. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

feat(agents): detached (background) agent-tool runs — durable completion, progress & milestones (#1758) * design: add RFC for detached (background) agent-tool runs Adds the design record for first-class detached sub-agent runs with a durable named-method completion hook and progress/milestone signaling, in response to cloudflare/agents#1752. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(agents): core detached agent-tool runs with durable completion hook Implements the core of rfc-detached-agent-tools (cloudflare/agents#1752): - `runAgentTool(cls, { detached })` dispatches a sub-agent without awaiting, returning `{ runId, status: "running" }`. Fire-and-forget (`detached: true`) or a durable per-run callback (`detached: { onFinish: "methodName" }`). - Durable, eviction-surviving completion delivery via a single guarded funnel with two independent ledger slots (finish / give-up) using a claim+lease, so delivery is exactly-once on the happy path and at-least-once under failure — a premature give-up can never dedupe a child's real late completion (the #1752 production incident). - Warm fast path (waitUntil) + durable self-scheduling reconcile backbone (this.schedule) that self-cancels once no detached run remains. - Reconcile fork: detached runs are never sealed `interrupted` on a lost observer (the normal state for a background run); the backbone owns them and re-arms on restart. - Absolute `maxBudgetMs` give-up ceiling (default 24h, finite because a detached run has no observer to notice a leak) surfaced as `interrupted`/`budget-exceeded`. - `cancelAgentTool(runId)` by-id cancellation through the same guarded path. Schema bumped to v10 with detached + ledger columns (idempotent migrations). Co-authored-by: Cursor <cursoragent@cursor.com> * test(agents): cover detached delivery ledger (exactly-once + two-slot) Drives the detached delivery funnel directly: exactly-once on terminal, dedupe under concurrent fast-path/backbone deliveries, and the independent finish/give-up slots so a budget give-up never dedupes a child's real late completion (#1752). Also switches the ledger claim to rowsWritten() since UPDATE ... RETURNING row counts are not a reliable claim signal on Workers SQLite. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(agents): document detached runs + inspectAgentToolRun null contract Adds a changeset (minor) and a "Detached (background) runs" section to the agent-tools doc covering the detached handle, durable onFinish, budget give-up, explicit cancellation, and the inspectAgentToolRun null-means-not-yet contract from #1752. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(agents): satisfy oxlint + oxfmt for detached changes Co-authored-by: Cursor <cursoragent@cursor.com> * feat(think): detached runAgentTool notify convenience Adds `detached: { notify: true }` sugar: when a detached sub-agent run finishes, a Think agent injects the result back into the chat via submitMessages (idempotent per run + status) so the model reacts, without hand-wiring onFinish. Override formatDetachedCompletion() to customize. Wired generically in the base Agent by resolving the conventional notify hook by name so the core stays decoupled from the chat layer. Co-authored-by: Cursor <cursoragent@cursor.com> * example(agents-as-tools): demonstrate detached background runs Adds a `research_background` tool that dispatches a Researcher with `detached: { notify: true }` (returns immediately, result posted back into the chat on completion) and a `cancelBackground(runId)` callable built on cancelAgentTool. Updates the system prompt and README to cover the background flow. Co-authored-by: Cursor <cursoragent@cursor.com> * test(agents): bump expected schema version to 10 for detached columns Co-authored-by: Cursor <cursoragent@cursor.com> * feat(agents): ephemeral progress + durable milestones for agent-tool runs Adds two complementary signaling channels to long-running agent-tool runs, plus a round of detached-run hardening from the deep review pass. Progress (4a) — ephemeral: - `reportProgress({ fraction, message, phase, data? })` emits transient `data-agent-progress` parts; coalesced, latest-snapshot-only, persisted as `progress_json`. Surfaced via `AgentToolRunState.progress`, the `onProgress` hook, and `inspectAgentToolRun`. - Shared `AgentToolProgressEmitter` centralizes coalescing/persistence so Think and AIChatAgent share one code path; `forget(runId)` clears coalescing state on terminal cleanup (no per-run map leak). - No-progress budget (`detachedNoProgressBudgetMs`) gives up a detached run that reports initial activity then goes silent; resets on any progress/milestone. Milestones (4b) — durable: - `reportProgress({ milestone, data })` persists an ordered, observable record (new `cf_*_agent_tool_milestones` tables), broadcasts a `data-agent-milestone` part (never coalesced), and surfaces `AgentToolRunState.milestones`. - `detached: { onMilestones }` injects an at-most-once chat notification on both the warm path and the reconcile cold path. Two modes — the `string[]` shorthand defaults to "narrate": - "narrate" (default): synthetic assistant message, no model turn — a cheap status line for milestones the agent needn't act on. - "react": user-role turn so the model responds (steer / start dependent work). Costs a turn. Override prose via `formatDetachedMilestone()`. Client/UX: - Reducer projects milestones (deduped by sequence) and advances the progress snapshot monotonically. Synthetic notify messages render as "Agent event" with milestone/result badges instead of a raw user/assistant bubble. - agents-as-tools example emits a "sources-gathered" milestone and uses the narrate shorthand; tray renders progress + milestone chips. Detached-run hardening (review pass): - Terminal `agent-tool-event` is always broadcast on cancel (synthesize seq). - Generic `onFinish` delivery serialized against the turn queue via `_runDetachedDelivery` (Think/AIChatAgent enqueue on `_turnQueue`). - `_armDetachedBackbone` race guarded with an arming mutex. - Failed detached deliveries re-throw and emit `agent_tool:detached:delivery_failed`; accumulating live runs emit `agent_tool:detached:live_count_warning`. Schema bumped to 11. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(think): explain why submission drain uses allowNested Document at the call site that the submission drain runs fire-and-forget, so a nested submission turn (e.g. a detached-finish notify calling submitMessages mid-turn) is admitted safely without deadlock — and why the flag is applied to all submissions rather than scoped to detached notify. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(ai-chat): detached notify + onMilestones parity with Think AIChatAgent now implements the parent-side detached chat conveniences that were previously Think-only, so `detached: { notify }` and `detached: { onMilestones }` work on an AIChatAgent parent instead of being silently dropped (the docs and the base _deliverDetachedMilestone JSDoc already claimed support). - formatDetachedCompletion + _cfDetachedNotifyFinish (auto-wired by name) inject the completion as a user turn the model reacts to; idempotent per (runId, status) via a deterministic message id. - formatDetachedMilestone + _deliverDetachedMilestone override deliver milestone notifications: "narrate" (default) persists an assistant line with no model turn; "react" injects a user turn + reply. Idempotent per (runId, name). AIChatAgent has no durable-submission layer and TurnQueue has no re-entrancy bypass, so the shared _injectDetachedNotification helper avoids self-deadlock by dispatching the react turn three ways: - inside our own serialized finish-delivery slot -> run inline + awaited (the ledger only marks delivered after it resolves, so the reaction is eviction-safe); - inside a FOREIGN active turn (e.g. cancelAgentTool mid-turn) -> fire-and-forget a turn that runs once the slot frees (inline would interleave, enqueue-await would deadlock); the persisted message makes it best-effort-durable; - otherwise -> enqueue + await normally. An in-flight id set dedupes a concurrent warm-tail + backbone delivery of the same milestone within one isolate (milestones have no ledger claim), while messages.some() dedupes re-delivery against persisted history across eviction. Tests: ai-chat detached-notify covers notify react + idempotency, narrate (no model turn), and react milestones. Docs/changesets/RFC updated to reflect that notify + onMilestones are chat-host conveniences (Think + AIChatAgent). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com>

Harden Think and AIChat recovery foundations (#1611) * test: cover Think recovery failure surfaces Add characterization coverage for the production recovery gaps before changing runtime behavior. These tests capture the current failure surfaces so the follow-up fixes can prove that they improve behavior rather than only reshaping APIs. The Think recovery e2e harness now covers repeated restart churn around an interrupted turn, the post-persist/pre-turn request failure path, and parent agent-tool recovery interruption after restart. The test worker persists recovery observations across restarts and exposes test-only controls for forced turn failures and retained agent-tool rows. The focused regression tests also document two non-restart reliability issues: poisoned transcripts with orphan tool calls fail later turns, and createCompactFunction can skip summarization when tool-heavy histories are under-counted by the heuristic tail budget. This commit intentionally contains only tests and test harness changes. Runtime recovery behavior is left unchanged for the subsequent fix commits. Co-authored-by: Cursor <cursoragent@cursor.com> * test: cover AIChatAgent recovery failure surfaces Mirror the Think restart-churn characterization coverage in the AIChatAgent recovery e2e harness before changing shared recovery behavior. Both chat packages use the same underlying fiber recovery pattern, so the shared fix work needs coverage that proves AIChatAgent receives the same protection rather than only improving Think. The e2e worker now persists recovery observations in Durable Object storage so restart churn does not lose the evidence we need to assert on. The new e2e test starts a slow recovered chat turn, repeatedly kills and restarts wrangler around the interrupted fiber, and verifies recovery still fires and stale fiber rows are cleaned up. This commit is intentionally limited to characterization coverage. It excludes Think-only concerns such as Session compaction and durable submissions, which do not apply to AIChatAgent. Co-authored-by: Cursor <cursoragent@cursor.com> * feat: add recovery observability channels Introduce dedicated observability event types and diagnostics channels for the recovery work that follows. Fiber recovery, chat recovery, transcript repair, and agent-tool reconciliation now have stable event names instead of being mixed into unrelated message or lifecycle streams. The channel routing tests lock in the public names for the new channels and ensure chat:transcript:* events land on the transcript channel while chat recovery events stay on the chat channel. Co-authored-by: Cursor <cursoragent@cursor.com> * feat: bound and observe fiber recovery Make the generic fiber substrate safer before adding chat-specific retry policy. Recovered fibers now carry a recoveryReason, interrupted run rows emit fiber recovery lifecycle events, and internal framework recovery hooks are bounded by a default timeout so startup cannot wedge forever on a broken internal recovery path. Managed fibers whose recovery hook throws are now marked terminal error instead of staying indefinitely interrupted with only an error message attached. Unmanaged run rows continue to be pruned after recovery handling so the same broken stale row does not re-trigger forever across boots. This also exposes the agentTool observability channel under the camelCase subscribe key while preserving the diagnostics channel name agents:agent_tool. Co-authored-by: Cursor <cursoragent@cursor.com> * feat: add bounded chat recovery incidents Add shared chat recovery configuration and incident context so Think and AIChatAgent recovery are bounded by framework-owned attempt state. Recovery hooks now receive incidentId, attempt, maxAttempts, and recoveryKind, with attempt === 1 representing the first handling of an incident. Both chat runtimes persist recovery incidents in Durable Object storage, emit chat recovery lifecycle events, use configurable stable-state timeouts, and stop scheduling more recovery work once maxAttempts is exceeded. Exhaustion emits a terminal chat recovery event and sends a user-visible terminal chat error frame; Think also marks any matching running submission as error. The default behavior is enabled for existing chatRecovery=true users with maxAttempts 6 and stableTimeoutMs 10000, while custom policy can be supplied via chatRecovery object configuration. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: surface Think chat request failures Extend Think's existing onChatError hook with optional stage context and route post-persist request failures through it. The request path now emits chat:request:failed with request id, stage, persistence state, and sanitized error text before sending the terminal chat error frame. This gives applications a server-side hook for failures that occur after user messages have been accepted but before or during the model turn, without routing those chat failures through the generic Agent onError hook. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: bound agent-tool recovery scans Add a total recovery deadline for parent-side agent-tool reconciliation so a restart cannot spend unbounded time inspecting stale child rows. Per-child inspection remains bounded, and any rows reached after the total deadline are terminalized as interrupted with an explicit recovery-deadline error. The recovery loop now emits structured agent_tool recovery lifecycle events for begin, per-row outcome, deadline, completion, and unexpected failure, giving operators visibility into exactly which child run was finalized and why. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: repair poisoned chat history before model calls Repair incomplete tool-call transcript shapes before Think converts UI messages for provider calls. Persisted orphan tool calls are stripped from the active Session history, clients are rebroadcast with the repaired transcript, and a chat:transcript:repaired event records the repair counts and tool call ids. Also let createCompactFunction use an optional tokenCounter for tail-budget selection. Callers with tokenizer or model-reported accounting can avoid the heuristic under-count that otherwise protects too much tool-heavy history and skips summarization. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: add recovery foundation changeset Record the public recovery API, observability, and behavior changes across the published packages so the release notes explain the new default recovery bounds and transcript/compaction fixes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: export chat recovery lifecycle types Expose the new chat recovery config and exhaustion context types from agents/chat so downstream packages can typecheck against the shared lifecycle surface. Update the Think test fixture to construct the enriched fiber recovery context. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: harden recovery edge cases and docs Stabilize chat recovery incident accounting across retry fibers, preserve failed internal recovery rows for later scans, and avoid stale continuation targets after orphan persistence. Tighten transcript repair behavior and document the new recovery and observability surfaces so users can configure and debug bounded recovery. Co-authored-by: Cursor <cursoragent@cursor.com> * docs: clarify recovery UX surfaces Document the distinction between stream resumption and durable chat recovery so users can configure and observe recovery behavior correctly. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: persist normalized transcript repairs Track length-preserving part replacements so repaired tool inputs are written back before provider calls. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: remove dead transcript repair counter Align transcript repair observability with AI SDK v6 message parts by dropping the unused removedToolResults field. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: harden chat recovery incident bounds and fiber retry limits Follow-up hardening from a full review of the recovery foundation. These close edge cases where bounded recovery could still leak storage, exceed its attempt budget, or fail to surface a stuck turn. Chat recovery incidents (Think + AIChatAgent): - Drop `recoveryKind` from the incident identity so a single interrupted turn that flips between `retry` (no chunks persisted) and `continue` (partial chunks exist) across restarts shares one attempt budget. The kind is still tracked as a mutable field on the incident record. - Delete the incident record on `completed` (success) and add a TTL sweep (1h inactivity) on each new incident so durable storage no longer grows without bound. Exhausted/failed/skipped records are retained for inspection until they age out. - Guard a throwing `onExhausted` hook so the terminal error frame (and Think's running-submission interruption) is always delivered. - Wrap the post-begin recovery dispatch (onChatRecovery, orphan persist, scheduling) so any throw flips the incident to a terminal `failed` state and emits `chat:recovery:failed` instead of leaking in `attempting`. Generic fiber recovery (Agent): - Add `fiberRecoveryMaxAgeMs` (default 24h). A repeatedly-throwing unmanaged `onFiberRecovered()` row is still retried while fresh but is evicted with a `fiber:recovery:skipped` / `max_age_exceeded` event once it ages out, so a poison row cannot re-trigger forever across boots. - Note that the fiber recovery hook timeout bounds the wait but does not cancel the underlying internal operation. Observability: - Replace the hand-coded `agentTool` subscribe special-case with a `CHANNEL_DIAGNOSTIC_NAME_OVERRIDES` lookup to prevent future drift between camelCase keys and snake_case diagnostics channel names. Tests: - Think + AIChatAgent: shared attempt budget across retry/continue flip, incident deletion on completion, stale incident sweep, `failed` transition when onChatRecovery throws, and terminal UX delivery when onExhausted throws. - Agent fibers: a fresh throwing unmanaged row is retained (retryable), an aged throwing row is evicted with `max_age_exceeded`. - Update existing incident-id assertions to the kind-less format. All unit and e2e suites pass (Think, AIChatAgent, Agent fiber recovery, observability routing). Co-authored-by: Cursor <cursoragent@cursor.com> * docs: document fiber recovery max-age bound Clarify that an `onFiberRecovered()` hook which always throws is retried only until the row exceeds `fiberRecoveryMaxAgeMs` (default 24h), after which it is discarded with a `fiber:recovery:skipped` / `max_age_exceeded` event. Previously the bullet implied a thrown hook kept the row indefinitely. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

Add first-class Agent Skills (`agents/skills`) with Think integration (#1584) * Add first-class Think agent skills Introduce a first-class Agent Skills integration for Think so agents can declare skill sources with getSkills(), advertise a compact skill catalog in the session prompt, and let the model activate matching skills through dedicated tools instead of generic Session context loading. This adds the core SkillSource model, manifest/frontmatter helpers, a SkillRegistry, and Think wiring for activate_skill and read_skill_resource. Skill fingerprints are stored in Think config and used to refresh cached prompts when bundled or runtime skill catalogs change. Skill names now fail fast on duplicates across sources so ambiguous registrations do not silently pick the first provider. Add bundled local skills support to agents/vite via import attributes such as import skills from "./skills" with { type: "skills" }. The Vite plugin parses SKILL.md YAML frontmatter, bundles allowed Agent Skills resource directories, emits a deterministic manifest-backed SkillSource, and keeps resource collection limited to references/, scripts/, and assets/ to avoid leaking unrelated local files. Add an R2-backed Think skill source with skills.r2(bucket, options). It reads standard Agent Skills directory layouts from R2, parses SKILL.md files, exposes resource descriptors, lazily fetches individual resources, supports metadata or content fingerprinting, and refreshes mutable bucket indexes on an interval so prompt catalogs can update without restarting the Durable Object. Add an agent-skills example demonstrating bundled skills, including release notes, debug planning, brand voice, and pirate voice skills. Document the design decisions in design/skills.md and add focused tests for parsing, manifest sources, registry behavior, R2 directory discovery, resource reads, duplicate detection, and fingerprint refresh behavior. Also make the affected Think and AI Chat vitest worker suites retry consistently to reduce transient worker test flakes. Co-authored-by: Cursor <cursoragent@cursor.com> * Expand Think skills with script execution Add first-class skill script execution to Think so agent skills can bundle and run task-specific helper scripts alongside SKILL.md instructions. This introduces the run_skill_script tool, a SkillScriptRunner contract, and a workerScriptRunner implementation with explicit capability boundaries for workspace, tools, network, and timeouts. Support JavaScript and TypeScript scripts through the existing DynamicWorkerExecutor path, using @cloudflare/worker-bundler to compile TypeScript skill scripts before execution. Add Bash support through just-bash and Python support through Python Dynamic Workers with a host bridge for tool calls and workspace access. Tighten the runtime UX and safety model by validating script paths, restricting executable resources to scripts/ with supported extensions, defaulting omitted script input to {}, defaulting script timeout to 30 seconds, and granting read-only workspace access when a workspaceInstance is provided while keeping writes, network, and tools as explicit opt-ins. Update the agent-skills example to demonstrate bundled skills, the Vite skills import attribute, worker_loaders, TypeScript/Python/Bash release-note scripts, and the simplified default runner configuration. Refresh README and design documentation so the first-class Think skills API is the recommended path going forward. Add focused test coverage for skill registry script tools and worker script execution across TypeScript, Bash, and Python, including context/input handling, tool invocation, workspace read/write permissions, validation failures, and script error surfacing. Co-authored-by: Cursor <cursoragent@cursor.com> * Update Think skills RFC status Mark the implemented skills MVP as complete and leave Git-backed sources plus R2 write/delete helpers as explicit follow-up work. Co-authored-by: Cursor <cursoragent@cursor.com> * Improve Think skill resource and script compatibility Broaden Agent Skills compatibility by making bundled and R2 resources binary-safe, adding encoding and MIME metadata to resource descriptors, and supporting qualified cross-skill resource reads. This lets skills expose non-text assets without corrupting content while giving model-visible tools clearer metadata and diagnostics. Tighten resource path handling across manifest, R2, and script execution so malformed paths cannot escape the intended skill resource namespace. R2 content fingerprints now hash binary resources through base64 content instead of lossy text decoding, preserving correctness when resources include images, fonts, PDFs, or other binary files. Make skill script execution friendlier to CLI-style skills without committing to a JavaScript filesystem shim yet. Python and Bash scripts receive /input.json, /context.json, and mounted /skill resources, while JavaScript and TypeScript keep top-level execution and function-style compatibility. TypeScript and JavaScript scripts can now import sibling script resources because worker-bundler receives all text script files and bundles multi-file script packages when needed. Update the agent-skills example, README, design note, and changeset to describe the new resource behavior, script defaults, and the deferred JavaScript filesystem compatibility design. Add regression coverage for binary resources, unsafe resource paths, qualified reads, sibling script imports, Python CLI-style scripts, and CPU-bound Python timeouts. Co-authored-by: Cursor <cursoragent@cursor.com> * Add skill script filesystem compatibility Add a stable worker-bundler virtualModules option so callers can provide generated modules for exact import specifiers without reaching for esbuild plugin internals. This lets framework integrations alias modules like node:fs, fs/promises, or other virtual runtime APIs while preserving the existing virtual filesystem resolver and transform-only warnings. Use that capability in Think skill scripts to provide a partial fs/node:fs/path compatibility layer for JavaScript and TypeScript skills. Skill-local files, input, and context can be read synchronously; workspace access remains async-only through fs.promises because it crosses the host Worker boundary; /output writes are returned as scratch artifacts instead of mutating durable workspace state; and /workspace writes require explicit read-write workspace permission. Keep the authoring model aligned with Agent Skills by continuing to bundle sibling script imports through worker-bundler, supporting both static fs imports and dynamic import("node:fs"), and preserving the function-style run(input, ctx) compatibility path while surfacing any output artifacts it writes. Document virtualModules in worker-bundler, update Think and skills design docs for the partial filesystem contract, and extend the agent-skills example with a bundled release-notes style guide read via node:fs. Add regression coverage for virtual module aliases, JS/TS fs reads and writes, output artifacts, workspace permission boundaries, dynamic imports, and Node-like workspace readdir/stat behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * Polish Think skills runtime and docs Tighten the skill script runtime before PR by making Bash resource handling and exit semantics match the path-based script contract, adding Python /output artifact collection, and improving dev-mode invalidation for bundled skill imports. This also expands Think skills documentation so the public docs explain skill sources, tool exposure, script execution requirements, and runtime dependencies instead of leaving the feature mostly in examples and package README text. Co-authored-by: Cursor <cursoragent@cursor.com> * Add default Think workspace bash tool Expose a sandboxed bash tool from Think's built-in workspace tools so agents can use shell-style workflows for multi-file operations without each app wiring its own executor. The tool mounts a bounded snapshot of the workspace into just-bash, runs with network disabled by default, and syncs created, updated, deleted, and empty-directory changes back to the durable workspace. Make the write-back path conservative: directory snapshots are paginated, oversized or unreadable files are reported as skipped and treated as protected paths, /tmp and system-like paths are ignored for new-file sync, and write/delete failures are returned as structured tool errors instead of being reported as successful changes. Preserve binary writes when the workspace supports writeFileBytes, fall back to text writes when safe, and return structured stdout, stderr, exitCode, changedFiles, skippedFiles, and errors for both successful and failed bash execution. Add workspaceBash as an opt-out/option property on Think and keep createWorkspaceTools typed with a concrete WorkspaceTools shape so callers can discover the optional bash tool cleanly. Document the default behavior, snapshot limits, tuning options, and opt-out path in the Think README and docs. Cover the new behavior with assistant tool tests for persisted file changes, non-zero exits, skipped-file protection, empty directory sync, paginated snapshots, and structured timeout/error output. Verified with: - npm run check - npx vitest --run -c packages/think/src/tests/vitest.config.ts packages/think/src/tests/assistant-tools.test.ts packages/think/src/tests/skill-runner.test.ts Co-authored-by: Cursor <cursoragent@cursor.com> * Harden Think skills and promote the engine to `agents/skills` This builds on the initial first-class Think skills work, hardening the runtime, simplifying the script API, and relocating the engine so it is framework-agnostic rather than Think-specific. Engine moved to `agents/skills` - Move the skills engine (types, frontmatter parser, `SkillRegistry`, the bundled-manifest and R2 sources, and the script runner) from `@cloudflare/think` into a new framework-agnostic `agents/skills` export. The runner depends only on a local structural `SkillWorkspace` interface, so the engine no longer couples to Think or `@cloudflare/shell`. - `@cloudflare/think` now re-exports the engine as `skills` (so `import { skills } from "@cloudflare/think"` is unchanged) and keeps only the integration wiring: `getSkills()`, `getSkillScriptRunner()`, the Session catalog context block, and the fingerprint refresh. - Any AI SDK caller (including `@cloudflare/ai-chat` in `onChatMessage`) can now build a `SkillRegistry` and merge `registry.tools()` + `registry.systemPrompt()` directly. - Move the skill/runner/r2 tests into `packages/agents/src/tests` and add the `LOADER` worker-loader binding to the agents test worker so script execution runs in the agents workers pool. Import API: `agents:skills` specifier - Replace the `import x from "./skills" with { type: "skills" }` import attribute (and its per-project `.ts` type shim) with an explicit `agents:skills` virtual specifier resolved by `agents/vite`. The path is optional and defaults to a `./skills` directory next to the importer; `agents:skills/<dir>` targets a sibling directory. - Ship ambient types from `agents` (`skills-module.d.ts`), referenced from the built `dist/index.d.ts`, so importing `agents` (directly or via `@cloudflare/think`) types the specifier with no per-project shim. Graceful skill loading (never throw in the turn path) - `SkillRegistry` skips duplicate skill names (first source wins) and sources that fail to list, recording diagnostics in `warnings` instead of throwing; warnings reset each load. `Think` wraps init/refresh and logs warnings deduped by message. The Vite plugin warns on duplicate bundled names at build time. Experimental, simpler script runner - Rename `skills.workerScriptRunner` to `skills.runner`, flag it `@experimental`, and log a one-time warning on first use. - JS/TS scripts are now function-style only: `export default run(input, ctx)` with `ctx = { skill, files, workspace, tools, output }`. Removed the hand-rolled `node:fs`/`path` compatibility shim; bundled text resources are exposed via `ctx.files` and scratch artifacts via `ctx.output.writeFile`. - Unify capabilities and permission enforcement behind a single `SkillScriptHostBridge`, constructed fresh per `run()` so `/output` artifacts never leak between concurrent runs. JS providers, Python RPC, and Bash commands all delegate to it. Python and Bash keep the path-based `/skill` / `/input.json` / `/output` contract. Bundled asset guardrails - The Vite plugin warns when a bundled skill asset (or the total) exceeds size thresholds and recommends `skills.r2()` for large assets. Example (`examples/agent-skills`) - Switch to `import bundledSkills from "agents:skills"` and `skills.runner`, delete the type shim. - Keep script execution TypeScript-only (function-style, reading the style guide from `ctx.files`); drop the Python and Bash demo scripts. - Replace the `brand-voice` persona skill with a procedure-style `test-plan` skill. - Render skill tool activity (`activate_skill` / `run_skill_script`) inline and light up activated skills in the sidebar. Docs and changesets - Update `docs/think/index.md`, `packages/think/README.md`, `design/skills.md`, and `packages/agents/AGENTS.md` for the new home, specifier, function-style `ctx` API, and ordering/first-source-wins semantics. - Split the changeset: skills bumps `@cloudflare/think` + `agents`; the `worker-bundler` `virtualModules` option gets its own changeset. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix review nits: R2 fingerprint boundaries, Think deps, tool-merge docs - r2.ts: fold a part boundary into `stableHash` so different catalogs whose concatenated metadata/content streams would otherwise be identical (e.g. ["ab","cd"] vs ["abcd"]) now hash differently, preventing a missed catalog refresh. - think/package.json: `@cloudflare/codemode` and `@cloudflare/shell` were listed in both `dependencies` and `peerDependencies` (with codemode also flagged optional). They are required at runtime, so keep them as plain dependencies and drop the contradictory peer/optional entries. - docs/think/index.md: rewrite the dependency table — split provided peers (agents/ai/zod/telegram) from bundled deps (shell/codemode/just-bash), drop the stale `@cloudflare/worker-bundler` row, and note the skills engine lives in `agents/skills`. - docs/think/tools.md: correct the tool merge order to match code (extension tools before session tools) and add the missing skill-tools entry. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

fix(think,ai-chat,agents): harden recovery, transcript integrity & compaction under deploy churn (#1623) * fix(think,ai-chat): stop recovery falsely erroring a turn under repeated mid-turn deploys Under repeated real `wrangler deploy`s mid-turn, chat recovery runs a chain of continuations. Three bugs combined to mark a turn's durable submission `error` even when it actually completed every step (validated end-to-end with the deploy-churn harness + a recovery trace): 1. Lost ownership: the submission link (`recoveredRequestId`) was derived from each continuation's own fresh requestId, so chained continuations dropped it and the continuation that finally completed the turn could not mark the submission `completed`. Now keyed off the stable recovery root and threaded through the whole chain. 2. Stale-continuation clobber: a superseded continuation tripped the `conversation_changed` guard because the leaf had advanced via recovery's own forward progress (a new assistant message), not a new user turn, and overwrote the still-running submission to `error`. Now a superseded continuation skips benignly; only a genuinely newer user turn marks the submission `skipped` (never `error`). 3. Premature stable_timeout: a timeout while waiting for the isolate to settle (common while a deploy is in flight) failed the turn terminally at attempt 1. Now it reschedules within the `maxAttempts` budget. `@cloudflare/ai-chat` shares the recovery machinery but has no durable-submission layer, so it receives only the stable_timeout reschedule fix (mirrored in both `_chatRecoveryContinue` and `_chatRecoveryRetry`). Tests: 7 deterministic unit tests (5 in think, 2 in ai-chat) covering chained ownership, benign superseded skip, newer-user-turn -> skipped, stable_timeout reschedule within budget, and exhaustion. think 441 / ai-chat 475 green. Co-authored-by: Cursor <cursoragent@cursor.com> * test(deploy-churn): add tool-result rollback harness for real-deploy recovery testing Extends the deploy-churn example to drive a long, tool-using session via HTTP (no browser) against a REAL model (Workers AI or Anthropic) while firing real `wrangler deploy`s mid-turn, and measures whether completed tool calls re-run or the durable submission is wrongly errored. - `recordStep` tool: one ledger row per execution, so a re-run of a completed step shows up as a duplicate index (the "rollback" signal). - provider switch (workers-ai | anthropic) stored in a SQL config table so getModel()/getTools() observe it on a fresh post-deploy isolate. - `/drive/start|status|reset` HTTP routes driving `submitMessages` + ledger. - `scripts/deploy-rollback.ts` orchestrator: real deploys during the session, then a CLEAN / MINIMAL / ROLLBACK verdict plus submission status. This reproduced and validated both the #1621 tool-result durability fix and the recovery submission-status fix in the preceding commit. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think,ai-chat): reschedule stable-timeout recovery on a fresh schedule row The stable-timeout retry added in the previous commit used `schedule(..., { idempotent: true })` from INSIDE the currently-executing one-shot `_chatRecoveryContinue`/`_chatRecoveryRetry` schedule row. Because `alarm()` deletes that one-shot row only AFTER the callback returns, the idempotent reschedule deduped onto the still-present executing row and was then deleted with it — so the retry silently never fired and the turn STALLED (incident frozen at `stable_timeout_retry`, submission stuck `running`). A real-deploy repro with a 12s tool reproduced this: the turn stalled at 4/8. With the reschedule switched to a fresh (non-idempotent) delayed row it now completes 8/8 under the same churn. The unit tests previously passed only because they drive the callback directly (no executing row to dedup against). They now pre-insert a matching schedule row to simulate the executing one-shot, and assert the reschedule creates a NEW row. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think,deploy-churn): rollback-depth + task-amplification repros under churn Adds rigorous reproductions used to characterize recovery behavior under deploy churn (all show the framework is BOUNDED — re-runs at most the in-flight step, no deep rollback, no task amplification): - think e2e `tool-rollback.test.ts` + `ThinkToolRollbackE2EAgent`: a long deterministic tool loop with a non-idempotent ledger, rapid SIGKILL/restart; measures rollback DEPTH (re-runs vs evictions). - think e2e `task-amplification.test.ts` + `ThinkTaskParentE2EAgent`: a parent `runTask` tool driving a child agent; verifies an eviction mid-task does NOT re-run the whole child turn. - deploy-churn: configurable per-tool delay (`--delay-ms` / `delayMs`) so a real ~33s `wrangler deploy` lands DURING a tool execution (code-update reset mid-tool). This repro surfaced the stable-timeout reschedule stall fixed in the previous commit. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): assert re-reconstructing an interrupted stream is idempotent Pins the property that protects against the "disappearing/duplicated completed tool calls" failure mode under churn: re-running recovery on the same interrupted stream (e.g. a second eviction during the persist window) replaces the reconstructed assistant message by its stable id (taken from the stream's `start` chunk) rather than appending a duplicate or losing it. Verified the content is preserved across two recovery passes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): preserve interrupted tool calls as errored instead of deleting them `_repairToolTranscriptParts` deleted any tool call with no recorded output before the next turn. For a tool interrupted mid-execution (deploy/eviction) or an `ask_user` answered by the next message, that: - removed the call from the durable + broadcast transcript (it visibly "disappeared" — the exact symptom in the customer's deploy-churn video), and - let the model silently re-run it, duplicating non-idempotent side effects. Now the orphan is flipped to `state: "output-error"` with an explanatory message: the record is preserved, the model is told the tool errored (so it doesn't blindly re-run it), and conversion still gets a valid tool-result so the provider doesn't 400 with AI_MissingToolResultsError. Stringified `input`s are normalized in the same pass. Blast radius was a single test (which asserted the old deletion); it now asserts preservation. Full think suite green (442). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): ignoreIncompleteToolCalls at convert as a last-line backstop After `_repairTranscriptForProvider` heals orphan tool calls (preserving them as errored results), pass `ignoreIncompleteToolCalls: true` to `convertToModelMessages` so any incomplete tool call that still slips through (compaction edges, addToolOutput races, unrecognized part shapes) is dropped at conversion instead of throwing AI_MissingToolResultsError and wedging the turn. No-op in the common path (the repair runs first); verified no test churn. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agents): flow Session tokenCounter into compaction boundary logic (#1593) A tokenCounter set on Session.compactAfter() only drove the fire/no-fire trigger; createCompactFunction's tail-budget boundary still used the chars/4 heuristic. On tool-heavy histories that under-counts ~4-5x, so the tail budget covered the whole history and compaction fired every turn but returned null — never shortening history (worse than not configuring it). The Session now passes its counter to the compaction function via a new CompactContext argument; createCompactFunction uses it for the tail walk when no explicit CompactOptions.tokenCounter was given. One counter on compactAfter() now drives both "should we compact?" and "what should we compact?". If the trigger fires but compaction still returns null, the Session logs a one-time warning instead of looping silently. CompactFunction gains an optional second context?: CompactContext arg (backward compatible). Session suite green (74). Co-authored-by: Cursor <cursoragent@cursor.com> * compaction: re-arm no-op warning on success + document per-message counter caveat - Reset the one-time auto-compaction no-op warning when compaction succeeds, so a later regression is surfaced again instead of staying silent. - Document that the Session counter flowed into createCompactFunction is invoked per-message: usage-only counters degrade the tail budget to minTailMessages, and the counter runs O(n) per compaction. Recommend an explicit per-message CompactOptions.tokenCounter for precise budgeting. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: format changeset files with oxfmt * fix(think): give getScheduledChatRecoveryPayloadForTest a serializable return type A `Record<string, unknown>` return collapses to `never` across the Durable Object RPC stub boundary (Workers RPC drops `unknown`-valued records as non-serializable), so the chained-continuation test saw `payload` as `never` and failed typecheck. Return the concrete recovery-link fields instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): default a missing tool input to {} during transcript repair Transcript repair already parsed stringified-JSON tool inputs, but a tool call with a missing or null `input` was left unrepaired — Anthropic rejects a `tool_use` block whose `input` is absent, so the turn 400s forever. A new `_normalizeToolInput` helper now also defaults a missing/null input to `{}` on both the orphan-healing and settled-part paths. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(agents): structured retryable failure envelope for agentTool() agentTool() collapsed every non-completed sub-agent run to an opaque { ok: false, error: string }, so a parent agent could not tell a transient interruption (child reset/superseded by a deploy or parent recovery) apart from a terminal failure or an intentional cancellation — and would often parrot the interruption text to the user as final. Failures now return AgentToolFailure { ok: false, status, error, retryable }: interrupted -> retryable: true (and surfaces the interruption reason), while aborted and error -> retryable: false. Backward compatible for consumers reading ok/error; AgentToolFailure is exported from `agents`. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(think): opt-in inactivity watchdog for the streaming read loop A model stream that parks without ever throwing (no chunk, no error, no `done`) left the chat read loop waiting forever — an infinite spinner with no terminal state. There was no detection for a silently hung turn. Add `chatStreamStallTimeoutMs` (default 0 = off): if no UI-message-stream chunk arrives within the window, the watchdog aborts the turn so the loop exits with a terminal stream error (routed through onChatError stage "stream") and emits a new `chat:stream:stalled` observability event. Applies to both the WebSocket turn loop and the chat()/sub-agent callback loop. The watchdog aborts the turn's signal (not a reader cancel) so the AI SDK pipeline tears down without writing to an already-cancelled readable; the abandoned read's rejection is pre-caught to avoid an unhandled rejection. It measures inter-chunk inactivity (which includes tool execution), so it must be set above the slowest expected model TTFT and tool latency. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(think): address PR #1623 review — dedup reschedule, retry-path test, sharper diagnostics - Extract the stable-timeout reschedule into a shared `_rescheduleRecoveryAfterStableTimeout` helper (mirrors @cloudflare/ai-chat), removing the two inlined near-identical copies in `_chatRecoveryRetry` / `_chatRecoveryContinue` so a future fix can't diverge between them. - Add a `_chatRecoveryRetry` stable-timeout reschedule test (the path previously only covered for `_chatRecoveryContinue`). - Surface when an incomplete tool call survives transcript repair and is about to be dropped by `ignoreIncompleteToolCalls` (warns + emits), so the backstop can't silently mask a repair gap. - Make the compaction no-op warning distinguish a per-message vs whole-prompt (usage) tokenCounter, since "configure a tokenCounter" was misleading when one was already configured. - Document the single-field `_activeChatRecoveryRootRequestId` serialization invariant (safe only because turns are serialized by the turn queue). Co-authored-by: Cursor <cursoragent@cursor.com> * docs: align with PR #1623 behavior changes - chat-agents: transcript repair now heals orphaned tool calls (preserved as errored results) and normalizes malformed/missing inputs — was "removing". - observability: add the new chat:stream:stalled event (agents:chat channel) and clarify chat:transcript:repaired counts (preserved-as-errored + backstop). - think README: document the opt-in chatStreamStallTimeoutMs inactivity watchdog. - agent-tools: document the AgentToolFailure shape and retryable semantics. - sessions: note the compactAfter tokenCounter now also drives the boundary walk (CompactContext), with the per-message/usage-counter caveat. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(think): add chatStreamStallTimeoutMs to the docs-site config reference Mirrors the package README so the developers.cloudflare.com Think config table (the target of the observability chat:stream:stalled link) documents the new inactivity watchdog. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): cancel the source stream when the stall-watchdog wrapper exits early The inactivity-watchdog generator wraps the model stream, but on an early consumer exit (a `break` on an in-band stream error, where the abort signal is NOT set) it never forwarded `.return()` to the source — leaking the wrapped ReadableStream that the old direct `for await` would have cancelled. Add a top-level finally that cancels the source on early termination, skipped after a watchdog stall (which already aborted the upstream, where a late cancel would make the AI SDK write to an already-cancelled readable). Tests: watchdog does not false-fire on a slow-but-steady stream (timer resets per chunk), and an in-band error under an armed watchdog terminates cleanly with no unhandled rejection. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): key submission abandon-paths off the recovery root, not the continuation id _markRecoveredSubmissionInterrupted was called with the per-continuation `requestId` in both terminal abandon paths — recovery exhaustion (`_exhaustChatRecovery`) and `{ continue: false }`. Under chained continuations (recoveryRootRequestId !== requestId) the durable submission row still carries the root id, so the `WHERE request_id = ?` lookup missed it and left the submission stuck `running` forever instead of `error`. Thread the recovery root through both paths (storing it on the incident record for the exhaustion path). Regression test drives a disabled-recovery chained continuation and asserts the root submission flips to `error` (verified to fail without the fix). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): treat output-error as settled in transcript repair _repairToolTranscriptParts' hasOutput check omitted `state === "output-error"`, so a tool part already healed to output-error (no `output` field) — or a tool that legitimately errored — re-entered the heal branch on every turn. That clobbered a real errorText with the generic "interrupted" message and emitted a spurious chat:transcript:repaired event + updateMessage write + broadcast each turn for the life of the conversation. Treat output-error as a settled terminal state (matching _incompleteToolCallIds). Regression test asserts a real errorText survives a follow-up turn (verified to fail without the fix). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): treat output-denied as a settled tool state; centralize the enumeration Sweep for the recurring "incomplete terminal-state enumeration" bug class found one more instance: `output-denied` (a user-denied tool approval) is a settled state the AI SDK converts into a denial tool-result, but transcript repair, the backstop detector, and the immediate-flush check all omitted it. Repair therefore flipped a denial into a generic "interrupted" error (losing the denial), and the denied result wasn't durably flushed. - Centralize the terminal-state check into `_toolPartHasSettledResult` (output-available | output-error | output-denied, plus legacy output/result), shared by `_repairToolTranscriptParts` and `_incompleteToolCallIds` so the two can no longer drift. - Flush `tool-output-denied` chunks immediately, like other settled results. - Regression test: an output-denied part survives a follow-up turn (verified to fail without the fix). Reviewed and confirmed complete (no change needed): _isTerminalSubmissionStatus, streamIsTerminal, _messageHasPendingInteraction, shouldMarkSkippedAfterGenerationChange. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agents): reconcile protects errored/denied tool results from stale-client clobber AIChatAgent sweep for the terminal-state-enumeration bug class: ai-chat's own tool-state handling already covers output-denied (it's the HITL home), but the shared reconciler did not. reconcileMessages (run at persist by both Think and AIChatAgent) only carried over the server's `output-available` result into a stale client part — so a client that persisted a stale `input-available` for a tool the server had already resolved to `output-error`/`output-denied` clobbered the resolved result, losing the error or the user's denial. Index all three terminal states and overlay the matching result field (output / errorText / approval). Tests assert a server output-error and output-denied survive a stale client input-available (verified to fail before). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor: address PR #1623 deep-review-2 hardening items - think: remove the now-dead message-deletion branch in _repairTranscriptForProvider (repair preserves every message, never deletes). - think: _normalizeToolInput now also parses a stringified-ARRAY input (`[...]`), not just objects. + test. - agents(reconciler): make the server-state overlay state-driven so only the field matching the terminal state is carried (a stray `output` on an output-error part can't ride along). + test. - ai-chat: document why _chatRecoveryContinue's conversation_changed skip does NOT split assistant-leaf vs user-leaf like Think (no submission layer to protect) — guards against a future regression if submissions are added. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: ask-bonk[bot] <ask-bonk[bot]@users.noreply.github.com>

Make nightly e2e trustworthy + harden chat-recovery / agent-tool coverage (#1797) * fix(ai-chat): make nightly e2e trustworthy and deterministic The ai-chat Playwright nightly had been red for weeks, timing out at the 30-minute job limit with no actionable output. Root cause was twofold: - Specs navigated to `about:blank` (a null origin) before opening WebSockets to `localhost`, which Chromium's Private Network Access rules reject — so every test failed to connect, and with no globalTimeout the job ran until the CI ceiling. - The remote `ai` binding made `wrangler dev` slow to become ready, and Playwright's port-only readiness check raced ahead of it. Fixes: - Add a `/__health` endpoint to the e2e worker and make all specs navigate there (same-origin) before connecting WebSockets. - Split the suite into a deterministic, AI-free config (`wrangler.mock.jsonc`, no credentials) and a bounded Workers-AI config, each with globalTimeout, retries, and real `/__health` readiness polling. - Run them as two nightly jobs so the deterministic suite is a green gate independent of remote AI. Also fix the 3MB compaction e2e test: it asserted the pre-refactor notice strings ("too large to persist" / "Preview:") instead of the current structured-truncation marker. Compaction itself works correctly on the setMessages path (verified against persistMessages + the row-size-guard unit tests); updated the assertions and un-quarantined it. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ai-chat): bump Workers-AI e2e retries to 3 The real-model LLM specs are inherently nondeterministic — the small Workers AI model occasionally varies wording or step boundaries and trips a strict assertion on the first attempt, but re-runs recover cleanly. Raise CI retries from 1 to 3 for the LLM-only suite (deterministic suite stays at 1). Retries only fire on failure, so green runs are unaffected. Co-authored-by: Cursor <cursoragent@cursor.com> * ci(nightly): run deployed recovery suites on the nightly schedule The two Layer-5 DEPLOYED suites (ai-chat recovery on real edge, Think recovery probe on real edge) were gated behind a `RUN_DEPLOYED_E2E` repo variable that was never set, so they had never run in CI — only locally. That left the real-edge recovery paths unverified and free to silently rot. Enable them on the nightly `schedule` event while keeping manual `workflow_dispatch` runs opt-in (via the `run_deployed` input or the repo variable), so ad-hoc debugging dispatches don't deploy billable Workers unintentionally. Each suite still uniquely names and always deletes its throwaway Worker. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think): re-apply channel policy on recovered pre-stream retry turns Adds recovery × channels coverage (previously zero) and fixes a real bug it surfaced: `continueLastTurn` re-resolved the channel from the persisted user message on recovery, but `_retryLastUserTurn` (the pre-stream retry path used by `_chatRecoveryRetry`) admitted the recovered turn without re-resolving the channel. So a turn interrupted before streaming was retried with the default policy instead of its channel's instructions / tool narrowing, even though the `metadata.channel` stamp survived. The new workers-pool test `channel-recovery.test.ts` asserts both the stamp survives and per-channel policy is re-applied across BOTH recovery paths (continue + retry); the retry case reproduced the bug before the fix. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ai-chat): sub-agent SIGKILL agent-tool recovery e2e (Think parity) ai-chat had full SIGKILL chat-recovery e2e but no sub-agent coverage, while Think tests an agent-tool child recovering across a process kill. Closes that parity gap: adds a `ChatRecoveryHelperParent` (plain Agent) that drives a `runAgentTool` run whose child is an `AIChatAgent` (`ChatRecoveryHelperChild`, slow finite stream + chatRecovery + default continue recovery). The test starts the run, confirms it is in-flight, SIGKILLs + restarts wrangler against the same persist dir, and asserts the parent re-attaches to the still-running child and collects its real terminal (`completed`) instead of abandoning it as `interrupted` (#1630). Auto-runs in the nightly e2e-ai-chat-recovery job (matches the e2e include glob); no CI change. Test-only, no changeset. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think,ai-chat): re-bind agent-tool child run_id across recovery A recovered agent-tool child turn minted a fresh request id but left the child-run row's request_id pointing at the pre-eviction turn, breaking frame attribution. A long-running recovered child then forwarded nothing to the parent's re-attach tail and was sealed `interrupted` once the no-progress budget elapsed, even while still advancing. Both recovery paths (continueLastTurn / _retryLastUserTurn) now re-bind the child-run row and the in-memory attribution map to the current turn's request id, keeping frames flowing across recovery so the parent re-attaches to the real terminal. - think: _rebindAgentToolChildRunRequestId + wiring; un-skip the reattach-budget e2e (now green, 199s) and add deterministic workers-pool unit coverage. - ai-chat: parity fix + deterministic unit coverage. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think): cover nested agent-tools, concurrency cap, runTurn × recovery, action-pause × recovery Adds deterministic Workers-pool coverage for agent-tool composability and recovery interactions that were previously untested: - nested-agent-tools: 3-level chain (grandparent → middle → grandchild) completes and grandchild observation does not bridge to the grandparent - max-concurrent-agent-tools: over-cap runs are rejected, freed slots reuse, Infinity allows unbounded concurrency - run-turn-recovery: runTurn wraps execution in a recovery fiber and cleans up, and a fresh runTurn composes cleanly on a recovery-resolved transcript - action-pause-recovery: a parked durable pause is not seen as a pending interaction by recovery and stays approvable across an active incident Test-only; no public API change. Co-authored-by: Cursor <cursoragent@cursor.com> * test(think,ai-chat): harden agent-tool recovery coverage + unify child-run predicate Production (behavior-preserving hardening, no migration): - Unify the in-flight child-run predicate across @cloudflare/think and @cloudflare/ai-chat on an explicit `status IN ('starting','running')` in both `_rebindAgentToolChildRunRequestId` and `_agentToolRunForRequest`. Both child-run tables already carry `status` AND `completed_at`, and terminal rows set them together (the lifecycle invariant), so this is equivalent to the prior `completed_at IS NULL` / `status = 'running'` checks but states intent and stays consistent. Document the one-child-run-row-per-DO invariant (child DOs are addressed by runId) that makes the rebind unambiguous. Coverage: - rebind no-op safety (think + ai-chat): no table, settled row, and defensive newest-of-many-active-rows selection — locks the "no-op on non-child recovery" contract. - maxConcurrentAgentTools: soft-terminal `interrupted` runs do NOT occupy a slot (re-issue after recovery is never cap-blocked); in-flight `running` runs do. - channel × recovery: per-channel TOOL policy (not just instructions) is re-applied on a recovered turn; a turn with no channel stamp falls back to default policy; channel re-resolution AND agent-tool request_id rebind compose on the SAME recovered turn. - nested agent-tools: each nesting level enforces its own maxConcurrentAgentTools independently of its parent. - action-pause × recovery: a parked durable pause stays approvable across a CONTINUE recovery incident (the retry path was already covered). - ai-chat agent-tool recovery e2e: document the coverage split (deterministic unit suite is the tight rebind gate; SIGKILL e2e is the integration smoke) and why we don't replicate Think's slow no-progress-budget e2e. Behavior-preserving + test-only; no changeset. Co-authored-by: Cursor <cursoragent@cursor.com> * Add test-coverage matrix and useAgent tests Introduce a living design doc `design/test-coverage-matrix.md` and link it from design/AGENTS.md and design/README.md to track feature × test-layer coverage and quarantined tests. Add a new browser React test `packages/agents/src/react-tests/useAgentToolEvents.test.tsx` to exercise live-vs-replay dedupe and terminal guards for agent-tool events. Also set a 15-minute Playwright `globalTimeout` in `packages/codemode/e2e/playwright.config.ts` so slow/flaky e2e runs produce reports instead of silent CI cancellations. * test: harden browser/hook coverage, resolve skip debt, add nightly e2e jobs Workstream A — browser/hook depth: - New real-Worker agent-tool replay-on-reconnect suite (agents react-tests) backed by a deterministic, LLM-free TestAgentToolStubChild; proves live-vs-replay dedupe + typed interrupted-cause survival across a real socket reconnect. - New deterministic Think resume-handshake suite (think react-tests) driving the real useAgentChat: single ACK per RESUMING (#1733), replay-no-dup, and idle transcript restore. Workstream B — coverage matrix: - design/test-coverage-matrix.md per-cell precision + A1/A2 rows; cross-linked from root + agents AGENTS.md; recorded idle-connect transcript-delivery as a keep-divergent row in the chat-recovery RFC convergence matrix. Workstream C — nightly hygiene (.github/workflows/nightly.yml): - Add codemode Playwright e2e job and agents browser-connector job; enable the gated Workers-AI tanstack leg. Workstream D — skip debt (all resolved): - basepath HTTP custom-path: replaced the meaningless "not 404" skip with two real positive tests (method+body forwarding; instance resolution). - workflow-error-reporting step.do: un-skipped via introspector disableRetryDelays() (real throw + retry exhaustion, no wall-clock waits). - workflow-integration x5: stale comments + one stale assertion; all un-skipped. - shell git clone: reclassified to an opt-in Group A gate (RUN_GIT_CLONE_E2E, forwarded into the pool via vitest.config.ts), repinned to octocat/Hello-World. - useAgent async query: fixed a missing `await` on the suspending render. Test-only + CI + docs; no package public API change (no changeset needed). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

fix(chat): orphaned-stream recovery no longer merges a new turn into the previous message (#1691) (#1693) * fix(chat): orphaned-stream recovery no longer merges a new turn into the previous message (#1691) When an AIChatAgent stream is interrupted before its assistant message is persisted (Durable Object hibernation, deploy churn, isolate restart, reconnect), orphan recovery reconstructs the message from stored chunks. If the chunks carry no provider `start.messageId` — the common case with `streamText(...).toUIMessageStreamResponse()`, where the id is assigned client-side — recovery used to fall back to the LAST assistant message in history. That is correct for a continuation, but wrong for a normal new turn after a later user message: the recovered chunks were appended onto the PREVIOUS assistant message, corrupting both the persisted transcript and future model context. Core fix - ResumableStream now persists the allocated assistant message id in stream metadata (`message_id` column, added via a one-time, schema-checked migration) and exposes `getStreamMessageId()`. - `_persistOrphanedStream` keys recovery on that stored id when the chunks carry no provider `start.messageId`, so a new turn becomes its own message and a continuation still merges into the message it was extending (it stored the cloned last-assistant id). A provider `start.messageId` still wins when present. Pre-migration rows keep the legacy last-assistant fallback. - Dropped the now-unused `is_continuation` metadata column. Two related variants of the same corruption on the durable (chatRecovery) continuation path, found during review and fixed here: - Early-persist + recovery (e.g. a tool-approval pause) re-appended chunks it had already stored, duplicating a tool call's parts. Recovery now skips reconstructed parts whose `toolCallId` already exists on the message. - A new turn interrupted before any assistant part was persisted — cut off before the first chunk materialized, or discarded via `onChatRecovery` returning `{ persist: false }` — was "continued" by cloning the previous assistant message and merging into it. `_handleInternalFiberRecovery` now detects that the conversation leaf is still the unanswered user message (no partial to continue) and re-runs the turn fresh, so it becomes its own message. @cloudflare/think is unaffected — its session-tree recovery already allocates a distinct message id per orphan and never falls back to the last assistant message. Tests - New regression + wiring tests in durable-chat-recovery, resumable-streaming, and the test worker, including the fiber-continuation happy path and the two edge cases (empty partial, persist:false) that previously merged. Verification - Verified live against real LLMs (Workers AI, OpenAI, Anthropic) and Think via a SIGKILL-mid-stream / restart harness (wip/issue-1691-live): the recovered turn always lands as its own message and the previous turn is untouched. - Cross-model continuation with large partials is clean (no duplication, no restarts); OpenAI and Anthropic resume a truncated partial to completion. The harness and its methodology notes are documented in its README. * chore(chat): address PR review nits on #1691 recovery fix - Report `recoveryKind: "retry"` to `onChatRecovery` and the incident record for an empty-partial new turn (interrupted before any chunk), since that case is deterministically a retry — it's knowable before the hook runs. The `persist: false` sibling case still reports "continue" (it only becomes a retry based on the hook's own return value) and the comment documents why. - Await `_persistOrphanedStream` in the `triggerInterruptedStreamCheck` test helper so it matches the production fiber-recovery path (latent test-only race, harmless in practice but now correct). - Rename the two `wip/` package.json names to the `@cloudflare/agents-*` prefix so changesets' ignore glob excludes them from versioning/release.

AI Gateway resumable streaming: RFC + experimental harnesses + workers-ai-provider@3.2.0 bump (#1764) * design: RFC + harness for AI Gateway resumable streaming in workers-ai-provider Adds the design record and the empirical harness behind merging ai-gateway-provider's capabilities into workers-ai-provider, centered on AI Gateway native resumable streaming (cf-aig-run-id + /resume). design/rfc-workers-ai-gateway-merge.md (+ index entry in design/AGENTS.md): Capability-driven dual-transport design. The run path (env.AI.run) gives resumable streaming; the gateway path (env.AI.gateway(id).run([...])) gives server-side fallback + caching. They are disjoint, so a delegate selects the transport from the requested options and warns/errors on incompatible mixes. Covers: dispatch mechanics (run-path forwarding fetch + gateway-path capture/redispatch), slug canonicalization, routing-layer providers (OpenRouter), fallback modes, a tiered resume-expiry recovery ladder, an error taxonomy with a nested attempt tree, Agents SDK / Think chat-recovery integration, provider-specific interactions (OpenAI Responses API scoped out; model-agnostic user-message continuation instead of deprecated Anthropic prefill), and a two-tier (unit + live e2e) test strategy. experimental/gateway-resume/ — the harness that validated it against a live account with unified billing: - Resume contract: cf-aig-run-id is issued for dash-catalog models on the new run API (not @cf/* yet); resume(from=0) reproduces the full stream byte-for-byte; `from` is an SSE EVENT index, not a byte offset; replay is provider-native SSE (so a provider-matched parser is required). - Transport split: cf-aig-run-id only on env.AI.run; cf-aig-step (server fallback) + cf-aig-cache-status (caching) only on env.AI.gateway().run([]). - Request-side passthrough (/passthrough): real @ai-sdk/openai (.chat) and @ai-sdk/anthropic bodies dispatched through env.AI.run parse cleanly (text, tools, usage, finish); cf-aig-run-id surfaces on result.response.headers; anthropic-version header survives. This is the gate the whole run-path delegate design rested on. - Delegate engine (src/delegate.ts, /delegate): reference implementation of the capability matrix + transport selection + forwarding fetch, exercised end-to-end through streamText. All scenarios verified: run-path resume, server fallback (cf-aig-step increments on a real bad-primary fallthrough), caching, conflict errors, escape hatch, both providers. - Buffer TTL: ~330-360s (~5.5 min); expiry contract is a clean 404 {"error":"Request not found"} (vs 200/0-bytes past-end, vs 500 for a malformed id) — the signal the tiered recovery ladder branches on. Reproduced by ttl-sweep.sh / ttl-sweep-fine.sh. The harness is the prototype that the workers-ai-provider/gateway-delegate feature (in the cloudflare/ai repo) was ported from. Co-authored-by: Cursor <cursoragent@cursor.com> * design: validate transparent resume reconnect/replay (RFC §7.1) Prototype + live-validate the resume reconnect layer in the harness, then mark RFC §7.1 BUILT + VALIDATED. experimental/gateway-resume: - src/resumable.ts: createResumableStream — SSE event-boundary buffering (emit only complete events, buffer the trailing partial), terminator counting for the resume `from` index, reconnect via env.AI.fetch(/resume?from=N), 404 -> ResumeExpiredError. Includes a dropAfterEvents fault injector for testing. - src/index.ts: /resume-stream endpoint runs streamText through a run-path @ai-sdk model whose body is wrapped in the resumable stream, with ?dropAfter=N fault injection, reporting reconnect count + completion. Live results (proves transparent in-stream recovery, the previously unproven part): clean run -> 0 reconnects, finishReason stop; injected drop after 20 and after 80 events (openai) -> 1 reconnect each, complete parse, no error; anthropic native SSE drop after 5 events -> 1 reconnect, complete parse. Byte alignment holds because partial events are discarded and resume realigns on the boundary. RFC: §7 now carries a BUILT + VALIDATED status note, §7.1 documents the boundary-buffering correctness detail (not just a counter), and History records the implementation landing in cloudflare/ai (branch feat/workers-ai-provider-gateway-delegate). Co-authored-by: Cursor <cursoragent@cursor.com> * design: validate cross-invocation re-attach (RFC §7.1 / §9) Generalize the harness resumable stream + add a /reattach endpoint, and update the RFC to reflect the built re-attach primitive + the concrete Layer B seam. experimental/gateway-resume: - resumable.ts: optional `initial` + `fromEvent` (re-attach with no initial body, starting from resume?from=fromEvent) + onProgress(eventOffset) hook. - index.ts: /reattach simulates invocation #1 starting a run, then invocation #2 (after eviction) re-attaching with no initial body. Asserts (a) from=0 reproduces the full response through the @ai-sdk parser, and (b) from=mid is byte-exact against the known tail. Live results: openai (152 events) and anthropic (21 events) both pass — from=0 parses to finishReason stop with full text; from=mid is byte-equal to the tail (openai 22481==22481, anthropic 1814==1814). RFC: §7.1 status note now documents both modes (in-stream wrap + cross-invocation re-attach) and the onProgress capture surface; §9 plug-in steps updated to the concrete delegate hooks (onDispatch -> runId, onProgress -> eventOffset, stash in the chat fiber, re-attach via createResumableStream({ fromEvent }) on onChatRecovery). Co-authored-by: Cursor <cursoragent@cursor.com> * feat(experimental): gateway-resume-think — Layer B re-attach recipe for Think A self-contained Think agent demonstrating the missing "Layer B" recovery from RFC §9: on Durable Object eviction mid-turn, re-attach byte-exactly to the same AI Gateway run (cf-aig-run-id) instead of regenerating (which re-spends tokens). Pattern: getModel() capture cf-aig-run-id (onRunId) + live SSE offset (onProgress); this.stash({ runId, eventOffset }) — survives eviction (throttled to every 8th event). ‹DO evicted› onChatRecovery(ctx) planResume(ctx.recoveryData) — re-attach if the buffer is still live (TTL window), else fall back to the default; arms the next continuation + returns { continue: true }. continueLastTurn() getModel() returns a re-attach model that replays the exact tail from the stashed offset via createResumableStream ({ runId, fromEvent }) — zero new tokens. Files: src/plan.ts pure Layer-B decision (re-attach vs fallback; TTL + shape guards) src/resume.ts resumable stream (vendored from workers-ai-provider), re-attach mode src/gateway-model.ts AI SDK model over env.AI.run: buildCaptureModel / buildReattachModel src/server.ts GatewayResumeAgent extends Think wiring it together src/layer-b.test.ts 9 hermetic tests (planResume decisions + re-attach stream) scripts/driver.mjs live end-to-end (start → ctx.abort → assert reattach) README.md the pattern, how to run hermetic tests + live deploy Modeled on experimental/chat-recovery-probe (ctx.abort eviction, deploy/driver validation). Vendors a copy of the resume primitive so the experiment is self-contained; the shipping version lives in workers-ai-provider. Validation: tsc clean; oxlint/oxfmt clean; 9 unit tests green; wrangler deploy --dry-run bundles with AI + DO bindings. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(gateway-resume-think): deterministic live driver + validated re-attach Validated the Layer-B recipe against a real deploy (gateway "default", openai/gpt-5.4): a mid-stream ctx.abort() leaves a stashed { runId, eventOffset }; onChatRecovery re-attaches byte-exactly from the stashed offset and the turn converges with zero new tokens. ✓ captured run 47e67890a911… at event 86 → interrupt (ctx.abort, mid-stream) ✓ re-attached to run 47e67890a911… from event 88 ✓ turn converged driver.mjs: wait until the run-id is CAPTURED before interrupting (no fixed sleep racing the gateway round-trip — the earlier 2s interrupt beat the first byte and fell back), tolerant JSON reads across the abort/reboot window, and assert both the reattach decision and convergence. wrangler vars: GATEWAY "default" (the real resume-capable gateway) instead of the placeholder. README: document the live run + the buffered-tail semantics (resume replays what was buffered up to the abort; it does not regenerate). Co-authored-by: Cursor <cursoragent@cursor.com> * design: prove gateway runs are detached (keep generating after disconnect) Add a /detach probe to the gateway-resume harness: start a run, read a few SSE events, then reader.cancel() to simulate the originating request dropping mid-stream, and sample resume?from=0 over time. Finding: the run is server-driven / detached. The first resume?from=0 BLOCKS while tailing the live run (openai/gpt-5.4 6.7s -> 513 events; anthropic claude-opus-4.7 8.6s -> 24 events) and replays the COMPLETE stream including the terminal event ([DONE] / message_stop), despite only 3 events consumed before the disconnect. Upstream generation is not tied to the originating socket -> re-attach is genuinely zero-loss. Also extend /reattach to parse a mid-offset re-attach through @ai-sdk: from=0 -> full message (772 chars, stop); from=83/167 -> byte-exact tail, parses cleanly (stop), 392 chars (the tail text, not corruption). This corrects a wrong caveat in the gateway-resume-think recipe: the demo's 510 chars was the correct tail (events 88->end), to be concatenated with the prefix Layer A already delivered = full message, zero new tokens. Docs: RFC §7.1/§9 (detachment finding + prefix+tail merge model), both harness READMEs. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(gateway-resume-think): prove zero-loss reconstruction + fix offset throttle Add a ground-truth verification path and prove, end-to-end on a real DO eviction, that recovery reconstructs the FULL answer with zero regenerated tokens. - /gw/verify (+ parseReattachText): parse resume?from=0 to the full run text and compare to the recovered assistant message. Live result: recovered === full run, byte-identical (e.g. 17303 chars), zero tokens. - Recovery now re-attaches from event 0 (full replay) instead of the stashed tail offset. The run is detached (completes server-side after the originating disconnect, per experimental/gateway-resume /detach), so from=0 replays the complete buffer for zero tokens and continueLastTurn REPLACES the partial leaf with it -> provably whole. A tail re-attach risks dropping the prefix under replace semantics + the Layer-A<->SSE offset-space mismatch (RFC §9.4). - Fix the stash throttle: SSE offsets JUMP (a chunk can carry several events), so `eventOffset % N === 0` never lands and only the offset-0 stash survived. Switch to a delta-based throttle (advanced >= N since last stash). Verified: 38 stashes, offset 305 captured across eviction. - Driver: poll verify over time (distinguishes a still-streaming continuation from a real stall) and report pre-eviction stash diagnostics. - Hermetic test: full(from=0) === prefix(0..k) + tail(from=k) through the openai parser (seam regression guard). - Docs: recipe README + RFC §9 updated with the validated from=0 design and the throttle/seam learnings. Co-authored-by: Cursor <cursoragent@cursor.com> * design: reconcile RFC with as-built implementation (cloudflare/ai#573) Update the AI Gateway merge RFC to match what actually shipped: - Status → accepted/implemented (#573); Decision → accepted with the deliberate deviations from the original sketch called out. - §3: add an "As built" callout (single createWorkersAI entry, internal delegate, wire-format-keyed plugins, optional default gateway, per-call fallback, per-slug autocomplete, per-provider runWireFormat, metadata). - §3a: mark the generic openai-compatible wrapper resolved — folded into the wire-format-keyed openai plugin. - Open questions: resolve the ones #573 answered (wrapper descriptor shape, per-slug autocomplete typing, fallback ergonomics, conflict granularity, openai-compatible). TTL configurability, OpenRouter internal-routing recoverability, Responses API, continuation boundary, and the framework cf-aig-run-id capture seam remain open follow-ups. - History: add the 2026-06-16 consolidation entry (single entry, literal types, metadata, alibaba/minimax, runWireFormat, default gateway, hardened e2e). Co-authored-by: Cursor <cursoragent@cursor.com> * update workers-ai-provider --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com>

feat(think): add create-think package and starter templates (#1695) * feat(think): add create-think package and starter templates Introduce `create-think` (`npm create think`) and a top-level `think-starters/` directory of complete, runnable starter apps, and rework `think init` to scaffold from them via a `--template` flag. Templates - New `think-starters/` workspace members (added to pnpm-workspace.yaml): - basic — minimal Think chat agent + small React UI - personal-assistant — persistent memory (configureSession) + scheduled tasks - coding-agent — workspace file tools + a coding skill (Worker Loader) - customer-support — custom tools + an escalation skill - Each is a self-contained, deployable Workers app (own package.json, wrangler.jsonc, vite.config.ts, agents/**, generated think.d.ts) and uses `workspace:*` deps so they build/test in CI as in-repo examples. think init / @cloudflare/think/cli - Add `--template` (default `basic`) and `--ref` flags to `think init`. - Replace the single inline scaffolder with a template-fetch model: copy from the local `think-starters/` dir when in-repo, otherwise use an injected remote fetcher. On fetch, set the package name and rewrite `workspace:*` deps to published ranges so the app installs standalone. - Expose `initCommand` and template helpers via a new side-effect-free `@cloudflare/think/cli` export (added to build entries + package exports). create-think - New `create-think` package: a thin bin that forwards argv to `initCommand` and injects a degit (tiged) fetcher pulling starters from `cloudflare/agents/think-starters`. Tests / housekeeping - Rewrite CLI init tests for the template model: default template, all templates + workspace-version rewrite, unknown template, injected fetcher (ref/name handling), non-empty/outside-root guards, existing-app no-op, dry-run, and inspect/types on a generated app. - Normalize pending changesets to patch bumps; add changesets for `@cloudflare/think` and the new `create-think` package. * fix(think): rewrite Worker name on scaffold and drop dead --route-prefix - finalizeTemplate now also rewrites the `name` field in the scaffolded wrangler config to the user's project name (targeted replacement that preserves JSONC comments/formatting), so apps no longer all deploy under the shared template Worker name (e.g. "think-basic-starter"). Renamed finalizeTemplatePackageJson -> finalizeTemplate. - Remove the `--route-prefix` option from `think init`: the template-based scaffolder no longer generates config, so the flag was accepted but silently ignored. Also drop the now-unused `routePrefix` from InitCommandOptions and refresh the stale init command description. - Extend the all-templates init test to assert the Worker name is rewritten.

fix(agents): progress-keyed agent-tool re-attach so a deploy can't abandon a still-running child (#1630) (#1670) * fix(agents): progress-keyed agent-tool re-attach so a deploy can't abandon a still-running child (#1630) A deploy that interrupted an in-flight `runAgentTool` child used to abandon the still-running child as `interrupted` and re-run its already-completed work. Parent recovery re-attaches to the child and tails it to its real terminal, but the old re-attach used a flat 120s wall-clock budget that was NOT reset by the child's forward progress — so a healthy, actively-streaming child whose recovery legitimately ran longer than the budget got sealed `interrupted`. Core fix - Re-attach budget is now progress-keyed: it bounds time spent with NO forward progress (resets on every forwarded chunk) and is hard-bounded by the child's own recovery ceiling, so a genuinely hung child still seals and can never block recovery forever. - `_forwardAgentToolStream` returns a discriminated end reason (`done | idle | aborted`). The re-attach loop re-arms (opens a fresh tail) ONLY on a clean stream-close while the child is still advancing (a re-evicted-but-progressing child). A full no-progress window seals `no-progress` immediately even if the child streamed earlier in that window — no bonus window — which also guarantees at most one pending tail reader per re-attach (fixes per-cycle reader accumulation). - think/ai-chat finalize a child facet's own run row as soon as its recovered turn settles, so a re-attached parent collects the terminal result immediately instead of waiting out a full no-progress window. Follow-ups - Typed interrupted cause: `RunAgentToolResult`, the `agentTool()` `AgentToolFailure` envelope, the `onAgentToolFinish` lifecycle result, and the `agent-tool-event` wire event now carry a machine-readable `AgentToolInterruptedReason` (`no-progress | window-exceeded | not-tailable | inspect-timeout | inspect-failed | recovery-deadline`) plus a `childStillRunning` boolean, propagated through the chat reducer into `AgentToolRunState` for UIs. `retryable` stays coarse (always true for interrupted); refine via `reason` / `childStillRunning`. - Configurable budgets: public `AgentStaticOptions` `agentToolReattachNoProgressTimeoutMs` (default 120000) and `agentToolReattachMaxWindowMs` (default 900000). - Give-up teardown (ceiling only): a `window-exceeded` give-up cancels the child (`childStillRunning: false`) so it stops consuming a fiber/keep-alive; `no-progress` give-ups stay soft (`childStillRunning: true`) to preserve repair-on-re-issue. think `cancelAgentToolRun` also aborts an in-flight chat-recovery turn and releases live tails. Tests - New behavioral/unit coverage: scripted-adapter re-arm matrix (rearm-then-complete, idle-after-progress), public-knob resolution, window-exceeded teardown, and a not-tailable unit seam (defensive branch is unreachable through a real RPC child, so exercised via a plain adapter). - Deterministic e2e repro (`reattach-budget.test.ts`) verifies a still- progressing child is collected `completed` (not `interrupted`) after a deploy. - Type-level test pins the public interrupted-cause surface. Verified: agents workers (1296), ai-chat (617), think agent-tools (22), e2e repro, and full `npm run check` across 91 projects. * fix(agents,ai-chat,think): close re-attach parity gaps in chat-recovery (#1630 review follow-ups) Two review findings against the progress-keyed re-attach PR, both parity gaps between the continue and retry recovery paths / between Think and AIChatAgent. 1. Retry-path reconcile (think + ai-chat). The eager child-side terminalization was only wired into `_chatRecoveryContinue`'s `finally`. A child facet evicted *before* producing any assistant content recovers via the pre-stream retry path (`_chatRecoveryRetry` → `_retryLastUserTurn` / a fresh user turn), which — like `continueLastTurn` — never flows through `startAgentToolRun`'s finalizer. Without the reconcile there, the child's run row strands `running` with its tailers open, forcing the re-attached parent to wait out a full no-progress window before collecting an already-settled result — the exact delay the PR set out to remove. Mirrored the `_reconcileOwnStaleAgentToolChildRuns()` call into both retry `finally` blocks and updated the doc comments to name both recovery paths. 2. AIChatAgent teardown now cancels the in-flight recovery turn. Think's `cancelAgentToolRun` sweeps `_submissionAbortControllers` to tear down an orphaned chat-recovery turn on give-up; AIChatAgent's did not, so a `window-exceeded` teardown left the recovered turn grinding (and holding a keep-alive) even though the row was already sealed `aborted` and tailers closed. AIChatAgent has no submission layer but does have a per-request `AbortRegistry` with `abortAllRequests()` — purpose-built for single-purpose sub-agent facets — so this was a fixable parity gap, not a design limitation. `cancelAgentToolRun` now calls `abortAllRequests(reason)`, and `AbortRegistry.destroyAll` / `abortAllRequests` gained an optional `reason` (backward-compatible) so the abort carries the give-up cause for diagnostics, matching Think. Verified: npm run check (91 projects), ai-chat 617, think 513, think agent-tools 22, ai-chat agent-tools 20, and the e2e reattach-budget repro (child collected `completed`). * test(agents,ai-chat,think): uncap re-attach ceiling by default + cover child self-finalize/cancel (#1630 review) Review follow-ups for the progress-keyed agent-tool re-attach work. Behavior: - Default `agentToolReattachMaxWindowMs` to `Infinity` (was 15min). Post-#1672 the child no longer has a wall-clock recovery ceiling, so a finite parent ceiling would tear down a healthy, still-streaming child that simply outran a fixed window under deploy churn — the exact bug #1672 removed, reintroduced at the parent layer. A hung child is already bounded by the no-progress budget; a content-runaway is bounded uniformly by the child's own `maxRecoveryWork` / `shouldKeepRecovering`. Integrators can still set a finite ceiling (which keeps the `window-exceeded` teardown). Docs/comments: - Rewrite docs/agent-tools.md "Interrupted runs and recovery" to describe re-attach + tail-to-terminal, the progress-keyed budget, the uncapped-by- default ceiling, and the soft (`no-progress`) vs hard (`window-exceeded`, torn down) give-up mapping; add `reason`/`childStillRunning` to the `AgentToolFailure` block. - Fix stale comments/JSDoc that referenced the removed child ceiling and the old 15min default; clarify `window-exceeded` only fires with an opt-in cap. Tests: - P1: child facet self-finalizes its own run row via the reconcile in BOTH `_chatRecoveryContinue` AND `_chatRecoveryRetry` finally blocks (the pre-stream retry path the earlier review flagged), in Think and ai-chat. - P2: assert the default re-attach ceiling resolves uncapped (no finite cap). - P3: assert `applyAgentToolEvent` propagates `reason`/`childStillRunning`. - P4: assert `cancelAgentToolRun` aborts an in-flight recovery turn (Think `_submissionAbortControllers` / ai-chat `AbortRegistry`) and seals `aborted`. * fix(agents): persist typed interrupted cause so it survives reconnect replay (#1630 follow-up) The #1630 re-attach work added a typed interrupted cause (`reason` / `childStillRunning`) on RunAgentToolResult and emitted it on live wire events. But those fields were never persisted to `cf_agent_tool_runs`, so a client that reconnected and replayed a stored `interrupted` run saw `reason`/`childStillRunning` as `undefined` — silently regressing any UI told (by our docs) to branch on them instead of the `error` prose. Persist and faithfully reconstruct the cause: - Schema v9: add additive `interrupted_reason TEXT` + `child_still_running INTEGER` columns to `cf_agent_tool_runs` (CREATE TABLE + idempotent addColumnIfNotExists migrations; bump CURRENT_SCHEMA_VERSION 8 -> 9). - _updateAgentToolTerminal writes both columns unconditionally, so repairing a soft `interrupted` row to a hard terminal (e.g. a re-attach that finally collects `completed`) CLEARS the stale cause instead of leaving it dangling. childStillRunning maps undefined->NULL, true->1, false->0. - New shared helper _agentToolInterruptedExtrasFromRow reconstructs the typed cause from a row, guarding on status === 'interrupted' and treating each column's NULL independently. Wired into every row->result / row->event rebuild: _resultFromAgentToolRow, _readAgentToolRun, _replayAgentToolRuns (reconnect replay), and _reconcileAgentToolRuns. Tests: new TestAgentToolReplayAgent fixture drives the real persist path and a capture-connection replay. agent-tool-replay.test.ts covers no-progress (childStillRunning true), torn-down window-exceeded (childStillRunning false), reason-only (the reconcile path: no childStillRunning), legacy rows (both columns NULL -> bare interrupted event), and clear-on-repair to completed. All green: build, npm run check (90 projects typecheck), and the agents / ai-chat / think agent-tool suites pass. * fix(agents): Infinity no-progress re-attach budget means 'never seal on no-progress', not 'skip waiting' Review follow-up. `agentToolReattachNoProgressTimeoutMs` is typed `number?` (which includes `Infinity`), and the sibling `agentToolReattachMaxWindowMs` uses `Infinity` as its 'off / uncapped' default — so a user setting the no-progress budget to `Infinity` reasonably expects 'never give up on a silent child'. Instead, the guard in _reattachAgentToolRunToTerminal short-circuited on `!Number.isFinite(...)`, returning immediately with an `interrupted`/`no-progress` seal — identical to passing `0`, the exact opposite of the intended semantic. - Drop the `!Number.isFinite` clause from the early-return guard so a positive (incl. Infinity) budget falls through to the tail loop; only a non-positive budget still means 'do not wait'. - In _forwardAgentToolStream, require Number.isFinite(idleTimeoutMs) for idleEnabled so an Infinity budget disables the idle timer entirely (it is never armed -> never fires). This also avoids handing Infinity to setTimeout (which clamps to ~1ms and fires almost immediately in the Workers/Node runtime). A silent-but-alive child is now followed until its stream closes or the finite hard ceiling fires. `0` remains the 'don't wait, collect only an already-terminal child' sentinel; semantics for finite budgets are unchanged. Tests: new `infinite-no-progress-ceiling` scenario in the Think scripted re-attach seam drives a silent, never-closing stream with an Infinity no-progress budget + a finite ceiling and asserts it tails (tailAttempts 1) and seals `window-exceeded` (never `no-progress`) — pre-fix it short-circuited with zero tail attempts. Docs + changeset updated. (Note: the ai-chat _reconcileOwnStaleAgentToolChildRuns 'misses starting rows' review item is NOT a bug — ai-chat inserts child run rows directly as 'running' and has no 'starting' status, unlike Think's child-run table, so `WHERE status = 'running'` is complete. No change made.) All green: build, npm run check (90 projects), agents/ai-chat/think agent-tool suites.

fix(think,agents): opt-in proactive + reactive recovery for mid-turn context-window overflow (#1662)

Launch first-class Think messengers (#1587) * Launch first-class Think messengers Add a first-class messenger surface to Project Think so a root Think agent can own Chat SDK webhook ingress, route messenger turns into Think conversations, and stream replies back through provider-specific delivery policies. The new API centers on getMessengers(), defineMessengers(), chatSdkMessenger(), and provider subpath entrypoints so applications can opt into only the adapters they use instead of pulling every Chat SDK provider into the base package. The messenger runtime creates one Chat SDK instance per root Think agent, registers all configured adapters together, and routes webhook requests after Think internal routes but before user onRequest fallbacks. Incoming Chat SDK threads and messages are normalized into provider-neutral MessengerEvent objects, then delivered through durable idempotent fibers that call Think's streamed chat() path on either the root agent or a per-thread subagent. Telegram is the first provider implementation. It lives under @cloudflare/think/messengers/telegram with defaults for webhook verification, Chat SDK state sharding, Telegram streaming soft limits, long-message follow-up splitting, and expected final-edit no-op handling. The generic @cloudflare/think/messengers entrypoint remains provider-neutral and does not import Telegram adapter code. Harden the launch behavior around production footguns: webhook verification must be explicit for every messenger definition, Telegram requires a secret token unless verification is intentionally disabled, external delivery failures use generic user-facing text, recovery snapshots strip provider raw payloads and attachment fetch/data functions, and the default respondTo policy replies to direct messages and mentions without responding to every ordinary subscribed-thread message. Update the existing Chat SDK messenger example to consume the shared Think messenger delivery/state helpers while keeping its advanced manual ingress architecture. The example now uses ThinkMessengerStateAgent, imports TextStreamCallback and Telegram helpers from @cloudflare/think/messengers, requires TELEGRAM_WEBHOOK_SECRET_TOKEN, and refreshes its README to distinguish the advanced manual pattern from the new getMessengers() path. Document the launch across the Think README and docs: add messenger exports, peer dependency notes, configuration overrides, a dedicated Think messengers guide, route/default/security behavior, conversation targeting, state facets, delivery recovery, messenger context, and custom adapter guidance. Also update Chat SDK state docs to mention the Think-specific state agent wrapper used by the example. Add worker-runtime tests for messenger defaults, path and adapter validation, explicit verification posture, verifier response short-circuiting, event-to-message conversion with attachments, serializable recovery snapshots, recovery/failure classification, interrupted fiber recovery through revived Chat SDK threads, streamed delivery overflow handling, active messenger context delivery, sanitized external errors, Think request precedence, and Telegram helper behavior. Verified with: - npm run test --workspace @cloudflare/think -- src/tests/messengers.test.ts - npm run build --workspace @cloudflare/think - npm run test --workspace @cloudflare/think - npm run check Co-authored-by: Cursor <cursoragent@cursor.com> * Polish Think messenger foundation Tighten the Think messenger runtime so Chat SDK is a direct dependency, webhook verification does not consume adapter input, streaming checkpoints are only written after visible output starts, and recovery replay can checkpoint through durable fiber resolution. This also adds opt-in action event handling, Telegram adapter-name support for multi-bot setups, a default Telegram provider export, and tests for the new routing, recovery, verification, and action behavior. Add a Vite-based think-chat-sdk example that exercises the high-level getMessengers() path with Telegram webhook setup and a websocket-backed conversation dashboard. Update the Think docs, README, and changeset to present the intended API shape and clarify when to use the Think-native example versus the lower-level manual Chat SDK ingress example. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix messenger self-target cancellation Allow messenger delivery targets to cancel chats synchronously so self-mode replies use Think.cancelChat without masking delivery failures. This preserves the original delivery error classification when post streaming fails and adds regression coverage for local self targets. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix Think messenger action idempotency Include the source message, action id, user, and value in action-event idempotency keys so distinct button interactions do not collapse while webhook retries remain deduplicated. Also align the Think Chat SDK example favicon with the examples convention. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

fix(chat-recovery): exempt human-in-the-loop turns from recovery budgets; close think hibernation gap (#1684) * experimental: add chat-recovery-probe harness for #1672 A headless @cloudflare/think harness that validates the durable chat-recovery assumptions from #1672 against the real production runtime, using a deterministic synthetic in-DO model (no LLM) so the only interruption is a real deploy or ctx.abort(). Validated in prod: - happy-path: a progressing turn completes (submission `completed`). - A1 invariant (headline): a progressing turn survives >15min of real deploy churn with monotonic progress and ZERO seals — no `max_recovery_window_exceeded` (which the pre-#1672 build emitted at the ~15min ceiling). Findings recorded for follow-up (not exercised cleanly by this harness): - `ctx.abort()` opens fresh retry-incidents (`conversation_changed`) instead of incrementing one incident's attempt. - Under churn, `submitMessages` submission-recovery proliferates incidents and can error submissions; the per-attempt guards (maxRecoveryWork / shouldKeepRecovering / no-progress) need a single accumulating incident with same-message continuation — best covered by #1672's deterministic unit tests. Experimental; not a product example. * fix(chat-recovery): exempt human-in-the-loop turns from recovery budgets; close think hibernation gap A turn parked on a pending CLIENT interaction (an `input-available` part for a client-side tool with no server `execute`, or an `approval-requested` part) is waiting on the human, not stuck. After a mid-turn Durable Object restart the in-memory pending-interaction promise is gone, so `waitUntilStable()` repeatedly times out until the client reconnects and replays the tool-result/approval. The recovery loop was treating those timeouts as deploy churn and sealing perfectly healthy turns with `stable_timeout` / `no_progress_timeout` — the "session interrupted" banner users hit when they simply took a while to answer a prompt. Core behavior (packages/ai-chat, packages/think — each carries its own copy of the recovery engine): - While a client interaction is pending the turn is budget-free: `_beginChatRecoveryIncident` suppresses the no-progress window, attempt cap, `maxRecoveryWork`, and `shouldKeepRecovering`, and keeps the no-progress clock fresh so the turn gets a full window once the human answers. - `_chatRecoveryContinue` / `_chatRecoveryRetry` PARK instead of rescheduling or exhausting: the incident is marked `skipped` with reason `awaiting_client_interaction`, the live "recovering…" indicator is cleared, and the client's eventual replay resumes the turn via the normal auto-continuation. A client that never returns is reclaimed by the incident TTL sweep and DO idle eviction. - think: a `submitMessages`-backed turn additionally has its durable submission row COMPLETED at park time. Recovery is that row's sole completion driver after a restart, so parking without completing would leave it `running` and the next boot's `_recoverSubmissionsOnStart` would sweep it to `error` (a false "session recovery error"). A fully-materialized client tool call in the leaf is exactly the terminal state a non-interrupted submission reaches, so `completed` is correct. - Server-tool orphans are deliberately excluded (their `execute()` died with the isolate, nothing will resolve them) and still recover via transcript repair. ai-chat's existing `hasPendingInteraction()` (used by `waitUntilStable`) does not distinguish client from server tools, so a narrower client-only predicate `hasPendingClientInteraction()` was added there to gate the exemption without changing `waitUntilStable` semantics. think's `hasPendingInteraction()` was already client-only. think hibernation-ordering fix: - The exemption depends on `_lastClientTools`. ai-chat restores them in its constructor (available when boot recovery evaluates budgets), but think restored them in `onStart()`, which the base `Agent` runs AFTER the boot-recovery path (`_handleInternalFiberRecovery` -> `_beginChatRecoveryIncident`). On a fresh wake the in-memory cache was empty, so a client-tool `input-available` orphan re-detected past the no-progress window was misread as "stuck" and wrongly sealed. - `_beginChatRecoveryIncident` now re-hydrates `_lastClientTools` from the durable `think_config` store (its own table, no Session init required, so the read is safe this early) before evaluating the budget; guarded so it is idempotent with the onStart restore and a no-op on the live-isolate stall path. `approval-requested` turns were never affected. agents: warn when `chatRecovery` is configured in `onStart()` - On every wake the SDK evaluates recovery budgets (and may seal a turn, firing `onExhausted`) BEFORE `onStart()` runs, so a `chatRecovery` produced there is read as the built-in defaults for the recovery that matters. The SDK now logs a one-time warning when `chatRecovery` is assigned during `onStart()` — for a config object or `chatRecovery = true` (disabling with `false` is a benign no-op and is not warned). Documented on `ChatRecoveryConfig`. - Also fixes an unsafe `as` cast in `Agent._rootAlarmOwner` (`binding as unknown as DurableObjectNamespace<Agent>`) so classic tsc agrees with tsgo. Tests: - ai-chat + think: retry-path park unit tests mirroring the continue-path ones (no reschedule, no budget spent, incident parked `awaiting_client_interaction`; think also asserts the submission row is `completed` and a follow-up `_recoverSubmissionsOnStart` sweep does not resurrect it as `error`). - think: boot-path hibernation regression test driving two fresh wakes with the in-memory client-tool cache cleared and tools only in the durable store — fails without the hydration guard (sealed `no_progress_timeout`), passes with it. New harness helpers: `seedDurableClientToolsForTest`, `clearInMemoryClientToolsForTest`; ai-chat `preScheduleRecoveryRetryForTest`. e2e hardening: - Add `packages/think/src/e2e-tests/harden-net.ts` (and inline equivalents in the ai-chat / agents chaos suites): best-effort wrap of `Socket.setTypeOfService` to swallow the benign write-time `EINVAL` undici throws synchronously when a socket is torn down mid-SIGKILL/restart, which otherwise surfaced as an unhandled exception and flaked otherwise-green runs. Imported by the 7 think e2e suites. - Rewrite the stale `should interrupt a stale parent agent-tool run` e2e to the progress-keyed re-attach semantics (#1630/#1670): the parent re-attaches to the still-running child (which self-heals via `continue`) and collects its real `completed` terminal instead of an abandoned `interrupted`. docs: - chat-agents.md: `onStart` guidance callout, and a "Turns waiting on a human are not sealed" subsection (parks with `awaiting_client_interaction`, client-only). - human-in-the-loop.md: "Surviving restarts while waiting for a human" subsection cross-linked to the stream-recovery reference. - NOTE: both still need a manual port to cloudflare/cloudflare-docs. experimental/chat-recovery-probe: - Extend the live-worker harness with `hitl`, `server-orphan`, and `approval` synthetic modes; add scenarios a6 (HITL exemption), a7 (server-orphan recovers, not exempt), a8 (approval exemption), rapid (onExhausted exactly once), and idem (submission idempotency), plus WebSocket priming/reply/approval helpers. a6/a7/a8 verified locally against real isolate resets. Changesets: chat-recovery-pending-client-interaction (think, ai-chat patch), chat-recovery-config-onstart-warning (agents, think, ai-chat patch).

fix(agents): reclaim resumable-stream buffers from an alarm (#1706) (#1712) * fix(agents): reclaim resumable-stream buffers from an alarm (#1706) Resumable-stream chunk buffers (cf_ai_chat_stream_*) were only swept lazily when a *subsequent* stream completed. A chat that received a single turn and then went idle never triggered that sweep, so its buffers lingered in the Durable Object's SQLite for the lifetime of the DO — an unbounded leak for one-off / low-traffic chats. Alarm-driven cleanup -------------------- AIChatAgent and Think now arm a scheduled cleanup alarm whenever a stream starts and whenever it finishes (completes or errors): - Arming on start is the safety net for the non-durable path (chatRecovery: false, the AIChatAgent default). Those turns don't run inside runFiber, so there's no leftover keepAlive alarm and no fiber-recovery scan; if the client never reconnects, nothing else wakes the DO. Arming on start guarantees a stream whose DO is evicted mid-flight still gets a future sweep instead of leaking. - Arming on finish covers the common completed/errored case. - Durable runFiber turns already self-heal (keepAlive survives eviction -> recovery finalizes -> arms cleanup), so arming on start is belt-and-suspenders there. The alarm sweeps aged buffers and re-arms only while reclaimable rows remain, so a fully-swept DO stops waking itself. Arming is idempotent so high-turn-count chats never accumulate cleanup schedules. The re-arm inside the fired callback is deliberately NON-idempotent: alarm() deletes the fired one-shot row after the callback returns, so an idempotent reschedule would dedup onto that doomed row and vanish with it, leaking any buffer that survived the sweep. Two short, purpose-specific retention windows --------------------------------------------- Replaces the single 24h threshold (which would have kept the leak alive for a day) with windows matched to what each buffer is actually for: - COMPLETED_RETENTION_MS = 10 min, measured from completion. The assistant message is persisted separately (cf_ai_chat_agent_messages), so a finished buffer is only a brief reconnect-and-replay grace: long enough to cover a client that dropped at the completion boundary, and to deliver a pending terminal error frame (#1645) on a resumed stream. After the window a late reconnect degrades gracefully to a clean done frame plus persisted history. - ABANDONED_STREAM_RETENTION_MS = 1 h, measured from LAST chunk activity. Generous so an interrupted turn has ample time to be resumed by a reconnecting client or healed by fiber recovery before its buffer is presumed dead. Keyed off last activity (not start time) so a long-running stream still emitting chunks is never swept mid-flight. Server-side reconstruction (recovery, resume-ACK via _persistOrphanedStream) only ever reads ACTIVE streaming rows, which live in this 1h window — never the 10min completed grace — so recovering a turn interrupted >10 min ago still works. No new public configuration: correctness never depends on the window (the durable message is the source of truth), so these are sane internal defaults rather than another knob to misconfigure. API --- ResumableStream gains cleanup(now?) (force a sweep, bypassing the lazy interval gate) and hasReclaimableStreams() to support alarm-driven cleanup. Tests ----- New stream-cleanup coverage in both ai-chat and think: - arms a single alarm on finish, deduping repeats - reclaims aged buffers when the alarm fires (no completeStream) - re-arms only while reclaimable buffers remain - survives the real alarm fire and re-arms when a younger buffer remains (guards the non-idempotent-reschedule footgun) - stops re-arming after the last buffer is swept - arms cleanup when a stream starts - window boundaries: completed grace (10m) vs abandoned stale (1h) - locks the arming delay at 10 min (regression guard against re-lengthening toward the old leak window) - keeps an in-flight buffer's chunks reconstructable past the completion grace (proves recovery reads the 1h streaming window) Docs: updated docs/resumable-streaming.md to describe the two windows (needs manual port to cloudflare/cloudflare-docs). * fix(think): arm cleanup on start in the RPC streaming path too (#1706) _streamResultToRpcCallback called this._resumableStream.start() raw, bypassing the _startResumableStream wrapper that arms the cleanup alarm on stream start. Its WebSocket sibling _streamResult already uses the wrapper; this RPC path (the sub-agent chat() entry point) was missed — an incomplete transformation. Normal finishes were already covered (every terminal path goes through _completeResumableStream / _errorResumableStream, which arm). The gap was the mid-flight-eviction case the start-time arm exists for: a sub-agent whose isolate resets mid RPC stream with chatRecovery: false has no fiber-recovery scan to re-arm on wake, so without start-time arming its buffer would never get a sweep — the exact #1706 leak. Now both streaming entry points share the same arm-on-start invariant.

fix(agents,think,ai-chat): re-attach to still-running sub-agent runs on parent recovery (#1630) (#1640) * fix(agents,think,ai-chat): re-attach to still-running sub-agent runs on parent recovery (#1630) When a parent agent was evicted (deploy / DO reset) while a child agentTool() run was in flight, recovery sealed the run `interrupted` within ~5s and the parent re-issued the task — re-running the child's already-completed work ("the agent went all the way back and lost the files it wrote"). - Stable child runId: agentTool() now derives `agent-tool:<toolCallId>` from the recovery-preserved tool call id instead of a fresh nanoid, so a turn re-run by recovery resolves to the SAME idempotent child facet rather than spawning a new one (the primary amplification fix). - Bounded re-attach: a duplicate non-terminal runId (runAgentTool) and a still-running child during startup reconciliation now tail the live child to its real terminal result, bounded by DEFAULT_AGENT_TOOL_REATTACH_TIMEOUT_MS (120s). A hung child still seals `interrupted` after the budget so recovery can never block forever. - think + ai-chat child tails are now read-only on consumer detach: a parent's re-attach budget expiring cancels only the read view, never the still-running child (so it keeps advancing toward its own terminal for a later collect). Adds an internal `agent_tool:recovery:reattach` observability event; no new public config. Reworked the reconcile unit tests (running child -> completed; new bounded-tail-able-stuck -> interrupted after budget), a stable-runId test, and a natural-agentTool() task-amplification e2e variant. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(think,ai-chat): collect a recovered sub-agent's real result after eviction (#1630) A child facet self-heals its interrupted agent-tool turn via its own chatRecovery, but that path never wrote the child's run row — so after a real eviction the row stranded `running` (think) / was force-errored (ai-chat) and the parent could only ever collect `interrupted`/`error`, never the recovered result. Both packages now reconcile a stale child-run row (running, no live abort controller = original isolate gone) from the durable transcript on inspectAgentToolRun, gated on recovery state via _classifyAgentToolChildRecovery (lists chat-recovery incidents on the 1:1 child facet): in-progress -> keep `running` so the parent's bounded re-attach keeps waiting; settled with a completed assistant response -> `completed`; failed/empty -> `error`. This keeps the child's own (working) saveMessages recovery path untouched. Validated by a real-eviction e2e: task-amplification now drives the natural agentTool() path through kill/restart churn and asserts the child reaches all 30 steps AND the parent's run row settles `completed`. Note: routing the child through the durable submission registry (submitMessages) was tried first and reverted — the e2e showed it regresses multi-restart, multi-step tool-loop recovery (_recoverSubmissionsOnStart errors a running submission whose messages were applied, fighting the chat-recovery fiber). The transcript reconcile avoids touching the recovery path entirely. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agents,think,ai-chat): parallel sub-agent re-attach + collect text-less recovered runs (#1630) Review hardening for the sub-agent recovery work: - Reconcile no longer starves siblings. _reconcileAgentToolRuns is now a two-pass sweep: a deadline-bounded inspect/classify, then re-attach of still-running children IN PARALLEL, each bounded by its own re-attach budget. Previously the shared total-recovery deadline was consumed by the first child's (up to 120s) re-attach, so a slow/hung child caused every later sibling to be abandoned `interrupted` without an attempt. - A settled recovery that produced an assistant turn is now collected as `completed` even when the turn ended on a tool result with no final text — keying off text alone (think _getAgentToolFinalText / ai-chat _extractLatestAssistantText) mis-sealed a legitimately-finished but text-less run as `error`. getAgentToolSummary still falls back to "". New reconcileParallelThinkChildrenForTest asserts a hung child (started first) no longer starves a fast sibling. Full think (457) + ai-chat (480) suites green; both real-eviction e2e scenarios pass. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(agents,think,ai-chat): polish sub-agent re-attach edge cases (#1630) Addresses the three deferred review items: (a) Re-attach now tails from the child's CURRENT max chunk sequence instead of afterSequence: -1. The client reducer appends chunks by arrival order (ignoring sequence/replay), and a reconnected client already has the stored chunks via _replayAgentToolRuns — so replaying them on re-attach duplicated the run's parts on a connected client under repeated re-attach. Forwarding only genuinely-new chunks keeps the live stream correct without dupes. (b) Documented that the tail reader is deliberately abandoned (not cancelled) on budget-abort: cancelling a remote child-facet RPC stream surfaces an unswallowable "Stream was cancelled" pump rejection (verified). The hold is already bounded — the child reaches terminal within its own chat-recovery ceiling, firing the tail's registered closer which releases the reader. (c) Extracted the inspect reconcile-persist into a named private helper (_reconcileStaleAgentToolChildRun) in both packages so read vs reconcile are separated. The persist is retained intentionally (enables prompt tail-close + cheap subsequent inspects) and documented as lazy materialization of the run's true terminal. think 457 / ai-chat 480 suites green; both real-eviction e2e scenarios pass. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agents): treat interrupted as a soft, repairable terminal (#1630) Deep-review finding: once a run was sealed `interrupted` (e.g. a reconcile exhausted its re-attach budget before a slow child finished), the parent could never recover it — `_updateAgentToolTerminal` excluded `interrupted` from its overwrite guard, and `runAgentTool` returned the cached row for any terminal status. So a re-issue (stable runId) got stale `interrupted`, the model saw a retryable failure, and retried with a NEW toolCallId → fresh child → re-ran the already-completed work, re-introducing amplification for slow children. #1630 explicitly called for `_updateAgentToolTerminal` to let a later child completion repair the parent row. - `_updateAgentToolTerminal` now overwrites `interrupted` (soft terminal); only completed/error/aborted are hard/immutable. - `runAgentTool` routes an existing `interrupted` run through the re-attach path (like a non-terminal run) instead of returning the cached interrupted, so a re-issue re-attaches and repairs the row to the child's real result. Also: corrected a stale comment in the reconcile defer branch (re-attach now tails from the child's max chunk sequence, not -1). New reissueInterruptedThinkChildForTest locks in the repair. think 458 / ai-chat 480 green; both e2e scenarios pass. Co-authored-by: Cursor <cursoragent@cursor.com> * test(deploy-churn): add sub-agent re-attach dimension under real deploys (#1630) Extends the deploy-churn reliability harness (real `wrangler deploy`s mid-turn, higher fidelity than the SIGKILL e2e) to cover sub-agent re-attach: - `DeployChurnSubAgentChild` — a long, recoverable `recordStep` ledger child (mirrors the proven packages/think `ThinkToolRollbackE2EAgent`). - A `"subagent"` harness mode on `DeployChurnAgent` that drives the child via `agentTool()` (natural stable-runId path), plus `configureSubAgentRun` / `getSubAgentStatus` RPCs. - `churn.ts --mode subagent` with a RE-ATTACHED-vs-AMPLIFIED verdict. Run against the deployed worker (2 real mid-loop deploys) it confirms the #1630 fix holds under real deploys — the child completed all 30 steps with 1 re-run (no amplification, no data loss) — and surfaced a deeper follow-up: an orchestrating parent that only `await`s a sub-agent makes no forward progress of its own, so its chat recovery can exhaust before the child finishes (parent collected `interrupted` despite the child completing). That's the plan's N1 "apply recovery to the sub-agent path" item, beyond this PR's re-attach scope. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

fix(chat-recovery): bound Durable Object memory-limit (OOM) crash loops (#1825) (#1826) * fix(chat-recovery): bound Durable Object memory-limit (OOM) crash loops (#1825) A chat-recovery turn whose Durable Object isolate exceeds its 128 MB memory limit could loop forever, re-running the (billable) turn on every platform alarm retry. The isolate streams a little content before the reset, which bumps the durable progress counter; on the next wake recovery reads that as forward progress and resets both progress-keyed bounds (maxAttempts, noProgressTimeoutMs), and because each crash lands inside the alarm-debounce window the attempt counter is pinned too. With maxRecoveryWork defaulting to Infinity, no instrument could ever seal the turn, so the model ran forever. This lands a layered fix: 1. Finite maxRecoveryWork default (1000, was Infinity). The work meter is the one signal that keeps climbing across the loop, so a finite default seals a runaway with reason="work_budget_exceeded". 2. OOM-specific in-DO budget (chatRecovery.maxOomRetries, default 3). A memory reset re-OOMs on re-run (the turn's working set, not the platform, is the cause), so it is classified as a distinct deterministic failure rather than a deploy-style transient: it is NOT deferred and retried forever. Each crash bumps a durable per-incident oomAttempts counter; after a small number of tries it seals with reason="out_of_memory". Fast and attributable. 3. Alarm-boundary circuit breaker (Agent.alarm()) as the universal backstop for OOMs that bypass the in-DO budgets entirely - thrown before the budget code runs (boot-time state hydration), or whose own small writes also OOM under memory pressure. Left unhandled such an error propagates out of alarm() and the platform auto-retries forever. alarm() now intercepts ONLY Durable Object memory-limit resets at the outermost frame, where the heavy turn has unwound and GC has reclaimed its footprint, so the seal/purge writes can land where mid-turn ones OOMed. A durable strike counter (static maxAlarmMemoryLimitStrikes, default 3) tolerates a few resets - backing off the looping rows so the retry is not a hot loop - then seals the recovery (out_of_memory) and surgically purges ONLY the looping schedule rows, leaving unrelated scheduled tasks intact. Emits a new alarm:memory_limit_reset event. Everything except memory-limit resets re-throws exactly as before. Supporting changes: - Broaden + export isDurableObjectMemoryLimitReset(error): matches the shared "exceeded its memory limit" fragment so truncated/reworded surfacings observed in real #1825 logs still classify. Sibling to isDurableObjectCodeUpdateReset / isPlatformTransientError. - _executeScheduleCallback now DEFERS (re-throws) memory-limit resets for one-shot rows instead of swallowing them after in-process retries, so the error reaches the alarm-boundary breaker; track the executing row id so the breaker can purge the exact looping row. - think/ai-chat override _cf_recoveryAlarmCallbacks() and _cf_sealMemoryLimitedRecovery() to target their recovery continuation callbacks and terminalize active incidents (banner + onExhausted + seal). - Remove the redundant result-path OOM handling in continueLastTurn: those turns are already terminalized, so it only risked wasteful reschedules and duplicate terminal signals. Adds unit + integration coverage (predicate, listActiveChatRecoveryIncidents, alarm circuit breaker), an RFC follow-up section, docs, and changesets. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(chat-recovery): reset alarm OOM strike counter on a clean alarm The alarm-boundary memory-limit strike counter (maxAlarmMemoryLimitStrikes, #1825) is documented as counting CONSECUTIVE alarm OOM resets, but it was only ever deleted when the breaker sealed — never after a clean alarm — so it actually tracked LIFETIME resets. A Durable Object hitting rare, non-consecutive transient spikes (e.g. one a month) would eventually reach the strike budget and wrongly seal healthy recovery work. alarm() now best-effort clears cf_agents:oom_alarm_strikes after a clean _cf_runAlarmBody() so strikes must be consecutive to seal. The clear reads first and only writes when a strike is recorded, so the common no-strike path costs no write. Adds a regression test (strike recorded -> clean alarm resets to 0 -> next OOM starts at strike 1). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Add Chat SDK messenger example with managed fiber durability (#1563) * Add Chat SDK messenger example Demonstrates Chat SDK ingress on Agents with subagent-backed state and Think-owned conversation replies. Co-authored-by: Cursor <cursoragent@cursor.com> * Stream Chat SDK messenger replies Adds Think chat streaming with RPC-safe cancellation so messenger delivery failures can stop the corresponding sub-agent turn. Co-authored-by: Cursor <cursoragent@cursor.com> * Add managed fiber jobs Introduce managed fiber jobs on top of runFiber so agents can durably accept idempotent background work, inspect retained status, cancel running jobs, explicitly resolve interrupted jobs, and record recovery policy decisions. This adds the cf_agents_fibers ledger, schema v8 migration, status/list/delete/resolve APIs, cooperative cancellation signals, and waitForCompletion support that waits on terminal ledger state instead of only the callback promise. Tighten crash recovery semantics for managed work by reconciling stale run rows, recovering ledger-only pending/running rows, skipping recovery for already-terminal fibers, settling setup failures, and letting onFiberRecovered return a FiberRecoveryResult to move interrupted fibers to completed, error, aborted, or intentionally interrupted. The implementation also tracks active managed executions and terminal waiters so duplicate requests can join in-memory work when possible while post-restart retries drive the same recovery path. Use the new managed fiber API in the Chat SDK messenger example for AI replies. Telegram messages now get a stable per-message idempotency boundary, completion waiting preserves Chat SDK per-thread visible reply serialization, and recovery policy is explicit: accepted replies are replayed while mid-stream interruptions post a concise apology and settle the retained job. Expand coverage across unit, sub-agent, schema, and real eviction tests. The E2E harness now starts wrangler dev with persisted SQLite state, kills it mid-managed-fiber, restarts it, and verifies interrupted retention, recovery-result settlement, duplicate waitForCompletion retries after restart, and sub-agent managed fiber recovery through the parent alarm. Document the new durable job surface in the Agent and durable execution docs, including waitForCompletion, cancellation behavior, retained terminal records, explicit recovery outcomes, and how this differs from Think message admission. Co-authored-by: Cursor <cursoragent@cursor.com> * Polish managed fiber cleanup API Rename the public managed-fiber terminal timestamp from completedAt to settledAt, and rename the cleanup filter from completedBefore to settledBefore. These names better describe terminal rows across completed, error, aborted, and interrupted states while keeping the existing SQLite completed_at column internal. Make default deleteFibers() cleanup preserve interrupted rows. Interrupted managed fibers often need inspection or explicit application-level resolution, so callers must now opt in to deleting them by passing status: "interrupted". Clarify FiberContext.snapshot documentation so it does not imply callbacks are automatically re-entered with recovered snapshots; recovery snapshots are delivered through onFiberRecovered(). Add a regression test that default cleanup deletes completed rows while preserving interrupted rows, then verifies explicit interrupted cleanup still works. Co-authored-by: Cursor <cursoragent@cursor.com> * Document managed fiber adoption patterns Add practical guidance for using managed fibers around webhook-style application jobs, including retained cleanup with settledBefore, interrupted recovery, resolveFiber, and waitForCompletion behavior. Clarify the boundary between Think submissions and managed fibers across the Think docs, package README, server-driven messaging docs, webhook docs, and examples so users can distinguish durable Think turn admission from app-owned side-effect jobs. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix PR install after main package bumps Use the workspace dependency for the Chat SDK messenger example's Think package so npm ci can resolve the merged branch after main's version-package release. Always run npm ci in the shared GitHub install action while relying on setup-node's npm package cache, avoiding stale node_modules cache hits that can mask lockfile drift. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix managed fiber review issues Correct the malformed Think changeset frontmatter so Changesets can parse the release metadata. Ensure waitForCompletion waits for a terminal managed fiber status even when duplicate calls race with an already-running recovery pass, and cover the race with a regression test. Also document and test the Chat SDK state adapter's list-level TTL behavior. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

fix(chat): keep AI SDK status correct when reconnect races the pre-stream window (#1784) (#1803) * fix(chat): keep AI SDK status correct when reconnect races the pre-stream window (#1784) A chat turn spends a window between "request accepted" and "first chunk produced" in neither the active-stream nor terminal-replay state: it is queued, debouncing, waiting on waitForMcpConnections, or running async setup inside onChatMessage before a stream object exists. A client that reconnected or re-mounted in that window was answered with cf_agent_stream_resume_none and gave up, so the turn the server went on to stream never drove the client's AI SDK `status` — the UI stayed stuck at "ready" until a full remount. Server (agents/chat): - New shared PreStreamTurns tracker (pre-stream-turns.ts) for accepted-but-not- yet-streamed turns and the connections parked waiting on them, mirroring ContinuationState (pure data + send-through-callback). - New server->client cf_agent_stream_pending frame (protocol + wire-types + golden builder). - ResumeHandshake now parks resume requests that arrive during the pre-stream window and emits STREAM_PENDING ("keep waiting") instead of RESUME_NONE, then flushes parked connections into the normal STREAM_RESUMING handshake on stream start. Continuation affinity is relaxed via an optional isConnectionPresent host hook so a transparent reconnect (whose connection id changed) can resume a continuation whose original owner connection is gone. Client: - ws-chat-transport: handleStreamPending() extends the resume probe from the 5s fast path to a 60s backstop so the probe stays open across the gap. - useAgentChat re-probes the stream on a transparent socket reopen (e.g. a 1006 reconnect that does not remount the component) so status recovers. Hosts: - Wired the begin/park/flush/settle lifecycle into both AIChatAgent and @cloudflare/think. - Skipped turns (supersede / queue generation change) settle WITHOUT releasing parked connections (releaseParked: false), so a client parked during the window survives onto the successor turn instead of being cut loose by a premature RESUME_NONE in the supersede/settle microtask race. Hibernation: PreStreamTurns is in-memory only and is safe because the pre-stream window cannot overlap hibernation — a turn between begin() and stream start is an unresolved message-handler promise that pins the DO in memory, so eviction only happens once a durable stream exists (resumed via ResumableStream) or the turn finished. Documented as an invariant on each _preStream field. Tests: - Unit: PreStreamTurns (incl. skip-path contract), handshake park/pending/flush/ affinity, golden STREAM_PENDING frame. - Transport: STREAM_PENDING keep-waiting timeout extension (incl. tool continuation path). - Integration (workers runtime, real AIChatAgent): onConnect/resume-request park -> pending -> resuming, and parked-client-survives-overlapping-submits. - React hook: transparent reconnect re-probe + STREAM_PENDING keepalive. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ai-chat): guarantee pre-stream turn settles if a pre-turn-body step throws (#1784) AIChatAgent.begin()'d the pre-stream turn before persistMessages, _mergeQueuedUserMessages, and the queued mcp.waitForConnections / _setRequestContext steps. A throw in any of those never reached the settle in chatTurnBody's finally, so the request id stayed stuck in _preStream (hasInFlight() true forever) and every later client was parked on STREAM_PENDING (60s) instead of getting an immediate STREAM_RESUME_NONE — until chat clear or DO eviction. Wrap the whole post-begin() scope in a top-level try/finally that always calls _settlePreStreamTurn, mirroring @cloudflare/think. Idempotent with the existing happy-path settle. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Sunil Pai <18808+threepointone@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Add durable Think submissions API (#1511) * Add durable Think submissions API Introduce submitMessages() for Think so RPC callers can durably accept a programmatic chat turn without waiting for model execution to finish. The new API persists a submission row before execution, supports idempotent retries through idempotencyKey, exposes inspection/list/cancel/delete helpers, and emits submission lifecycle observability events. The implementation uses cf_think_submissions as a SQLite-backed ledger and an idempotent scheduled drain as the wakeup mechanism. Submissions move through a small pending/running/terminal state machine, append messages to Session only after being claimed, and use messages_applied_at as the replay boundary so hibernation recovery never duplicates already-applied messages. Pending submissions are synchronously skipped during turn reset, terminal states are protected with conditional updates, and recovered chat continuations stay running until they reach a terminal outcome. Add focused coverage for fast acceptance, idempotent retries, FIFO draining, cancellation, reset races, startup recovery, chat recovery, malformed durable rows, cleanup filters, and recovered-continuation cancellation. Also add user docs, a contributor-facing design doc, a changeset, observability tests, and a dedicated think-submissions example that demonstrates durable submission, retry, status inspection, cancellation, and cleanup flows. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix submission stream error retention Only durable submissions need to capture stream errors in the programmatic stream error map so their terminal row can be marked as error after _streamResult returns. Other callers such as saveMessages, WebSocket chat turns, continuations, and auto-continuation already handle stream errors through the normal response hook path and should not leave request-scoped entries behind for the isolate lifetime. Scope _programmaticStreamErrors writes behind an explicit capture option used by _runSubmission, and add regression coverage that a non-submission stream failure does not retain an entry. Also align the think-submissions example with the examples directory convention by using public/favicon.ico from the standard example favicon instead of a custom favicon.svg. Co-authored-by: Cursor <cursoragent@cursor.com> * Preserve aborted submission status after stream errors A captured programmatic stream error should only turn an otherwise completed submission into an error. If the underlying programmatic turn reports aborted or skipped, those explicit terminal outcomes must win even when abort/reset also surfaced as a stream iterator error. Factor submission final status selection into a helper, clear error_message for non-error terminal states, and add regression coverage for completed+error, aborted+error, and skipped+error precedence. Co-authored-by: Cursor <cursoragent@cursor.com> * Make submission recovery staleness configurable Keep the stale-evidence safety net for recovered durable submissions, but expose the recovery freshness window as a protected static setting that Think subclasses can tune for legitimate long-running turns. The default remains 15 minutes, preserving the existing behavior for normal agents while avoiding a hardcoded limit for providers or workloads that can validly run longer before a Durable Object restart. Update the recovery check to read submissionRecoveryStaleMs from the concrete subclass, document the override point in the durable submissions design doc, and add coverage proving that an older recoverable fiber remains running when a subclass extends the stale window. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

fix(think): serialize parallel client-tool result/approval applies (#1649 follow-up) (#1657) * fix(think): serialize parallel client-tool result/approval applies (#1649 follow-up) ## Problem The auto-continuation barrier added in #1651 stopped *premature* continuation, but a deeper data race remained in Think. Each `cf_agent_tool_result` / `cf_agent_tool_approval` WebSocket message triggers a read-modify-write of the whole assistant message via `_applyToolUpdateToMessages`, which awaits a storage read (`await this.session.getHistory()`) before writing. When the model fans out parallel tool calls in a single assistant step, the per-result applies run concurrently: 1. apply A and apply B both `await` the storage read and resolve with the same snapshot — every sibling still `input-available`. 2. A patches only its own part and writes the full message back. 3. B (holding the stale snapshot) patches only *its* part and writes the full message back, clobbering A's result to `input-available`. The clobber lands in both durable storage and the `this.messages` cache, so `_hasIncompleteToolBatch()` keeps seeing an unanswered sibling. The continuation barrier then times out and the transcript-repair backstop errors the lost call with "The tool call was interrupted before a result was recorded." ## Fix Introduce a per-instance serialization tail (`_interactionApplyTail`) and route every result/approval apply through `_enqueueInteractionApply`, which chains each apply off the previous one. This makes each read-modify-write atomic and ordered, so siblings can no longer clobber each other. - `.then(run, run)` is used so a rejected apply can't poison the rest of the batch. - `_pendingInteractionPromise` still tracks the newest link; because the chain is serial, the barrier's single-slot wake-up transitively waits for every predecessor. - `resetTurnState` drops the tail so a new turn never queues behind a stale (possibly hung) apply from a reset turn. ## ai-chat (defensive) The same serialization is added to `@cloudflare/ai-chat` for symmetry. ai-chat does **not** exhibit this bug today: `_findAndUpdateToolPart` performs the read-modify-write synchronously (no await between reading `this.messages` and the synchronous SQLite write), so concurrent applies cannot interleave. The queue is a forward-guard that keeps the invariant safe if that apply ever gains an await (e.g. async storage). ## Tests - Added a deterministic serialization test in both packages (`testInteractionApplySerialization`) that drives two overlapping read-modify-writes through the queue. Without serialization the result is 1 (clobbered); serialized it is 2. Verified the Think test FAILS without the fix (`expected 1 to be 2`). - Added an integration test in Think asserting three concurrently-sent parallel results all settle to `output-available`, none is repaired, and exactly one continuation runs. - `_enqueueInteractionApply` is `protected` + `@internal` so the test agents can drive it deterministically (the vitest-workers harness can't reproduce the real storage interleave because DO SQLite is synchronous). All checks pass: format, oxlint, typecheck (91/91), think (486) and ai-chat (566) suites. * fix(think): apply client-tool results/approvals that arrive mid-stream (#1649 follow-up) ## Problem The serialization fix (prior commit) stopped parallel client-tool results from clobbering each other, but the reporter still hit "The tool call was interrupted before a result was recorded." on every client-resolved tool call — even single (non-parallel) ones. Root cause is an ordering inversion between streaming and persistence: - During a streaming turn the assistant message lives ONLY in the in-flight `StreamAccumulator`. It is not written to durable storage (or added to `this.messages`) until `_persistAssistantMessage` runs at the end of the stream. - The `tool-input-available` chunk, however, is broadcast to the client mid-stream. A fast client resolves the tool and sends `cf_agent_tool_result` BEFORE the message is ever persisted. - `_applyToolUpdateToMessages` only scanned durable storage, so the apply found no matching message and silently no-op'd. - The end-of-stream persist then wrote the tool part as `input-available`, and the auto-continuation's transcript-repair backstop errored it as "interrupted". This is why "parallel sometimes works": a slow sibling (e.g. an RPC tool) resolves AFTER the stream has finished and persisted, so its apply finds the message in storage. ## Fix `_applyToolUpdateToMessages` now applies the update to BOTH: 1. the in-flight `StreamAccumulator` (in place), so the result rides into the eventual end-of-stream persist instead of being dropped; and 2. durable storage (unchanged), which still handles already-persisted messages — including partials written mid-stream by stall recovery and cross-message tool results that target an earlier message. Applying to both is deliberately monotonic: a later persist of `accumulator.toMessage()` (e.g. stall-recovery partial persist) carries the settled result and cannot downgrade it back to `input-available`. The accumulator is exposed via a new `_streamingAssistant` field for the duration of each streaming turn and cleared on every exit path (`_streamResult` normal + stall-scheduled/exhausted early returns, `_streamResultToRpcCallback` finally) and in `resetTurnState`. The field is intentionally ephemeral and never persisted; it is only ever set while the isolate is held alive by `keepAliveWhile`, so hibernation cannot strand it. This mirrors `@cloudflare/ai-chat`'s existing `_streamingMessage` handling, generalized to also cover the post-persist path. It fixes both client tool RESULTS and APPROVALS, since both flow through `_applyToolUpdateToMessages`. ## Why correctness holds for the auto-continuation barrier Mid-stream, the in-flight message is not yet in `this.messages`, so `_hasIncompleteToolBatch()` inspects a stale (prior) leaf. This does NOT break correctness: the continuation is enqueued behind the active streaming turn on the turn queue, and that turn persists the now-settled result before the continuation's transcript repair runs. The ordering guarantee is the turn queue, not the barrier's cache view. Documented inline. ## Tests - Deterministic reproduction (`simulateMidStreamClientToolResult`): a tool call exists only in the accumulator when the result is applied, then the message is persisted. Asserts the part ends `output-available`. Fails on the pre-fix code (persists `input-available`). - Deterministic approval variant (`simulateMidStreamClientToolApproval`): an `approval-requested` part reaches `approval-responded` via the same path. - Full-pipeline integration test over the real WS protocol: a new slow client-tool mock model finalizes the tool call (emits `tool-call`) then holds the stream open with trailing text. The test waits via `streamingToolCallState` until the call is exposed, sends `cf_agent_tool_result` while the stream is still open, then asserts the part is `output-available`, carries the output, and that NO `chat:transcript:repaired` fired for it. E2E (wrangler-spawning) was assessed and deferred: the integration test already exercises the full server pipeline in the real Workers runtime and times the mid-stream send deterministically; the e2e harness here is built for process-kill/eviction scenarios where a timing-based test would be flaky and largely redundant. ## Verification - format, oxlint (0/0), typecheck (91/91 projects) - full think suite: 488 passed (pre-additions); client-tools: 61 passed including the 3 new tests