Abhinav

feat(app-server): add history_mode to thread (#29927) ## Description This PR adds a new `historyMode = "legacy" | "paginated"` to `Thread`. This will be stored in `SessionMeta` in the JSONL rollout file and as a new column in the SQLite thread_metadata table, and exposed on `thread/start` and on the `Thread` object in app-server. ## What changed - Added canonical `ThreadHistoryMode` with `legacy` and `paginated`, defaulting old and new SessionMeta to `legacy`. - Carried `history_mode` through core session config, ThreadStore stored metadata, local/in-memory stores, rollout metadata extraction, and the existing SQLite `threads` table. - Added experimental `historyMode` to app-server v2 `Thread` and `thread/start`. - Made paginated stored threads metadata-discoverable but unsupported for legacy full-history reads, `load_history`, live resume, and create paths. - Regenerated app-server schema fixtures and added protocol/state/thread-store/app-server coverage for persistence and fail-closed behavior. ## Compatibility floor Because users may be running various versions of Codex binaries on the same machine (TUI, Codex App, etc.), we will need to establish a compatibility floor for upcoming paginated threads, which will change how thread storage reads and writes work. The overall plan here: ``` Release N: - Add historyMode to SessionMeta / Thread / SQLite metadata. - Teach binaries to understand paginated threads. - If a binary sees `historyMode="paginated"` but does not support the paginated contract, it refuses to resume/mutate the thread. - Default remains `"legacy"`. Release N+1: - First-party clients start opting into paginated threads where appropriate. - Internal dogfood / staged rollout. - Measure old-client usage and paginated-thread unsupported errors. Release N+2: - Only after Release N+ is overwhelmingly deployed, make paginated the default. - Accept that a small tail of N-1-or-older binaries may not understand paginated threads. ``` The important behavior change is fail-closed handling for a binary that encounters a persisted `paginated` thread before it knows how to fully support paginated history. In app-server, if a thread is `paginated`, we will: - allow metadata-only discovery paths like `thread/list` and `thread/read(includeTurns=false)`, so clients can still see the thread and inspect its `historyMode` - reject legacy full-history/live-thread paths like `thread/read(includeTurns=true)` and `thread/resume` with an unsupported JSON-RPC error - avoid silently treating an unknown or future `historyMode` as `legacy` Under the hood, the ThreadStore layer also rejects legacy operations that would need to load or replay the full thread history for a paginated thread. That gives us the behavior we want for Release N: future paginated threads are visible, but this binary fails closed instead of trying to operate on them as if they were legacy threads.

core: Consolidate Responses API Codex metadata (#27122) ## What Introduce a `CodexResponsesMetadata` struct that defines all the core metadata we send to Responses API. Example fields are `thread_id`, `turn_id`, `window_id`, etc. Going forward, `client_metadata["x-codex-turn-metadata"]` will be the canonical way Codex sends metadata to Responses API across both HTTP and websocket transports. For now, we continue to emit the existing top-level HTTP headers and top-level `client_metadata` fields from the same `CodexResponsesMetadata` struct for compatibility reasons. Also, app-server clients who specify additional `responsesapi_client_metadata` via `turn/start` and `turn/steer` will have those fields merged into `client_metadata["x-codex-turn-metadata"]`, but cannot override the reserved fields that core uses (i.e. the fields in `CodexResponsesMetadata`). ## Why Responses API request instrumentation is the source of truth for downstream Codex analytics that join requests by Codex IDs such as session, thread, turn, and context window. Before this change, those values were assembled through several request-specific paths: HTTP request bodies, websocket handshake headers, websocket `response.create` payloads, compaction requests, and the rich `x-codex-turn-metadata` envelope all had their own wiring. That made metadata propagation easy to drift across API-key/direct Responses API requests, ChatGPT-auth/proxied requests, websocket requests, and compaction requests. It also made additions like `window_id` error-prone because a field could be added to one transport projection but missed in another. ## What changed - Added `CodexResponsesMetadata` as the core-owned snapshot for Codex metadata sent to ResponsesAPI. - Render `client_metadata["x-codex-turn-metadata"]`, flat `client_metadata` projections, and direct compatibility headers from that same snapshot. - Include the known Codex-owned fields in the turn metadata blob, including installation/session/thread/turn/window IDs, request kind, lineage, sandbox/workspace metadata, timing, and compaction details. - Treat app-server `responsesapi_client_metadata` as enrichment for the Codex turn metadata blob while preventing those extras from overriding Codex-owned fields. - Use the same metadata path for normal turns, websocket prewarm, local compaction, remote v1 compaction, and remote v2 compaction. - Keep websocket connection-only preconnect metadata separate so handshakes carry compatibility identity headers without inventing a fake turn metadata blob. ## Verification - `cargo check -p codex-core` - `just fix -p codex-core`

feat(core, mcp): cache codex_apps tools in memory (#29003) ## Description This makes Codex Apps tool reads use a shared in-memory snapshot instead of rereading the disk cache every time `list_all_tools()` runs. Disk still seeds the cache on startup and gets updated after successful fetches, but it is no longer the live read path. The core change is that `McpManager` now owns a process-scoped `CodexAppsToolsCache`. Codex threads in the same app-server process now share this Codex Apps in-memory tools snapshot. The snapshot is keyed by the Codex home plus the Codex Apps identity: the active Codex auth user/workspace and the effective Codex Apps MCP source config. There's already code to hard-refresh the cache, so we respect it in this PR. ## Local benchmark I ran a local steady-state microbenchmark of the exact repeated Codex Apps cached-tools read this PR removes, using the same real local cache payload in both trees: `3,678,138` bytes and `381` tools. The cache file was already warm in the OS page cache, so this measures same-process reread/deserialization work rather than cold-disk latency or full turn latency. Each run is 25 iterations (mimicking a turn that makes 25 inference calls). | Version | Run 1 | Run 2 | Avg | |---|---:|---:|---:| | `origin/main` disk read + JSON deserialize + `filter_tools` | `50.755 ms` | `52.894 ms` | `51.825 ms` | | This branch in-memory `current_tools` + `filter_tools` | `0.740 ms` | `0.778 ms` | `0.759 ms` | That removes about `51 ms` from each repeated Codex Apps cached-tools read on this machine, roughly `68x` faster for that subpath. It is useful evidence for the hot path this PR changes, but not a claim that every production turn gets `51 ms` faster; end-to-end impact also depends on the rest of `list_all_tools()` and tool-payload construction. This is on my M2 Max macbook, so with a slower disk this would be much worse (and indeed we did see this really blew up turn runtime with a slow disk).

feat(core): add metadata field to ResponseItem (#28355) ## Description This PR adds an optional `metadata` field to `ResponseItem` for Responses API calls. Only mechanical plumbing, no actual values populated and sent yet. Turns out just adding a new field to `ResponseItem` has quite a large blast radius already. This change is backwards compatible because `metadata` is optional and omitted when absent, so existing response items and rollout history without it still deserialize and requests that do not set it keep the same wire shape. For provider compatibility, we strip out `metadata` before non-OpenAI Responses requests so Azure and AWS Bedrock never see this field. My followup PR here will actually make use of it to start storing and passing along `turn_id`: https://github.com/openai/codex/pull/28360 ## What changed - Added `ResponseItemMetadata` with optional `turn_id`, plus optional `metadata` on Responses API item variants and inter-agent communication. - Preserved item metadata through response-item rewrites such as truncation, missing tool-output synthesis, compaction history rebuilding, visible-history conversion, rollout/resume, and generated app-server schemas/types. - Strip item metadata from non-OpenAI Responses requests while preserving it for OpenAI-shaped requests. - Updated the mechanical fixture/test construction churn required by the new optional field.

feat(rollout): persist TurnItems for paginated thread rollouts (#30188) ## Description This PR makes new threads with `history_mode = "paginated"` persist `ItemCompleted(item: <turn_item>)` in their rollout JSONL file. Legacy threads keep persisting the existing legacy events. Because the format is selected per thread, a rollout is either legacy or paginated; we do not need to support mixed rollouts containing both representations. This PR depends on [#31473](https://github.com/openai/codex/pull/31473). ## Why Paginated thread history needs stable turn/item IDs and completed item snapshots so the later SQLite projector can materialize appended rollout JSONL without rebuilding the whole thread. Keeping the legacy persistence policy unchanged avoids changing historical rollouts or the readers that still consume them. ## What changed - Made rollout filtering history-mode aware. Paginated threads keep completed canonical `ItemCompleted` events and drop their redundant legacy projections; legacy threads keep the existing event set. - Made forks inherit the source thread history mode, so copied legacy history is never filtered as paginated. - Made paginated threads assign IDs to locally-created response items even when `Feature::ItemIds` is off, and reject streamed output items that arrive without server IDs. - Updated legacy turn replay, rollout list/search, and SQLite metadata extraction to understand completed canonical user-message items.

core: support extension-owned turn items (#31283) ## Description This PR adds a `codex-extension-items` crate for extension-owned `TurnItem` schemas, and updates standalone image generation to start using it via `TurnItem::Extension`. This gives us a way to prevent Core from having to be aware of all extension items. App-server still exposes the existing public `ThreadItem::ImageGeneration` shape, now by wrapping the same shared `image_generation::ImageGenerationItem` type. The new `codex-extension-items` crate is necessary because the image gen extension item is used by: - `codex-image-generation-extension`, which produces it. - `codex-tools / core`, which carry it generically. - `codex-protocol`, which serializes it into lifecycle events and rollouts. - `app-server protocol`, which wraps it in public `ThreadItem::ImageGeneration` ``` extension implementation ↓ codex-extension-items ↓ protocol / tools / app-server ``` We keep the hosted Responses API image generation as `TurnItem::ImageGeneration` because core still owns its persistence and legacy fanout. ### Before Standalone image generation is implemented as an extension, but its item representation previously lived in the core protocol. This sets the precedent that core is aware of all extension items, which would be good to avoid. ``` image-gen extension → constructs codex_protocol::ImageGenerationItem → emits ExtensionTurnItem::ImageGeneration → core matches ImageGeneration specially → protocol stores TurnItem::ImageGeneration ``` ### After ``` image-gen extension → constructs extension-owned ImageGenerationItem → emits generic ExtensionItem → core transports/persists it generically → app-server wraps ImageGenerationItem as ThreadItem::ImageGeneration ``` Future extension items can have typed app-server APIs without adding a new `TurnItem` variant, `ExtensionTurnItem` variant, or core emitter match arm. ## What changed - Added `codex-extension-items` with the closed `ExtensionItem` enum and shared `image_generation::ImageGenerationItem` schema. - Added generic `TurnItem::Extension(ExtensionItem)` and `ExtensionTurnItem::Extension { item, legacy_events }` paths. - Updated standalone image generation to emit a typed extension item and provide its existing legacy `ImageGenerationBegin` / `ImageGenerationEnd` events explicitly. - Kept canonical lifecycle ordering: core emits `ItemStarted` / `ItemCompleted` before extension-provided legacy events. ## Follow-up Standalone web search still uses its typed special-case path. Migrating it later would let `ExtensionTurnItem` collapse into a single extension-item struct.

feat(core): store turn_id on ResponseItem metadata (#28360) ## Description This PR is a followup to https://github.com/openai/codex/pull/28355 and starts assigning `internal_chat_message_metadata_passthrough.turn_id` to durable Responses API items created during a turn. The goal is that those items keep the `turn_id` that introduced them when Codex resends stateless HTTP context, reconstructs history for resume/fork paths, or reuses websocket response state. ## What changed - Set `internal_chat_message_metadata_passthrough.turn_id` when missing as response items enter durable history, initial/replacement history, inter-agent communication history, and local compaction summaries. - Preserve existing item turn IDs instead of overwriting them during persistence, resume reconstruction, compaction, forked history, and websocket incremental reuse. - Keep `compaction_trigger` fieldless because it is a request control, not a durable response item. - Update focused history/request assertions and fixtures for stateless requests, websocket incrementals, compaction, thread injection, prompt debug, and related CI coverage.

ensure thread.history_mode is immutable (#30261) ## Description This PR makes `thread.history_mode` immutable after the thread's canonical first `SessionMeta` has been written. Later same-thread `SessionMeta` lines are compatibility metadata writes, not a new thread definition. Without this, an older binary could append a `SessionMeta` that omits `history_mode`; when a newer binary replays it, serde defaults that missing field to `legacy` and SQLite could downgrade a paginated thread. ## Why `history_mode` is the persisted thread storage contract. Paginated-thread fail-closed behavior and SQLite memory filtering depend on it staying aligned with canonical rollout metadata, especially when multiple Codex binary versions can touch the same local rollout. ## What changed - Stop generic rollout metadata replay from overwriting `history_mode` from later `SessionMeta` items. - Remove `history_mode` from `ThreadMetadataPatch`, so mutable metadata sync and app-server metadata updates cannot rewrite it. - When local metadata sync has to recreate a missing SQLite row, recover `history_mode` from the rollout's canonical first `SessionMeta` instead of from a mutable patch. - Keep the in-memory thread store using the created thread's canonical `history_mode` instead of metadata patches. - Fill the one remaining core test `CreateThreadParams` initializer with the new `history_mode` field; Bazel CI caught this after the parent history-mode PR landed. ## Validation - `just fmt` - `just test -p codex-thread-store` - `just test -p codex-state session_meta_does_not_set_model_or_reasoning_effort`

core: migrate standalone web search to extension-owned turn items (#31525) ## Description This PR migrates standalone web search onto the extension-owned turn-item path introduced in #31283. Standalone web search now emits `ExtensionItem::WebSearch` through generic `TurnItem::Extension`, while app-server still exposes the existing typed `ThreadItem::WebSearch` JSON shape. Hosted Responses API web search stays on core-owned `TurnItem::WebSearch`. ## What changed - Added `web_search::WebSearchItem` and `WebSearchAction` to `codex-extension-items` under the stable `web.search` kind. - Collapsed `ExtensionTurnItem` to generic `{ item, legacy_events }` now that no typed extension special cases remain. - Kept the existing `WebSearchBegin` / `WebSearchEnd` compatibility events and canonical-first ordering. - Updated app-server projection/history and generated TypeScript; the app-server JSON schema is unchanged.

feat(core): emit canonical review mode items (#31473) ## Description This PR moves review-mode markers onto canonical `TurnItem` lifecycle: - `TurnItem::EnteredReviewMode` - `TurnItem::ExitedReviewMode` Core now emits `ItemStarted` / `ItemCompleted` for both. The completed items map back into the existing `EnteredReviewMode` / `ExitedReviewMode` events, so raw core event consumers and legacy rollout persistence keep seeing the old events. This is the compatibility layer needed before paginated rollouts persist review markers as `ItemCompleted(TurnItem)`. ## Why Review markers were one of the remaining app-server thread items created directly from legacy events. Giving them canonical items lets paginated history persist stable turn/item IDs without changing legacy rollouts. ## What changed - Added canonical review-mode `TurnItem`s and switched review flow to emit their lifecycle. - Added completed-item → legacy review event mappings with stable turn/item IDs. - Switched app-server live notifications to the generic canonical item path and kept legacy replay compatible with old payloads. - Updated `ThreadHistoryBuilder` to replay canonical review items even though review turns still do not emit `TurnStarted`.