Anton Panasenko

fix(remote-control): surface websocket task stalls (#24473) ## Why When the app-server remote-control websocket path stalls during connection setup or teardown, the existing logs do not show where the task stopped, and several awaits can keep the task from returning promptly. That makes offline or stale-host incidents hard to distinguish from expected shutdown or disable flow. Issue: N/A (internal incident investigation) ## What Changed Added structured lifecycle and status logging around remote-control enable/disable requests, websocket task startup and exit, connection cycles, enrollment context, and status/environment transitions. Bound websocket connect, transport-event forwarding, and connection-worker shutdown waits. On timeout, the code logs the stalled operation and stops or aborts workers so the loop can reconnect or exit instead of waiting indefinitely. Ping sends now also observe shutdown cancellation.

feat(app-server): persist remote-control desired state (#27445) ## Why Remote-control runtime enablement and persisted enrollment preference were represented by separate flags. That made startup rehydration, RPC persistence, and new-enrollment seeding race with one another, and it did not cleanly distinguish runtime-only CLI or daemon starts from durable app-server RPC changes. ## What Changed - Replace the parallel enablement, seed, and rehydration flags with one transport-owned `RemoteControlDesiredState`. - Add nullable enrollment-scoped persistence and preserve existing preferences during enrollment upserts. - Rehydrate plain startup only after auth and client scope resolve, without overwriting a concurrent RPC transition. - Make ordinary `remoteControl/enable` and `remoteControl/disable` durable while retaining `ephemeral: true` for runtime-only callers. - Have the daemon explicitly request ephemeral enablement and regenerate the app-server schemas. ## Verification - Covered migration and `NULL`/`0`/`1` persistence round trips. - Covered plain-start rehydration and runtime-only versus durable enrollment seeding. - Covered durable enable, durable disable, and ephemeral enable through app-server RPC. - Covered the daemon's exact `{ "ephemeral": true }` request payload. Related issue: N/A (internal remote-control persistence architecture change).

fix(remote-control): avoid server token refresh retry storms (#30201) ## Why Remote-control websocket reconnects and pairing requests proactively refresh their server token. When `/server/refresh` returns a transient error such as `502`, the still-valid token was discarded as a usable connection path, causing reconnect failures and repeated refresh attempts that could amplify an upstream incident. ## What Changed - Start proactive refresh five minutes before token expiry and distinguish it from a required refresh for missing or expired tokens. - Continue websocket and pairing operations with the existing valid token after `429`, `5xx`, or timeout failures. - Share an in-memory `next_refresh_at` throttle across websocket and pairing callers, honoring both `Retry-After` formats and otherwise using a jittered 24–36 second delay. - Keep required refreshes strict, preserve `404` enrollment replacement, and clear token/throttle state for `401` and `403` auth recovery. - Preserve refresh response metadata internally and add focused wire-level and integration coverage. ## Verification Added behavioral coverage proving that: - a valid near-expiry token still completes websocket and pairing requests after transient refresh failures; - `Retry-After` suppresses a subsequent refresh across websocket and pairing callers; - request and response-body timeouts are classified as transient; - an expired token, including one that expires during refresh, cannot proceed to websocket connection; - auth failures clear the attempted token without overwriting a concurrently rotated token.

feat(app-server): add remote control client management RPCs (#25785) ## Why Remote-control clients need to list and revoke controller-device grants without enabling or enrolling the local relay. These are signed-in account-management operations, so coupling them to websocket, pairing, enrollment, or persisted relay state would prevent clients from managing stale grants from the picker. Related enhancement request: N/A. This adds the Codex app-server surface for the planned upstream environment-scoped revoke endpoint. ## What Changed - Added experimental app-server v2 RPCs: - `remoteControl/client/list` - `remoteControl/client/revoke` - Added picker-oriented protocol types and standard generated schema fixtures. The list response intentionally omits backend account id, enrollment status, and location fields. - Added `app-server-transport/src/transport/remote_control/clients.rs` for environment-scoped GET and DELETE requests. It builds escaped URL path segments, forwards optional pagination query fields, sends ChatGPT auth plus `chatgpt-account-id`, converts RFC3339 `last_seen_at` values to Unix seconds, accepts `204 No Content` revoke responses, and retries once after a `401`. - Extracted shared ChatGPT auth loading and recovery into `app-server-transport/src/transport/remote_control/auth.rs` so websocket, pairing, and client management use the same account-auth boundary. - Retained the configured remote-control base URL on `RemoteControlHandle` and resolve management URLs lazily, preserving deferred validation while relay startup is disabled. - Registered list as `global_shared_read("remote-control-clients")` and revoke as `global("remote-control-clients")`. ## Verification - Added transport coverage proving list and revoke work while relay state is disabled, IDs are escaped, picker-only fields are returned, timestamps are converted, revoke accepts `204`, auth headers are forwarded, `401` retries exactly once, `403` is not retried, and malformed list payloads retain decode context. - Added an app-server integration test proving both JSON-RPC methods work before relay enablement and successful revoke returns `{}`. - Regenerated and validated experimental and standard app-server schema fixtures.

feat(app-server): enforce managed remote control disable (#27961) ## Why Managed deployments need a reliable deny gate for remote control. Persisted enablement and explicit startup requests currently remain able to start the transport, while the removed `features.remote_control` key is intentionally only a compatibility no-op. This adds a dedicated requirement that administrators can use to force remote control off without deleting the user's persisted preference. Removing the requirement and restarting restores the prior choice. ## What Changed - Added top-level `allow_remote_control` requirements parsing, sourced layer precedence, debug output, and `configRequirements/read` exposure as `allowRemoteControl`. - Added a typed transport policy captured from the startup requirements snapshot. Managed disable forces the initial state to disabled and prevents enrollment, refresh, connection, and persisted-preference mutation. - Rejected every `remoteControl/*` RPC before parameter deserialization with JSON-RPC `-32600` and `remote control is disabled by managed requirements`. - Preserved the existing disabled status notification and the previous behavior when the requirement is `true` or omitted. - Regenerated app-server protocol schemas and documented the new requirement. ## Verification - Confirmed all remote-control RPCs, including a malformed request, return the managed-policy error while the initial status notification remains `disabled`. - Confirmed explicit ephemeral startup and persisted enablement make no backend connection and leave the SQLite preference unchanged. - Confirmed `allow_remote_control = true` does not enable or block remote control and `configRequirements/read` returns `allowRemoteControl: false` for the deny policy. Related issue: N/A (managed-policy hardening).

feat(remote-control): add pairing start (#25675) ## Why Remote control enrollment authorizes a desktop server, but app-server v2 did not expose the follow-up pairing operation needed to mint a short-lived controller pairing artifact from that enrolled server. Clients need a narrow RPC that starts pairing without exposing the backend `serverId` or conflating pairing with websocket connection state. Issue: N/A; internal remote-control pairing API change. ## What Changed Added experimental app-server v2 `remoteControl/pairing/start` with `manualCode` input and `pairingCode`, nullable `manualPairingCode`, `environmentId`, and Unix-seconds `expiresAt` output. The method serializes under its own `global("remote-control-pairing")` scope and is documented in `app-server/README.md`. Extended the remote-control transport with private `/server/pair` request/response types and normalized `pair_url` handling. Pairing uses the current enrolled server bearer, refreshes that bearer when needed, keeps backend `server_id` private, validates returned `server_id` and `environment_id` against the current enrollment, and preserves backend status/header/body context for failures and malformed responses. Wired the request through `RemoteControlRequestProcessor` and `MessageProcessor`, mapping unavailable/disabled pairing to `invalid_request` and backend failures to internal errors. ## Verification - `just test -p codex-app-server-transport` - `just test -p codex-app-server remote_control_pairing_start_returns_pairing_artifacts`

feat(app-server): migrate remote control to server tokens (#24141) ## Why `codex-backend` now authenticates remote-control server websocket connections with short-lived server tokens instead of the user's ChatGPT access token. `app-server` needs to mint and refresh those server tokens without persisting them, so a restart can reconnect from durable enrollment identity while keeping the bearer token memory-only. ## What Changed Updated the remote-control transport to consume `remote_control_token` and `expires_at` from server enroll responses and added `/server/refresh` support for persisted enrollments or expiring cached tokens. Websocket handshakes now send `Authorization: Bearer <remote_control_token>` with the existing server identity headers, and no longer send the ChatGPT bearer token or `chatgpt-account-id` on that websocket path. The in-memory enrollment state now owns the ephemeral server token cache, while SQLite still persists only `server_id`, `environment_id`, and `server_name`. Websocket `401`/`403` clears only the cached token for refresh on reconnect; websocket or refresh `404` clears stale persisted enrollment and re-enrolls. Response body previews redact `remote_control_token` before surfacing parse errors. ## Verification - `just test -p codex-app-server-transport` - Manual prod smoke with an isolated `CODEX_HOME`: `codex remote-control --json -c 'chatgpt_base_url="https://chatgpt.com/backend-api"'` reached `status:"connected"` with `environmentId:"env_i_6a17d9f1d764832986da2e80f4554f1b"`.

feat(exec-server): use protobuf relay frames (#22343) ## Why Remote exec-server now needs one executor websocket to serve multiple harness JSON-RPC sessions. Rendezvous routes by `stream_id`, and the exec-server side needs to use the same stable relay frame contract instead of a hand-rolled JSON shape. The relay protocol also needs to make ownership boundaries clear: harness and executor endpoints own sequencing, acks, retries, duplicate suppression, segmentation, and reassembly; rendezvous only routes frames. ## What Changed - Add the checked-in `codex.exec_server.relay.v1.RelayMessageFrame` proto plus generated prost bindings for `codex-exec-server`. - Encode remote harness/executor relay traffic as binary protobuf websocket frames while keeping local websocket JSON-RPC unchanged. - Demux executor-side relay streams into independent `ConnectionProcessor` sessions keyed by `stream_id`. - Add a programmatic `RemoteExecutorConfig::with_bearer_token(...)` constructor for non-CLI callers and integration tests. - Add an integration test that starts the remote executor against a fake registry/rendezvous websocket and verifies two virtual streams share one executor websocket without cross-talk, including per-stream reset behavior. - Document the remote relay envelope, sequence ranges, `ack`/`ack_bits`, and endpoint responsibilities in `exec-server/README.md`. ## Verification - `cargo test -p codex-exec-server --test relay multiplexed_remote_executor_routes_independent_virtual_streams -- --exact` - `cargo test -p codex-exec-server --test relay` - `cargo test -p codex-exec-server` passed outside the sandbox. The sandboxed run hit macOS `sandbox-exec: sandbox_apply: Operation not permitted` in filesystem sandbox tests.

feat(remote-control): allow pairing while disabled (#26215) ## Why `remoteControl/pairing/start` creates authorization for future remote-control connections, so it should not require the live websocket to already be enabled. Requiring enable first made pairing depend on presence instead of the persisted server enrollment that pairing actually uses. Pairing also needs to recover when that persisted server row is stale. If `/server/pair` returns `404`, making the first pairing attempt fail forces a manual retry even though the client can clear the stale row and create a replacement enrollment immediately. ## What Changed - Allow `remoteControl/pairing/start` to reuse or create the persisted remote-control server enrollment while remote control is disabled. - Keep the selected in-memory enrollment across disable and share it with websocket connect so a later enable uses the same selected server. - Thread the app-server client name through pairing so stdio persistence keeps using the websocket-owned enrollment key. - Recover pairing server-token auth failures through the existing refresh/auth-recovery path. - Recover stale pairing enrollment on `/server/pair` `404` by clearing the stale selected enrollment, re-enrolling once, and retrying pairing once. - Add focused disabled-pairing and stale-pairing recovery coverage. ## Verification - `remote_control_pairing_start_returns_pairing_artifacts_while_disabled` exercises pairing before enable. - `remote_control_handle_reenrolls_after_stale_pairing_enrollment` exercises stale `/server/pair` `404` recovery without a manual retry. Related: N/A

feat(exec-server): add Noise rendezvous environment (#28774) ## Why Codex can run a remote exec server through the Noise relay, but the normal environment-manager path could not establish an environment-registry-backed harness connection. Signed rendezvous URLs and harness authorizations are short-lived, so reconnects must fetch a fresh bundle instead of retaining stale connection credentials. A stalled registry request must also fail within the regular remote connection deadline, without exposing these credentials in debug logs. Issue: N/A (internal environment-service integration). ## What Changed - Add environment-manager configuration for a registry-backed Noise rendezvous environment. - Request a fresh bundle from `/cloud/environment/{environment_id}/connect` for every physical harness connection, using the existing 10-second remote connection timeout. - Share the Environment Registry register, connect, and validate wire payloads through `codex-exec-server` and `codex-core-api`. - Redact the signed rendezvous URL and harness authorization from the public connect response's `Debug` output. - Add focused coverage for registry bundle retrieval, stalled requests, and credential redaction.

fix(app-server): avoid blocking connection cleanup (#26852) ## Why Remote-control app-server sessions can reconnect every 5-7 seconds when the shared transport-event queue fills. The queue's consumer handled `ConnectionClosed` by awaiting all in-flight RPCs for the disconnected connection. A stuck RPC therefore blocked processing of replacement connection and initialize events until remote-control forwarding hit its five-second timeout and reconnected again. Related issue: N/A (internal remote-control incident investigation). ## What Changed - Split fast RPC admission closure from draining: `ConnectionRpcGate::close()` rejects queued and future RPCs, while `shutdown()` continues waiting for RPCs that already started. - Close a disconnected connection's RPC gate before spawning the existing RPC drain and resource cleanup in a tracked background task, so the transport-event consumer remains available without waiting for active RPCs. - Reap completed cleanup tasks during normal operation, drain them during graceful shutdown, and abort them during forced shutdown. - Add regression coverage for closing with an active RPC, rejecting post-close requests without polling them, and preserving the existing shutdown wait behavior. ## Verification `just test -p codex-app-server --lib connection_rpc_gate` passes all 6 tests, including the new close-versus-drain regression coverage.

fix(remote-control): preserve enrollment on generic websocket 404s (#26741) ## Why A remote-control WebSocket handshake can receive a generic HTTP 404 when an intermediary routes the request without preserving the WebSocket upgrade. Treating every 404 as proof that the remote app server is gone clears valid enrollment and causes repeated re-enrollment, new environment and server IDs, Habitat churn, and noisy `/server/enroll` traffic. ## What Changed - Clear enrollment only when a 404 JSON response explicitly contains `{"detail":"Remote app server not found"}`. - Preserve enrollment for empty, plain-text, malformed, or otherwise unrecognized 404 responses, return the transport error, and retry with the existing reconnect backoff. - Log the status, correlation headers (`request-id` or `x-oai-request-id`, plus `cf-ray`), and bounded/redacted response body for unrecognized 404s. - Cover both explicit missing-server re-enrollment and generic 404 enrollment preservation/reconnect behavior. ## Verification `just test -p codex-app-server-transport` passes all 114 tests on the rebased branch, including the targeted explicit and generic WebSocket 404 scenarios. Related issue: N/A

Support staging OAuth client ID overrides (#28257) ## Summary - allow app-server ChatGPT login to use a configured OAuth client ID - reuse the same client ID for refresh and revoke requests - cover staging login, refresh, and revoke request payloads ## Tests - `just test -p codex-login` - `just test -p codex-app-server login_account_chatgpt_uses_debug_oauth_overrides` - `just test -p codex-login logout_with_revoke_revokes_refresh_token_then_removes_auth` - `just fix -p codex-login` - `just fix -p codex-app-server` - `just fmt`

fix(remote-control): retry after auth recovery (#23775) ## Why When remote control hits an auth failure such as a revoked or reused refresh token, the websocket loop falls into reconnect backoff. If the user fixes auth while that loop is sleeping, remote control can stay offline until the old retry timer expires because nothing wakes the loop or resets its exhausted auth recovery state. ## What Changed Added an auth-change watch on `AuthManager` for refresh-relevant cached auth updates. The remote-control websocket loop now subscribes to that signal, resets `UnauthorizedRecovery` and reconnect backoff when auth changes, and retries immediately instead of waiting for the previous delay. Updated the remote-control transport test to verify that reloading auth with the now-available account id wakes enrollment before the prior retry delay. ## Verification `cargo test -p codex-app-server-transport remote_control_waits_for_account_id_before_enrolling`

fix(remote-control): cap reconnect backoff (#24164) ## Why Remote-control websocket reconnects currently use the shared exponential backoff helper without a local ceiling, so a long failure streak can stretch retries out indefinitely and leave the runtime behavior hard to inspect from logs. ## What Changed Cap the remote-control reconnect delay at 30 seconds, then reset the reconnect attempt counter once that capped delay is emitted so the next failure starts from the initial jittered delay again. The reconnect failure log now records the attempt number, chosen delay, and whether the cap triggered a reset, with a separate info log when the backoff counter is reset after the cap. ## Verification `just test -p codex-app-server-transport` Related issue: N/A

feat(remote-control): add daemon pairing command (#29913) ## Why Users who run Codex remote control through daemon mode can keep the daemon running, but they do not have a CLI path to mint the short-lived manual pairing code needed to connect another device. Without this command, they need to speak app-server JSON-RPC directly. Related: #25675 ## What Changed - Added `codex remote-control pair`, which connects to the existing daemon control socket and calls `remoteControl/pairing/start` with `manualCode: true`. - Kept the command non-lifecycle-mutating: it does not start, enable, or restart the daemon. - Human output labels the manual code as `Pairing code: ...`; `--json` preserves the full pairing response. - Added daemon socket-client, CLI formatting, and parser coverage. ## Verification - `remote_control_client::tests::start_pairing_requests_manual_code` verifies the daemon client sends `{ "manualCode": true }` and parses the complete response. - `remote_control_cmd::tests::remote_control_pairing_human_output_labels_the_manual_code` verifies the human-facing output.

Refresh signed exec-server URLs on reconnect (#28374) ## Summary - add a provider API that supplies a fresh signed WebSocket URL for each remote exec-server connection - refresh the signed URL after disconnects and retry once when a handshake returns `401 Unauthorized` - allow `EnvironmentManager` consumers to register remote environments backed by the URL provider ## Tests - `just test -p codex-exec-server -E 'test(remote_websocket_client_refreshes_url_after_unauthorized_handshake) | test(remote_websocket_client_refreshes_url_after_disconnect)'` — 2 passed - `cargo check -p codex-core-api` — passed - `just fix -p codex-exec-server` — passed - `just fix -p codex-core-api` — no test targets; no-op - `just fmt` — passed - `just test -p codex-exec-server` — 187 passed; 32 unrelated macOS sandbox tests could not invoke nested `sandbox-exec` (`Operation not permitted`)

Preserve zsh tied PATH exports in shell snapshots (#34293) ## Why zsh prints tied scalar/array exports such as `PATH` and `path` with `-T` metadata. The snapshot filter rejected that syntax, so commands available only through the configured `PATH` could not be resolved from a restored snapshot. ## What changed - Recognize writable tied exports and retain them in zsh shell snapshots. - Skip readonly exports so restoring a snapshot does not make variables such as `PATH` immutable. ## Testing Added macOS coverage for restoring a tied `PATH`, handling entries with spaces, deduplicating path entries, excluding readonly tied exports, and resolving a command available only through the restored snapshot. GitOrigin-RevId: 297cfae3368230c7f6fe5ef84f0a10b35a2c802c

Add context to sampling retry logs (#33963) ## What changed - Add structured `turn_id`, `retries`, `max_retries`, and `sampling_error` fields to sampling-stream retry warnings. - Add a focused test that captures the warning and verifies its retry and error context. GitOrigin-RevId: dea868875d162b3f97b3c8a256359278f7267c1c

fix(core): instrument stalled tool-listing handoff (#24667) ## Why When a turn needs a follow-up request after tool output is recorded, Codex can still appear stuck in `Thinking` before the next `/responses` request is opened. The existing local trace showed the last completed response and the absence of a new backend request, but it did not show whether the stall was in tool-router preparation or later request setup. Issue: N/A (internal incident investigation) ## What Changed Added trace spans around the pre-stream tool-router handoff in `core/src/session/turn.rs`, including the `built_tools` phase and the MCP manager read lock. Added per-server MCP tool-listing spans and trace breadcrumbs in `codex-mcp/src/connection_manager.rs` with startup snapshot / startup-complete state so a pending MCP client is visible in feedback logs instead of looking like a silent hang. ## Verification - `just fmt` - `just test -p codex-mcp` - `just test -p codex-core` (prior full rerun fails in this workspace on unrelated integration tests: code-mode output length expectations, one shell timeout formatting assertion, and shell snapshot timeouts; latest review-fix rerun compiled and passed 1160 tests before I stopped the abnormally slow unrelated suite)