Chris Tolliday

Fail instead of hang when a paged-out value can't be hydrated Summary: Currently paging deserialization failures cause DICE to hang, which seems to be because `WorkerCancelled` finishes the task *without ever producing a result*, so every awaiting computation blocked forever. Surface the failure as a cancelled computation instead, so awaiters resolve promptly. The `MatchPagedOut` / `CheckDepsPagedOut` lookup arms now finish the task with a result via `finish_hydration_failure` on hydration failure. A paged-out value is a cache entry and is always recomputable, so a hydration failure can and should degrade to a recompute rather than a hard error, which is done in the next diff. This also reports a soft_error propagated from DICE via the DiceEvent tracker. As is, this doesn't propagate the error message for deserialization failures except via the soft_error because the next diff will need another channel for propagating deserialization errors if they don't fail builds (and making this diff propagate the error correctly is not straightforward). Reviewed By: Nero5023 Differential Revision: D113503770 fbshipit-source-id: 815eb7ac62213da8ef8508b1bf3610d39df3e66f

Measure the memory paging actually moves, from the allocator Summary: The DataKey byte counters say how much paging wrote and read, not how much memory it freed. Those differ in ways that never wash out: an arc shared with a still-resident value is serialized but not dropped, an arc already on disk is dropped but not written, and `evict_keys` skips nodes that are missing, vacant or injected after their bytes were already counted. Differencing two growing counters accumulates all of that. Measure the memory instead, at the two points it actually moves: - page-out, on the core state thread in `evict_keys`. The graph holds the last reference — `page_out_value` consumed and dropped the worker's copy before the eviction was queued — so that is where the free happens, and jemalloc charges a free to the thread performing it whichever thread allocated it. - page-in, around the synchronous deserialize in `hydrate`. No await inside the window, so tokio cannot migrate us mid-measurement. Both take the net of allocated and deallocated so transient buffers cancel. An arc that is not actually freed contributes nothing, which is the case the byte counters get wrong. The counters come from jemalloc's `thread.{allocated,deallocated}p`, reached through buck2's own weak-symbol jemalloc bridge in `facebook/mem_frag` — the folly Rust binding only exposes `bool`. The pointers are resolved once per thread and `volatile`, since the compiler otherwise assumes an allocation cannot change global state. dice takes the reader as an injected `fn` so it stays allocator-agnostic and OSS-clean; the daemon installs it under `fbcode_build`. `offloaded - restored` is an upper bound on what paging is currently saving: it does not subtract values paged out and later invalidated, whose memory would have been reclaimed anyway. Bounding that is left for a follow-up. Reviewed By: Nero5023 Differential Revision: D117141211 fbshipit-source-id: 31139583123beb9857dc2814aa5cd36e730b9807

Create single choke point for VersionedGraph node mutation Summary: Structural refactor that funnels all `VersionedGraph` node mutation through 2 choke points - a `NodeMut` drop and `retain_injected`. This enables the next diff to maintain a list of page-out candidates that is kept in sync with the state of the actual values in `nodes`. The raw node map becomes private to the store module and is reached only through: `node_mut` (an in-place mutation guard over a present node), `node_entry` (mutate-or-insert resolved in one `entry()` lookup, yielding a `NodeMut` guard when present or a `VacantSlot` when absent), and `nodes()` (read-only access for iteration, lookup, and count). Reviewed By: JakobDegen Differential Revision: D112270660 fbshipit-source-id: b47e741da13cd40eeedef5b0c6a4e9a6765465f4

Page out DICE to disk when the daemon goes idle Summary: Adds a `buck2_hydration.page_out_on_idle` buckconfig. When set, the `finalize` of a finishing command schedules a background task that pages the DICE graph out to disk to reclaim memory once the daemon is idle; paged-out values hydrate back in on demand. Only the sole active command triggers a page-out: `finalize` consults the active-command registry (keyed by trace id) and schedules the page-out only when no other command is active, so of a set of concurrent commands just the last to finish triggers it. The task first waits for the residual DICE work of the finished command to drain (`wait_for_idle`) before paging out. Cancellation: a new `Dice::page_out_cancellable` threads a cancel check through the page-out workers so they yield promptly, leaving a partially paged-out graph (a valid state). Any new command cancels a running page-out (`cancel_active_page_out`, called from `ActiveCommand::new`), so it yields CPU, I/O, and the DICE state thread back to real work. One lock-free `AtomicU8` state (Idle/Running/Cancelled) serves as both the single-flight guard and the cancel flag; the cancel check passed into DICE is a capture-free `fn() -> bool` (`PageOutCancel`), so it needs no allocation. `buck2 debug hydration status` reports whether an idle page-out is in progress, and the invocation record carries a `page-out-on-idle` tag with the config value. With this initial diff there is overhead to scanning the DICE graph to see if there are any nodes to be paged out; later diffs make this essentially free to run on every command. Reviewed By: jtbraun Differential Revision: D110380917 fbshipit-source-id: 33b9789ebbf1f247cfd9cdc2f742e79915b33ec4

Categorize MISSING_INTERNAL_PATH errors Summary: Main motivation is to give a specific tag to the error that caused S693548 since it was obscured by other unrelated errors, some of which are input/environment errors. Also giving the rest of these specific tags since these errors are all being hit in production with some frequency. May or may not be worth investigating some of these tier0 errors depending on how often they are being hit. Reviewed By: Nero5023 Differential Revision: D115633388 fbshipit-source-id: 1338ac1dcccf77ea83de8244654d37cbe801b170

Skip re-paging-out DICE values that can't be serialized (NonPageable) Summary: When an idle page-out considers a value but can't serialize it (e.g. `NoValueSerialize`, or a serialization error), mark the node `PagedState::NonPageable` so it isn't offered as a page-out candidate again until it is recomputed. Otherwise every idle page-out would retry the same unserializable values indefinitely. Adds the `NonPageable` variant, `mark_non_pageable` (plumbed through the state thread), and routes the page-out serialization-failure path to mark them (works similarly to `evict_keys`). `keys_to_page_out` now selects candidates by `paged_state == NeverPagedOut` rather than the absence of a `DataKey`, which also excludes paged-back-in nodes. Reviewed By: jtbraun Differential Revision: D112056837 fbshipit-source-id: 56d73de7eef666b9b494c033960b3edefa80d6fd

Meter DataKey-level page-out/page-in I/O in the storage layer Summary: Add a `MeteredPagableStorage` decorator over the `PagableStorage` backend that tallies every DataKey blob written (`store_data`) and read (`fetch_data` / `fetch_data_blocking`) into a shared `StorageIoMetrics` (four running atomics: `data_keys_out`/`bytes_out`/`data_keys_in`/`bytes_in`). `bytes_*` sum each DataKey's serialized value payload (`PagableData::data`). Counting at the trait boundary — rather than in `DiceStorage::page_out_value` / `hydrate` — is what captures nested `PagableArc` sub-values: they are written by the default `page_out_item` and read lazily via `PagableStorageHandle`, both of which route back through the decorator. This closes the gap where the existing per-DICE-value `page_in_*` metrics count only the top-level blob. `DiceStorage::new` wraps its backend in the decorator and holds the shared metrics. `Dice::storage_io_metrics()` returns `Option<StorageIoSnapshot>` — `None` when pagable storage is not configured, so that downstream logging can distinguish "paging is off" from "paging is on and moved nothing". `store_data` records only after the inner write succeeds, matching the read path. The byte totals are a proxy for allocated in-memory bytes moved (not RSS), not guaranteed-freed memory: a `PagableArc` shared with a still-resident value is serialized here without being dropped, so `bytes_out` over-counts under partial page-out. Assuming arcs are referenced only by DICE values, a full page-out drops every arc and a subsequent full page-in re-allocates matching bytes, so `bytes_out - bytes_in` approximates the allocated bytes currently offloaded. Wiring these totals into the invocation record and `PageOutSummary` is a follow-up. Reviewed By: Nero5023 Differential Revision: D117141213 fbshipit-source-id: 800fc1b3aa2882145cbc55918a12335456c8575d

Let `--overall-timeout` interrupt a worker request Summary: `LocalExecutor` passes the command's `LivelinessObserver` into the non-worker branch of `exec` and not into the worker branch, four lines apart: let r = if let Some(worker) = worker { Ok(worker.exec_cmd(request.args(), env, request.timeout()).await) } else { self.exec(..., request.timeout(), ..., liveliness_observer, ...).await }; So `WorkerHandle::exec_cmd`'s `tokio::select!` has exactly two arms: the gRPC response, and the worker process dying. A command dispatched to a persistent worker is therefore uninterruptible — cancellation cannot reach it, and neither can `--overall-timeout`, whose whole purpose is to make buck2 stop and report before its caller loses patience. This is not theoretical. Citadel's Reality Labs on-device tests run through a persistent worker that leases physical hardware, and roughly 10,700 jobs a week reach their 3h Sandcastle step timeout having written zero test results. Pulling the event log for one (`f3f9198e-98c4-42a7-86b3-99244e0936f1`) and diffing span starts against span ends shows exactly five spans open at SIGKILL: t+0s Command (buck2 test) t+4s TestDiscovery suite=...tesser/console/test:ConsoleTest t+4s WorkerQueued -> WorkerInit (worker spawned fine) t+8s ExecutorStage Local/WorkerExecute (request sent) ... nothing for 10,791 seconds ... t+10,799s SIGKILL `actions_running_count` is 0 in the final snapshot — buck2 was doing nothing but waiting on that one RPC. A second job (`738615cc...`, a different suite) is identical to the second. The `--overall-timeout` deadline fired at t+10,200s and changed nothing, because the only thing consulting the observer was code this command was no longer running. Adding the observer as a third `select!` arm makes the worker branch behave like the non-worker branch: the command comes back as `GatherOutputStatus::Cancelled` and the existing `manager.cancel_claim` path handles it. Sandcastle's step timeout then finds a buck2 that has already exited and reported, instead of one to SIGKILL. Note this does not tell the worker to stop. The `Execute` RPC has no cancellation and the `Exec`/`ExecuteCancel` pair in `worker.proto` that does is unimplemented on both sides (buck2 still calls the deprecated `Execute`; the RL worker returns `unimplemented` for `Exec`). Dropping the request sends RST_STREAM, so a worker serving it inline sees its handler dropped, but one that has already queued the command internally keeps going until the `WorkerPool` tears its process down at the end of the buck2 command. Returning early is still strictly better than the status quo — today buck2 waits forever *and* the worker keeps going. Scope: this only changes what happens once a command is already being cancelled, so healthy runs are unaffected — the observer stays alive and the arm never fires. The per-command `request.timeout()` is still unenforced by buck2 for worker commands; that is the next diff in this stack. Reviewed By: NavidQar, scottcao Differential Revision: D116740783 fbshipit-source-id: 1269d6df9b603da00ee31263198e8d3e4a22a21e

Idle page-out: cancellation policy, page_out_triggered, status --wait, and e2e test Summary: Refinement, observability, and testing on top of the idle page-out core (previous diff). Cancellation policy: the core diff cancels a running idle page-out from `ActiveCommand::new` for every command. This gates that on `StreamingCommandOptions::cancels_idle_page_out()` (default true), fired in `run_streaming`. Commands that do no DICE graph work opt out. This isn't strictly necessary but is needed for `debug hydration status --wait` to work. `debug hydration status --wait`: blocks until any in-progress idle page-out finishes before reporting, so callers (and tests) observe the settled state without polling. Implemented with a `Notify` the page-out guard signals on drop. `page_out_triggered`: the invocation record reports whether a command triggered an idle page-out (triggering, not completion), via a `PageOutTriggered` instant event emitted from `finalize`. e2e test `test_page_out_on_idle`: a cold build triggers the idle page-out (asserts `page_out_triggered`), `status --wait` confirms it finished and actually paged values out, and the next build pages them back in. Reviewed By: jtbraun Differential Revision: D111861750 fbshipit-source-id: 18d89dafab190770a7d25b677ac5e7da94cfd2ba

Move idle page-out mechanism into paging module Summary: Leave `hydration.rs` with only the `buck2 debug hydration` command code (`hydration_command`, `HydrationServerCommand`, `format_status_summary`) and move the page-out mechanism it drives into `paging.rs`, alongside the existing paging telemetry (`PagingManager`). This only moves code. Later diffs consolidate/simplify more of this logic into `PagingManager`, which is easier when it's in one place. Reviewed By: Nero5023 Differential Revision: D113718192 fbshipit-source-id: 97cc255ca75fc52b66ed7241130a971d50be2c07