Chenhao Zuo

Reuse resident heaps during page-in Summary: This completes the shared-heap paging fix started by the previous diff. In the X/Y example, after X is paged out and evicted, Y may still keep heap A alive. When X is paged back in, reconstructing a second allocation of A would give both allocations the same `HeapRefId`. A subsequent page-out can register the chunks of one allocation and then hit the pointer-not-found panic when it serializes a value from the other. Track each registered heap as a weak `HeapRefId` to `FrozenFrozenHeap` association. Page-in reuses A when Y still keeps it alive and resolves the serialized `value_index` for X through the existing chunk index for A, while still consuming the serialized heap body to keep the deserializer cursor aligned. If A is no longer resident, the existing lazy heap reconstruction path remains unchanged. Pointer-identity-checked cleanup prevents an older allocation from unregistering a newer resident entry with the same ID. Reviewed By: christolliday Differential Revision: D113883882 fbshipit-source-id: 4b4345dbbe733ae68e7addc8bd1d5dba991db45a

Make storage-lifetime state explicit Summary: The former `SessionContext` name hid the fact that its state survives individual serialization and deserialization operations. Rename it to `StorageContext`, make storage-owned extensions opt in through `StorageState`, and return shared typed `Arc`s from the registry. Update every backend and user to obtain this state from the storage instance, including making the in-memory owner and handle share one context. This establishes a clear storage-lifetime boundary before introducing the separate root page-in lifetime. Reviewed By: christolliday Differential Revision: D114793222 fbshipit-source-id: 5045510474865c5ad58b42af80fabe3731f1f02a

Introduce root-scoped page-in identity Summary: In deserialization, Starlark must resolve serialized heap references through a `HeapRefId -> exact heap` binding. That logical ID is expected to be unambiguous within one stored DICE root's reachable heap graph, but it is not unique across independent roots or different incarnations retained by DICE. The binding state therefore needs the lifetime and identity of one top-level page-in, rather than the whole storage backend. Add `PageInScope`, created exactly once by `PagableStorageHandle::root_deserializer` for each top-level hydration. Nested Arc deserializers and deferred recipes inherit the same scope, while a separate root deserializer receives a distinct scope even when it reads the same `DataKey`. Make concrete deserializer construction internal so Pagable users define only the root boundary and cannot accidentally substitute a new scope for nested work. The scope carries the root key and provides the lifetime anchor for root-specific state added next. Reviewed By: christolliday Differential Revision: D114793227 fbshipit-source-id: 3bdfe1c243c7f7aa0dd857e02185a06cdb9ad7a4

Add PartialPagableArc with canonical stored identity Summary: Introduce `PartialPagableArc<T>` for values whose restored in-memory form can remain incomplete. It provides a stable Arc allocation, a write-once canonical `DataKey`, shared identity across clones, and a weak counterpart that does not extend the value's lifetime. Before a key is assigned, the Arc can serialize a new value normally. After restoration, page-out must reuse the stored key and direct inner serialization is rejected. Migrate `FrozenHeapRef` to this representation so a lazy frozen-heap skeleton can materialize values on demand without ever replacing its complete stored representation with a partial one. Reviewed By: christolliday Differential Revision: D114793220 fbshipit-source-id: 22f97722137cbf7a1f9bb58b4dd065d79f0c8424

Cross-process pagable ser/de for BcInstrs Summary: `BcInstrs` is a packed `Box<[u64]>` of `BcInstrRepr` records with embedded `FrozenValue`s; copying the raw words is unsound across processes because the pointers don't relocate. Give `BcInstrs` a manual `StarlarkSerialize`/`StarlarkDeserialize` that walks the instruction stream and routes each instruction's `I::Arg` through its own ser/de, so embedded `FrozenValue`s relocate via the normal mechanism. Wire format: a u8 tag followed by a `(u32 opcode, arg)` stream terminated by `End`. `'static`-backed args re-resolve on read instead of being serialized (`BcNativeFunction` from its frozen value, `KnownMethod` by name); `BcOpcode` serializes as a u32 (there are >256 opcodes). This unblocks paging any `FrozenDef`. Reviewed By: christolliday Differential Revision: D108842732 fbshipit-source-id: acc9b5836a003e287ffe2341b58a2b73a19679f2

Scope heap deserialization state by root Summary: Move Starlark's `HeapRefId` bindings out of storage-lifetime state and into a scope-owned `StarlarkDeserScope`. Each restored `FrozenFrozenHeap` retains its exact `HeapDeserializationState`, recipe, and owning scope; the scope indexes heaps weakly and upgrades the exact owner before accessing arena-backed state. Nested Starlark deserializers now recover their scope structurally from the Pagable recipe instead of accepting arbitrary shared state. Keep only cross-thread wait coordination storage-scoped, because an Arc-cached restored heap can be shared across roots, and key that graph by exact `FrozenHeapPtr` plus `value_index`. This is the commit that makes the DICE cross-root regression pass normally. Reviewed By: christolliday Differential Revision: D114793221 fbshipit-source-id: d17014591ba929b28ebd62b21457eb0c68a7051a

Model DICE page-ins on the critical path Summary: Builds on `ActivationTracker::key_paged_in` to put DICE page-in latency on the build critical path as a typed `PageIn` node, so critical-path output and invocation telemetry attribute hydration cost to the key it gated instead of leaving an unexplained gap. The graph distinguishes the three hydration paths reported by DICE: 1. **Exact cache hit (`Match`)**: DICE already knows the cached entry applies, so hydration is the key's only work and no `key_activated` callback follows. Emit the timed `PageIn(key)` immediately and a zero-duration completion with `key -> PageIn(key)`. 2. **Reuse after dependency validation (`AfterDependencyValidation`)**: DICE validates the cached dependencies, then hydrates the still-valid old value for reuse. Pair the page-in with the subsequent activation and emit `key -> PageIn(key) -> dependencies`. 3. **Hydration after recomputation (`AfterRecompute`)**: DICE recomputes the key first, then hydrates the old value for equality comparison and downstream cutoff. Separate evaluation work from completion and emit `key -> PageIn(key) -> EvaluationWork(key) -> dependencies`, preserving that recomputation happened before hydration. `BuildSignalSender` records the phase until the activation arrives, and `BuildSignalReceiver` pairs the timed page-in with that evaluation data. Constructing the relationships at pairing time preserves phase ordering and avoids duplicated dependency edges from a graph-finalization transform. Page-ins render as typed `CriticalPathEntry2::PageIn` entries carrying the DICE key type. Reviewed By: NavidQar Differential Revision: D111943501 fbshipit-source-id: 611cc66c5fd7e13660072dfd2c2212cea379cdd8

Reproduce cross-root heap scope collision Summary: Add a DICE/Starlark integration regression with two independently stored roots whose dependency heaps have the same logical `HeapRefId` but different contents. The test pauses hydration of the old root, hydrates the new root, and then resumes the old root to demonstrate that storage-global Starlark state can replace H0 with H1 and resolve B0 through the wrong heap. The unfixed revision uses a narrow expected panic at the H0/H1 assertion. The marker is removed when Starlark heap bindings move into the root's `PageInScope`. Reviewed By: christolliday Differential Revision: D114793226 fbshipit-source-id: 791f5377c9b10a58ea2f71261342051b348a073f

lock-free per-slot init state + heap registry Summary: Two related changes prepare `HeapDeserializationState` for partial deserialization's concurrent traversal. **Per-slot init state.** The old `Mutex<HeapDeserializationState>` + `bool initialized` couldn't distinguish "claimed but still being deserialized" from "done": `initialized` flipped to `true` before `starlark_deserialize` ran and the mutex was dropped, so a second thread observing `initialized=true` would read the value before it was fully constructed. Per-slot atomics now encode `0` = not started, `1` = in progress, `2` = failed, and a non-zero aligned header pointer = done. A thread that loses the claim race gets the slot's pre-allocated header pointer back via `ClaimResult::InProgress` (no blocking at this commit); if the winner errors before publishing done it stores `FAILED`, so a `(heap_id, value_index)` resolves to a deterministic failed-slot error. (Blocking on an in-progress slot until its claimer finishes is added later, in 111.) **Single cross-heap registry.** `StarlarkDeserState.heap_deser_states[heap_id]` becomes the canonical source for any heap's state. The implicit "current heap" plumbing — `current_heap_deser_state` field, `CurrentHeapDeserState` session-context entry, `find_by_frozen_value` / `ptr_to_index` reverse-map — all go away. Every `FrozenValue` reference is fully qualified by `(heap_id, value_index)` on the wire, so `ensure_initialized` resolves directly through the registry. Reviewed By: christolliday Differential Revision: D105131368 fbshipit-source-id: 66a4587991e87ac3c77fb7c009275db6c30fb71b

Remove `skip_pagable` from `#[starlark_value]` Summary: `skip_pagable` defaulted to `true` and could only be set to `true` (there was no way to set it to `false`), making the flag and its associated `impl_starlark_pagable` method dead code. Remove the attribute, its parsing, the dead method, and all ~100 call sites. Reviewed By: christolliday Differential Revision: D106569109 fbshipit-source-id: ec41d2867c9dc544da10839664ceaf214085076b

Diagnose unresolved frozen pointers Summary: When pagable serialization cannot resolve a `FrozenValue` pointer, include the target type, nearby chunk-index entries, and a scan of live registered heaps in `FrozenValueNotRegistered`. Classify whether the evidence points to an inconsistent chunk index or a missing heap ownership/registration edge. Add structured, `#[cold]` diagnostic searches for one heap, the source heap dependency graph, and all live heaps registered in the serialization state. Keep this error-only implementation grouped before the tests so the normal heap and serialization paths remain easy to follow. Reviewed By: christolliday Differential Revision: D115329718 fbshipit-source-id: 7f54b88bc9cb4323f5d542c4518606cf51a8fb5a

Reuse stored Arc keys across page-out backends Summary: A restored `FrozenHeap` may be only partially deserialized in memory even though its complete canonical representation already exists in storage. Serializing the partial in-memory heap directly is difficult because unmaterialized values are represented only by deferred recipe state, not ordinary resident values that the serializer can traverse. Record the stored `DataKey` on erased Arcs during deserialization, and make the generic, in-memory, and Sled recursive page-out paths reuse that key instead of invoking the inner serializer. This establishes the storage-identity contract required by `PartialPagableArc` in the next commit. Reviewed By: christolliday Differential Revision: D114793219 fbshipit-source-id: 83e68eb68d3d46759d97b5adc0c55e0a37c77fb8

introduce PagableDeserializerRecipe Summary: Pagable-framework groundwork for partial deserialization. Cross-heap resolution will need to reopen a specific heap's bytes long after the outer `deserialize_arc` callback returns — `self.pagable` may be pointing at the wrong stream by then. **New trait — `PagableDeserializerRecipe`.** A reconstructable handle to a `PagableDeserializer`. Callers stash a recipe and call `open(storage)` later to get a fresh deserializer over the same bytes, without holding the original deserializer live. **Storage is passed to `open()`, not held on the recipe.** A recipe that stored its own `PagableStorageHandle` would form an `Arc` cycle: storage owns a `SessionContext` that may stash `Arc<dyn PagableDeserializerRecipe>` (e.g. for cross-heap pointer resolution in later diffs), which would point back at the same storage. Passing storage in at `open()` time keeps the recipe pure data — no cycle is possible by construction. Callers source storage from the active deserializer via its `storage()` method. **Recipe for `PagableDeserializerImpl` — `PagableDeserializerRecipeImpl`.** Backed by `Arc<PagableData>` (owned bytes + nested-arc `DataKey`s). The data is owned, so a recipe can be parked in a long-lived map and reopened after the `deserialize_arc` call that handed it out has returned. **Wired into `deserialize_arc`.** The callback signature now receives an `Arc<dyn PagableDeserializerRecipe>` alongside the sub-deserializer. Most callbacks ignore it; consumers that need deferred reads — `FrozenHeapRef` in the next diff — stash it for later. Both `PagableDeserializerImpl` and `TestingDeserializer` are updated to construct the right recipe shape. Reviewed By: christolliday Differential Revision: D105131367 fbshipit-source-id: 4aca2cef46ebf6e90ebcfd61bd60aa99b295abda

Add `debug hydration status` to report DICE hydration Summary: Adds a `status` subcommand under `buck2 debug hydration` that reports how many DICE graph node values are resident in memory vs paged out to pagable storage, with a per-key-type breakdown sorted by total (resident + paged-out) descending. It complements the existing `page-out` / `page-in` subcommands. Unlike `page-out` / `page-in`, `status` is read-only: it does not require an idle DICE and runs non-exclusively, so it can be issued during a build without blocking, or being blocked by, that build. It reports a consistent snapshot taken on the DICE core-state thread. Wiring: a new `STATUS` value on `HydrationSubcommand` and a new `HydrationResponse { summary }` proto message (the `Hydration` RPC now returns `HydrationResponse` instead of `GenericResponse`); `CoreState::hydration_status` classifies occupied nodes as resident (value in memory) vs paged out (only a `DataKey` remains) behind a new `StateRequest::HydrationStatus`; `Dice::hydration_status` resolves keys to per-type counts via the key index; the server renders the summary via `format_status_summary` and the client prints it. Reviewed By: christolliday Differential Revision: D109364561 fbshipit-source-id: 79f9240121b5930fbaf64ab2e6bf352c25b66d6c

Drop `starlark` dep from `buck2_core`, `buck2_fs`, `buck2_util` Summary: `buck2_common` can't depend on `starlark` (banned in `app_dep_graph_rules/rules.bzl`) — the buck2 client binaries depend on `buck2_common` and shouldn't pull in the starlark interpreter. Adding `starlark` to any foundation crate (`buck2_core`, `buck2_fs`, `buck2_util`) creates a banned transitive path via `buck2_common`. This drops the `starlark` dep from those three crates: removes the `StarlarkPagableViaPagable` derive from their types, and removes the orphan `ThinBoxSlice<T>: StarlarkSerialize/StarlarkDeserialize` impls from `buck2_util`. Affected downstream `StarlarkPagable` derives are updated at the use sites — either with `#[starlark_pagable(pagable)]` (pagable-only fields) or `#[starlark_pagable(serialize_with = "...", deserialize_with = "...")]` helpers (mixed pagable/starlark fields). The `StarlarkPagable` derive in `starlark_derive` is extended to accept `serialize_with`/`deserialize_with` on enum variant fields, matching the existing struct-field behavior. Foundation types touched (kept `Pagable`, dropped `StarlarkPagableViaPagable`): - `buck2_core`: `ImportPath`, `ProjectRoot`, `ProjectRelativePathBuf`, `ProviderId`, `ProvidersLabel`, `ConfiguredTargetLabel` - `buck2_fs`: `ForwardRelativePathBuf` - `buck2_util`: removed the `StarlarkSerialize`/`StarlarkDeserialize` orphan impls on `ThinBoxSlice<T>` Consumer bridging is added in `buck2_build_api` (artifact/value, cmd_args/options, provider/collection, resolve_query_macro) and `buck2_transition`. Reviewed By: christolliday Differential Revision: D104865040 fbshipit-source-id: f3b2d79caf9bd581219782fcace59cd3c3c18fea

wait for in-progress slots to fix concurrent/cyclic sentinel access Summary: Partial deserialization (110) allocates each value's header with a sentinel vtable and fills it in lazily. When a slot is mid-deserialization, `ensure_initialized` handed back that not-yet-materialized value; using it then panics with "accessing a frozen value that has not been deserialized yet" (the sentinel vtable in `values/traits.rs`). This shows up during concurrent DICE page-in, where one `buck2-rt` worker hashes a value (e.g. as a SmallMap key) while another worker is still deserializing it. The `ClaimResult::InProgress` path now blocks on the claimer (`wait_for_slot` / `wait_for_init` on a per-slot condvar) and reads the materialized value, instead of returning the sentinel pointer. To avoid deadlocking on a genuine cycle — a same-thread re-entry or a cross-thread cycle — a `WaitForGraph` (the `claimers` / `waiters` edges walked by `has_cycle`) detects when blocking would deadlock and returns the sentinel only in that unavoidable case. The per-claim and per-wait edges are tracked with RAII guards so every exit path unwinds them exactly once. Adds `test_cross_thread_cycle_does_not_deadlock`, which concurrently deserializes a cross-referencing cycle (`a = [b]; b = [a]`) on two threads and asserts the round-trip neither deadlocks nor panics. Also adds `test_concurrent_page_in_does_not_hash_sentinel_key`, a regression for the sentinel-hash above — one thread hashes a key while another is still deserializing it (setup explained in the test's comment). Reviewed By: christolliday Differential Revision: D107338958 fbshipit-source-id: 14f7307a26b13fca48f130bf33f69cdbd77b4e3a

Defer DICE hydration until cached values are needed Summary: `CheckDepsPagedOut` currently hydrates the previous value before validating its dependencies, even when dependency changes force recomputation and make that value unnecessary. This eager page-in adds avoidable storage fetch and deserialization work to incremental builds. This diff carries the paged-out candidate through dependency validation and hydrates it only when the old value is required. Unchanged dependencies hydrate and reuse the cached value; changed dependencies with the same structure recompute first and hydrate only to preserve equality-based version reuse; changed dependency structure skips the page-in entirely. Delayed hydration also preserves a newer resident value if another in-flight update wins the race. By placing `CheckDepsPagedOut` hydration after dependency validation, this also makes the causal ordering used by the later critical-path diff straightforward: `PageIn(A)` can sit between A and its validated dependencies (`A -> PageIn(A) -> dependencies`). With eager parent hydration, accurately ordering nested page-ins would instead require non-obvious cross-page-in edges such as `PageIn(B) -> PageIn(A)` and `PageIn(C) -> PageIn(A)`. The remaining stack records this page-in work and surfaces it through build telemetry and critical-path reporting. Reviewed By: christolliday Differential Revision: D113098463 fbshipit-source-id: 510212e2847c5fb254a94864c3d8f20b58f7a48d

`FrozenDynamicLambdaParamsStorageImpl` real `StarlarkPagable` via `#[starlark_pagable_typetag]` Summary: `FrozenDynamicLambdaParamsStorageImpl` participates in pagable typetag dispatch via `dyn FrozenDynamicLambdaParamsStorage`, but its body holds `FrozenValue`s in `lambda_params` that must resolve against the currently-(de)serializing starlark heap. `#[starlark_pagable_typetag]` emits the recovery bridge alongside the typetag registration, so starlark-domain typetag impls don't need a hand-written `PagableSerialize`/`PagableDeserialize` bridge. `#[starlark_pagable_typetag]` mirrors `#[pagable::pagable_typetag]`: - **Trait def**: wraps `#[pagable_typetag]` and emits a sealed marker `StarlarkTypetagTraitMarker for dyn Trait`. - **`impl Trait for Foo`**: wraps `#[pagable_typetag]`, emits the recovery bridge (`recover_from_pagable` + delegate to `StarlarkSerialize`/`StarlarkDeserialize`), and asserts the trait carries the marker. Reviewed By: christolliday Differential Revision: D102558111 fbshipit-source-id: c5be176554c42df25fb42ebdb79f2447460e8b6d

Add per-DICE-key-type page-in telemetry to the invocation record Summary: Adds per-DICE-key-type page-in telemetry so we can measure and improve DICE page-in (reading a paged-out value back in) overhead. Each page-in's fetch time, deserialize time, and byte count is accumulated per DICE key type on the daemon and rides on the periodic `Snapshot`. At the end of a command the client recorder diffs the first vs last snapshot to record that command's page-in cost in the `InvocationRecord`: aggregate `page_in_count` / `page_in_fetch_us` / `page_in_deser_us` / `page_in_bytes` plus a per-key-type breakdown. Retrieve it with `buck2 build --unstable-write-invocation-record <path>`. Reviewed By: NavidQar Differential Revision: D109629093 fbshipit-source-id: 519ac1ff0b1d9a8b5f163a6f2c3450cead639cb6

add memory + speed benchmark for `ptr_to_location` chunk index Summary: The shape change in [112/n] makes a real trade between memory and per-op cost. Without a benchmark we can't catch regressions in either direction or speak concretely about the production-scale win, so this diff adds `state_benchmark.rs` with side-by-side comparisons of the new chunk-indexed `StarlarkSerState` against the previously shipped per-value `DashMap` shape across memory, build time, and per-lookup latency. Memory is reported two independent ways and cross-checked: 1. **Linux `/proc/self/statm` RSS delta** around each construction window — page-granular but reports actual physical memory pinned. 2. **Closed-form analytic estimate** from documented hashbrown / `BTreeMap` internals — drives the assertions. The OLD DashMap's RSS-vs-analytic agreement of 1.00× at 10M values validates the methodology. `allocative::size_of_unique` was tried first but isn't useful here — allocative's std `HashMap` / `BTreeMap` impls don't tag the bucket array as `Unique`, so they return only the struct shell (~48 B for a 20K-entry HashMap). The doc comment in the file calls this out so future readers don't go down the same path. At the 10M-value sample (2K heaps × 5K values, ~16K chunks) the bench shows: - **memory**: chunk index ~50 MB RSS / ~40 MiB analytic vs DashMap ~427 MB RSS / ~400 MiB analytic (~10× smaller analytic, ~9× smaller RSS); - **build**: ~1.78 s vs ~9.0 s (~5× faster); - **lookup**: ~814 ns vs ~727 ns (~1.12× slower). The bench prints all the numbers and asserts the structural invariants (chunk count ≪ value count, memory ratio ≥ 8×, lookup envelope ≤ 500 µs) so a regression to per-value indexing or a lookup pathology trips on the assertion instead of silently re-inflating session-resident memory. Also includes `test_chunk_index_lookup_is_correct_and_fast` for per-pointer correctness and `test_lookup_misses_when_ptr_not_in_any_chunk` for the negative case. Reviewed By: christolliday Differential Revision: D105235514 fbshipit-source-id: 4b37cfd7322d2b70e3e5a940b6da8a588c2f6963