Chenhao Zuo

Cross-process pagable ser/de for BcInstrs Summary: `BcInstrs` is a packed `Box<[u64]>` of `BcInstrRepr` records with embedded `FrozenValue`s; copying the raw words is unsound across processes because the pointers don't relocate. Give `BcInstrs` a manual `StarlarkSerialize`/`StarlarkDeserialize` that walks the instruction stream and routes each instruction's `I::Arg` through its own ser/de, so embedded `FrozenValue`s relocate via the normal mechanism. Wire format: a u8 tag followed by a `(u32 opcode, arg)` stream terminated by `End`. `'static`-backed args re-resolve on read instead of being serialized (`BcNativeFunction` from its frozen value, `KnownMethod` by name); `BcOpcode` serializes as a u32 (there are >256 opcodes). This unblocks paging any `FrozenDef`. Reviewed By: christolliday Differential Revision: D108842732 fbshipit-source-id: acc9b5836a003e287ffe2341b58a2b73a19679f2

pagable_typetag for TypeMatcher dispatch Summary: `TypeMatcherBox` is `Box<dyn TypeMatcherDyn>`. Deserialize needs to know the concrete type behind the `dyn` so it can pick the right `pagable_deserialize`. Wire up `pagable::typetag` to provide that dispatch: - **`TypeMatcherDyn`** becomes a `#[pagable_typetag]` trait with a `PagableTagged` supertrait, so a per-trait registry exists for it. - **Each concrete matcher** is tagged so it lands in the registry: `#[pagable_typetag(TypeMatcherDyn)]` for non-generic ones (`IsStr`, `EnumTypeMatcher`, `UserProviderMatcher`, …) and `#[pagable_tagged(TypeMatcherDyn)]` + `register_type_matcher!` for generic wrappers (`IsListOf<I>`, `IsDictOf<K, V>`, `IsTupleElems2<A, B>`, …). - **`TypeMatcherBox`** gets a manual ser/de that writes `tag + payload` on the way out and reads the tag to dispatch through `<dyn TypeMatcherDyn>::deserialize_box` on the way in. A later diff replaces this with the `Pagable` derive once tag handling is generalized. - **`TypeMatcherAlloc`** allocators (`any_of_two_matcher`, `list_of_matcher`, `dict_of_matcher`, `set_of_matcher`) switch from `impl TypeMatcher` to explicit generic params with `IsXxx<…>: PagableRegisteredFor<dyn TypeMatcherDyn>` bounds — this forces callers to register the concrete instantiations they construct. Reviewed By: christolliday Differential Revision: D103624117 fbshipit-source-id: 7efc071cd349c5282ce8b9aa1ed166f7584c94ef

Add `recover_from_pagable` bridges between starlark and pagable serialization layers Summary: When two starlark values share an `Arc<T>`, we want that `Arc` to dedup on the wire — round-trip back to a single allocation. Pagable already has an Arc-identity dedup mechanism, but it lives in the lower `pagable::PagableSerialize` / `PagableDeserialize` layer, not in the starlark `StarlarkSerialize` / `StarlarkDeserialize` layer. So for an `Arc` field on a starlark value, we have to switch that field's serialization off the starlark layer and onto the pagable layer. The catch: the inside of that `Arc<T>` may itself contain `FrozenValue`s that need to resolve against the currently-(de)serializing starlark heap. Once we've switched into the pagable layer, the starlark heap context is gone — so the body can't be serialized correctly without recovering it. This diff adds two associated-function bridges: - `StarlarkSerializerImpl::recover_from_pagable(serializer)` - `StarlarkDeserializerImpl::recover_from_pagable(deserializer)` Each reads the heap context that was stashed in the `SessionContext` before the pagable hop and returns a fully-wired `StarlarkSerializerImpl` / `StarlarkDeserializerImpl`. A type's `PagableSerialize` / `PagableDeserialize` impl can therefore recover the starlark context and delegate its body back to `StarlarkSerialize` / `StarlarkDeserialize`, completing the round-trip. Plumbing changes that fall out of this: - `current_heap_deser_state` becomes `Arc<Mutex<HeapDeserializationState>>` so a nested deserializer entered through `recover_from_pagable` shares the **same** forward-reference work queue as the outer flow. - New `CurrentHeapId` / `CurrentHeapDeserState` wrappers stored in the `SessionContext` so the heap id (and deser state) survives trips through pure pagable layers. This is a prerequisite for the next diff. Reviewed By: cjhopman Differential Revision: D102555402 fbshipit-source-id: e6ccc281473e80f43b6718d2c91170549c7314ea

lock-free per-slot init state + heap registry Summary: Two related changes prepare `HeapDeserializationState` for partial deserialization's concurrent traversal. **Per-slot init state.** The old `Mutex<HeapDeserializationState>` + `bool initialized` couldn't distinguish "claimed but still being deserialized" from "done": `initialized` flipped to `true` before `starlark_deserialize` ran and the mutex was dropped, so a second thread observing `initialized=true` would read the value before it was fully constructed. Per-slot atomics now encode `0` = not started, `1` = in progress, `2` = failed, and a non-zero aligned header pointer = done. A thread that loses the claim race gets the slot's pre-allocated header pointer back via `ClaimResult::InProgress` (no blocking at this commit); if the winner errors before publishing done it stores `FAILED`, so a `(heap_id, value_index)` resolves to a deterministic failed-slot error. (Blocking on an in-progress slot until its claimer finishes is added later, in 111.) **Single cross-heap registry.** `StarlarkDeserState.heap_deser_states[heap_id]` becomes the canonical source for any heap's state. The implicit "current heap" plumbing — `current_heap_deser_state` field, `CurrentHeapDeserState` session-context entry, `find_by_frozen_value` / `ptr_to_index` reverse-map — all go away. Every `FrozenValue` reference is fully qualified by `(heap_id, value_index)` on the wire, so `ensure_initialized` resolves directly through the registry. Reviewed By: christolliday Differential Revision: D105131368 fbshipit-source-id: 66a4587991e87ac3c77fb7c009275db6c30fb71b

Remove `skip_pagable` from `#[starlark_value]` Summary: `skip_pagable` defaulted to `true` and could only be set to `true` (there was no way to set it to `false`), making the flag and its associated `impl_starlark_pagable` method dead code. Remove the attribute, its parsing, the dead method, and all ~100 call sites. Reviewed By: christolliday Differential Revision: D106569109 fbshipit-source-id: ec41d2867c9dc544da10839664ceaf214085076b

TyStarlarkValue: foundation, HasTyVTable trait, register_ty_starlark_value! macro Summary: ## Why we need this `TyStarlarkValue` is the typing-system handle to a `StarlarkValue` impl. Before this diff it held a raw `&'static TyStarlarkValueVTable`. To make `TyStarlarkValue` pagable round-trippable, the inner pointer has to become a `pagable::StaticValue<TyStarlarkValueVTable>`. That gets us most of the way, but creates a new problem: `TyStarlarkValue::new::<T>()` needs the right `StaticValue` *for that specific `T`*. This diff goes with: the **`HasTyVTable`** trait. Each `T` that can flow through `TyStarlarkValue::new` is required to implement `HasTyVTable` with a `const TY_VTABLE_STATIC: StaticValue<TyStarlarkValueVTable>` filled in by the per-`T` registration. `TyStarlarkValue::new::<T>` now reads `T::Canonical::TY_VTABLE_STATIC` and any unregistered type is a compile error. A `register_ty_starlark_value!` macro is the standard way to populate the trait — it emits the static, the `pagable::static_value!` registration, and the trait impl in one block. Two macro forms exist because V-parameterized types (`Foo<'v, V: ValueLike<'v>>`) instantiate to two distinct Rust types — `Foo<'v, Value<'v>>` and `Foo<'static, FrozenValue>` — that share a single `'static` vtable content but need separate `HasTyVTable` impls. `#[starlark_value]` is updated to emit the macro automatically for the common cases, so most types pick up registration without a manual call. The cases the macro can't handle — types generic over non-`ValueLike` parameters (`StarlarkAny<T>`, `AnyArray<T>`, `StarlarkAnyComplex<T>`, `CallEnter<'v, D>`, `StarlarkTargetSet<Node>`) — get a temporary `UNREGISTERED_VTABLE_STATIC` placeholder so the trait bound compiles; following diffs then replace each placeholder with proper per-`T` registration, and in diff 67 deletes the placeholder. ## What changes - **`HasTyVTable` trait** + `TyStarlarkValue::new::<T>` requires `T::Canonical: HasTyVTable` and reads the vtable from the trait const. - **`register_ty_starlark_value!` macro** — simple `($ty)` form for non-generic and lifetime-only types; `(generic = <…>, elided_ty = …, impl_ty = …)` form for V-parameterized types so live and frozen impls share one content static. - **`#[starlark_value]` macro** — auto-emits the registration for cases it can resolve. - **`PagableSerialize`/`PagableDeserialize` for `TyStarlarkValue`** — delegate to the inner `StaticValue<TyStarlarkValueVTable>`. - **`UNREGISTERED_VTABLE_STATIC` sentinel** — temporary placeholder pointing at `NoneType`'s vtable, used by the placeholder `HasTyVTable` impls for `StarlarkAny<T>` / `AnyArray<T>` / `StarlarkAnyComplex<T>` / `CallEnter<'v, D>` / `StarlarkTargetSet<Node>`. Removed in diff 67. - One-line `Self: HasTyVTable` bound additions on a handful of types (`partial.rs`, `list/value.rs`, `compiled.rs`, `any_array.rs`, `any.rs`, `any_complex.rs`, `targetset.rs`) to satisfy the new trait-bound contract. Reviewed By: cjhopman Differential Revision: D102509848 fbshipit-source-id: b02d06e7aa59f4537f8bba68016d4905403409f2

introduce PagableDeserializerRecipe Summary: Pagable-framework groundwork for partial deserialization. Cross-heap resolution will need to reopen a specific heap's bytes long after the outer `deserialize_arc` callback returns — `self.pagable` may be pointing at the wrong stream by then. **New trait — `PagableDeserializerRecipe`.** A reconstructable handle to a `PagableDeserializer`. Callers stash a recipe and call `open(storage)` later to get a fresh deserializer over the same bytes, without holding the original deserializer live. **Storage is passed to `open()`, not held on the recipe.** A recipe that stored its own `PagableStorageHandle` would form an `Arc` cycle: storage owns a `SessionContext` that may stash `Arc<dyn PagableDeserializerRecipe>` (e.g. for cross-heap pointer resolution in later diffs), which would point back at the same storage. Passing storage in at `open()` time keeps the recipe pure data — no cycle is possible by construction. Callers source storage from the active deserializer via its `storage()` method. **Recipe for `PagableDeserializerImpl` — `PagableDeserializerRecipeImpl`.** Backed by `Arc<PagableData>` (owned bytes + nested-arc `DataKey`s). The data is owned, so a recipe can be parked in a long-lived map and reopened after the `deserialize_arc` call that handed it out has returned. **Wired into `deserialize_arc`.** The callback signature now receives an `Arc<dyn PagableDeserializerRecipe>` alongside the sub-deserializer. Most callbacks ignore it; consumers that need deferred reads — `FrozenHeapRef` in the next diff — stash it for later. Both `PagableDeserializerImpl` and `TestingDeserializer` are updated to construct the right recipe shape. Reviewed By: christolliday Differential Revision: D105131367 fbshipit-source-id: 4aca2cef46ebf6e90ebcfd61bd60aa99b295abda

Extend #[derive(StarlarkPagable)] with skip, bound, and enums Summary: Three orthogonal gaps in the derive macro, all hit by later diffs in this stack: - **`#[starlark_pagable(skip)]` / `skip = "expr"`**: some fields — caches, profiling data, handles to non-serializable runtime state — have no meaningful serialized form. On serialize they become a no-op (with a `let _ = &self.field;` to silence unused-field lints); on deserialize they are reconstructed via `Default::default()` or a caller-supplied expression. This is the Starlark analog of `#[pagable(skip)]`. - **`#[starlark_pagable(bound = "...")]`**: when a generic type has impl-level constraints that don't match the literal type parameters (e.g. `DictGen<T>` actually needs `'v, T: DictLike<'v>`), the user can override the `impl<...>` parameter list verbatim. Without this, the derive emits bounds copied from the type definition and those don't always typecheck. - **Enum support**: previously the macro rejected enums. Now it emits a `u8` discriminant tag (variant index, capped at 255) followed by each variant's payload fields, reusing the same `pagable` / `skip` / `skip_expr` handling as structs. Required by `CodeMapImpl` and `SerializedFrozenValue`-like enums downstream. - **`#[starlark_pagable(impl_for = "...")]`**: Imp starlark pagable for some specific generic T. Reviewed By: cjhopman Differential Revision: D102186975 fbshipit-source-id: a2d3dd0ef92167814dfc14a748a135b26588909c

Migrate ArcStr/ArcOrStatic to StaticStr identity Summary: Fields holding `&'static str` (e.g. native function parameter names) lose their identity across pagable round-trips. To fix this: - `ArcOrStatic<T>` switches its `Static` arm from `&'static T` to `pagable::StaticValue<T>`, so the static lives in the registry and round-trips by index. - `ArcStr::new_static` now takes `StaticStr` instead of `&'static str`. - `ArcOrStatic<T>` gets manual `PagableSerialize`/`PagableDeserialize`: a 1-byte tag picks between the indexed-static path and `serializer.serialize_arc(...)` for the heap-allocated path. - All call sites — native function param specs (via the `starlark_module` derive macro), `ArcStr::EMPTY_STR`, internal/test usages — are updated to register strings via `pagable::static_str!` and pass the resulting `StaticStr`. - `TyCallable` drops the `ArcOrStatic` wrapping in favor of plain `Arc<TyCallableInner>` plus a `OnceLock<Arc<...>>` for the static-`any()` case. Same end result, but without the now-unneeded enum. This is the load-bearing change that makes the rest of the stack possible. Reviewed By: cjhopman Differential Revision: D102487030 fbshipit-source-id: 7ebea25fdea44a69ae9a1eb67e436a9733c28195

Add `debug hydration status` to report DICE hydration Summary: Adds a `status` subcommand under `buck2 debug hydration` that reports how many DICE graph node values are resident in memory vs paged out to pagable storage, with a per-key-type breakdown sorted by total (resident + paged-out) descending. It complements the existing `page-out` / `page-in` subcommands. Unlike `page-out` / `page-in`, `status` is read-only: it does not require an idle DICE and runs non-exclusively, so it can be issued during a build without blocking, or being blocked by, that build. It reports a consistent snapshot taken on the DICE core-state thread. Wiring: a new `STATUS` value on `HydrationSubcommand` and a new `HydrationResponse { summary }` proto message (the `Hydration` RPC now returns `HydrationResponse` instead of `GenericResponse`); `CoreState::hydration_status` classifies occupied nodes as resident (value in memory) vs paged out (only a `DataKey` remains) behind a new `StateRequest::HydrationStatus`; `Dice::hydration_status` resolves keys to per-type counts via the key index; the server renders the summary via `format_status_summary` and the client prints it. Reviewed By: christolliday Differential Revision: D109364561 fbshipit-source-id: 79f9240121b5930fbaf64ab2e6bf352c25b66d6c

Drop `starlark` dep from `buck2_core`, `buck2_fs`, `buck2_util` Summary: `buck2_common` can't depend on `starlark` (banned in `app_dep_graph_rules/rules.bzl`) — the buck2 client binaries depend on `buck2_common` and shouldn't pull in the starlark interpreter. Adding `starlark` to any foundation crate (`buck2_core`, `buck2_fs`, `buck2_util`) creates a banned transitive path via `buck2_common`. This drops the `starlark` dep from those three crates: removes the `StarlarkPagableViaPagable` derive from their types, and removes the orphan `ThinBoxSlice<T>: StarlarkSerialize/StarlarkDeserialize` impls from `buck2_util`. Affected downstream `StarlarkPagable` derives are updated at the use sites — either with `#[starlark_pagable(pagable)]` (pagable-only fields) or `#[starlark_pagable(serialize_with = "...", deserialize_with = "...")]` helpers (mixed pagable/starlark fields). The `StarlarkPagable` derive in `starlark_derive` is extended to accept `serialize_with`/`deserialize_with` on enum variant fields, matching the existing struct-field behavior. Foundation types touched (kept `Pagable`, dropped `StarlarkPagableViaPagable`): - `buck2_core`: `ImportPath`, `ProjectRoot`, `ProjectRelativePathBuf`, `ProviderId`, `ProvidersLabel`, `ConfiguredTargetLabel` - `buck2_fs`: `ForwardRelativePathBuf` - `buck2_util`: removed the `StarlarkSerialize`/`StarlarkDeserialize` orphan impls on `ThinBoxSlice<T>` Consumer bridging is added in `buck2_build_api` (artifact/value, cmd_args/options, provider/collection, resolve_query_macro) and `buck2_transition`. Reviewed By: christolliday Differential Revision: D104865040 fbshipit-source-id: f3b2d79caf9bd581219782fcace59cd3c3c18fea

Fix asymmetric Box/Arc<dyn Trait> wire format via pagable_serialize_body split Summary: # The `Arc<dyn Trait>` / `Box<dyn Trait>` round-trip bug ## Setup A struct deriving `Pagable` holds a trait-object field: ```rust #[derive(Pagable)] pub struct AnimalHolder { pub animal: Arc<dyn Animal>, } ``` The `Animal` trait is a `#[pagable_typetag]` trait with `PagableTagged` as a supertrait, and (before the fix) `PagableTagged: PagableSerialize` as a further supertrait. A round-trip of `AnimalHolder` corrupts the wire and panics on deserialize: - "Hit the end of buffer, expected more data". ## Serialize path ```rust AnimalHolder::pagable_serialize (from `derive(Pagable)`) → calls pagable_serialize on each field Arc<dyn Animal>::pagable_serialize (blanket Arc<T> impl) → serializer.serialize_arc(self) TestingSerializer::serialize_arc → write Arc identity (usize) → for first occurrence: arc.serialize(self) ArcEraseDyn::serialize for Arc<dyn Animal> → ArcErase::serialize_inner(self, ser) ArcErase::serialize_inner for Arc<T>, T = dyn Animal → T::pagable_serialize(self, ser) → i.e. <dyn Animal as PagableSerialize>::pagable_serialize(self, ser) ← KEY HOP <dyn Animal as PagableSerialize>::pagable_serialize (compiler auto-generated) → dynamic-dispatch to the concrete type's PagableSerialize::pagable_serialize Wrapper<Cat>::pagable_serialize (from `derive(Pagable)`) → writes body fields only — NO TAG ``` ## Deserialize path ```rust AnimalHolder::pagable_deserialize (from `derive(Pagable)`) → calls pagable_deserialize on each field Arc<dyn Animal>::pagable_deserialize (blanket Arc<T> impl) → deserialize_arc::<Self, _>(de) deserialize_arc → reads Arc identity → for first occurrence: ArcErase::deserialize_inner(...) ArcErase::deserialize_inner for Arc<T>, T = dyn Animal → T::deserialize_box(deser) → i.e. <dyn Animal>::deserialize_box(deser) <dyn Animal>::deserialize_box (from `#[pagable_typetag]` on the trait) → reads tag (string) + body via the typetag registry ``` ## Where the bug is The asymmetry sits at one logical hop, present in both the `Box<dyn T>` and `Arc<dyn T>` paths: - **Serialize**: vtable-dispatched call to `<dyn Animal as PagableSerialize>::pagable_serialize`. - **Deserialize**: explicit call to `<dyn Animal>::deserialize_box`, which reads `tag + body`. The deserialize side does what's expected. The bug is on the serialize side: the vtable dispatch went somewhere wrong. ### Why the serialize side wrote body-only The chain of supertraits before the fix: - `trait Animal: PagableTagged + Send + Sync + Debug` - `trait PagableTagged: PagableSerialize + Send + Sync` So transitively `Animal: PagableSerialize`. The Rust language feature that bites: > For any object-safe trait `T`, the compiler automatically generates `impl T for dyn T`. By extension, also `impl Supertrait for dyn T` for each supertrait in the chain. So the compiler silently emits, in addition to `impl Animal for dyn Animal` and `impl PagableTagged for dyn Animal`, also: ```rust // Compiler-generated, no source code anywhere. impl PagableSerialize for dyn Animal { fn pagable_serialize(&self, ser: &mut dyn PagableSerializer) -> Result<()> { // dynamic-dispatch to the concrete type's PagableSerialize::pagable_serialize } } ``` So the serialized data would not have tag. ## The fix Three coordinated changes: ### 1. **Drop `PagableSerialize` from `PagableTagged`'s supertraits.** ```rust pub trait PagableTagged: Send + Sync { ... } // was: + PagableSerialize ``` Now `Animal: PagableSerialize` is no longer transitively required. The compiler stops auto-generating `impl PagableSerialize for dyn Animal`. ### 2. **Have `#[pagable_typetag]` emit an explicit `impl PagableSerialize for dyn Trait`.** This is now legal because the auto-impl is gone (otherwise E0371 — "the object type automatically implements the trait" — would fire). The explicit impl writes `tag + body` via `serialize_tagged`: ```rust impl PagableSerialize for dyn Animal { fn pagable_serialize(&self, ser) -> Result<()> { PagableTagged::serialize_tagged(self, ser) // writes tag + body } } ``` Now the "key hop" in the serialize path lands here, writing the matching wire format. ### 3. **Add `pagable_serialize_body` as a required method on `PagableTagged`.** `serialize_tagged` writes the tag, then needs to write the body. The naive approach — `<Self as PagableSerialize>::pagable_serialize(self, ser)` — recurses forever on `&dyn Animal`: that path resolves to the macro's impl from step 2, which calls `serialize_tagged` again. Adding `where Self: Sized` to dodge it doesn't help — `serialize_tagged` is meant to be called on `&dyn Animal`. Fix: give `PagableTagged` its own body-writing slot. Each concrete type forwards `pagable_serialize_body` to its own `PagableSerialize::pagable_serialize` (body only). `serialize_tagged` calls `self.pagable_serialize_body(...)`, which on `&dyn Animal` dispatches through the `PagableTagged` vtable to the concrete type's forwarder — bypassing `<dyn Animal as PagableSerialize>::pagable_serialize` entirely. No recursion; the wire is exactly `tag + body`. ## Before this change The test `test_pagable_derive_arc_dyn_trait_field_roundtrip` would fail Test UI: https://www.internalfb.com/intern/testinfra/testrun/11258999238093744 Reviewed By: cjhopman Differential Revision: D102513000 fbshipit-source-id: 2626f7a7229a9d68929600fbc3480d6477ee4595

`StarlarkAny<T>` ser/de Summary: Adds `StarlarkSerialize`/`StarlarkDeserialize` for `StarlarkAny<T>` (delegating to `T`'s impl) and tightens `StarlarkAnyRegistered` to require `T: StarlarkPagable`. Derives `StarlarkPagable` on the codemap/frame types that flow through `FrozenAnyValue` (`FrameSpan`, `FrozenFileSpan`, `InlinedFrame`, `InlinedFrames`, `CopySlotFromParent`) and `Pagable` on bc index types (`BcSlot`, `BcSlotOut`, `LocalSlotIdCapturedOrNot`). Adds a manual `StarlarkSerialize`/`Deserialize` bridge for `CodeMap` (foreign type). Tightening `StarlarkAnyRegistered` forces a few transitive holdouts (`Globals`, `FrozenModuleData`, `DefInfo`, `UserProviderCallableData`) to satisfy `StarlarkPagable`; this diff stubs them with `PagablePanic` placeholders. `UserProviderCallableData` gets a real impl in [87/n], `FrozenModuleData` in [91/n], and `DefInfo` in [92/n]; `Globals` remains `PagablePanic` and will be handled in a later diff. Reviewed By: cjhopman Differential Revision: D102558116 fbshipit-source-id: 1abc6bb2d58a3cba118424fa99919cf55886b96f

wait for in-progress slots to fix concurrent/cyclic sentinel access Summary: Partial deserialization (110) allocates each value's header with a sentinel vtable and fills it in lazily. When a slot is mid-deserialization, `ensure_initialized` handed back that not-yet-materialized value; using it then panics with "accessing a frozen value that has not been deserialized yet" (the sentinel vtable in `values/traits.rs`). This shows up during concurrent DICE page-in, where one `buck2-rt` worker hashes a value (e.g. as a SmallMap key) while another worker is still deserializing it. The `ClaimResult::InProgress` path now blocks on the claimer (`wait_for_slot` / `wait_for_init` on a per-slot condvar) and reads the materialized value, instead of returning the sentinel pointer. To avoid deadlocking on a genuine cycle — a same-thread re-entry or a cross-thread cycle — a `WaitForGraph` (the `claimers` / `waiters` edges walked by `has_cycle`) detects when blocking would deadlock and returns the sentinel only in that unavoidable case. The per-claim and per-wait edges are tracked with RAII guards so every exit path unwinds them exactly once. Adds `test_cross_thread_cycle_does_not_deadlock`, which concurrently deserializes a cross-referencing cycle (`a = [b]; b = [a]`) on two threads and asserts the round-trip neither deadlocks nor panics. Also adds `test_concurrent_page_in_does_not_hash_sentinel_key`, a regression for the sentinel-hash above — one thread hashes a key while another is still deserializing it (setup explained in the test's comment). Reviewed By: christolliday Differential Revision: D107338958 fbshipit-source-id: 14f7307a26b13fca48f130bf33f69cdbd77b4e3a

`FrozenDynamicLambdaParamsStorageImpl` real `StarlarkPagable` via `#[starlark_pagable_typetag]` Summary: `FrozenDynamicLambdaParamsStorageImpl` participates in pagable typetag dispatch via `dyn FrozenDynamicLambdaParamsStorage`, but its body holds `FrozenValue`s in `lambda_params` that must resolve against the currently-(de)serializing starlark heap. `#[starlark_pagable_typetag]` emits the recovery bridge alongside the typetag registration, so starlark-domain typetag impls don't need a hand-written `PagableSerialize`/`PagableDeserialize` bridge. `#[starlark_pagable_typetag]` mirrors `#[pagable::pagable_typetag]`: - **Trait def**: wraps `#[pagable_typetag]` and emits a sealed marker `StarlarkTypetagTraitMarker for dyn Trait`. - **`impl Trait for Foo`**: wraps `#[pagable_typetag]`, emits the recovery bridge (`recover_from_pagable` + delegate to `StarlarkSerialize`/`StarlarkDeserialize`), and asserts the trait carries the marker. Reviewed By: christolliday Differential Revision: D102558111 fbshipit-source-id: c5be176554c42df25fb42ebdb79f2447460e8b6d

Add per-DICE-key-type page-in telemetry to the invocation record Summary: Adds per-DICE-key-type page-in telemetry so we can measure and improve DICE page-in (reading a paged-out value back in) overhead. Each page-in's fetch time, deserialize time, and byte count is accumulated per DICE key type on the daemon and rides on the periodic `Snapshot`. At the end of a command the client recorder diffs the first vs last snapshot to record that command's page-in cost in the `InvocationRecord`: aggregate `page_in_count` / `page_in_fetch_us` / `page_in_deser_us` / `page_in_bytes` plus a per-key-type breakdown. Retrieve it with `buck2 build --unstable-write-invocation-record <path>`. Reviewed By: NavidQar Differential Revision: D109629093 fbshipit-source-id: 519ac1ff0b1d9a8b5f163a6f2c3450cead639cb6

add memory + speed benchmark for `ptr_to_location` chunk index Summary: The shape change in [112/n] makes a real trade between memory and per-op cost. Without a benchmark we can't catch regressions in either direction or speak concretely about the production-scale win, so this diff adds `state_benchmark.rs` with side-by-side comparisons of the new chunk-indexed `StarlarkSerState` against the previously shipped per-value `DashMap` shape across memory, build time, and per-lookup latency. Memory is reported two independent ways and cross-checked: 1. **Linux `/proc/self/statm` RSS delta** around each construction window — page-granular but reports actual physical memory pinned. 2. **Closed-form analytic estimate** from documented hashbrown / `BTreeMap` internals — drives the assertions. The OLD DashMap's RSS-vs-analytic agreement of 1.00× at 10M values validates the methodology. `allocative::size_of_unique` was tried first but isn't useful here — allocative's std `HashMap` / `BTreeMap` impls don't tag the bucket array as `Unique`, so they return only the struct shell (~48 B for a 20K-entry HashMap). The doc comment in the file calls this out so future readers don't go down the same path. At the 10M-value sample (2K heaps × 5K values, ~16K chunks) the bench shows: - **memory**: chunk index ~50 MB RSS / ~40 MiB analytic vs DashMap ~427 MB RSS / ~400 MiB analytic (~10× smaller analytic, ~9× smaller RSS); - **build**: ~1.78 s vs ~9.0 s (~5× faster); - **lookup**: ~814 ns vs ~727 ns (~1.12× slower). The bench prints all the numbers and asserts the structural invariants (chunk count ≪ value count, memory ratio ≥ 8×, lookup envelope ≤ 500 µs) so a regression to per-value indexing or a lookup pathology trips on the assertion instead of silently re-inflating session-resident memory. Also includes `test_chunk_index_lookup_is_correct_and_fast` for per-pointer correctness and `test_lookup_misses_when_ptr_not_in_any_chunk` for the negative case. Reviewed By: christolliday Differential Revision: D105235514 fbshipit-source-id: 4b37cfd7322d2b70e3e5a940b6da8a588c2f6963

improve `StarlarkSerState` `ptr_to_location`: per-chunk index Summary: `StarlarkSerState` previously mapped every Starlark pointer individually. A typical Buck2 graph carries ~570K heaps × ~196 values/heap ≈ 112M values, so the `DashMap<usize, (HeapRefId, u32)>` lookup table sat near 4 GB resident throughout each serialize session. This diff replaces that with a `RwLock<BTreeMap<usize, ChunkEntry>>` keyed by **chunk base address** instead of per-value pointer. Memory scales as `O(chunks)` rather than `O(values)` — the arena hands chunks out in tens, not millions. Lookup uses `range(..=raw_ptr).next_back()` to find the owning chunk in `O(log C)`, then `binary_search`es the chunk's sorted `payload_offsets` to recover the within-chunk index `k`. The wire's `value_index` is `chunk.values_before + k`. ### Alternative considered: chunk walk (no per-chunk payload_offsets) Before settling on `binary_search`, tried storing **only** `(base, size, heap_id, values_before)` per chunk — no per-value data at all — and recovering the within-chunk index at lookup time by walking the chunk via `Arena::iter_chunk` and counting headers until one matches `raw_ptr`. That variant gets to `O(chunks)` memory (~470 KiB at 10M values, an 870× reduction), but per-lookup becomes `O(V_c)` linear over the chunk and runs ~91 µs in debug — **125× slower** than DashMap, which makes the encode hot path unworkable. Storing `payload_offsets` is a `4 B`-per-value tax that brings lookup back to ~814 ns (≈ DashMap parity) while keeping memory dramatically below the per-value baseline. ### Benchmark (10M values, 2K heaps × 5K values, ~16K chunks) | | OLD (DashMap per-value) | walk variant | **binary_search (this diff)** | |---|---|---|---| | Build time | 9.0 s | 1.49 s | **1.78 s** | | Memory (analytic) | 400 MiB | 471 KiB | **39.8 MiB** | | Memory (RSS) | 427 MB | 2.7 MB | **49.0 MB** | | RSS/analytic agreement | 1.02× | 5.6× (noisy, small) | **1.20×** | | Lookup avg | 727 ns | 91 µs | **814 ns** | | Lookup ratio (NEW/OLD) | 1.0× | 125× slower | **1.12× slower** | | Memory ratio (OLD/NEW) | 1.0× | 870× smaller | **10.3× smaller** | Extrapolated to production scale (~570K heaps × ~196 values/heap ≈ 112M values): memory ~4.8 GB → ~450 MB; total lookup wall time over 112M ops ~80 s → ~90 s; build ~100 s → ~20 s. Reviewed By: christolliday Differential Revision: D105235516 fbshipit-source-id: 28598712bfa65afef0286a4d2c5a259cbb446820

BcInstrs round-trip tests + cyclic-deser fix Summary: Adds the first tests that page a module of functions out and back in and then run the deserialized bytecode — covering const/`FrozenValue`, named and `*args` calls, list/str methods, native calls, loops, comprehensions, and many locals — plus a determinism check (compile the same source twice → identical bytes). These surfaced a blocker: a `FrozenDef` references its module, which references the def back, so during deserialize the def's typed module pointer was constructed (`FrozenValueTyped::new_unchecked`) while the module slot was still an in-progress sentinel, tripping its type `debug_assert`. Fix: add `FrozenValueTyped::new_allow_uninitialized` for the deserialize path. The two-phase deserializer fills the slot in phase 2, so checking the type mid-cycle is a false positive; correctness rests on ser/de symmetry (the value was serialized through the same type). Reviewed By: christolliday Differential Revision: D108842734 fbshipit-source-id: 48309df746ef1830435f191833e953f95b60129e

Pagable for TypeMatcher and concrete matchers Summary: Tightens `trait TypeMatcher: TypeMatcherBase + Pagable` so every concrete matcher must round-trip. Adds `#[derive(Pagable)]` to every concrete matcher implementation: `StarlarkTypeIdMatcher`, `UserProviderMatcher`, `ProviderMatcher`, `TransitiveSetMatcher`, `StructMatcher`, `EnumTypeMatcher`, `NamespaceMatcher`, `RecordTypeMatcher`, and `DummyTypeMatcher`. All are simple data records (`TypeInstanceId` or `TyStarlarkValue` payloads, or unit), so the derive is enough. Reviewed By: cjhopman Differential Revision: D102512991 fbshipit-source-id: 26d8dc3e97174ae4b192c8f685ece7510c8f5e3c