buck2 — Engineering Performance

lifetimes: Brand the compiler IR and the bytecode Summary: The compiler's products - `ExprCompiled`, `StmtsCompiled`, `DefInfo`, the spans, and the `Bc` they compile to - held `FrozenValue`s, so everything the evaluator read out of bytecode was erased and re-typed at the read: `Def::at_brand`'s vtable compare per `CallFrozenDef`, the `FrozenValue::to_value` reads in the interpreter, and the erasures at the ends of `demote`. This diff puts the brand on the IR and the bytecode. The IR lives at `'fm`, the module frozen heap the `Compiler` allocates into (`ExprCompiled<'fm>`, `ResolvedIdent::Global(Value<'fm>)`, which ripples through `CstPayload<'fm>` into the scope resolver and the typechecker), and a `Bc<'v>` is at the brand of the heap it runs against. The two meet in one place: `eval_regular_top_level_stmt` rebrands the statement's `Bc<'fm>` to `Bc<'v>` through the module edge, once per top-level statement, and the module's `DefInfo` handle is rebranded once in `eval_module`. Everything downstream is plain reads. Where a compiler product has to be a heap value - `DefInfo`, `FrameSpan`, `InlinedFrame`, `FrozenFileSpan` - it is allocated as a `StarlarkAnyComplex<T<'f>>`, the `FrozenModuleData` idiom, with a hand `FreezeBranded` that is unreachable because these are only ever allocated in frozen heaps. `Def`'s fields move to its brand, `Def::at_brand` goes, and `post_freeze` gets its edge from `ModuleHeaps::seal_with`, which is the relocated mint: the closure gets the same edge `frozen_heap` hands out, for the same reason. `FrozenFileSpan` keeps its name only because `codemap::FileSpan` already has the good one. The bytecode needed one structural move. Opcodes dispatch by `TypeId`, so instruction types must stay `'static` while their operands now carry `'v`. Call instructions therefore take `'static` marker types (`FullArgs`, `PosArgs`, `AnyCallable`, `NativeCallable`) whose GAT names the branded operand; the instruction body is unchanged and dispatch does no more work than before. `BcNativeFunction<'v>` is at the bytecode's brand rather than immortal, since a native function value can live in any frozen heap the module references, not only in a static. Smaller things this forces or frees: - `LocalAsValue` placeholders were a process-wide singleton heap indexed by slot; they are now a per-`Compiler` cache in the module frozen heap, so `alloc_simple_typed_static` goes. - `BuiltinFn::frozen()` becomes `at()` over `HeapEdge::immortal()`: `Constants` is a static. - `TypecheckProfile` is keyed by `String`: the profile outlives the evaluation, and the def names it was keyed by are now at the brand. - `UnusedLoad` carries spans instead of CST nodes, since the CST is branded and the result outlives the heap scope. - `Module::get_slot_frozen`, `frozen_value`, `Arguments::frozen_to_v` (a transmute) and `FrozenAnyArray` are dead and go. `FrozenAnyValue`/`alloc_any_value` stay for buck2's `UserProviderCallable`, and `TypeCompiled::to_frozen` is branded with a `to_frozen_unbranded` next to it for the same consumers; both are 4-C's to delete. - `is_builtin`, `speculative_exec_safe` and `eq_is_ptr_eq` move from `FrozenValue` onto `Value`, where the compiler now calls them. No new `unsafe` beyond the `seal_with` mint; the tuple `ProvidesStaticType` impl is of the same kind as its neighbours. The pagable wire format is unchanged. Reviewed By: jtbraun Differential Revision: D118970674 fbshipit-source-id: 62668fb6d7cdd982e66918d4cc93fd02d2700f1c

Move starlark_fmt into buck2/tools/ Summary: Relocates the `starlark_fmt` crate (an opinionated ruff-based Starlark/BUCK formatter) from `fbcode/starlark_fmt/` to a new shared `fbcode/buck2/tools/starlark_fmt/` directory as the first step toward open-sourcing it into the public `facebook/buck2` repo, and makes the crate open-source-buildable. Sources are moved with `sl mv` to preserve internal history. Relocation: - New shared `fbcode/buck2/tools/` home for buck2-adjacent tools. - Renamed `mod.rs` files to sibling modules (`autofixes.rs`, `formatting.rs`, `subcommands.rs`) to satisfy the buck2 `rust-no-mod-rs` lint. - No local `PACKAGE`; the crate inherits `fbcode/buck2/PACKAGE`. Target-level `visibility` is kept on `starlark_fmt_lib`. - A forwarding `alias` is left at `fbcode//starlark_fmt:starlark_fmt` so the generated DotSlash manifest (`tools/lint/starlark/starlark_fmt`) keeps resolving; removing it is a follow-up gated on the MSDK schedule repoint. - Updated the sole external Buck consumer `fbcode//autodeps2/autodeps2`, the `README.md` build/test paths, and dropped the now-stale `starlark_fmt` row from `fbcode_enable_pyrefly_targets.csv`. The ~20 path-based consumers of `tools/lint/starlark/starlark_fmt` are unaffected (that path is unchanged). Open-source readiness (so the OSS `cargo`/`buck2-oss` build works). Because this diff registers the crate in the OSS cargo workspace, it must itself be OSS-buildable, so all of the following live here: - Applied the dual MIT/Apache license header to all `.rs` files. - autocargo-generated `Cargo.toml` for the crate and registered it in the OSS workspace (`fbcode/buck2/Cargo.toml` members). - Gated the internal `lint` subcommand out of the OSS build: it uses `//common/rust/lint_message` (not in OSS), so `mod lint`, the `lint_message` import, the `Command::Lint` variant/arm, and the `lint_files` use are `#[cfg(fbcode_build)]`, and the dep is `# oss-disable`. `fmt`/`diff`/`stdin` are unaffected; the internal build is unchanged. - Gated the Meta-internal `quickcheck_arbitrary_derive` (used only by the fbcode_build-gated fuzz tests) out of the OSS build: `mod fuzz_starlark` is `#[cfg(all(test, fbcode_build))]`, and an autocargo `extra_buck_dependencies` removal drops it from the generated `Cargo.toml`. - Pinned `get-size2 = "=0.10.1"` into the OSS `[workspace.dependencies]` (via an autocargo `extra_buck_dependencies` reference) so the OSS cargo build resolves `ruff_python_ast` against `compact_str 0.9.1` (0.10.2 would need compact_str ^0.10 and fail the `CompactString: GetSize` bound). - Fixed lints that `buck2-oss-lint` denies but internal lint doesn't: `clippy::str_to_string` (28x `.to_string()` -> `.to_owned()` on `&str`) and `let_underscore_drop`. Reviewed By: d16r Differential Revision: D113069398 fbshipit-source-id: d724baad1d7b60ea545a335961b6bd36ceca933c

hypermodern: Add dice_core Summary: The core state of `docs/incrementality.md` as its own crate: keys, revisions and untracked-input revisions as opaque identities, branches and versions, one claim per key and branch pinning its witnessing certificate, assertion histories for injected keys, exact per-branch rdep maps, and the four operations `commit`, `fork`, `lookup` and `write`, plus `take` for today's `unstable_take`. Nothing in dice uses it yet; the next diff replaces `core/graph` with it. It is a separate crate because we will want the algorithm usable outside dice eventually, and drawing the line now is what keeps it pure: everything environment-specific comes in through one `Env` trait with three associated types, the dependency payload of a certificate (dice's series-parallel deps), data per claim (dice's invalidation paths) and data per assertion (dice's invalidation priority). The core stores these and hands them back. The one thing it does with them itself is compare payloads, since whether two certificates are the same one is a question about their traces, not about which allocation holds them; environments hand back the allocation they were given, so the comparison is by address almost always. A key that is both asserted and certified is a hard panic on both paths, in release too; dice's public API only allows that through test mocks. The one-word `triomphe` `Arc` dice uses moves in here, because certificates are shared between the core's claims and the environment's candidates; dice re-exports it and so depends on the crate already. Lookups that cannot answer offer a candidate for the environment to re-establish only if it was stamped with the version's untracked-input revision; one that was not can never cover the version. `nearest_certificate` is there for environments that want that certificate's value regardless, as dice does for its equality cutoff. ## Where the code departs from the doc - Seqs are 1-based (`NonZeroU32`), so that `Option<Version>` is free and the root's first commit is seq 2, matching the version numbers dice's tests already use. - Fork builds the child's rdep map from the certificates of the attached set instead of copying the parent's and supplementing it. With exact maps that is the same set of edges, and it lets the checker assert Invariant 3 as an equality rather than a superset. - The install policy for a branch that already has a claim is "replace iff the new window reaches further" (§5.2 leaves it as a policy). One visible consequence, pinned by a test: a branch's own attach replaces a closed copy it inherited for earlier seqs, so those seqs fall back from `Valid` to `Unknown` with a candidate. - §5.2 step 2.4, the same certificate still covering the child's fork seq, turns out to be unreachable through `write` under one-window-per-slot semantics (a re-issue after a close always restarts the window at or after the close). The arm is kept; the test that expected it documents what happens instead, a re-home. ## Testing Three layers, all in the crate. Unit tests, one module per part of the doc: resolution rules 1 to 3, commit and its BFS, write and coverage including non-contiguous covers of returning injected premises, each cascade rule of §5.2 step 2, fork including the round-6 counterexample and the support worklist, the untracked-input semantics ported from `check_that_force_dirty_*`, retention, and the constructions that shaped the design (the certificate island and cyclic writes of Appendix C, the fork worklist's revision check, a grandchild below a shadow, dirties on both sides of a fork, a re-homed claim's ε history, the G4 chain, fill-in from a closed parent window, the displaced-claim precision loss, certificates written after `take`). Every mutating call in a test re-checks the invariants. `check_invariants` is a full scan of everything the state promises about itself: the branch tree, history ordering, Invariants 1 to 3, `closed_index` exactness, and no key both asserted and certified. The master invariant is not a property of the state alone, so it is left to the oracle. The fuzzer runs random operation sequences against an oracle that computes §3.2's judgment by brute force over the recorded history. Two generators: arbitrary certificates, random keys, revisions, premises and ε, for which soundness must still hold because the theorem is unconditional; and an honest environment with fixed dep graphs and a deterministic value function, driven the way dice's worker drives the core, which additionally checks from-scratch consistency, that a write at a head attaches, and that a commit closes nothing outside the changed keys' support. Failing cases shrink by deleting operations and print with their seed. To check the harness has teeth I planted bugs one at a time. A first round of five (BFS skipping inherited claims, no cascade, closed covers installed open, stale edges kept, no support filter at fork) was caught within 200 cases, every time by the invariant checker. A second round of five aimed past the checker (off-by-ones in `History::at` and `Window::covers`, a child resolving at its parent's head instead of the fork point, inherited injected cover denied, no-op asserts decided against the parent's view) was caught by the unit tests and both fuzz modes; the last one only by the oracle, so it is a unit test now too. Reviewed By: christolliday Differential Revision: D118978185 fbshipit-source-id: f81aef9a831e5cd6edbb0e4dd9afbb4f625334de

mem: Add mini_vec crate and type Summary: `MiniVec` is a `Vec` but is a single pointer on the stack; the length and capacity are bitpacked into the *top* bits of the pointer if they're small, otherwise allocated in the heap. This does introduce high-bit pointer packing to buck2 which we have not had so far. Claude assures me that modern kernels do not hand out high addresses without a user explicitly asking for them and no one complained when I asked, so I guess the team is on board with this. I'll use this in a couple places, but the first thing we'll be able to do is replace all the `ThinBoxSlice` Reviewed By: jtbraun Differential Revision: D105465807 fbshipit-source-id: 65663fb277c1301be55a3b1e94ef831ae80d4f78

unsafe: Split `heap_type.rs` into a module per heap Summary: The file held the unfrozen heap and its garbage collector, the frozen heap builder, the sealed heap with its paging wire format and diagnostics, the heap names, and the `OwnedFrozen` family: 2500 lines, five unrelated safety arguments. The `OwnedFrozen` invariant (the value is kept alive by the heap in `heap_ref`) is protected by field privacy, and that privacy extended over all of it. Now each heap is a module and the privacy boundaries coincide with the safety boundaries: - `unfrozen.rs`: `Heap`, `Tracer`, GC - `frozen.rs`: `OwnedFrozenHeap`, `FrozenHeap`, the shared reference set - `sealed.rs`: `FrozenFrozenHeap`, `FrozenHeapArc`, paging, diagnostics - `name.rs`: heap names - `owned_frozen.rs`: `OwnedFrozen`, `OwnedFrozenRef`, `OwnedFrozenReconstructor`, and only the impls that touch the pairing; the conveniences that were in `owned_frozen.rs` move to `owned_frozen_ext.rs` so that they stay unable to reach the fields `HeapKind` goes to `arena.rs`, next to the walk that takes it. Two accessors on `FrozenHeapArc` (`new_sealed`, `is_empty`) replace the field accesses that crossed the new boundaries. Code moves otherwise. Reviewed By: jtbraun Differential Revision: D121315380 fbshipit-source-id: 17d01833ffe78d2721c0fbdda38232cd5591f23e

Codemod `ok_or_else(|| internal_error!(...))` to `Option::internal_error` Summary: Mechanical adoption of the `BuckErrorOptionContext` API from the previous diff: 345 call sites across 142 files convert `.ok_or_else(|| internal_error!(...))` into the documented spelling — `.internal_error("...")` for plain messages, `.with_internal_error(|| format!(...))` where the message has format arguments. Message shape (`... (internal error)`), `InternalError` tag, and caller source location are identical before and after. Import fallout handled: 109 now-dead `use buck2_error::internal_error;` lines removed (40 files keep it for remaining non-`Option` macro uses); imports scoped into `mod tests` where the only converted sites are `cfg(test)`-gated, so the lib build stays warning-free. One `#[cfg(windows)]` site (`buck2/src/check_user_allowed.rs`) is deliberately left unconverted since it cannot be compile-verified on Linux. Reviewed By: christolliday Differential Revision: D116804875 fbshipit-source-id: 879fcb502cb449cca4e7e7cf608c08ccff0e1819

lifetimes: Brand the frozen heap Summary: `FrozenHeap` becomes a `Copy` handle `FrozenHeap<'fh>` onto an `OwnedFrozenHeap`, the way `Heap<'v>` is a handle onto its owned heap: `OwnedFrozenHeap::with` opens a scope with a fresh brand, `seal` turns the builder into the `OwnedFrozen<()>` owner, `FrozenHeap::temp` is the scratch form, and `OwnedFrozen::build` converges on the `temp` shape (closure returns at the heap's own brand; the `'fv`/`'h` split and its "foreign `FrozenValue`" clause are gone). Every allocator on the handle returns at the brand, and `AllocFrozenValue<'fv>` takes the handle and returns `Value<'fv>`. One diff because there is no compiling seam: once `&FrozenHeap` stops being a type, everything that held one has to move at once, and app/ has to come along for the same reason. The two `AllocFrozenValue` impls outside buck2 - metalos's `starlark_util.rs` and skycastle's `text_match.rs` - are in this diff for the same reason: the signature change forces them. They change signature only; `alloc_simple` does the same allocation it did before. ## The module's frozen heap `Module::frozen_heap` (and `Evaluator::frozen_heap`, same name so a legacy caller fails at the call site rather than compiling to something else) is now closure-scoped: `|fh: FrozenHeap<'fm>, edge: HeapEdge<'v, 'fm>|`. The module's frozen heap cannot share `'v` - values of the value heap could then be allocated into a heap that outlives it - so it gets its own brand, and the edge is how its allocations reach `'v`. That relocates the one `HeapEdge::unchecked_new` from `Evaluator::frozen_heap_edge` (deleted); the keep-alive that comment hand-waved is now structural: `freeze_impl` gives the value heap a reference to the sealed heap. The SAFETY comment names the remaining gap honestly (a module dropped without freezing), which predates this diff. `eval_module` compiles and executes in one such scope; the `Compiler` carries the handle, and `OptCtxEval` is implemented for `Compiler` (instead of `Evaluator`) and `OptimizeOnFreezeContext`, both of which know their frozen heap's brand. ## What stays `FrozenValue` The compiler IR and bytecode, and the frozen list/tuple storage. They reach the branded heap through crate-private, greppable erasures on the handle - `alloc_frozen` and `alloc_str_intern` - or an explicit `unpack_frozen()` where a branded value has to be stored erased. That census is the worklist for retiring `FrozenValue`. `alloc_simple_typed_static` survives crate-privately for the `'static` typed handles (`FrozenAnyValue`, the native method tables), now with a checked downcast instead of the `&'static FrozenHeap` casts, which are gone. Two smaller API notes: the frozen `alloc_simple` bound is now `AValueSimpleBound<'fh>` like `alloc_simple_typed`, since a `Send + Sync + 'static` bound cannot be met by branded values that live in the heap (the `HeapSendable`/`HeapSyncable` supertraits carry the same guarantee for `'static` types); and `Value<'fv>: AllocFrozenValue<'fv>`, the twin of `Value<'v>: AllocValue<'v>`. `OwnedFrozen::build` erases the brand before sealing rather than through `unchecked_new` afterwards, because the closure's result is branded with the borrow of the heap that `seal` consumes; the transmute moved into a shared `erase_brand`, no new unsafe operation. The pagable tests keep feeding `FrozenValue`s to the ser/de plumbing through a small `ErasingHeap` test wrapper rather than restructuring eighty tests around closures; they get rewritten when `FrozenValue` goes. Reviewed By: jtbraun Differential Revision: D118970647 fbshipit-source-id: c6993b65ed87685d7bdad224504617fdb7ed1081

Require tag for all buck2_error_derive enum types Summary: Force an ErrorTag for all current and future enum types that are derived from buck2_error_derive. Can be as simple as `Input`, `Environment`, and `Tier0`. Note that I tagged all the current ones to the best of my knowledge and could be wrong. Please fix or let me know anyone finds anything that seems wrong Reviewed By: JakobDegen Differential Revision: D67954001 fbshipit-source-id: 17798faa4d7365e1d24c3eddac2bcc55d1e66b95

fix: a lot of instances of clippy::needless_lifetimes (#944) Summary: This doesn't get all of them because: 1. there are so many of them 2. the lint is pretty jank Part of https://github.com/facebook/buck2/issues/943 Pull Request resolved: https://github.com/facebook/buck2/pull/944 Reviewed By: JakobDegen Differential Revision: D74271655 Pulled By: dtolnay fbshipit-source-id: c7e41c47248d4be1f5d33a14b676ab35fc5180e2

update platform010 & platform010-aarch64 symlinks Summary: * `__rust_no_alloc_shim_is_unstable_v2` replaces `__rust_no_alloc_shim_is_unstable`. It's a no-op function rather than a static data member. * Fix clippy errors from if-let chains * Sprinkle `'_` in places to make ambiguous lifetimes lint happy * `nonnull_provenance` feature is stable * `result_flattening` feature is stable * `file_lock` feature is stable * `extract_if` now accepts a range as the first argument * `duration_constructors` split into `duration_constructors_lite` Reviewed By: diliop Differential Revision: D80307486 fbshipit-source-id: 6285ea8416d34d1a9fa7bff1d564176a7542c42c