Jeremy Braun

Codemod `ok_or_else(|| internal_error!(...))` to `Option::internal_error` Summary: Mechanical adoption of the `BuckErrorOptionContext` API from the previous diff: 345 call sites across 142 files convert `.ok_or_else(|| internal_error!(...))` into the documented spelling — `.internal_error("...")` for plain messages, `.with_internal_error(|| format!(...))` where the message has format arguments. Message shape (`... (internal error)`), `InternalError` tag, and caller source location are identical before and after. Import fallout handled: 109 now-dead `use buck2_error::internal_error;` lines removed (40 files keep it for remaining non-`Option` macro uses); imports scoped into `mod tests` where the only converted sites are `cfg(test)`-gated, so the lib build stays warning-free. One `#[cfg(windows)]` site (`buck2/src/check_user_allowed.rs`) is deliberately left unconverted since it cannot be compile-verified on Linux. Reviewed By: christolliday Differential Revision: D116804875 fbshipit-source-id: 879fcb502cb449cca4e7e7cf608c08ccff0e1819

Fix clippy warnings for fbcode//buck2/scripts:buck2-oss-lint Summary: Was breaking oss lints: https://www.internalfb.com/intern/test/562950276062710 `useless_borrows_in_formatting` is now included in clippy::all under the Rust 1.97 toolchain. Our internal clippy lints are -W only for this, but OSS are error all, so they need fixed. Reviewed By: NavidQar Differential Revision: D113775014 fbshipit-source-id: 5c05fda200a7154d8ca8668d8b9d9dbfa90dc9bd

aquery: build tset nodes with a worklist instead of recursion Summary: aquery fails with `stack overflow (internal error)` on tset graphs nested deeper than ~1,500 levels — graphs that `buck2 build` handles fine. Builds are depth-safe by construction: every tset projection is its own spawned DICE computation (`EnsureTransitiveSetProjectionKey`), so awaiting a child suspends the parent task and each node polls from the base of a fresh worker stack; the Starlark-side tset iterators are all explicit-stack iterative. aquery's shadow-graph construction was the one exception: `compute_tset_node → convert_inputs → get_tset_node → compute_tset_node` awaits nested `BoxFuture`s inline within a single task, so polling stacks one set of native frames per nesting level on a fixed 2 MiB worker stack until `check_stack_overflow()` trips. This replaces the recursion with two iterative phases in `compute_tset_node`: - **Discovery**: a breadth-first worklist resolves each reachable projection key one level deep (`convert_inputs_shallow`), probing the existing `NodeCache` so cached subgraphs are not re-walked. Waves are processed with `try_compute_join`, preserving the old code's sibling parallelism. The root key is never probed, since the caller's `get_or_compute` has already registered its in-flight entry. - **Assembly**: an explicit postorder stack builds the interned `SetProjectionInputs` children-before-parents, seeding each completed node into the `NodeCache` (via a new non-blocking `NodeCache::seed`). Construction now imposes no depth limit, but a new hard cap remains necessary: `SetProjectionInputs` owns its children recursively through immutable `ArcIntern`s, so *dropping* a deep chain still recurses and was observed to SIGSEGV the daemon between depth 8,000 and 12,000 once construction stopped being the bottleneck (`ArcIntern` is immutable, so the drop cannot be made iterative without restructuring the type). Each node's subtree depth is therefore tracked in the cache value — counting depth accumulated across separately-computed cached subgraphs — and queries over tsets nested deeper than `MAX_TSET_NESTING_DEPTH` (4,096, ~3x margin below the observed crash point) fail with a clean `Transitive set nesting depth exceeds the maximum of 4096 supported by aquery` error instead. Net effect on a linear tset chain of depth N (`aquery "deps(...)"`, chained `run` actions each consuming the accumulated projection): | depth | before | after | |---|---|---| | 1,000 | works | works, byte-identical output | | 2,000–4,096 | `stack overflow (internal error)` | works | | 4,097+ | `stack overflow (internal error)` | clean depth-limit error, daemon unaffected | Reviewed By: JakobDegen Differential Revision: D117688476 fbshipit-source-id: d2f47f5765070df21381148bf36a0a3aece0cc16

buck2: add `buck2 log snoop` to watch in-progress commands' event logs Summary: Adds a tail-mode event log reader and a `buck2 log snoop` subcommand that tails the event log of a running command and renders it in a Superconsole, catching up on past events and then following along live — a way to watch a build's progress from another terminal. The tail reader (`TailReader` in `buck2_event_log`) retries on EOF instead of ending the stream, relying on the writer's per-tick flushes making the compressed stream incrementally decodable. It only reads through the already-open handle and never re-opens or locks the file, so the owning client remains free to delete the log (including on Windows, where the Rust standard library opens files with delete sharing). A deleted or abandoned log ends the stream after `--idle-timeout` seconds without growth. Event content always comes from log files, but when a daemon is already running, snoop additionally opens a read-only `SubscribeToActiveCommands` subscription (via `BuckdConnectOptions::ExistingOnly` — it never spawns a daemon) to know which commands are live: - The Superconsole header shows `(snoop <i>/<n> running: <command line>)` for the attached invocation, and the `Buck UI` link shows its trace id. - The replay speed keys switch invocations instead: `k` attaches to a newer running command and `j` to an older one (implemented via a snoop-specific `Clock` that reinterprets `scale_speed`, so no changes to the shared console input handling are needed). Without daemon info, or when viewing a finished invocation, they navigate the retained event logs instead. - By default snoop attaches to the newest running command; when nothing is running it waits for the next command to start rather than instantly replaying the newest finished log, and when the snooped command ends it keeps waiting indefinitely (Ctrl-C to exit) for whatever runs next. With an explicit `--trace-id`/`--recent`/path selector it replays that invocation and exits when it ends. `--idle-timeout` only bounds how long a non-growing log is retried before its command is treated as gone. Reviewed By: JakobDegen Differential Revision: D115659505 fbshipit-source-id: f808b7b8901f1dc03621316b840857177ab2e68b

Add FreezeDynamic that allows for frozen-type determination at runtime Summary: Add `FreezeDynamic`, `FreezePlan`, `FreezeTarget`, and `FreezeSlot` to let a complex value choose its concrete frozen AValue layout at runtime. Bridge existing `FreezeStatic` implementations through the same protocol, so existing types keep their fixed-layout behavior. Prepare and reserve the destination before replacing the source with its forwarding pointer, then publish the final vtable only after the complete payload is initialized. The reservation header from the preceding diff keeps failed or unwound freezes walkable without consuming payload bytes. `FreezeSlot::write` reports a destination that does not match the selected target as a `FreezeError` instead of panicking, so a misbehaving plan cannot abort the process. The reserved destination is never exposed as a value: sources forward through the slot's pointer, and `publish` returns the `FrozenValue` only after the final header is written. The existing `try_freeze_directly` shortcut remains in place alongside `FreezeTarget::direct`; the next diff consolidates it into `FreezeStatic::prepare_freeze`. Add coverage for selecting different frozen types, for a failed freeze leaving the frozen heap usable, and for a mismatched freeze target reporting an error. An unwind-path test is gated to `cfg(panic = "unwind")` builds, since Buck builds tests with panic=abort. Reviewed By: JakobDegen Differential Revision: D98794639 fbshipit-source-id: 29d68aa730b9d4ea39510861a9d0bce15da544a7

Encode heap reservations in entry headers, remove Blackhole Summary: Add `Reservation(ValueAllocSize)` as the third `AValueHeapEntryState`, encoded with tag `10` in the allocation header. Arena walking can recover the allocation size without reading payload bytes, leaving the complete payload available for incremental initialization. Remove the `BlackHole` value, its generated and runtime vtables, and the associated payload-based sentinel handling. Preserve reentrant downcasts by treating reservations as not-yet-observable values. Reviewed By: JakobDegen Differential Revision: D114738940 fbshipit-source-id: 27f6584d8f951d15e47276ee71a4dcd52364e5fb

aquery: extend mixed-graph traversal to rdeps, allpaths, somepath, and bounded deps Summary: Unbounded unfiltered `deps()` traverses aquery's mixed action/tset graph in O(nodes + edges); every other aquery traversal still consumes the flattened `QueryTarget::deps()` view and pays O(n^2). Worst is `rdeps` (and `allpaths`, which delegates to it): it materializes the entire flattened universe graph via `Graph::build_stable_dfs` and then `reverse()`s it — quadratic in memory as well as time. This moves the remaining unfiltered traversals onto the mixed graph, in a new `mixed_graph` module (`deps.rs` renamed, since it no longer only serves `deps`): - `rdeps`/`allpaths` (unbounded and depth-bounded): materialize the mixed graph of the universe closure with the existing postorder traversal machinery, reverse its O(n + e) edge list, and emit actions in postorder. Bounded depth uses a level-synchronized reachability pass where entering an action node costs one hop and passing through tset projection nodes is free — which is exactly the flattened depth, since a flattened edge corresponds to a mixed path with a single action entry. - `somepath`: reuses the generic `async_bfs_find_path` (now `pub`) over mixed nodes, with the target predicate matching action keys only; tset nodes are projected out of the returned path, leaving consecutive actions connected by flattened edges as before. - Depth-bounded `deps()`: a level-synchronized frontier walk where each level's actions expand their tset closures directly from the inline `SetProjectionInputs` structure. Tsets expanded at an earlier level are skipped: level-order guarantees their member actions were already discovered at a depth no greater than they would get here. Filtered forms of all of these fall back to the flattened implementations, which their semantics require: a `deps(..., filter)` filter expression observes each node's flattened first-order deps (`first_order_deps()`), and DOT output renders the flattened edges themselves. To let the aquery overrides fall back, the `rdeps` and `somepath` trait default bodies move to free functions in `buck2_query`, mirroring `deps`. This also fixes unfiltered aquery `rdeps`/`allpaths` erroring with `Not an action: <target>` whenever the universe contains an analysis node — which any plain target-literal universe does (e.g. `rdeps(deps(//foo), //bar)`). The default implementation resolves universe roots through `QueryEnvironmentAsNodeLookup`, which cannot look up analysis nodes; the mixed traversal resolves roots from the universe set first, the same way `dfs_postorder`'s `AqueryNodeLookup` always has. The new `test_rdeps` integration test covers this case. Measured on the synthetic worst case from D117623866 (M producers under one shared tset, N chained consumers, M = N = 5,000, opt binary vs prod buck2, result sets verified identical; `rdeps`/`allpaths` compared with the analysis node subtracted from the universe since prod errors otherwise): | query | before | after | |---|---|---| | `rdeps(universe, //:p0)` | 4.74s | 0.44s | | `rdeps(universe, //:p0, 1)` | 4.78s | ~0.5s | | `allpaths(//:c4999, //:p0)` | 4.69s | 0.42s | | `somepath(//:c4999, //:c0)` (full-depth search) | 2.28s | 0.41s | | `deps(//:c4999, 2)` | 0.11s | 0.23s | At M = N = 20,000, `rdeps` drops from 83.9s to 1.1s. Reviewed By: JakobDegen Differential Revision: D117691207 fbshipit-source-id: f191934791f723d5c36d852b3c1ab0a1c29af8da

Degrade corrupt-span resolution to diagnostics instead of panicking Summary: Starlark call-stack frames and compiler spans can, rarely, carry a `Span` that does not lie within their own `CodeMap`. Resolving one goes through `CodeMap::find_line` (an `assert!`) or `CodeMap::source_span` (a raw slice), and either panics — killing the buck2 daemon mid-command: in production while compiling BUCK files (`Compiler::eval_module`), and locally while formatting an unrelated PACKAGE-file error. P2457607755 catalogs the production occurrences and the open root-cause leads. The root cause is unknown, so this diff has three goals: stop the crashes, make each occurrence visible and auditable, and record enough detail to be a clue later. Stop the crashes. `CodeMap::find_line`, `find_line_col`, and `source_span` clamp instead of panicking: a position past the end resolves to the last line, and a span that is not a character range grows to the enclosing character boundaries rather than yielding nothing. Make them visible. Each clamp reports through a new reporter hook in `starlark_syntax`, which `starlark` bridges to a process-global `SoftErrorHandler` slot (`set_global_soft_error_handler`), which buck2 installs in `init_late_bindings` as the existing `Buck2StarlarkSoftErrorHandler` — so occurrences land in Scuba rather than vanishing. Contexts holding an `Evaluator` keep using the evaluator-scoped handler; the global slot exists for span resolution and freeze-time optimization, where none is in reach. `FrozenFileSpan::new` reports at construction too, catching corruption at the moment a bad pairing is built rather than when it is next read, and `Frame::write_two_lines` prints a self-describing line in place of the source snippet. Record clues. Reports name the resolution path, the offending position or span, the file length, and whether the `CodeMap` is a real source file or a native one — that last distinction matters, because a span resolved against a native codemap it cannot belong to is what a pointer landing on unrelated memory looks like, `<native>` being a fixed 8-byte source. Reporting is capped at eight occurrences per process, since resolution is hot and one corrupt span is typically resolved many times, but is not capped at one, so several distinct corruptions stay visible. Debug builds still fail fast at `FrozenFileSpan::new`. This intentionally does not fix the corruption itself. This is a mitigation, not a feature: a `TODO(jtbraun)` at the reporting machinery in `codemap.rs` marks the whole diff for back-out once the invalid spans are root caused. That is also why resolution clamps rather than surfacing "invalid position" through the resolvers' return types — sentinel-bearing signatures would thread through every consumer and make this hard to remove. Reviewed By: JakobDegen Differential Revision: D115579787 fbshipit-source-id: ec52e040c65e842e31fc91154823eb65c53f1a43

Move SQLite page-out writes to shard writers Summary: Today, the page-out workers will flush a sqlite shard if it fills. That stops that worker from finishing anything else that may be in it's queue. When we move the serialization to tokio futures, we do NOT want the tokio async workers pausing for long periods of time to flush sqlite, we want them to continue serializing into the next output buffer so that the sqlite threads can flush on their own later. -- Replace synchronous sqlite flushes from page-out workers with per-shard `page-sql-N` writer threads and bounded active/spare/pending write buffers. Default the sqlite shard count to 10 while preserving `BUCK2_PAGABLE_SQLITE_SHARDS`, and keep aggregate writer/backpressure counters for validation. Reviewed By: christolliday Differential Revision: D109892236 fbshipit-source-id: 589d64d1ba0cc00413f2b5c2f1bbfc18615fd16b

Use stable type names for pagable typetag tags Summary: Typetags are persisted with serialized data, but generic registrations built them with `std::any::type_name`, whose output has no stability guarantee across compiler releases. A toolchain upgrade could silently change every generic tag and invalidate previously written data. The pre-existing `#[pagable_tagged]` wrapper path had the same problem. Introduce `PagableStableName`, which composes a name from `module_path!()` and source identifiers — for generics, recursively through type arguments plus `Display`-formatted const arguments. The result changes only when a type is renamed or moved, never with a compiler upgrade. `#[derive(Pagable)]` / `#[derive(PagableSerialize)]` implement the trait automatically (skipped for types with lifetime parameters, since composed names are cached by `TypeId`). Generic monomorphizations cannot hold the composed name in a per-instantiation static, so `memoized_stable_name` interns it, leaking exactly one `String` per instantiation per process. Generic `#[pagable_typetag]` impls get the `PagableStableName` requirement added to their where clause by the macro; hand-written impls of `#[pagable_tagged]` wrappers need `Self: PagableStableName` spelled out alongside the existing `PagableRegisteredFor` bound. The starlark `TypeMatcher` combinators and `TypeMatcherAlloc` factory methods get the bound threaded through accordingly. Reviewed By: JakobDegen Differential Revision: D115599856 fbshipit-source-id: 8d63133f67290a4bff1f06a1b38334c4ab0aecd0