github.com-facebook-buck2

Move starlark_fmt into buck2/tools/ Summary: Relocates the `starlark_fmt` crate (an opinionated ruff-based Starlark/BUCK formatter) from `fbcode/starlark_fmt/` to a new shared `fbcode/buck2/tools/starlark_fmt/` directory as the first step toward open-sourcing it into the public `facebook/buck2` repo, and makes the crate open-source-buildable. Sources are moved with `sl mv` to preserve internal history. Relocation: - New shared `fbcode/buck2/tools/` home for buck2-adjacent tools. - Renamed `mod.rs` files to sibling modules (`autofixes.rs`, `formatting.rs`, `subcommands.rs`) to satisfy the buck2 `rust-no-mod-rs` lint. - No local `PACKAGE`; the crate inherits `fbcode/buck2/PACKAGE`. Target-level `visibility` is kept on `starlark_fmt_lib`. - A forwarding `alias` is left at `fbcode//starlark_fmt:starlark_fmt` so the generated DotSlash manifest (`tools/lint/starlark/starlark_fmt`) keeps resolving; removing it is a follow-up gated on the MSDK schedule repoint. - Updated the sole external Buck consumer `fbcode//autodeps2/autodeps2`, the `README.md` build/test paths, and dropped the now-stale `starlark_fmt` row from `fbcode_enable_pyrefly_targets.csv`. The ~20 path-based consumers of `tools/lint/starlark/starlark_fmt` are unaffected (that path is unchanged). Open-source readiness (so the OSS `cargo`/`buck2-oss` build works). Because this diff registers the crate in the OSS cargo workspace, it must itself be OSS-buildable, so all of the following live here: - Applied the dual MIT/Apache license header to all `.rs` files. - autocargo-generated `Cargo.toml` for the crate and registered it in the OSS workspace (`fbcode/buck2/Cargo.toml` members). - Gated the internal `lint` subcommand out of the OSS build: it uses `//common/rust/lint_message` (not in OSS), so `mod lint`, the `lint_message` import, the `Command::Lint` variant/arm, and the `lint_files` use are `#[cfg(fbcode_build)]`, and the dep is `# oss-disable`. `fmt`/`diff`/`stdin` are unaffected; the internal build is unchanged. - Gated the Meta-internal `quickcheck_arbitrary_derive` (used only by the fbcode_build-gated fuzz tests) out of the OSS build: `mod fuzz_starlark` is `#[cfg(all(test, fbcode_build))]`, and an autocargo `extra_buck_dependencies` removal drops it from the generated `Cargo.toml`. - Pinned `get-size2 = "=0.10.1"` into the OSS `[workspace.dependencies]` (via an autocargo `extra_buck_dependencies` reference) so the OSS cargo build resolves `ruff_python_ast` against `compact_str 0.9.1` (0.10.2 would need compact_str ^0.10 and fail the `CompactString: GetSize` bound). - Fixed lints that `buck2-oss-lint` denies but internal lint doesn't: `clippy::str_to_string` (28x `.to_string()` -> `.to_owned()` on `&str`) and `let_underscore_drop`. Reviewed By: d16r Differential Revision: D113069398 fbshipit-source-id: d724baad1d7b60ea545a335961b6bd36ceca933c

mem: Add mini_vec crate and type Summary: `MiniVec` is a `Vec` but is a single pointer on the stack; the length and capacity are bitpacked into the *top* bits of the pointer if they're small, otherwise allocated in the heap. This does introduce high-bit pointer packing to buck2 which we have not had so far. Claude assures me that modern kernels do not hand out high addresses without a user explicitly asking for them and no one complained when I asked, so I guess the team is on board with this. I'll use this in a couple places, but the first thing we'll be able to do is replace all the `ThinBoxSlice` Reviewed By: jtbraun Differential Revision: D105465807 fbshipit-source-id: 65663fb277c1301be55a3b1e94ef831ae80d4f78

Add profile options to all commands Summary: This adds a common set of flags to all commands that support enabling profiling based on regexes that match stringified forms of StarlarkEvalKind. There's more documentation on the flags on the --profile-patterns flag in CommonStarlarkOptions. But basically: --profile-patterns: accepts regex patterns of evaluations to enable profiling --profile-patterns-mode: profiling mode to use --profile-patterns-output: root path to write profiling data. every individual evaluation gets its profiling data written to a separate file. we write to a subdir of this with similar format to buck-out/log (i.e. datetime+uuid) we additionally write an all_keys.list file listing all the evaluation names we profiled. This works by (1) setting up profile configuration that StarlarkEvaluatorProvider uses to enable profiling as appropriate and then (2) adding a ProfileEventListener that gets profile_collected calls as evaluations are finished. We already consistently use StarlarkEvaluatorProvider to get the enabling of profiling. The remaining diffs in the stack ensure that we consistently finalize profiles for all evaluations. Reviewed By: JakobDegen Differential Revision: D64281578 fbshipit-source-id: b09824a94d1a596216fa9d30e761436492e63b39

Add errorformat lib Summary: rfc: https://fb.workplace.com/groups/buck2dev/permalink/4110348932586481/ This diff adds the errorformat parsing lib. Since I did this development first locally using git, it is hard for me to split the diffs. However, I added comprehensive tests, it should be ok. Reviewed By: Will-MingLun-Li Differential Revision: D82085318 fbshipit-source-id: 4a5f0cde6bb37a504e126bd70f5002de7d950167

fix: a lot of instances of clippy::needless_lifetimes (#944) Summary: This doesn't get all of them because: 1. there are so many of them 2. the lint is pretty jank Part of https://github.com/facebook/buck2/issues/943 Pull Request resolved: https://github.com/facebook/buck2/pull/944 Reviewed By: JakobDegen Differential Revision: D74271655 Pulled By: dtolnay fbshipit-source-id: c7e41c47248d4be1f5d33a14b676ab35fc5180e2

cli: Allow `--isolation-dir` globally Summary: Let's see what CI says at least? Reviewed By: IanChilds Differential Revision: D87866823 fbshipit-source-id: c04e23ac1dc764a35f58e50286bba436e51740c0

Add `--agent-context` CLI flag with buckconfig-driven validation Summary: RFC: https://fb.workplace.com/groups/buck2dev/permalink/4302138863407486/ Adds a new `--agent-context key=value` CLI flag (modeled after `--client-metadata`) that lets AI agents pass structured context like `intent=fix`, `attempt=2`, or `prior_error=missing_target`. On the server side, a buckconfig-driven schema (`[agent_context]` / `[agent_context#field]` sections) allows enforcing required fields, allowed values, and unknown-key rejection for specific client IDs, while non-enforced clients pass data through without validation. Reviewed By: jtbraun Differential Revision: D99735398 fbshipit-source-id: 1c3e7d0eacae1d92b18068ad5e8fd472fd4d2d39

Replace Dice task internals Summary: This mostly rewrites the management of the per-key dice task state that manages cancellation, dependents, spawning the worker, etc. Chris initially did this because it unblocked an approach to paging that he was considering (described in the next diff). My understanding is that they have since picked a different direction, so this needs new justification. I do think that despite that change, the next diff is a well-motivated simplification on its own. It fixes a small memory issue I encountered elsewhere, but also having just the single map does seem quite a bit easier to reason about. As to whether this diff would be well-motivated on its own even without the next one is a bit less obvious (claude and I talked and interestingly we disagree somewhat). The new data structure is subtler and more complicated than any one thing that existed before, but in exchange for that there's only one thing; previously there were three differently synchronized things (an atomic and two locks) managing information equivalent to what is now the `DiceTaskInternal` contents. I tend to think that's an improvement. The current version has some sketchy things going on, but at least they're all in one place and I think a person has better odds of reasoning correctly about it (claude tells me the previous version of this was actually probably bug-free but idrk if I believe it). In other words, I like this because I think one explicitly complicated component is better than three implicitly complicated ones. There's one somewhat significant wart here in the form of the `AtomicList` type, which is simple but pretty poorly optimized. I may try to revisit that. ...and then after writing all that explanation I ran benchmarks on the next diff and saw it was -2.5% memory. So now this is all easily justified Reviewed By: jtbraun Differential Revision: D103355606 fbshipit-source-id: f8616468c3e8f94ed2c3a8fe8e5d0276a56b7748

errors: Remove test-related anyhow usage Summary: We still have a bunch of anyhow usage in test related code. This commit removes ~all of that This is overwhelmingly find-and-replace and other very simple stuff, nothing really interesting here Differential Revision: D88890953 fbshipit-source-id: 8d5299acf25590299314528925a3e46cc2a04560

Introduce pagable serialization framework Summary: This introduces the `pagable` serialization framework, which extends serde with the ability to preserve Arc sharing semantics across serialization boundaries. This is a foundational component of the "Infinite Graph" initiative to enable graph hydration for buck2. The framework allows buck2 to: 1. Serialize/deserialize its internal computation state while preserving the extensive Arc sharing in its data structures 2. Initialize daemon state from previously computed results (bulk graph hydration) 3. Load partial graphs on-demand to reduce memory usage 4. Share computation data across machines for daemon reuse Unlike standard serde which deep-copies Arc contents, pagable tracks Arc identity during serialization and restores the same sharing structure during deserialization. This is critical for buck2's memory efficiency and correctness. **pagable crate:** Core traits in `traits.rs`: - `PagableSerializer` / `PagableDeserializer` - context traits that implementations must provide, offering both serde serialization and pointer stashing - `PagableSerialize` / `PagableDeserialize` - traits for types that can be serialized/deserialized with the framework - `Pagable` - convenience trait combining both with Send + Sync + Debug + 'static Built-in implementations for: - Primitives (u8-u128, i8-i128, f32, f64, bool, String, Duration) - Standard types (Option, Box, PhantomData, tuples up to 5 elements) - Collections (Vec, Box<[T]>, BTreeMap, arrays up to 32 elements) - serde_json::Value Testing utilities: - `TestingSerializer` / `TestingDeserializer` - in-memory implementations with TypeId tracking to catch type mismatches during stash/unstash **pagable_derive crate:** Derive macros: - `#[derive(Pagable)]` - derives both PagableSerialize and PagableDeserialize - `#[derive(PagableSerialize)]` - derives just serialization - `#[derive(PagableDeserialize)]` - derives just deserialization Field attributes: - `#[pagable(flatten_serde)]` - serialize field using serde directly - `#[pagable(skip)]` - skip field during serialization - `#[pagable(discard = "expr")]` - skip serialization, use expr for deserialization - `#[pagable(bound = "...")]` - custom trait bounds for generics Reviewed By: christolliday Differential Revision: D88400546 fbshipit-source-id: 85e9baf8b313e6217a68efd47cf64ae599d9540a

update platform010 & platform010-aarch64 symlinks Summary: * `__rust_no_alloc_shim_is_unstable_v2` replaces `__rust_no_alloc_shim_is_unstable`. It's a no-op function rather than a static data member. * Fix clippy errors from if-let chains * Sprinkle `'_` in places to make ambiguous lifetimes lint happy * `nonnull_provenance` feature is stable * `result_flattening` feature is stable * `file_lock` feature is stable * `extract_if` now accepts a range as the first argument * `duration_constructors` split into `duration_constructors_lite` Reviewed By: diliop Differential Revision: D80307486 fbshipit-source-id: 6285ea8416d34d1a9fa7bff1d564176a7542c42c

mem: Change how `DiceTransaction` -> `DiceComputation` works Summary: We want to have `&'c mut DiceComputations<'d>` return `&'d Value`s. Right now, it can't do that because `DiceTransaction` derefs to `DiceComputations<'static>` and obviously we can't get back `'static` things. We instead need a conversion like `&'d DiceTransaction -> DiceComputations<'d>` and this diff does that. We remove the deref impl (since the signature is no longer right), add `.ctx() -> DiceComputations<'d>` instead, and then go and fix all the compiler errors. As a part of that, we can actually change the representation of `BaseComputeCtx` to just directly hold a `CoreCtx`, which makes a bit more sense. I'll do some renames towards the end of this stack to help justify why things are the way they are (note also that the use of `ModernComputeCtx::Parallel` is fine here and actually has nothing to do with `Parallel` specifically) Reviewed By: NavidQar Differential Revision: D110626399 fbshipit-source-id: 4ba0db607a88e3456ccb7b091b6c2bf5402566c7

feat: implement bytes type Summary: Implement the Starlark `bytes` type per the spec at https://github.com/bazelbuild/starlark/blob/master/spec.md#bytes. The `bytes` type is an immutable sequence of bytes (integers in range 0-255). It supports: - Literal syntax: `b"hello"`, `b'hello'`, `b"""hello"""`, `rb"\n"` (raw) - Constructor: `bytes(b"...")`, `bytes("str")` (UTF-8 encode), `bytes([int...])` (from iterable) - Indexing: `b"abc"[0]` returns `97` (integer, not 1-byte bytes) - Slicing: `b"hello"[1:3]` returns `b"el"` - Containment: `97 in b"abc"`, `b"bc" in b"abcd"` - Concatenation (`+`), repetition (`*`), comparison, hashing - `str(b"...")` decodes UTF-8 with U+FFFD replacement for invalid sequences - `repr(b"...")` returns byte literal syntax - `ord(b"A")` returns `65` - `.elems()` method returning iterable of 1-byte bytes objects - Not directly iterable (use `.elems()`) Reviewed By: JakobDegen Differential Revision: D105587378 fbshipit-source-id: 3b7d2444101c94e94444d428577213ddec8cc5c9

Require error tag for buck2_error! macro Summary: Last part of the code where `buck2_error` can be created. Have `buck2_error!` macro take in an ErrorTag instead of an array, making it possible to enforce an ErrorTag (Since ErrorTag can't be set as empty but we can't enforce a non-empty array at compile time unless there's something I'm not aware of). All usage of this macro have at most 1 ErrorTag anyways so this doesn't make a difference at the moment, and tags can be added as an array through the `.tag()` function Reviewed By: JakobDegen Differential Revision: D67998689 fbshipit-source-id: 397fbc272f05d97713ec66284c563035e516e46a

Add PagableArc and storage abstraction Summary: Add smart pointer types that support transparent paging of data to/from storage: - `PagableArc<T>`: Reference-counted pointer where data can be paged out to storage when memory pressure is high - `PinnedPagableArc<T>`: Reference-counted pointer that guarantees data stays in memory, with direct Deref access - `PinnedPagableArcBorrow<'a, T>`: Borrowed reference avoiding refcount manipulation for short-lived access The storage abstraction includes: - `DataKey`: 128-bit unique identifier for stored data - `PagableStorageHandle` trait: Interface for storage backends to persist and retrieve paged-out data - `PagableDeserializer::storage()`: Allow deserializers to access storage for nested paging Also removes unused dependencies from the BUCK file (bytemuck, dashmap, futures, fxhash, sled, libc). Differential Revision: D88924819 fbshipit-source-id: b4f31b2f0e8b4a330a481de638c642ac9d81f3f2

artifacts: Attach a `'v` lifetime to `DeclaredArtifact` Summary: Declared artifacts have shared mutability during an analysis evaluation. This isn't all that uncommon, but what is uncommon is that they achieve that by just storing an `Rc`: https://www.internalfb.com/code/fbsource/[72d70f64f6a9b1869a176e70a2ad2d6d95eb5729]/fbcode/buck2/app/buck2_artifact/src/artifact/artifact_type.rs?lines=341 We're not normally in the business of doing that. Shared mutability in starlark is generally represented by having some starlark value own a `RefCell`, allocating that starlark value, and then using references to that allocation to achieve sharing. That approach is generally a bit better because it cleanly ties the "duration" of the shared mutability to the starlark evaluation - once the starlark evaluation has completed, the shared mutable thing is gone - this helps with correctness a bit (important, given the standard implications of shared mutability) More concretely though, we have this thing: https://www.internalfb.com/code/fbsource/[72d70f64f6a9]/fbcode/buck2/app/buck2_core/src/unsafe_send_future.rs?lines=38-47 Used here, for example: https://www.internalfb.com/code/fbsource/[72d70f64f6a9]/fbcode/buck2/app/buck2_analysis/src/analysis/env.rs?lines=233 This justifying for Sendness is incorrect in cases like this. Even if you send an entire heap with all of its values across a thread, there's nothing guaranteeing that the `Rc` hasn't been cloned into something that isn't being sent across threads. A future diff does a better job of using the type system (instead of unsafe) to justify `Send` in a situation like that, and it turned up this problem. So the plan is to replace the `Rc` with an allocation that's correctly tied to the starlark heap. That requires putting a `'v` lifetime onto the `DeclaredArtifact` type, which in turn requires putting it onto a great many other places. This diff does that without yet using the `'v` lifetime for anything interesting, just a `PhantomData` for now. All changes in this diff are trivial(ish). The interesting stuff was pulled out earlier in the stack Differential Revision: D77119488 fbshipit-source-id: dbefb6b28a13fa166e0e88e4bda5fba73b329460

Make PagableSerialize/Deserialize traits object safe Summary: Making these traits object safe simplifies typetag implementation later in this stack, as well as more pagable implementations later on. This means incurring the cost of dynamic dispatch for each deserialize call, but let's not worry about that yet. (code mostly commandeered from cjhopman) Reviewed By: cjhopman Differential Revision: D90884770 fbshipit-source-id: 7b94e603fa9473222bd2ab838a91ca27246925dd

mem: Implement improved `join_all` and `try_join_all` futures Summary: We currently use `futures` for this which is extremely inefficient for little good reason. Reimplement it better. Details are in the code as always, but I'll say that I actually am pretty happy with where this landed in terms of complexity to benefit ratio. The one little index table trick is basically the only clever thing going on here, and in exchange we have memory use that's like `80 + N * (max(future size, return size) + 1)` (compared to `futures` which allocates like 80 bytes of bookkeeping for each future). I'll send some follow ups later to use this in other parts of buck2 as well, but will stick to the current stuff in dice for a bit longer first. Reviewed By: jtbraun Differential Revision: D111267627 fbshipit-source-id: bb3891e360b50afa622a482c74b4c5c1600900ec

open source kosabi/common Summary: open sourcing kosabi/common is a dependency for stubsgen, so open sourcing it required for open sourcing stubsgen Reviewed By: IanChilds Differential Revision: D82041656 fbshipit-source-id: 527cd0467159b09d06f96f8187bdeab73b88e690

Require tag for all buck2_error_derive enum types Summary: Force an ErrorTag for all current and future enum types that are derived from buck2_error_derive. Can be as simple as `Input`, `Environment`, and `Tier0`. Note that I tagged all the current ones to the best of my knowledge and could be wrong. Please fix or let me know anyone finds anything that seems wrong Reviewed By: JakobDegen Differential Revision: D67954001 fbshipit-source-id: 17798faa4d7365e1d24c3eddac2bcc55d1e66b95