Chris Tolliday

Pagable for DICE test keys Summary: Needed to add Pagable/PagableTagged bounds on Key/DiceKeyDyn, not because these need to be pagable, which is why this uses PagablePanic for many of the non-trivial types. Reviewed By: cjhopman Differential Revision: D92463610 fbshipit-source-id: 50e1715af69198a5687218f5b89d2f6b7a909ff4

Add value_serialize for DICE test keys Summary: These don't need to be serializable, just adding this so the default implementation of value_serialize can be removed. Reviewed By: scottcao Differential Revision: D94842640 fbshipit-source-id: a18bb5bc0c320d0eb84e7d281d207b076c8a20a1

Pagable for DICE keys 2/2 Summary: Make remaining DICE keys in buck2/app pagable Reviewed By: cjhopman Differential Revision: D92463612 fbshipit-source-id: 4c382fe42b2319b4701bb6b9eaca44c17e08341e

Generalize static_string to static_value Summary: Generalizes StaticStr added in D92094775 to work for arbitrary types. Adds a new macro `declare_static_value_type!` to add the boilerplate and newtypes for a new static value type. `static_value!` can be used to declare a static instance of any declared type. Reviewed By: Nero5023 Differential Revision: D101062636 fbshipit-source-id: 11a0ee7d0cedfb44c74baf08853a2dbb8a223d97

Simple sled-backed PagableStorage Summary: This is the most naive of possible disk-backed storages. This will be replaced by a SQLite backend later but doesn't hurt to have this available. Reviewed By: cjhopman Differential Revision: D97558558 fbshipit-source-id: 7826e8a24b78c65462f1c9f714ccf66528ceaf24

Add value_serialize for buck2 DICE values Summary: Only using `TodoValueSerialize` for `ReadFileKey`, to be implemented later. Differential Revision: D94842639 fbshipit-source-id: c8650206cd9aab49ef5ac079b64057ddcbb6f023

Pagable for DICE keys 1/2 Summary: Make some DICE keys pagable using `derive(Pagable)` and `#[pagable_typetag(DiceKeyDyn)]` Reviewed By: cjhopman Differential Revision: D92463609 fbshipit-source-id: acbe5649f1d088b2d45ea83ac3a5cda833115afb

Derive Pagable for more core types Summary: Derive pagable for some common types, used by DICE values in later diffs Reviewed By: cjhopman Differential Revision: D94290768 fbshipit-source-id: 82f1a4d758a8e560469fab8e2ca897e8f68e1603

Add pagable::StaticStr Summary: To support Pagable for &'static str. Doing this without requiring a new type would require using an interner, which would work but not be as efficient. There should be relatively few places where we declare static strings and potentially a lot of places where we serialize the same string so it seems worth making this efficient. Reviewed By: cjhopman Differential Revision: D92094775 fbshipit-source-id: c521d516d671ce4ae6ddac4c1a65417a373f3f84

Support registering generic types with typetag Summary: Adds support for registering generic type instantiations with `#[pagable_typetag]`'d traits. - `register_typetag!(Trait, Wrapper, InnerType)` — registers `Wrapper<InnerType>` for deserialization as dyn Trait, and implements `PagableRegistered<dyn Trait>` for `InnerType` - `#[pagable_tagged(Trait)]` — attribute macro that generates a PagableTagged impl with `PagableRegistered` bounds on generic params, enforcing registration at compile time - TypetagRegistration<T> / TypetagRegistry<T> — updates registry to be usable by both `#[pagable_typetag]` and `register_typetag` Reviewed By: Nero5023 Differential Revision: D101228018 fbshipit-source-id: be2cf361aef3f2558a751ed03bcd03115a4d2c55

[buck2 Use DashMap for SessionContext Summary: The need to lock the `Mutex<SessionContext>` for serialization makes it not possible to serialize anything in parallel. This still uses a Mutex for the inner StarlarkSerContext/StarlarkSerdeContext, which is addressed in the next diff. Reviewed By: cjhopman Differential Revision: D104714165 fbshipit-source-id: b28437a8270ac6998a0b5425bf61337c6b5ab844

Add value_serialize on key types Summary: Used to optionally serialize a DICE value, most DICE keys in buck2 can use `OkPagableValueSerialize` for cases where the DICE value is a result and the error case should not be serialized. For now adding a default implementation that returns `TodoValueSerialize`, to be removed later. Differential Revision: D94772188 fbshipit-source-id: 03b805e84b2e23e64c52e42692a3ec17f5e18fc3

Add SqliteBackedPagableStorage Summary: Initial implementation of SQLite pagable storage backend. Already faster and more memory efficient than sled, but will be faster with later diffs. Many keys, small values. Sled vs SQLite: | num keys | value | num deps | storage | dice nodes | dice edges | | -------- | ----- | -------- | ------- | ---------- | ---------- | | 1M | 10B | 0 | sled | 1000002 | 2M | | stage | duration | cpu user | cpu system | rss anon | Δ rss anon | peak rss | Δ db size | Δ jemalloc allocated | Δ jemalloc resident | | -------- | -------- | -------- | ---------- | -------- | ----------- | -------- | ---------- | --------------------- | -------------------- | | compute | 7.39s | 26960ms | 8870ms | 862MB | +862MB | 2.79GB | +158B | +799MB | +940MB | | page_out | 37.37s | 29290ms | 6180ms | 931MB | +68MB | 1.08GB | +403MB | -18MB | +31MB | | page_in | 22.18s | 18710ms | 3480ms | 1.12GB | +187MB | 1.12GB | +24MB | +200MB | +187MB | | total | 66.94s | 74960ms | 18530ms | 1.12GB | 1.12GB | 2.79GB | 427MB | 980MB | 1.16GB | | num keys | value size | num deps | storage | dice nodes | dice edges | | -------- | ---------- | -------- | ------- | ---------- | ---------- | | 1M | 10 | 0 | sqlite | 1000002 | 2M | | stage | duration | cpu user | cpu system | rss anon | Δ rss anon | peak rss | Δ db size | Δ jemalloc allocated | Δ jemalloc resident | | -------- | -------- | -------- | ---------- | -------- | ----------- | -------- | ---------- | --------------------- | -------------------- | | compute | 5.41s | 25260ms | 5150ms | 854MB | +854MB | 2.46GB | +16KB | +772MB | +894MB | | page_out | 25.46s | 9470ms | 15710ms | 791MB | -63MB | 966MB | +114MB | -91MB | -63MB | | page_in | 10.28s | 4910ms | 5300ms | 1.00GB | +210MB | 1.01GB | 0B | +209MB | +211MB | | total | 41.15s | 39640ms | 26160ms | 1.00GB | 1.00GB | 2.46GB | 114MB | 889MB | 1.04GB | Fewer keys, larger values. Sled vs SQLite: | num keys | value | num deps | storage | dice nodes | dice edges | | -------- | ----- | -------- | ---------- | ---------- | ---------- | | 100K | 10KB | 0 | sled | 100002 | 200K | | stage | duration | cpu user | cpu system | rss anon | Δ rss anon | peak rss | Δ db size | Δ jemalloc allocated | Δ jemalloc resident | | -------- | -------- | -------- | ---------- | -------- | ----------- | -------- | ---------- | --------------------- | -------------------- | | compute | 0.63s | 4110ms | 1560ms | 1.10GB | +1.10GB | 1.24GB | +158B | +1.12GB | +1.18GB | | page_out | 15.00s | 10750ms | 8590ms | 301MB | -804MB | 1.20GB | +1.90GB | -868MB | -846MB | | page_in | 34.65s | 30740ms | 4780ms | 1.50GB | +1.20GB | 1.51GB | +34MB | +1.21GB | +1.20GB | | total | 50.28s | 45600ms | 14930ms | 1.50GB | 1.50GB | 1.51GB | 1.93GB | 1.46GB | 1.54GB | | num keys | value size | num deps | storage | dice nodes | dice edges | | -------- | ---------- | -------- | ------- | ---------- | ---------- | | 100K | 10K | 0 | sqlite | 100002 | 200K | | stage | duration | cpu user | cpu system | rss anon | Δ rss anon | peak rss | Δ db size | Δ jemalloc allocated | Δ jemalloc resident | | -------- | -------- | -------- | ---------- | -------- | ----------- | -------- | ---------- | --------------------- | -------------------- | | compute | 0.56s | 4070ms | 1110ms | 1.12GB | +1.12GB | 1.25GB | +16KB | +1.09GB | +1.16GB | | page_out | 12.27s | 6680ms | 5610ms | 95MB | -1.02GB | 1.12GB | +1.06GB | -1.03GB | -1.02GB | | page_in | 32.48s | 28280ms | 3990ms | 1.15GB | +1.05GB | 1.15GB | 0B | +1.04GB | +1.05GB | | total | 45.31s | 39030ms | 10710ms | 1.15GB | 1.15GB | 1.25GB | 1.06GB | 1.11GB | 1.18GB | Reviewed By: cjhopman Differential Revision: D103253125 fbshipit-source-id: 06e70f99ae477634a607142059d75e3b19e8f5b1

Always attach source line to SourceLocation Summary: Seeings some errors where the source line would be useful. Can't guarantee these are always the right source locations and not paths in some internal error handling code, but in places where the file! is correct, line! should be too. Reviewed By: AishwaryaSivaraman Differential Revision: D96191019 fbshipit-source-id: 964eb2a5dfac9db1c7a0eba9c408e2cc4cdb63a0

impl Pagable for ThinBoxSlice, ThinArcStr, PackageRelativePath Summary: A few manual impls, separating from derived impls in later diffs Reviewed By: cjhopman Differential Revision: D94290767 fbshipit-source-id: db6c3a8548079123ab292b6aa81d997b296b129b

Always attach both trimmed installer stderr and uploaded log path on installer error Summary: D94829200 shows a small amount of error context from the installer stderr on connection error, but it's not enough to debug some errors we are seeing. Changing this so that `handle_install_request` doesn't exit early and always uploads installer logs if they exist and increasing the size of the trimmed stderr shown in error messages. It's still confusing that we have both stderr logs and logs the installer writes to directly, these should also be unified, but that's a different problem. Reviewed By: NavidQar Differential Revision: D97808154 fbshipit-source-id: cc38051a56f910ef8be2bc1b9fee2acc0a1b8853

Detect invalid buck-out directory Summary: We're seeing a lot of errors that look like this: ``` Caused by: create_dir_all(/data/users/lbin/fbsource/buck-out/ig-codegen) Caused by: No such file or directory (os error 2) ``` `create_dir_all` shouldn't return "No such file or directory" just because parent dirs don't exist, but because they can't be created for some reason that is not explained by the ENOENT error code. If this happens, show a better error message and tag this as an environment error. Goal of this stack is to improve error messages and categorization for the bulk of the `MISSING_INTERNAL_PATH` errors we are seeing, which are a big source of Tier0 errors. Reviewed By: cjhopman Differential Revision: D101836718 fbshipit-source-id: 02d01615a41aaba6c623db6a434bf1e1d63b4658

Move page_out outside of CoreState Summary: Make this work more like page_in, so we can run async code outside of the core state thread without spawning new threads. At a high level, this calls `EvictCachedValues` to drop values that are already stored but have been paged back in, `KeysToPageOut` to get nodes that need to be serialized and stored, then `EvictKeys` to drop values that have been stored. The main downside of doing this outside of the core is that it's no longer trivial to immediately evict values from core state once they are paged out. We could send one `EvictKeys` message for every key but if a large number of these (1m+) are sitting in the queue for some reason the memory overhead is non-trivial (min size of a StateRequest is the largest enum variant/maybe 100 bytes+). Adding batching for `EvictKeys` to avoid adding unnecessary overhead to the core state while still being able to drop values as they are paged out. Reviewed By: cjhopman, Nero5023 Differential Revision: D105087599 fbshipit-source-id: 65bd24a6d28fd52406705c53e1a64202172c81d8

Make StarlarkEvalKind Pagable Summary: Moving `DynEvalKindKey` into `buck2_core` so that the concrete types that implement it can use it to register for type tag in the crates that define those types. It makes sense to have this in core since it's now used for more than just starlark debugging, it's also used as an identifier for sketches. Adding `StrongHash` impl for `AnonTarget` lets it use the blanket impl of DynEvalKindKey. `BxlDynamic` doesn't need to use `DynEvalKindKey` because it's just a String. We can't register String/ThinArcStr with typetag. Reviewed By: cjhopman Differential Revision: D104294122 fbshipit-source-id: 15b0a0d8e367355a9c1305c07ddb36dc9ed75cfa

Use pagable_typetag for BaseDeferredKeyDyn Summary: Support deserializing `Arc<dyn BaseDeferredKeyDyn>` Reviewed By: cjhopman Differential Revision: D92094776 fbshipit-source-id: 05cf4460db4a88d01959f75344c12ab271ff9c55