Alexey Spiridonov

`benchmark_ab.py`: multi-binary benchmark A/B reports Summary: `benchmark_ab.py` is a new tool to simplify iterating on changes that affect several benchmark binaries. It handles three recurring chores: - Finds benchmark binaries from Buck target patterns. - Aggregates repeated A/B runs into one report. - Prioritizes wins and regressions with absolute and relative thresholds. Read the file docblock for more. ```text $ folly/tool/benchmark_ab.py measure --before=bc56e16776 --after=0097974dca \ //folly/result/... ... High-priority regressions: 15.2+11.2ns (+73.5%): try_to_result_error fbcode//folly/result/test:result_bench 15.2+11.2, 15.2+11.3, 15.2+11.1, 15.2+11.2, 15.4+11.2 ``` Reviewed By: janondrusek Differential Revision: D112568029 fbshipit-source-id: 544b93f75660e791285743188150a3eec38b1e1e

Add repeatable backtests for rule changes Summary: Reading a rule diff cannot show whether it improves the artifact an agent produces. Add a backtest runner that stages a fixed scenario and explicitly selected rules, then preserves the artifact, trace, errors, and run metadata. The runner refuses local changes to the source files that shape the run and records their checkout revision, so a later comparison can identify and reproduce those inputs. Model, reasoning effort, and executable paths remain explicit metadata. ___ Differential Revision: D119108233 fbshipit-source-id: db92be24e462f47d7c8a92b9bb31b17b8ba80707

Add a coroutine-metadata backtest Summary: Add a backtest for writing a general-audience API contract from the middle of a real design conversation: - A partial implementation already exists but needs changes, much like a task resumed with truncated history. - Superseded discussion, an intermediate review, later decisions, and a frozen source snapshot must be reconciled. The resulting document must explain which coroutine work inherits metadata and where propagation stops, without exposing that history. ___ Reviewed By: yinglan98 Differential Revision: D119108234 fbshipit-source-id: 8ee2d0edc297074346f0da354c1124fca3838058

benchmark_ab.py: flag benchmarks with high run-to-run spread Summary: +488/-47 nonblank lines, excluding recorded `testdata/`. Large run-to-run variation can make an otherwise reportable effect unreliable, or deserve attention even when the estimated effect is small. Measure each complete before/after series by its range relative to its median. Calibrate extreme spread against the measured benchmark set: - Require at least 20 eligible series so one benchmark cannot move the cutoff too much. - Use Tukey's outer fence for the relative cutoff. - Keep the low-priority nanosecond and percentage thresholds as practical floors. Report the result without duplicating benchmarks: - Annotate priority rows in place. - Put unclassified outliers in a separate section with TSV class `high-spread`. - Add before/after ranges in ns and as a percentage of median to TSV. Series missing any round or with median at or below 2ns remain available for effect classification but do not influence spread calibration. ___ Differential Revision: D114015996 fbshipit-source-id: dc6f5f97d85da3b32fa6b9df89717501f307f7cb

Omit metadata markers from signal-safe stack traces Summary: `getAsyncStackTraceSafe()` returns code addresses for symbolization. A metadata marker stores a discriminator, not code, in its return-address field, so returning it would create a bogus frame. Skip markers while retaining their visible wrapper frames. If a marker has no parent, continue into the outer stack through the async-stack root stored on the wrapper. Reviewed By: yfeldblum Differential Revision: D117316299 fbshipit-source-id: 1dfa11b657bfbb82bd453c4824d21f290333709c

`benchmark_ab`: prebuild binaries; use a 20ps CI floor Summary: `benchmark_ab` previously invoked `buck run` for every target attempt. A build failure was therefore retried like a noisy benchmark, wasting the retry budget and obscuring that no measurement ran. Build both revisions once before measurement. Save the full build logs and a side-specific copy of each executable under `--out`, then run those copies for every round. Build failures now abort immediately, and each side stays tied to the binary built from that revision. Near zero, a percentage-based precision target shrinks toward zero even though timing noise does not. Tiny benchmarks can therefore be stable yet time out because their 95% confidence interval cannot become proportionally narrow enough. Use a 20 ps absolute floor for the confidence-interval width check. In both observed batches, this let every row pass that had failed only that check; 30 ps and 50 ps let no additional rows pass. Split-half stability remains separate, so early and late samples must still agree. Reviewed By: janondrusek Differential Revision: D117262537 fbshipit-source-id: 1af46980eeca42f9c54fd123552b13c91c5696aa

Stop async-stack walkers at buffer capacity Summary: Signal-safe async-stack walkers can run in a fatal-signal handler. Once their output buffer is full, further reads cannot add to the trace, but following a damaged chain can fault again and lose the diagnostic. Stop the Folly symbolizer and Android Breakpad walkers before reading another frame or root. Metadata-marker support adds reads that do not produce output: recognizing and skipping a marker follows its parent link. Output count will therefore no longer bound every chain read, so establish explicit capacity stops before adding marker handling. Reviewed By: yfeldblum Differential Revision: D117316296 fbshipit-source-id: 1b83a975b1453d632e2f83eab65c145232122948

Add signal-safe async-stack capture with scoped metadata Summary: Signal handlers and profilers need the values attached by `co_withMetadata()`, but `getAsyncStackTraceSafe()` returns addresses alone. Add `getAsyncStackTraceSafeWithMetadata()` to return the same address sequence with one optional metadata value per frame. Each `co_withMetadata()` value is returned beside its wrapper continuation address. Ordinary frames have no metadata, and internal marker frames stay hidden. Reviewed By: sonntex Differential Revision: D117316309 fbshipit-source-id: 442b87677a5a3dda3d4d2e6b75e569a324af2fda

Benchmarks taking ~0ns should converge in adaptive mode Summary: Adaptive mode stopped each benchmark row only when its confidence interval was small relative to the measured time. After subtracting the baseline, a row can measure as `0ns`; then the allowed interval is `0ns` too. Such benchmarks would keep sampling until timeout even after the absolute interval is tiny -- this is bad in two ways: - The output would claim convergence failure for a reliable run - The run would take a long time. Fix this by treating 10ps **absolute** differences as "precise enough". Reviewed By: swong15 Differential Revision: D112210828 fbshipit-source-id: 2bdb566d7737233fffb973f2c9ace44d94e3c882

Isolate Codex reviews and identify session models Summary: The agent rules need two runtime guarantees: independent reviewers must not inherit the author's private Codex configuration, and authorship delegation must know which model serves the current session. `codex-reviewer.py` resolves `codex` through `PATH` and runs it with a private home, user configuration disabled, and an explicit absolute work directory that controls repository context. Its fixed command surface keeps cold reviews read-only and preserves the prompt, trace, errors, and final response. A generated policy lets a fresh reviewer launch one cold review. Given a session UUID, `session_current_model_id.py` searches matching Codex and Claude traces from newest to oldest for a non-synthetic model name. It reports `UNKNOWN-NOTIFY-USER` when none is found. ___ Differential Revision: D117436460 fbshipit-source-id: ccc027b42bc9a1526109d837e0d8314ec5fb02f1