Lalit Maganti

tp: refactor *Engine -> Connection and split out Database (#5766) This CL does some prep work to make it much easier to distinguish between connection local objects and database global objects. Right now this doesn't matter os much because TP only has one connection and one database but this will not apply with multi-threaded trace processor.

tp: add syntaqlite-driven macro expansion behind a flag (#5615) **Stack:** 1. #5614 — `tp: prep PerfettoSqlParser for a second backend` 2. **#5615 — `tp: add syntaqlite-driven macro expansion behind a flag`** ← you are here 3. #5525 — `tp: replace sqlfluff formatter with syntaqlite` (parallel; based on #5615) 4. #5616 — `tp: route PerfettoSqlParser through syntaqlite macro expansion` 5. #5472 — `tp: delete the legacy PerfettoSqlPreprocessor` --- Adds a second PerfettoSqlParser backend that delegates macro expansion and statement splitting to the vendored syntaqlite library, behind a compile-time `PerfettoSqlParser::kUsesSyntaqliteMacros` constant (default: false). Both implementations live side by side; flipping the constant is a one-line change. * Bumps the vendored syntaqlite drop (`syntaqlite_perfetto.{c,h}` and the `--macro-style rust` regen flag) to expose the macro-lookup callback and rewrite-introspection APIs the new path needs. The bump tokenizes `!` as TK_BANG (188) instead of TK_ILLEGAL, so the existing preprocessor's macro detection and the SqliteTokenizer's re-exported token table are taught about kBang to keep the legacy path green. * Adds `IntrinsicMacroExpander` for the `__intrinsic_*` shims (stringify, token_apply, …) and a `MacroRewriteBuilder` that walks syntaqlite's flat rewrite list to rebuild a SqlSource with the same nesting as the authored macro calls (so SQLite-side error tracebacks resolve back to the call site). * Adds `SqlSource::FromMacroExpansion` for tagging rewrite nodes with their macro of origin. * Drops `SyntaqliteMacroImpl` next to `LegacyImpl` in `parser.cc`; the wrapper holds both as `std::optional<>` and the public methods dispatch via `if constexpr`. * Brings the new parser unittests for macro expansion. The single intrinsic-token-apply test that depends on the new path's exact whitespace is gated with `if constexpr (kUsesSyntaqliteMacros)`. No production behaviour change: the constant defaults to false so trace_processor still routes through the handwritten preprocessor. Sanity-flipped locally and verified parser/preprocessor unittests plus the PerfettoSQL diff tests pass on both paths.

tp: remove FindById and replace with operator[] (#5906) Now that ids are guaranteed to appear in tables, there's no point having std::optional on the API boundary. Just make it always return the RowReference instead.

tp: add IndexedFilterIn bytecode for In on indexed columns (#5158) ## Summary - Add `IndexedFilterIn` bytecode that uses binary search on index permutation vectors for `In` filters - When a column has an index and the query uses `In`, the planner now emits `IndexedFilterIn` instead of the generic `In` bytecode - For each value in the list, binary-searches the index permutation vector (O(log N) per value) and concatenates matching ranges - Reduces `In` filter cost from O(N) to O(k log N + matches) where k is the number of values ## Stack 1. #5154 - tp: add In filter support to TypedCursor and optimize In bytecode 2. **#5158 - tp: add IndexedFilterIn bytecode for In on indexed columns** (this PR) 3. #5155 - tp: migrate experimental_slice_layout to use In filter on track_id ## Test plan - 4 new bytecode interpreter tests (`IndexedFilterIn_Uint32_NonNull_MultipleValues`, `_NoMatch`, `_SingleValue`, `_String_SparseNull_MultipleValues`) - 1 new query planner test (`PlanQuery_SingleColIndex_InFilter_NonNullInt`) - 1 new end-to-end TypedCursor test (`TypedCursorInFilterWithIndex`) - All existing indexed filter tests updated and passing

tp: Decompose trace_processor_shell.cc into focused modules (#4959) Break the 2361-line monolith into 6 focused files under src/trace_processor/shell/: - query.{h,cc}: query execution and result printing - shell_utils.{h,cc}: stats printing and trace export - metrics.{h,cc}: metric extension loading and execution - sql_packages.{h,cc}: SQL package and stdlib loading - metatrace.{h,cc}: metatrace writing utilities - interactive.{h,cc}: interactive REPL and line editor This allows subcommands to reuse individual pieces (e.g. query execution, trace loading) without depending on the entire shell. No logic changes, purely mechanical extraction.

ui: add slice duration histogram when single selecting (#5659) Adds a reusable distribution panel and shows a duration histogram for matching slices from the single-slice details panel. The inline details view defaults to the selected slice track, supports switching to whole-trace scope, and links out to a full matching-slices tab.

tp: extract IncrementalState to fix CustomState UAF (#5593) Fixes the use-after-free flagged in `PacketSequenceStateGeneration::CustomState`. ## The bug Previously, each `CustomState` held a raw `generation_` back-pointer that was re-pointed every time a new generation was created via `OnNewTracePacketDefaults` (the `set_generation(this)` loop in the multi-arg ctor). The `TraceSorter` can hold a `RefPtr` to an older Generation `G1` while a newer `G2` — to which `G1`'s shared CustomState had been re-pointed — is dropped after `SEQ_INCREMENTAL_STATE_CLEARED`. `G1` stays alive (and so does the CustomState), but its `generation_` then dangles to freed memory; any subsequent lookup through the CustomState dereferences the freed `G2`. ## The fix Refactor the generation/CustomState relationship so that lifetime is correct **by construction**: - New `IncrementalState : RefCounted`. Owns the per-incremental-state-interval data: `interned_data_`, the array of `CustomState`s (now `unique_ptr` rather than `RefPtr`), and the persistent thread descriptor. A new `IncrementalState` is constructed only on `SEQ_INCREMENTAL_STATE_CLEARED`. - `PacketSequenceStateGeneration` becomes a thin per-`trace_packet_defaults` snapshot: holds `RefPtr<IncrementalState>` (shared with sibling generations within the same interval), the defaults blob, and the validity flag. All interned-data / custom-state / thread-descriptor accessors are forwarders to the `IncrementalState`. - `CustomState` is no longer `RefCounted` (it's owned uniquely by its `IncrementalState`). Its back-pointer is `IncrementalState*` and is set exactly once at lazy-allocation time inside `IncrementalState::GetCustomState<T>`. Because the `IncrementalState` owns the `CustomState`, the pointer is stable for the entire life of the CustomState — UAF impossible. - `OnNewTracePacketDefaults` no longer copies the `InternedFieldMap` or re-points CustomStates; it constructs a new Generation that shares the same `RefPtr<IncrementalState>`. This removes a per-defaults-change copy that could be O(map-size) in the hot path. - `OnPacketLoss` walks the IncrementalState's CustomState array and clears any slot whose `ClearOnPacketLoss()` opted in (TES today), then returns a new Generation referencing the same `IncrementalState` with the validity bit cleared. --- **Stack:** - #5588 — tp: stop mutating PacketSequenceStateGeneration on packet loss - #5590 — tp: split TrackEventSequenceState into descriptor + delta state - **#5593 — tp: extract IncrementalState to fix CustomState UAF** (this PR)

tp: import Firefox profiler markers as slices (#5686) Parse the `markers` table from preprocessed Gecko/Firefox profiler profiles and emit them as slices in trace_processor: - Pull `meta.categories` once at the profile level so marker `category` indices resolve to readable names. - Capture each per-thread `markers.{name,startTime,endTime,phase,category, data}` array; the `data` payload is captured as raw JSON via a new `Iterator::CollectCurrentScope` helper on the JSON parser and flattened into `data.*` args at parse time using `AddJsonValueToArgs`. - Phase 0 (Instant) becomes dur=0 slices; phase 1 (Interval) uses `endTime - startTime`; phase 2/3 (IntervalStart/End) are LIFO-matched per (utid, name) via a cookie stack. - Markers route through `TrackCompressor` with a blueprint dimensioned by `(utid, marker_name)` so each marker name on each thread gets its own track (matching the Firefox Profiler's marker chart layout) and same-name overlaps fan out across compressed lanes instead of being dropped. - Fix a pre-existing importer bug exposed while developing: when the top-level `shared` block carries only the string table (no shared frames), per-thread frames/stacks were silently dropped. Threading the strings through `Process{Legacy,Preprocessed}FramesAndStacks` fixes this. - Refactor `gecko_trace_tokenizer.cc` parsing into single-purpose helpers (`ParseFrameTable`, `ParseFuncTable`, `ParseStackTable`, `ParseSamplesTable`, `ParseMarkersTable`, `ParseMeta`, `ParseMetaCategories`, `ParseThreads`, `ParseMarkerDataArray`) so `ParseThread` and `ParseGeckoProfile` become a flat dispatch. - Add a dedicated `dev.perfetto.FirefoxProfilerMarkers` UI plugin (enabled by default) that lays out `firefox_marker` tracks under each thread, sub-grouped by category when a thread spans multiple categories. A single SQL query computes per-(utid, name) track groups with their dominant category and the depth needed for intra-name overlap. - Add diff tests covering all four phases, payload flattening, and same-name overlap fanning out across lanes. - Update other-formats.md to reflect markers being supported, and add generation instructions for samply and Python's `profiling.sampling --gecko` (3.15+).

tp: drive DebugAnnotation parsing on the proto-args work stack (#5587) ## Summary - Adds `ProtoToArgsParser::ParseDebugAnnotation` and `EnableDebugAnnotationParsing()`, building on the work-stack driver from #5589. - Pushes DebugAnnotation / NestedValue work items onto the same `work_stack_` as proto-message items (extends the variant); DebugAnnotation -> proto_value -> DebugAnnotation cycles are processed iteratively, so depth in the input does not grow the C++ stack. - `track_event_parser` opts in via `EnableDebugAnnotationParsing()`; the `.perfetto.protos.DebugAnnotation` type override is removed. - `DebugAnnotationParser` class is deleted; callers use `ProtoToArgsParser::ParseDebugAnnotation` directly. - Adds a regression test that parses a 1000-deep `DebugAnnotation -> proto_value(DebugAnnotation)` cycle. ## Stack - #5589 — prep refactor - **#5587 — security fix (this PR)** ## Test plan - [ ] `perfetto_unittests --gtest_filter="ProtoToArgsParserTest.*:DebugAnnotationParserTest.*"` passes (23 tests including new regression). - [ ] `tools/diff_test_trace_processor.py --name-filter="TrackEvent.*|DebugAnnotation.*"` passes.

tp: refactor TreeTransformer to use interpreter directly (#4948) ## Summary Refactor TreeTransformer to emit filter bytecodes directly instead of going through QueryPlanBuilder. This prepares the architecture for future tree operations (propagate up/down, merge, invert) by making TreeState the central owner of all tree + column data. ### Key changes - **Bytecodes emitted incrementally**: `FilterTree()` emits filter bytecodes + `FilterTreeState` immediately, not deferred to `ToDataframe()`. This allows chaining multiple operations on the same transformer. - **Column data compacted alongside tree**: Column storage is copied into TreeState and compacted in-place by `FilterTreeState`. Sparse null columns are densified on registration. Filter bytecodes always operate on compacted positions `[0..row_count-1]`. - **Removed QueryPlanBuilder dependency**: `FilterOnly` and its `FilterResult` struct removed — filter bytecodes emitted directly. - **BitVector::Compact()**: New `&&`-qualified method using PEXT (with portable `PextSlow` fallback) for efficient in-place bitvector compaction. Used by `FilterTreeState` to compact null bitvectors. - **BitVector::Clone()**: Word-level memcpy for deep copy. - **Performance**: Reusable `keep_bv` in TreeState (no alloc per filter), fused `old_to_new` + `original_rows` compaction pass, per-column compaction for cache-friendly access, column dedup when the same column is filtered multiple times. ### Architecture (prep for future ops) TreeState now owns: parent array, original_rows, column data copies, null bitvector copies, P2C cache, scratch buffers, and reusable keep_bv. Every tree bytecode (filter, and future propagate/merge) reads and writes through TreeState. Structural changes invalidate the P2C cache; traversal operations rebuild it lazily. ## Test plan - [x] 25 tree unit tests pass - [x] 13 tree diff tests pass (including chained filter tests) - [x] 90 dataframe unit tests pass (no regressions) - [x] 11 new BitVector::Compact unit tests added - [x] GN check clean

protovm: fix UAF in protovm::ParseSelect (#5795) ParseSelect snapshots `cursors_` (which holds raw Node* pointers in RwProtoCursor::node_ and parent_link_.map_node) before running nested instructions and unconditionally restores it afterwards. A nested `del` that frees the same node aliased by the snapshot would leave a dangling pointer on restore; a subsequent `del` would then double-free the node and corrupt the SlabAllocator freelist. Track one extra Node* in the parser: the dst.GetNode() of the innermost in-flight ParseSelect frame. When `del` is asked to free a non-root node whose address matches that snapshot, abort the program instead of producing a stale cursor on restore. Snapshots form a non-strict descending chain (each new ParseSelect either keeps dst at the same node or moves it deeper), so checking only the innermost is sufficient: if any outer snapshot aliases the deletion target, this one does too. Auxiliary space is O(1). Legitimate `del` patterns continue to work: top-level del; del under a select that walks dst to the target via path components; iterate-and- delete on a repeated field. Only `del` wrapped in a select that does not move dst is rejected, and those forms are always expressible by hoisting the del out of the wrapper: select(cursor=DST, path=[]) { del } -> del select(cursor=SRC, path=[X]) { del } -> select(SRC,X){...}; del select(path=[X]) { select(path=[]){del} } -> select(path=[X]){del} As defense in depth, also reserve TracePacket.trace_provenance (124) and TracePacket.protovms (125) in the SMB packet validator. Both fields are emitted only by TracingServiceImpl and have no legitimate producer-side use, so dropping them on the producer-incoming path prevents a compromised producer from injecting protovm programs or spoofing sequence provenance. Also, harden RwProtoCursor replacing the current PERFETTO_DCHECK with actual aborts. Regression tests added in vm_unittest.cc for both the empty-path and SRC-cursor-with-non-empty-path variants of the UAF, in packet_stream_validator_unittest.cc for the two newly reserved fields and in rw_proto_unittest.cc for the stricter checks. --------- Co-authored-by: Kean Mariotti <keanmariotti@google.com>

tp: Introduce NullBitvector struct to unify bv + popcount (#5255) Wrap the null bitvector pointer and its prefix popcount cache into a single NullBitvector struct, stored as one register instead of two separate ones. This reduces register bloat and makes the bv+popcount pair an atomic unit across all bytecode instructions, similar to how TreeState works. The struct is used for both SparseNull (bv + popcount) and DenseNull (bv only, popcount empty) columns. PrefixPopcount lazily fills the popcount field in-place.

tp: add documentation for symbolization/deobfuscation and improve bundling (#5535) Pull out symbolization/deobfuscation documentation into a separate page so that it's easy page to reference rather than being hidden inside the heap profiling docs (when it's not at all specific to heap profiling). Also some small improvements to traceconv bundle rounding off some sharp edges. And also write some tests for it too.

tp: add common_flags, query subcommand, and dispatcher (#5128) ## Summary - Shared infrastructure: common_flags.h/.cc (GlobalOptions, BuildConfig, SetupTraceProcessor, LoadTraceFile with symbolization/deobfuscation) - Query subcommand with -f, -c, -i, -W, -p flags - ClassicSubcommand: moves entire classic path into a Subcommand, uses shared infra - Thin dispatcher: finds subcommand or falls back to classic - Integration tests for query subcommand and classic backcompat Part 2/7 of the subcommand CLI implementation (RFC-0018). ## Stack 1. #5134 — SubcommandContext + GetLongOptions() interface 2. **#5128** ← this PR — common_flags + query + ClassicSubcommand + dispatcher 3. #5129 — repl + serve subcommands 4. #5130 — summarize + export subcommands 5. #5131 — metrics subcommand 6. #5132 — help system + file collision + classic edge case tests 7. #5133 — BUILD.gn consolidation

tp: add tree_propagate_down operator for tree transformations (#5257) ## Summary - Add `PropagateDown()` method to TreeTransformer for root-to-leaf BFS propagation - Support SUM, MIN, MAX, FIRST, LAST aggregate operations - Propagated columns become real columns enabling filter→propagate→filter chains - PropagateTreeDown bytecode takes spec_start/spec_count with separate source/dest columns ## Stack - #5262 tp: introduce TreeColumns and refactor TreeTransformer to use it - **#5257 tp: add tree_propagate_down operator for tree transformations** (this PR) ## Test plan - 13 new propagate diff tests including chained propagation and filter→propagate→filter - All 23 tree tests pass

tp: improve handling of includes to prepare for multi-connection (#5791) * add "claiming" system to allow for a single connection to "claim" the fact its including the module and have other connections which might also want to include at the smae time to wait for it. * add cycle detection of includes on same thread * change includes to "poison" on error: this allows removing mysterious errors on double includes or across-connections

perfetto: heap_profile: add host subcommand for local Linux profiling (#5049) Adds 'heap_profile host -- mybinary args' for profiling local Linux processes without an Android device, using LD_PRELOAD and the heapprofd glibc preload library. tracebox run without a subcommand with --system-sockets handles both the traced daemon and the session in a single invocation. --notify-fd blocks until the session is active before launching the profiled binary. PERFETTO_HEAPPROFD_BLOCKING_INIT=1 ensures the binary's first malloc waits for heapprofd to fully attach. The libheapprofd_glibc_preload.so prebuilt is downloaded from GCS automatically (linux-amd64/arm/arm64). A --preload-library flag allows overriding the path for local builds. Also adds heapprofd_glibc_preload to the roll-prebuilts manifest so it is updated alongside the other prebuilts on each release.

tp: drop ftrace based on raw timestamps not forged ones (#5809) We allow some events to change their timestamp value at tokenization time. This should however not be used to decide to drop the ftrace event or not, which should always use the raw ts. Properly plumb that through to prevent incorrect drops Moreover, improve performance of TracePacketData passing through the sorter queues by adding dedicated code instead of the same codepath as TrackEventData.

tp: Add subcommand infrastructure and classic CLI integration tests (#4958) Part of RFC 0018 (subcommand CLI for trace_processor_shell). Add comprehensive integration tests covering the entire classic CLI surface (version, help, query, export, parsing flags, dev flags, and mutual exclusion checks) to lock down backcompat before introducing subcommands. Also add the Subcommand base class and FindSubcommandInArgs function with unit tests as the foundation for subcommand dispatch.

tp: migrate winscope importer to be a plugin (#5867) This a straightshot move to integrate with the plugin system. Followup changes will consolidate all the winscope code into a single plugin for simple interaction with the rest of trace processor.