Lalit Maganti

tp: model trace types as importers, remove the TraceType enum (#6618) Today, all trace detection logic is centralized inside TP meaning it's impossible for plugins to register new formats and have them work without also changing the core code. As the number of formats has increased, this has gotten more and more unweildy Design it to instead be structured such that new trace types can be developed fully independently *without* making any changes to the core. This will be proved out in a followup change where perf text importer will be moved.

tp: add Arrow support for dataframes (#6827) Provide an Arrow serializer and deserializer for Dataframes using a standard Arrow file containing one record batch. Primitive columns map to Arrow numeric arrays, strings map to Utf8, nullable columns carry validity bitmaps, and sparse storage uses Arrow's logical row layout. The serializer computes metadata and variable-width sizes before streaming column contents with bounded scratch memory. The deserializer validates file framing, record batch metadata, and buffer extents before rebuilding column storage, null state, implicit Id columns, and strings in the target StringPool.

tp: refactor *Engine -> Connection and split out Database (#5766) This CL does some prep work to make it much easier to distinguish between connection local objects and database global objects. Right now this doesn't matter os much because TP only has one connection and one database but this will not apply with multi-threaded trace processor.

tp: add syntaqlite-driven macro expansion behind a flag (#5615) **Stack:** 1. #5614 — `tp: prep PerfettoSqlParser for a second backend` 2. **#5615 — `tp: add syntaqlite-driven macro expansion behind a flag`** ← you are here 3. #5525 — `tp: replace sqlfluff formatter with syntaqlite` (parallel; based on #5615) 4. #5616 — `tp: route PerfettoSqlParser through syntaqlite macro expansion` 5. #5472 — `tp: delete the legacy PerfettoSqlPreprocessor` --- Adds a second PerfettoSqlParser backend that delegates macro expansion and statement splitting to the vendored syntaqlite library, behind a compile-time `PerfettoSqlParser::kUsesSyntaqliteMacros` constant (default: false). Both implementations live side by side; flipping the constant is a one-line change. * Bumps the vendored syntaqlite drop (`syntaqlite_perfetto.{c,h}` and the `--macro-style rust` regen flag) to expose the macro-lookup callback and rewrite-introspection APIs the new path needs. The bump tokenizes `!` as TK_BANG (188) instead of TK_ILLEGAL, so the existing preprocessor's macro detection and the SqliteTokenizer's re-exported token table are taught about kBang to keep the legacy path green. * Adds `IntrinsicMacroExpander` for the `__intrinsic_*` shims (stringify, token_apply, …) and a `MacroRewriteBuilder` that walks syntaqlite's flat rewrite list to rebuild a SqlSource with the same nesting as the authored macro calls (so SQLite-side error tracebacks resolve back to the call site). * Adds `SqlSource::FromMacroExpansion` for tagging rewrite nodes with their macro of origin. * Drops `SyntaqliteMacroImpl` next to `LegacyImpl` in `parser.cc`; the wrapper holds both as `std::optional<>` and the public methods dispatch via `if constexpr`. * Brings the new parser unittests for macro expansion. The single intrinsic-token-apply test that depends on the new path's exact whitespace is gated with `if constexpr (kUsesSyntaqliteMacros)`. No production behaviour change: the constant defaults to false so trace_processor still routes through the handwritten preprocessor. Sanity-flipped locally and verified parser/preprocessor unittests plus the PerfettoSQL diff tests pass on both paths.

tp: add streaming table export (#6839) Add a bounded-memory Export API for statically registered Trace Processor tables. ExportOutput is a virtual streaming interface with an optional file path alternative for formats which need direct random-access output later. Runtime SQL tables and views are deliberately outside this API. Support two explicit tar formats. kArrowTar is a stable, forwards-compatible, export-only archive containing standard Arrow files, including implicit ID columns and empty static tables for external consumers. kPerfetto is a version-coupled archive with an internal manifest which can restore tables into a compatible fresh Trace Processor instance; its representation carries no cross-version compatibility guarantee. Expose export through RPC, HTTP, the shell export subcommand, and Python. Keep the existing SQLite export path unchanged; moving it behind this API waits for the separate file-I/O abstraction work.

tp: memoize proto arg keys to avoid re-interning per field (#6348) ProtoToArgsParser rebuilt and re-interned the flat_key/key strings of every arg on every field of every message. The flat_key for a given proto field path is invariant, so give the parser a StringPool and a memo trie keyed by field number. The Delegate API now takes already-interned StringIds instead of Key strings, so delegates are pure value sinks and never re-intern keys. A new proto_to_args_parser_benchmark isolates the key path and shows ~1.7-2x; end-to-end trace load gains are ~1-2% as key interning is a small slice of ingestion.

tp: remove FindById and replace with operator[] (#5906) Now that ids are guaranteed to appear in tables, there's no point having std::optional on the API boundary. Just make it always return the RowReference instead.

tp: add C++ flamegraph computation library (#6851) A flamegraph is the trie of key-paths: every path through the input forest with the same sequence of merge keys collapses into one merged node. The library computes that trie and returns it as a tree plus a flat dataframe (id, parentId, depth, name and per-metric value columns). Presentation (sibling ordering, x layout) is deliberately absent: it is cheap to compute over the far smaller merged output and belongs to the consumer, which can also skip it entirely. The three views are one algorithm parameterized by anchors, the frames the merged trees grow from and re-root at: top-down anchors at the kept roots, bottom-up at every counted frame, pivot at frames matching the pivot pattern. Downward, one forward scan drives a cursor through an exact (parent node, key) hash map; upward, each anchor's caller chain is walked carrying the anchor's weight: its counted subtree total stopping at other anchors, which for bottom-up reduces to the frame's own values. Nodes whose subtree is zero on every metric are dropped, which is also what gives show-stack filters their effect on the output. Filters (show/hide-stack, show-from-frame, hide-frame and the pivot pattern) are regexes evaluated once per distinct frame name through the input's name dictionary, then folded into per-frame kept/counted flags and a nearest-kept-ancestor index in a single pass along the paths. Hidden frames fold their values into the nearest kept ancestor. Frames carry any number of metrics, flattened row-major with a constant stride; every accumulation is metric-wide. Each output node retains its constituent input frames, so consumers can aggregate arbitrary per-frame properties without the library knowing about them (the benchmark exercises a ONE_OR_SUMMARY aggregation this way). Input frames may arrive in any order: parents-before-children is detected and used directly, anything else costs one reverse-index and DFS pass. FlexVector gains reserve() for the pre-sized node columns. On 1M-frame callstack-shaped forests (see flamegraph_benchmark.cc), top-down computes in ~90ms, ~120ms including the dataframe; with filters ~77ms, bottom-up ~200ms, pivot ~18ms. At 100k frames every view is below 10ms.

tp: replace GetExtensionSlowly with SelectiveTracePacketDecoder (#6218) Introduce SelectiveTracePacketDecoder, a wrapper around protozero's SelectiveTypedProtoDecoder. An allowlist mask covers the TracePacket metadata fields the pipeline reads by name; the data field and out-of-tree extensions fall into unknown_fields(), which drives module dispatch in one pass per packet instead of one buffer re-scan per registered extension id. This deletes GetExtensionSlowly(). Module hooks now take an args struct and receive the dispatched field as a TypedProtoField, read via generated field constants. The allowlist must stay disjoint from module-registered ids (DCHECKed). Wall time is neutral to ~+1% across the test traces, but extension dispatch no longer scales with the number of registered extension modules.

tp: add RemoteTraceProcessor and the `--remote` client (#6286) Add a RemoteTraceProcessor: a faithful, transport-only implementation of the TraceProcessor interface whose every method marshals to the corresponding TraceProcessorRpc message and talks to a `server unix` session.

tp: add support for multiple statements in query (#6794) Separate the queries by a single blank line. This has become necessary because AI these days really loves doing N queries simultaneously when gaining context.

tp: inline traceconv conversion into the shell subcommands (#6396) The `convert` subcommand used to rebuild an argv and hand it to TraceconvMain. Call the trace_to_text functions directly instead. While doing this, split the symbolization modes out of `convert`, which is now only about turning a trace into another artifact: - bundle becomes a top-level subcommand. - symbolize and deobfuscate move under a new `util` subcommand. - profile and java_heap_profile stay in convert. convert_helpers holds the stdin/stdout setup and the text-proto-to-binary path shared by convert and util. Output is unchanged from traceconv, except firefox and decompress_packets now exit 0 on success (they used to return the bool result directly, so success exited 1). TraceconvMain and the standalone binary are removed later in the stack.

ui: add slice duration histogram when single selecting (#5659) Adds a reusable distribution panel and shows a duration histogram for matching slices from the single-slice details panel. The inline details view defaults to the selected slice track, supports switching to whole-trace scope, and links out to a full matching-slices tab.

tp: add perfetto_manifest clock overrides (#6329) Allow overriding the clocks chosen by TP and their offsets and instead manually specify what the offset should be.

tp: add transport-neutral stack sampling protos and parsing (#6679) Add StackSample: a callstack captured for a thread, process, or cpu, measured against a primary counter timebase, emitted as TracePacket.stack_sample. Contexts and the primary descriptor can be inline or interned; interned context hangs off InternedData through the StackSampleInternedData extension. Parse into __intrinsic_stack_sample plus deduplicated task_context, execution_context, and timebase tables; each sample references its contexts by id. No tracks are minted. Frame in profile_common.proto gains a frame-kind field, parsed into a nullable stack_profile_frame.type column.

tp: extract IncrementalState to fix CustomState UAF (#5593) Fixes the use-after-free flagged in `PacketSequenceStateGeneration::CustomState`. ## The bug Previously, each `CustomState` held a raw `generation_` back-pointer that was re-pointed every time a new generation was created via `OnNewTracePacketDefaults` (the `set_generation(this)` loop in the multi-arg ctor). The `TraceSorter` can hold a `RefPtr` to an older Generation `G1` while a newer `G2` — to which `G1`'s shared CustomState had been re-pointed — is dropped after `SEQ_INCREMENTAL_STATE_CLEARED`. `G1` stays alive (and so does the CustomState), but its `generation_` then dangles to freed memory; any subsequent lookup through the CustomState dereferences the freed `G2`. ## The fix Refactor the generation/CustomState relationship so that lifetime is correct **by construction**: - New `IncrementalState : RefCounted`. Owns the per-incremental-state-interval data: `interned_data_`, the array of `CustomState`s (now `unique_ptr` rather than `RefPtr`), and the persistent thread descriptor. A new `IncrementalState` is constructed only on `SEQ_INCREMENTAL_STATE_CLEARED`. - `PacketSequenceStateGeneration` becomes a thin per-`trace_packet_defaults` snapshot: holds `RefPtr<IncrementalState>` (shared with sibling generations within the same interval), the defaults blob, and the validity flag. All interned-data / custom-state / thread-descriptor accessors are forwarders to the `IncrementalState`. - `CustomState` is no longer `RefCounted` (it's owned uniquely by its `IncrementalState`). Its back-pointer is `IncrementalState*` and is set exactly once at lazy-allocation time inside `IncrementalState::GetCustomState<T>`. Because the `IncrementalState` owns the `CustomState`, the pointer is stable for the entire life of the CustomState — UAF impossible. - `OnNewTracePacketDefaults` no longer copies the `InternedFieldMap` or re-points CustomStates; it constructs a new Generation that shares the same `RefPtr<IncrementalState>`. This removes a per-defaults-change copy that could be O(map-size) in the hot path. - `OnPacketLoss` walks the IncrementalState's CustomState array and clears any slot whose `ClearOnPacketLoss()` opted in (TES today), then returns a new Generation referencing the same `IncrementalState` with the validity bit cleared. --- **Stack:** - #5588 — tp: stop mutating PacketSequenceStateGeneration on packet loss - #5590 — tp: split TrackEventSequenceState into descriptor + delta state - **#5593 — tp: extract IncrementalState to fix CustomState UAF** (this PR)

tp: add minimal flatbuffer reader and writer (#6795) Trace processor needs to emit and parse small flatbuffer-encoded metadata (Arrow IPC framing) without taking a dependency on the flatbuffers library, which would have to be vendored for every embedder including the Wasm and Android builds. The writer builds buffers back-to-front so offsets are naturally forward-pointing. It tracks the maximum requested alignment and pads the final size to a multiple of it: element positions are aligned relative to the buffer end, so an aligned total size is what makes them aligned relative to the start, which is what strict flatbuffers verifiers check. String length prefixes are padded the same way. The reader is designed for untrusted input: every table, vtable, string and vector access is bounds-checked in 64-bit offset arithmetic and malformed or absent data uniformly reads back as defaults.

tp: import Firefox profiler markers as slices (#5686) Parse the `markers` table from preprocessed Gecko/Firefox profiler profiles and emit them as slices in trace_processor: - Pull `meta.categories` once at the profile level so marker `category` indices resolve to readable names. - Capture each per-thread `markers.{name,startTime,endTime,phase,category, data}` array; the `data` payload is captured as raw JSON via a new `Iterator::CollectCurrentScope` helper on the JSON parser and flattened into `data.*` args at parse time using `AddJsonValueToArgs`. - Phase 0 (Instant) becomes dur=0 slices; phase 1 (Interval) uses `endTime - startTime`; phase 2/3 (IntervalStart/End) are LIFO-matched per (utid, name) via a cookie stack. - Markers route through `TrackCompressor` with a blueprint dimensioned by `(utid, marker_name)` so each marker name on each thread gets its own track (matching the Firefox Profiler's marker chart layout) and same-name overlaps fan out across compressed lanes instead of being dropped. - Fix a pre-existing importer bug exposed while developing: when the top-level `shared` block carries only the string table (no shared frames), per-thread frames/stacks were silently dropped. Threading the strings through `Process{Legacy,Preprocessed}FramesAndStacks` fixes this. - Refactor `gecko_trace_tokenizer.cc` parsing into single-purpose helpers (`ParseFrameTable`, `ParseFuncTable`, `ParseStackTable`, `ParseSamplesTable`, `ParseMarkersTable`, `ParseMeta`, `ParseMetaCategories`, `ParseThreads`, `ParseMarkerDataArray`) so `ParseThread` and `ParseGeckoProfile` become a flat dispatch. - Add a dedicated `dev.perfetto.FirefoxProfilerMarkers` UI plugin (enabled by default) that lays out `firefox_marker` tracks under each thread, sub-grouped by category when a thread spans multiple categories. A single SQL query computes per-(utid, name) track groups with their dominant category and the depth needed for intra-name overlap. - Add diff tests covering all four phases, payload flattening, and same-name overlap fanning out across lanes. - Update other-formats.md to reflect markers being supported, and add generation instructions for samply and Python's `profiling.sampling --gecko` (3.15+).

ui: add multi-trace "at the same time" merge configurator (#6378) This CL introduces the first part of the final shape of the merge trace configuration tool in the UI. Right now we're focusing on exposing things who owns the global clock and alignment between them. Followup CLs will deal with multi-machine clocks and all the problems that come from that

tp: drive DebugAnnotation parsing on the proto-args work stack (#5587) ## Summary - Adds `ProtoToArgsParser::ParseDebugAnnotation` and `EnableDebugAnnotationParsing()`, building on the work-stack driver from #5589. - Pushes DebugAnnotation / NestedValue work items onto the same `work_stack_` as proto-message items (extends the variant); DebugAnnotation -> proto_value -> DebugAnnotation cycles are processed iteratively, so depth in the input does not grow the C++ stack. - `track_event_parser` opts in via `EnableDebugAnnotationParsing()`; the `.perfetto.protos.DebugAnnotation` type override is removed. - `DebugAnnotationParser` class is deleted; callers use `ProtoToArgsParser::ParseDebugAnnotation` directly. - Adds a regression test that parses a 1000-deep `DebugAnnotation -> proto_value(DebugAnnotation)` cycle. ## Stack - #5589 — prep refactor - **#5587 — security fix (this PR)** ## Test plan - [ ] `perfetto_unittests --gtest_filter="ProtoToArgsParserTest.*:DebugAnnotationParserTest.*"` passes (23 tests including new regression). - [ ] `tools/diff_test_trace_processor.py --name-filter="TrackEvent.*|DebugAnnotation.*"` passes.