Lalit Maganti

tp: overhaul error handling and diagnostics for the shell and bundle (#6969) Issue: https://github.com/google/perfetto/issues/6927 `trace_processor bundle` (formerly `traceconv bundle`) gave cryptic or misleading feedback when things went wrong: - Traces with nothing to enrich (e.g. a pure atrace trace, or a function_graph trace recorded **with** `symbolize_ksyms`) failed with a bare `bundle: failed to create bundle.` and no explanation. - A function_graph trace recorded **without** `symbolize_ksyms` got no feedback at all, even though its kernel addresses can never be symbolized offline. - Unwritable output paths (missing parent dir, read-only, path-is-a-dir) crashed the whole process with a `PERFETTO_CHECK`. - Extra positional arguments were silently ignored/misparsed (e.g. passing each `--symbol-paths` directory as a separate argument). - Conversion failures produced double error messages (an internal `PERFETTO_ELOG` + a generic "conversion failed" wrapper), and several paths crashed with `PERFETTO_FATAL` on ordinary user error. ## Changes - **bundle** now always produces the bundle and explains what could not be enriched, with actionable hints. It detects function_graph traces recorded without `symbolize_ksyms`, fails gracefully (instead of crashing) on unwritable output paths, and reports reading/enrichment progress. - **CLI**: every fixed-arity subcommand (`bundle`, `convert`, `util`, `export`, `query`, `server`, `metrics`) now rejects extra positional arguments with a hint, instead of silently misinterpreting them. - **traceconv**: the conversion entry points (`TraceToJson`, `TraceToSystrace`, `TraceToText`, `TraceToProfile`, `TraceToFirefoxProfile`, `UnpackCompressedPackets`, `SymbolizeProfile`, `DeobfuscateProfile`) now return descriptive `base::Status` errors instead of int/bool returns paired with internal `PERFETTO_ELOG`/`FATAL`, so failures are reported once with the actual reason. - **TarWriter**: fails gracefully when the output path cannot be opened (also fixes the same crash in `util merge`). - **Parse errors**: no longer append a redundant "The trace file is corrupt." sentence; the UI error-dialog extraction is updated for the new message format.

tp: model trace types as importers, remove the TraceType enum (#6618) Today, all trace detection logic is centralized inside TP meaning it's impossible for plugins to register new formats and have them work without also changing the core code. As the number of formats has increased, this has gotten more and more unweildy Design it to instead be structured such that new trace types can be developed fully independently *without* making any changes to the core. This will be proved out in a followup change where perf text importer will be moved.

tp: add Arrow support for dataframes (#6827) Provide an Arrow serializer and deserializer for Dataframes using a standard Arrow file containing one record batch. Primitive columns map to Arrow numeric arrays, strings map to Utf8, nullable columns carry validity bitmaps, and sparse storage uses Arrow's logical row layout. The serializer computes metadata and variable-width sizes before streaming column contents with bounded scratch memory. The deserializer validates file framing, record batch metadata, and buffer extents before rebuilding column storage, null state, implicit Id columns, and strings in the target StringPool.

tp: add streaming table export (#6839) Add a bounded-memory Export API for statically registered Trace Processor tables. ExportOutput is a virtual streaming interface with an optional file path alternative for formats which need direct random-access output later. Runtime SQL tables and views are deliberately outside this API. Support two explicit tar formats. kArrowTar is a stable, forwards-compatible, export-only archive containing standard Arrow files, including implicit ID columns and empty static tables for external consumers. kPerfetto is a version-coupled archive with an internal manifest which can restore tables into a compatible fresh Trace Processor instance; its representation carries no cross-version compatibility guarantee. Expose export through RPC, HTTP, the shell export subcommand, and Python. Keep the existing SQLite export path unchanged; moving it behind this API waits for the separate file-I/O abstraction work.

tp: memoize proto arg keys to avoid re-interning per field (#6348) ProtoToArgsParser rebuilt and re-interned the flat_key/key strings of every arg on every field of every message. The flat_key for a given proto field path is invariant, so give the parser a StringPool and a memo trie keyed by field number. The Delegate API now takes already-interned StringIds instead of Key strings, so delegates are pure value sinks and never re-intern keys. A new proto_to_args_parser_benchmark isolates the key path and shows ~1.7-2x; end-to-end trace load gains are ~1-2% as key interning is a small slice of ingestion.

tp: remove FindById and replace with operator[] (#5906) Now that ids are guaranteed to appear in tables, there's no point having std::optional on the API boundary. Just make it always return the RowReference instead.

tp: add C++ flamegraph computation library (#6851) A flamegraph is the trie of key-paths: every path through the input forest with the same sequence of merge keys collapses into one merged node. The library computes that trie and returns it as a tree plus a flat dataframe (id, parentId, depth, name and per-metric value columns). Presentation (sibling ordering, x layout) is deliberately absent: it is cheap to compute over the far smaller merged output and belongs to the consumer, which can also skip it entirely. The three views are one algorithm parameterized by anchors, the frames the merged trees grow from and re-root at: top-down anchors at the kept roots, bottom-up at every counted frame, pivot at frames matching the pivot pattern. Downward, one forward scan drives a cursor through an exact (parent node, key) hash map; upward, each anchor's caller chain is walked carrying the anchor's weight: its counted subtree total stopping at other anchors, which for bottom-up reduces to the frame's own values. Nodes whose subtree is zero on every metric are dropped, which is also what gives show-stack filters their effect on the output. Filters (show/hide-stack, show-from-frame, hide-frame and the pivot pattern) are regexes evaluated once per distinct frame name through the input's name dictionary, then folded into per-frame kept/counted flags and a nearest-kept-ancestor index in a single pass along the paths. Hidden frames fold their values into the nearest kept ancestor. Frames carry any number of metrics, flattened row-major with a constant stride; every accumulation is metric-wide. Each output node retains its constituent input frames, so consumers can aggregate arbitrary per-frame properties without the library knowing about them (the benchmark exercises a ONE_OR_SUMMARY aggregation this way). Input frames may arrive in any order: parents-before-children is detected and used directly, anything else costs one reverse-index and DFS pass. FlexVector gains reserve() for the pre-sized node columns. On 1M-frame callstack-shaped forests (see flamegraph_benchmark.cc), top-down computes in ~90ms, ~120ms including the dataframe; with filters ~77ms, bottom-up ~200ms, pivot ~18ms. At 100k frames every view is below 10ms.

tp: rebuild flamegraphs from typed trees (#6942) Consume the core Tree representation directly and preserve typed values while merging top-down, bottom-up, pivot, and filtered flamegraph views. _{Stack created with <a href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a href="https://gh.io/stacks-feedback">Give Feedback 💬</a>}

tp: replace GetExtensionSlowly with SelectiveTracePacketDecoder (#6218) Introduce SelectiveTracePacketDecoder, a wrapper around protozero's SelectiveTypedProtoDecoder. An allowlist mask covers the TracePacket metadata fields the pipeline reads by name; the data field and out-of-tree extensions fall into unknown_fields(), which drives module dispatch in one pass per packet instead of one buffer re-scan per registered extension id. This deletes GetExtensionSlowly(). Module hooks now take an args struct and receive the dispatched field as a TypedProtoField, read via generated field constants. The allowlist must stay disjoint from module-registered ids (DCHECKed). Wall time is neutral to ~+1% across the test traces, but extension dispatch no longer scales with the number of registered extension modules.

tp: add opt-in abstract filesystem access (#6894) Add minimal synchronous File and FileSystem interfaces for Trace Processor features which write named files. A serializable config knob gates SQL access, while a borrowed PlatformInterface supplies the filesystem capability. Let shell embedders provide the implementation and gate SQL file access behind --allow-sql-file-access, with an equivalent Python option. Route EXPORT_JSON and __intrinsic_file_write through the interface and remove the legacy file-descriptor form of EXPORT_JSON.

tp: add RemoteTraceProcessor and the `--remote` client (#6286) Add a RemoteTraceProcessor: a faithful, transport-only implementation of the TraceProcessor interface whose every method marshals to the corresponding TraceProcessorRpc message and talks to a `server unix` session.

tp: add support for multiple statements in query (#6794) Separate the queries by a single blank line. This has become necessary because AI these days really loves doing N queries simultaneously when gaining context.

tp: add SQLite VFS backed by abstract I/O (#6997) Add a random-access FileSystem interface and a SQLite VFS backed by it. Use process-wide local and no-op filesystem implementations, with reusable seek and truncate utilities.

tp: inline traceconv conversion into the shell subcommands (#6396) The `convert` subcommand used to rebuild an argv and hand it to TraceconvMain. Call the trace_to_text functions directly instead. While doing this, split the symbolization modes out of `convert`, which is now only about turning a trace into another artifact: - bundle becomes a top-level subcommand. - symbolize and deobfuscate move under a new `util` subcommand. - profile and java_heap_profile stay in convert. convert_helpers holds the stdin/stdout setup and the text-proto-to-binary path shared by convert and util. Output is unchanged from traceconv, except firefox and decompress_packets now exit 0 on success (they used to return the bool result directly, so success exited 1). TraceconvMain and the standalone binary are removed later in the stack.

tp: add perfetto_manifest clock overrides (#6329) Allow overriding the clocks chosen by TP and their offsets and instead manually specify what the offset should be.

tp: add transport-neutral stack sampling protos and parsing (#6679) Add StackSample: a callstack captured for a thread, process, or cpu, measured against a primary counter timebase, emitted as TracePacket.stack_sample. Contexts and the primary descriptor can be inline or interned; interned context hangs off InternedData through the StackSampleInternedData extension. Parse into __intrinsic_stack_sample plus deduplicated task_context, execution_context, and timebase tables; each sample references its contexts by id. No tracks are minted. Frame in profile_common.proto gains a frame-kind field, parsed into a nullable stack_profile_frame.type column.

tp: add minimal flatbuffer reader and writer (#6795) Trace processor needs to emit and parse small flatbuffer-encoded metadata (Arrow IPC framing) without taking a dependency on the flatbuffers library, which would have to be vendored for every embedder including the Wasm and Android builds. The writer builds buffers back-to-front so offsets are naturally forward-pointing. It tracks the maximum requested alignment and pads the final size to a multiple of it: element positions are aligned relative to the buffer end, so an aligned total size is what makes them aligned relative to the start, which is what strict flatbuffers verifiers check. String length prefixes are padded the same way. The reader is designed for untrusted input: every table, vtable, string and vector access is bounds-checked in 64-bit offset arithmetic and malformed or absent data uniformly reads back as defaults.

ui: add multi-trace "at the same time" merge configurator (#6378) This CL introduces the first part of the final shape of the merge trace configuration tool in the UI. Right now we're focusing on exposing things who owns the global clock and alignment between them. Followup CLs will deal with multi-machine clocks and all the problems that come from that

ui: complete multi-trace merging dialog (#6422) This CL completes support for all the complexity of trace merging both across clocks and across machines. This now makes it possible to open all sorts of combinations of traces in the UI.

tp: introduce the generic profiler_sample table (#6809) Callstack samples were spread across four parallel tables (perf_sample, cpu_profile_stack_sample, instruments_sample and the plugin-owned stack_sample family), each with its own flavour of the same schema and, in the StackSample case, a parallel reinvention of counters. This is the first step of collapsing them onto one model. Introduce __intrinsic_profiler_sample: one row per sample from any profiler source, with flat context columns (utid, upid, async context, ucpu, cpu_mode), an optional callstack, a session and an optional counter set. The perf mechanisms become the generic ones: - perf_session generalizes to profiler_session, one row per sampling data source instance of any source. Each session records its source and the unit of the timebase counter it sampled on, so the per-source views stay well formed (perf_session only exposes linux perf sessions) and consumers can tell time/cycle/instruction sampling apart from other sampling types. StackSample streams allocate one session per packet sequence. - perf_counter_set generalizes to profiler_counter_set, linking each sample to the counter rows (timebase and followers) recorded at its sample point. The linux perf importers (traced_perf proto and perf.data) write profiler_sample directly; the perf_sample and perf_session views keep their exact schemas, and the perf counter tracks, counter sets, stdlib functions and the perf_counter plugin keep working unchanged. The StackSample packet's counter descriptors now materialize as counter tracks instead of a bespoke descriptor table. A counter track models a counter instance, so the proto gains an explicit scope on CounterDescriptor: SCOPE_GLOBAL (a single instance, one track per sampling stream) and SCOPE_CPU (one instance per cpu, one track per stream and cpu). Other scopes are added on demand via feature requests. Per-sample values are pushed as counter rows and linked via the counter set, exactly like perf. A new stack_sample view exposes every sample which captured a callstack, across all sources; per-source views (perf_sample today, the cpu_profile and instruments tables in follow-ups) select their slice of the same table. Samples which did not capture a callstack (e.g. counter-only perf samples) stay visible through the per-source views.