sashwinbalaji

perfetto: add Zstd as a trace compression codec (#6459) The tracing service could already compress its output with deflate. This adds Zstd, which produces smaller traces at a similar speed. - Traces opt in with the new TraceConfig.compression field: set compression.zstd for Zstd or compression.deflate for deflate. If both are set, the higher field number wins (so zstd over deflate). - Zstd's level is configurable via compression.zstd.level (1 fastest to 22 smallest; 0 or unset uses zstd's default). - The old compression_type field is now deprecated but still honored, so services that predate `compression` keep compressing. - The service picks the codec from the config and runs it directly. A codec's library is only linked when its build flag (PERFETTO_ZLIB / PERFETTO_ZSTD) is on, so libz/libzstd stay out of builds that don't use compression, such as the SDK. - Each codec is configured by its own proto message, so the service never needs to know a codec's specific options. - Adding a codec later (e.g. lz4) is a new compressor file, one branch in the service, and a proto sub-message - nothing else changes. Bug: 529304320

tp: add zstd decompression support (#6461) This adds zstd across the decode paths so it can open the traces the tracing service now compresses with zstd. - Introduce a codec-agnostic StreamDecompressor with gzip and zstd implementations. The old gzip-only GzipTraceParser becomes a DecompressingTraceReader that works with any codec. - Whole-file compressed traces are detected by magic and streamed through the matching decompressor. - In-trace compressed bundles are decoded by the field that carries them: compressed_packets for deflate, zstd_compressed_packets for zstd. - zstd is only linked when PERFETTO_ZSTD is enabled, matching how the service gates the codec. Bug: 245929954 Bug: 529304320

tp: move heap graph, network and bluetooth traces out of tree (#6208) Move three more Android-owned TracePacket payloads out of the perfetto repo. * ART heap graph: TracePacket field 56 becomes an extension hole. The schema moves to protos/third_party/android/art/heap_graph.proto (package com.android.art.tracing, wrapper ArtHeapGraphTracePacket). The shared profiling protos it references (profile_common, deobfuscation) stay in perfetto. * Connectivity network trace: TracePacket fields 88/92 and InternedData field 30 become extension holes. The schema moves to protos/third_party/android/connectivity/network_trace.proto (package android.net.connectivity.tracing, wrappers ConnectivityTracePacket and ConnectivityInternedData). * Bluetooth trace events: TracePacket field 114 becomes an extension hole. The schema moves to protos/third_party/android/packages/ modules/bluetooth/tracing/bluetooth_trace.proto (package bluetooth.tracing, wrapper BluetoothTracePacket). The importers (heap_graph_module, heap_graph_tracker, network_trace_ module, android_probes_module/parser) read the fields as extensions via GetExtensionSlowly, the extension holes are registered in protos/perfetto/trace/extensions.json, the new protos are wired into the android_extensions descriptor, and the diff tests' textprotos are updated to extension bracket syntax.

descriptors: Match by type and fields rather than name (#5948)

perfetto: attribute TraceBufferV2 data loss by cause (#6082) When investigating data loss in the field, previous_packet_dropped only told us *that* packets were dropped, never *why*. This widens the field from a bool to a uint32 cause bitmask so a reported drop also carries its cause. Bit 0 (value 1) is the generic "dropped" flag and preserves the old bool semantics: any nonzero value means dropped, and it stays what TraceBufferV1 and producer-signalled drops emit. TraceBufferV2 additionally sets a bit identifying the cause: read gap, chunk corrupted, orphan continuation, reassembly gap, reassembly broken chain, overwrite, or trace-writer drop.

perfetto: add proto2 extension support to protozero and trace_processor (#6059) Adds the shared infrastructure needed to define and decode out-of-tree (OOT) proto extensions, independent of any specific extension: - protozero TypedProtoDecoderBase::GetExtensionSlowly() for resolving extension fields whose ids exceed the highest in-tree field id. - DescriptorPool / FieldDescriptor support for extension full names (ExtensionInfo carries the enclosing scope), with matching protozero_to_text / protozero_to_json formatting. - text_to_proto extension support and the tracing_proto_extensions tool, including a --base-descriptor flag so leaf extension protos can be compiled against a prebuilt FileDescriptorSet instead of the full source closure. - Refactor existing track_event and winscope extension decoding to use GetExtensionSlowly (behaviour-preserving for in-range fields).

docs: replace docs-site search with a local full-text index (#6565) The search box on perfetto.dev was backed by a Google Custom Search Engine, which only surfaced pages Google had already crawled and indexed. Newly added docs stayed invisible for weeks. Replace it with a self-contained client-side search: - gen_search_index.js builds assets/search_index.json at build time: full text for hand-written docs, and title + headings only for the large auto-generated reference pages (SQL tables, stdlib, protos) to keep the shipped index small (~370KB gzipped). - script.js loads the index on first use and ranks it with BM25 (title/heading/body field boosts, prefix matching for as-you-type, an exact-title boost, and jump-to-heading deep links). Result markup is built from DOM text nodes, so index content cannot inject HTML. | Keyword | Top result | Also in top 3 | |---|---|---| | data explorer | Data Explorer | Data Explorer Architecture; Heap Dump Explorer | | trace processor | Trace Processor (Python) | Trace Processor Architecture; Trace Processor (C++) | | perfetto ui | Perfetto UI | Perfetto UI Release Process; Perfetto UI Review Antipatterns | | tracing sdk | Tracing SDK | Track events (Tracing SDK); Trace packet interceptors (Tracing SDK) | | system tracing | Recording system traces with Perfetto | Advanced System Tracing on Android; Security model for system-wide tracing on Android/Linux | | heap profiling | Profiling memory usage and allocations with Perfetto | Memory: Callstack-based Allocation Profiling; heapprofd: Android Heap Profiler | | ftrace | Instrumenting the Linux kernel with ftrace | TraceConfig; Common tasks | | perfettosql | PerfettoSQL Syntax | PerfettoSQL Prelude; PerfettoSQL Style Guide | | trace config | Trace configuration | Advanced System Tracing on Android; PERFETTO(1) | | traced | TRACED(8) | TRACEDPROBES(8); Multi-machine recording | <img width="1975" height="537" alt="image" src="https://github.com/user-attachments/assets/0c2fe63c-0f8f-4115-a475-b71ddaccce0a" /> Fixes: #6513 Bug: 530934167

perfetto: unify trace attributes across config, packets and manifest (#6651) Trace attributes (key/value pairs describing a trace) could only be written as TraceAttributes packets. Unify the other ways of attaching them on the same message and query surface: * Replace TraceConfig.notes and `perfetto --add-note` with TraceConfig.trace_attributes and `--add-attribute`, embedding the TraceAttributes message in the config. Notes were never parsed by trace_processor so nothing can depend on them; field 46 is reserved. * Ingest config attributes through the same path as TraceAttributes packets; both become trace_attribute.<key> rows of the metadata table. * Let perfetto_manifest files annotate an archive (top-level or per-file) with an `attributes` section, stored as manifest_attribute.<key> rows: a separate namespace because they annotate the archive, not the recorded trace. * Make SetDynamicMetadata update duplicate keys in place so "last value wins" actually holds. Chrome importers keep their historical insert-always behavior via a new AppendDynamicMetadata. * Show both namespaces as sections on the UI's trace info page; update docs; add unit and diff tests for all paths. Fixes: https://github.com/google/perfetto/issues/3495

perfetto: prepare for OOT TrackEvent extensions (#5807) We're moving Android's TrackEvent extension protos out of this repo and into the Android tree (frameworks/base and frameworks/native) so Android can ship schema changes on its own cadence without a Perfetto release. See [discussions/4783](https://github.com/google/perfetto/discussions/4783). This PR is just the github-side scaffolding: - New proto files under protos/third_party/android/frameworks/{base,native}/.... Mirror copies; source of truth moves to the Android tree later. - track_event_extensions.json: split the android_system field-number range into android_frameworks_base and android_frameworks_native, preserving the historical 2000–2013 numbers. - Expose protozero_c_plugin as a Soong cc_binary_host so a later Android-tree genrule can generate .pzc.h headers at build time. Needed a small plugin option (multi-root imports) and one BUILD.gn guard widening so gen_android_bp can find the host target. Bug: 512378033

perfetto: list other tracing sessions in trace (#6627) Optionally record the lifecycle state changes of *other* tracing sessions active while a trace is being recorded, to help diagnose field issues caused by unexpected concurrent sessions. - Add the ConcurrentSessionEvent packet: state (mirrors the service's session state machine), unique_session_name, service-local session id (deliberately not the trace UUID), consumer uid and number of data sources. - Route every session state change through a single SetSessionState() helper. It broadcasts the change into the other opted-in sessions that are still recording (i.e. CONFIGURED, STARTED or DISABLING_WAITING_STOP_ACKS; read-only clones and already-disabled sessions are skipped), appending to a bounded per-session buffer that is drained on ReadBuffers(). - On creation, a session snapshots the current state of the other active sessions, so its trace records what was already running when it started. Each event is timestamped with when that session entered its current state. - Cloned sessions inherit the source's events and record a terminal DISABLED on teardown. - Opt-in via TraceConfig.BuiltinDataSource.enable_concurrent_session_events. - Add service unittests.

tp: keep parsing traces after wire-compatible schema changes (#6553) When a trace embeds its own proto descriptor, trace_processor merges it with the built-in schema. Two recent schema changes made that merge fail on older traces, even though the trace bytes were still readable. This tolerates both. - Widened field type (PR #6082): previous_packet_dropped went from bool to uint32. Both encode as a varint, so the bytes still decode. Accept a re-declared scalar when its wire type is unchanged; only reject a real wire-type change. - Removed field type (PR #6308): AndroidCameraFrameEvent and AndroidCameraSessionStats were deleted. Old traces still name them, so resolution can't find them. Skip such a field instead of failing the whole trace, but only for an allowlist of known-removed types so a genuine missing type is still an error. Both are temporary; they can go once traces predating these changes age out (b/524094370).

perfetto: move app_wakelock trace packets out-of-tree (#6060) Moves AppWakelockBundle and AppWakelockInfo out of the upstream trace schema into the Android platform tree (package com.android.internal) as frameworks/base out-of-tree extensions: - TracePacket.app_wakelock_bundle (116) and InternedData.app_wakelock_info (42) become `extensions`; the schemas move to protos/third_party/android/.../frameworks_base_trace_packet.proto and frameworks_base_interned_data.proto, registered in extensions.json. - app_wakelock_module, traceconv, protoprofile and additional_modules load the android_extension descriptor; the public C SDK drops the fields.

tbV2: relocate fully-read scraped chunks instead of rewriting them (#6767) Fixes a data-loss race seen in 24h field traces with sporadic writers. Fingerprint: orphan_continuation = overwrite = sequence_packet_loss = 1 on the buffer, one packet with previous_packet_dropped = 73. The race: - A flush scrapes a writer's half-full chunk; the incomplete copy is pinned in the buffer and read over time. - Hours later the writer commits the chunk. The recommit rewrote it in place, at the now-stale offset, with the write cursor almost a lap ahead: the never-read fragments could be overwritten before the next read pass. - If the chunk ends with kFragBegin, that eviction destroys the begin fragment and its continuation later reads back as an orphan. The fix: - If the scraped copy is fully consumed and the final commit brings new fragments, erase the stale copy and relocate the chunk to the write cursor, reusing the re-admit path for evicted chunks (#5372). - previously_consumed_payload skips what was already read, so nothing is duplicated; ChunkIDs stay consecutive, so no read gap. - Rescrapes, partially-consumed and out-of-order recommits keep the in-place rewrite, as do kDiscard buffers (they never lap, and the write path could hit DiscardWrite() and drop the fragments being recovered). New chunks_relocated stat (BufferStats field 20) counts these, mirroring chunks_rewritten. Field 20 was never assigned in BufferStats: the CL below added bytes_discarded_per_buffer = 20 to FilterStats but bumped the "Next id" above BufferStats, leaving a phantom hole. https://r.android.com/2882818 Four new tests cover the race itself, the rescrape / unread-payload / kDiscard exclusions, and the case where the continuation chunk is already buffered. Bug: 518755701.

Partial Revert tp: remove AndroidCameraFrameEvent and AndroidCameraSessionStats (#6322) Partial Revert: #6308 Re-add the protos as there are still some references to them internally. This reverts commit d78698066cc743b09109bf41a0045eae2f1ed4ae.

tp: Refuse to write binary trace output to a TTY (#6462) The traceconv/util conversion commands (decompress_packets, text_to_binary, symbolize, deobfuscate) emit raw binary protobuf. When no output path is given they write to stdout; if stdout is a terminal this dumps binary bytes (including control/escape sequences) straight to the TTY and corrupts it. The input side already guarded against this (it refuses to read from a TTY), but the output side had no equivalent check. Add the mirror-image guard so that when stdout is a TTY we error out with a clear message telling the user to pass an output file or redirect stdout. To avoid duplicating the platform-specific TTY detection (which several call sites open-coded via isatty()/STDIN_FILENO with per-file Windows ifdefs), introduce a shared base::IsTty(FILE*) helper that wraps the isatty()/_isatty() discrepancy in one place.

tp+ui: show concurrent tracing sessions as state tracks (#6647) Turn the ConcurrentSessionEvent packets emitted by traced into one state track per concurrent session. - Add ConcurrentSessionsModule: tracks are keyed by session id and named by unique_session_name ("Session <id>" when unnamed, a " (clone)" suffix for cloned sessions). STATE_DISABLED closes the track instead of drawing an open-ended state. consumer_uid and num_data_sources become args; unknown states are recorded in the trace import logs. - UI: register the tracks as state tracks, grouped under System > Concurrent tracing sessions. - Add TP diff tests.

proto_merger: keep oneof fields and enum values deleted upstream (#6189) When a field still exists in our local proto but was removed upstream, proto_merger is supposed to keep it under a "not present upstream" comment. That worked for plain message fields, but fields inside a oneof and enum values were dropped from the output: the merge step tracked them as deleted, the serializer just never wrote them back. - Write back deleted oneof fields and enum values, under the same comment used for the other deleted items. - Split the sources into a lib target so tests can link against them. - Add tests for the serializer and a merge round trip, hooked into perfetto_unittests.

perfetto: convert android_track_event.proto to shim (#5835) Schema for Android TrackEvent extensions moves out of protos/perfetto/trace/android/ and into protos/third_party/android/ under package com.android.internal. What moved where: - protos/perfetto/trace/android/android_track_event.proto → deleted. The AndroidTrackEvent container is gone too — its 16 extensions split into FrameworksBaseTrackEvent (fields 2004-2006, 2008-2016) and AndroidFrameworksNativeTrackEvent (2001-2003, 2007). - kAndroidTrackEventDescriptor (trace_processor's compiled-in descriptor) is replaced by two: kFrameworksBaseTrackEventDescriptor and kFrameworksNativeTrackEventDescriptor, each generated from its own .proto. Loaded side-by-side in track_event_module.cc. - include/perfetto/public/protos/trace/android/android_track_event.pzc.h → deleted. Consumers point at the new third_party paths directly. - The new third_party .proto files are not added to tools/gen_c_protos. The Android tree generates them at build time via the Soong genrule that calls protozero_c_plugin. - Force Protobuf Lite libraries to compile against public SDK stubs ( sdk_version: "current" ). This decouples them from private platform internals and breaks cyclic dependency blocks during Soong bootstrapping. Companion Android-tree CLs: - system/extras/tracing/ — installs /etc/tracing_descriptors.gz + holds a Java shim so unmigrated services keep building. - frameworks/base/proto/tracing/ and frameworks/native/tracing/ — get filled with the real proto content. Also fix protozero_c_plugin.cc: include-emission loop check order. Bug: 512378033

perfetto: unify OOT extension registries and reserve TracePacket range (#6039) Prep work for moving TracePacket and InternedData fields out-of-tree as proto2 extensions. This PR doesn't move any fields yet, it just sets up the shared infrastructure so each subsystem can land as its own PR later. What changes: 1. Folds track_event_extensions.json and interned_data_extensions.json into a single protos/perfetto/trace/extensions.json with three sections (TrackEvent, TracePacket, InternedData). TrackEvent content is copied verbatim; InternedData keeps perfetto_gpu and drops the never-used perfetto_tests reservation. 2. Teaches tools/tracing_proto_extensions to accept TracePacket and InternedData scopes (it already understood TrackEvent). Same validator, just more allowed values. 3. Reserves TracePacket tag range 1000-1999 for future OOT allocations, same pattern TrackEvent already uses for [1000, 10000].

perfetto: add CTS coverage for OOT tracing descriptors (#6556) On Android, traced loads out-of-tree proto extension descriptors from /etc/tracing_descriptors.gz and /vendor/etc/tracing_descriptors.gz (see RFC-0017) and emits them into every trace as TracePacket.extension_descriptor packets. Add two CTS tests that record a trace against the on-device service: - TestSystemExtensionDescriptor asserts the mandatory system descriptor is present, gzip-decodes into a non-empty FileDescriptorSet, and still declares the wire-locked payload fields that moved out-of-tree (heap_graph, protolog_message, app_wakelock_bundle, process_start_event, windowmanager), spanning the TracePacket, TrackEvent and WinscopeExtensions extension scopes. - TestExtensionDescriptorsWellFormed checks every emitted descriptor comes from a known path and decodes cleanly. The vendor descriptor is validated only when present, as it is not guaranteed on every device. Wire the new source into the hand-maintained test/cts/BUILD.gn and Android.bp, pulling in the gzip util and libz. Bug: 514799814 Test: ab/I10600010583161847