sashwinbalaji

tracing: add the v2 shared-ring chunk protocol (#7324) Add the shared-memory ring that tracing v2 producers will write packet fragments into, with a writer and a reader class on top of it: - One 64-bit atomic holds both the write and the read position. Writers reserve positions with a lock-free CAS on it; the reader publishes read_pos once per drain pass. - Every chunk starts with a 32-bit state word. The five states (Free, BeingWritten, Complete, RewriteRequested, RewriteAcknowledged) change hands only through exact-value CAS, so whoever loses a race sees the word that won. - A Free word carries a 16-bit wrap count, so a delayed writer cannot claim a chunk for a position the reader already resolved. Only the reader ever writes Free. - Writers append fragments to their chunk and publish by moving the state word. The reader may take the committed prefix of a chunk that is still being written; only the just-ended fragment can require relocation into a fresh chunk. - Finishing a writer releases its cached chunk, so later fragments require a new reservation. This establishes the flush watermark boundary. - Ring exhaustion follows BufferExhaustedPolicy: kDrop, kStall and kStallThenDrop. On Linux and Android a stalled writer waits on read_pos with a futex, under the same 30-second ceiling as v1. - Tests cover the ABI encoding, reader/writer race outcomes, position and wrap-count rollover, and multi-writer stress against a concurrent reader.

perfetto: support proto-group fragments in trace buffer v2 (#7565) Accept ordered batches of proto-group fragments and rewrite reassembled packets to length-delimited protobuf during readback. Record gaps before the next appended chunk so earlier buffered packets do not inherit the loss. Preserve loss across rejection, overwrite and cloning, with bounded state for inactive sequences. Reuse the existing buffer-space management and preserve legacy patch and recommit behavior. Reject unsupported flags, mixed input formats within a sequence and proto-group writes to ProtoVM destinations.

perfetto: add a trace writer and arbiter for the v2 ring buffer (#7567) Add the producer side that writes trace packets into the tracing v2 shared ring buffer. No production code creates these writers yet. The producer endpoints use them in a later change. Protocol: RFC 0046, https://github.com/google/perfetto/discussions/7120. TraceWriterV2Impl: - Implements TraceWriter on top of SharedRingBufferWriter. - Encodes nested messages as proto groups, so it never patches bytes that it already published. The service converts each packet back to length-delimited protobuf with ProtoGroupRewriter. - Decides when to ask for a drain: after a publication when ring buffer occupancy reaches 25%, after failed claims when it drops data, and before each wait for space. - Applies BufferExhaustedPolicy. Each wait for space requests at most 100 ms, then the writer tries again. Each chunk acquisition has a 30 s deadline, close to v1. After it, kStall aborts and kStallThenDrop drops. Without an attached reader, a full ring buffer drops data, also under kStall. - Sends the rest of a dropped packet to a per-writer drop buffer. The next publication sets kFlagDataLoss. - Flush(callback) does not wait for a service ack. The callback runs on the endpoint thread after the drain request. ProducerRingBufferArbiter, the producer side of one ring buffer: - Shares ownership of the mapping (shared_ptr<SharedMemory>), owns the SharedRingBuffer view and creates writers. - Tracks the service reader: kPending, kAttached, kDetached. After kDetached, new writers are NullTraceWriters. - RequestDrain(kCoalesced) merges the requests of all writers into one DrainV2RingBuffer() task on the endpoint thread. RequestDrain(kBeforeWait) sends at once on the endpoint thread, because a task posted there cannot run while the writer waits. - Flush() posts its own drain task, then the callback. SharedRingBufferWriter: - Accesses only the ring buffer. This change removes its Delegate and its BufferExhaustedPolicy argument. - A call that needs a chunk makes at most num_chunks attempts, then returns kFull or kClaimFailed (was kNoChunkAvailable) at once. - Only WaitForReadPosChange() blocks: on a futex until read_pos changes, or for one step of v1's sleep backoff without a futex. - A relocation that gets no replacement chunk stays pending until RetryRelocation() or DropRelocation(). Before, the writer dropped it (kRelocationDropped). - has_pending_data_loss() lets kStallThenDrop act as kDrop until a chunk carries the loss, as v1's drop_packets_ does. Ring buffer and ABI: - kMaxChunkSize is 64 KiB, so a chunk fits one TBChunk. - NumChunksForRingBufferLayout() moves to shared_ring_buffer_abi.h and returns StatusOr<uint32_t> with the reason. - SharedRingBuffer::LoadNumOutstandingPositionsRelaxed(), which TraceWriterV2Impl compares with its drain threshold. - New GN targets: v2:abi (the layout only) and v2:test_support. API changes: - ProducerEndpoint::DrainV2RingBuffer(), the v2 equivalent of CommitData(). The default does nothing. - SharedMemoryArbiter::AllocateTracingV2WriterID() and ReleaseTracingV2WriterID(). v1 and v2 writers of one producer share one ID pool, because the service keys sequences by (producer, WriterID).

perfetto: Make temporary file and shared memory creation fallible (#7869) Add fallible TempFile factories and an explicit sealed-memfd requirement for POSIX shared memory. Update callers to handle failures and use checked helpers in tests.

perfetto: attach and drain producer ring buffers in the service (#7568) perfetto: attach and drain producer ring buffers in the service Connect the service to the tracing v2 ring buffer that a producer allocates. The producer and the service agree on a set of protocol versions when the producer connects. With v2 in the set, the producer attaches its sealed memfd ring buffer, and the service drains it into the TraceBufferV2 buffers that the producer may write to. Tracing v2 is experimental. Its ABI and the new RPCs, introduced in v59, may still change. Design: RFC-0014, https://github.com/google/perfetto/discussions/4508. Ring buffer ABI: RFC-0046, https://github.com/google/perfetto/discussions/7120. Protocol versions: - New enum ProtocolAbiVersion, in tracing_service.h and producer_port.proto. The values are one-hot bits (kProtocolAbiV1 = 1, kProtocolAbiV2 = 2), so a uint32_t holds a set of versions. - Each version is an independent transport: v2 does not imply v1. The comments link the v1 shared memory docs and the v2 RFCs. - The producer sends the supported_protocol_abi_versions mask in InitializeConnection: V1, or V1|V2 when IPC supports v2. - The service replies with the mask of common versions. Unknown bits drop out. A missing or zero offer, from an older producer, means v1. With no common version, the service rejects InitializeConnection. - The producer disconnects if the reply has a version it did not offer, and logs both masks. A missing or zero reply, from an older service, means v1. - The service offers v2 only if tracing_v2::IpcSupportsTracingV2() and shmem emulation is off. - Each side keeps the common mask for the whole connection. ConnectProducer() takes it as a trailing uint32_t, v1 only by default, so existing callers do not change. - Ring buffer calls require v2 in the mask. The IPC client also requires a connection, like its other methods. Platform support: - tracing_v2::IpcSupportsTracingV2() and PERFETTO_TRACING_V2_IPC() are in src/tracing/v2/shared_ring_buffer_abi.h. IPC code checks them, not memfd directly. - PERFETTO_TRACING_V2_IPC() is the build-time check. It is PERFETTO_MEMFD_ENABLED(), which moves to memfd.h. It guards the PosixSharedMemory code, which Windows does not build. - IpcSupportsTracingV2() is the runtime check. It returns HasMemfdSupport(), which is false on builds without memfd. - v2:abi depends on ipc:common only in IPC builds. The in-process service also uses v2:abi on builds without IPC. Attach and drain: - New RPC AttachV2RingBuffer: the producer sends the chunk size and the memfd. The service takes the ring buffer size from the memfd. - ProducerEndpoint::AttachV2RingBuffer() takes the mapping as a const std::shared_ptr<SharedMemory>&. The producer owns it. The service takes a reference only when it accepts, so an in-process producer can share one mapping with the service. - The service rejects the ring buffer if v2 is not in the common mask, if one is already attached, if the memfd cannot be mapped or is larger than kMaxShmSize, or if the size and chunk size give no valid layout. A rejection is an IPC reject. The response has no fields. - On accept, the service creates a ServiceRingBufferDrainer, counts the ring buffer in its memory guardrail, and drains it once. - New one-way RPC DrainV2RingBuffer: the producer asks the service to read its ring buffer. Without an attached ring buffer, it does nothing. - Flush completion, tracing stop and producer disconnect also drain: ScrapeSharedMemoryBuffers() drains the ring buffer first, before its SMB checks. ServiceRingBufferDrainer: - It owns the reader and the view, and shares the mapping. - Each chunk names its target buffer in producer memory. The service writes a chunk only into a TraceBufferV2 that the producer may write to. GetBufferByID() takes an optional buffer type for this check. - Malformed chunks and chunks in an unknown format are discarded and counted in chunks_discarded. Loss that the producer flagged is recorded on the writer's sequence, not counted as a discard. - If the reader finds a protocol error, it stops for good and marks an ABI violation on every destination buffer of the producer. The producer stays connected. - If a drain loses a race with a writer, it posts one delayed retry. Destruction cancels it. Config: - New DataSourceConfig.supports_tracing_v2, field 147, next to protovm_config. The service sets it when v2 is in the common mask and the target buffer is a TraceBufferV2. It overwrites a consumer value. True allows v2 but does not select it. - The service skips an instance when v1 is not in the common mask and its target buffer cannot accept v2. - Startup session adoption ignores supports_tracing_v2, like the other fields that only the service sets. - ProtoVM on the same buffer is not supported yet.

perfetto: Add explicit tracing v2 producer APIs (#7569) Expose ring-buffer initialization and v2 writer creation so callers control transport selection. Make POSIX shared-memory creation fallible with an explicit sealed-memfd requirement.

perfetto: add tracing v2 stats (#7533) Add TraceStats.tracing_v2_producer_stats, with one entry for each connected producer that has a data source in the session and an attached ring buffer. The counters are totals since the producer connected. A cloned session keeps the entries from the time of the clone. The fields of an entry come from four places. - Identity and layout are producer_id, pid, producer_name, chunk_size_bytes and num_chunks. - The reader counts chunks_read and bytes_read. It also counts the chunks it discards. These are data_loss_chunks (the writer flagged them as lossy), malformed_chunks and unsupported_format_chunks. rewrite_requests and positions_skipped count chunks that were read while a writer still filled them, and positions that gave no chunk. - The drainer counts the chunks lost after the reader, which are invalid_writer_chunks, invalid_destination_chunks and trace_buffer_rejected_chunks. It also counts drain_passes, drain_time_ns and drain_max_time_ns, to show what v2 costs traced. The time uses CLOCK_MONOTONIC. - The writers count writer_dropped_packets, writer_stalls and writer_stall_time_ms when the ring buffer is full. They keep them in the ring buffer header and the service reads them through its own mapping. A writer adds its drops when it leaves drop mode and on Flush(), so the value can lag. WriterStats also gets protocol_abi_versions, which says whether a sequence came through the SMB or the ring buffer.

perfetto: Integrate tracing v2 with the SDK (#7700) Select v2 once per instance in the muxer and initialize its ring buffer before publishing the instance. Preserve that choice on failure and propagate packet encoding through the C SDK.

perfetto: Add shift-left compaction to TraceBufferV2 (#7750) Consumed chunks leave pages resident during streaming traces. Add MaybeCompact() to move live TraceBufferV2 chunks to the front after a read cycle, once the returned packets are no longer in use. Compaction zeroes the unused tail and retains the previous used size plus 25% spare capacity to reduce page faults on later writes. Pages beyond that are released where supported. Compaction is skipped if live chunks span both sides of the write cursor. Gate this behind experimental_trace_buffer_v2_compaction, default off, for writable buffers with the overwrite policy. Add per-buffer counters for completed and skipped compactions.

perfetto: add proto group encoding to C and C++ messages (#7524) Tracing v2 writes packets into an append-only ring buffer, so a writer cannot go back and fill in the length of a nested message. Add a second encoding for nested messages to protozero and to the C SDK message builder. - Proto group encoding opens a nested message with a start-group tag and closes it with one 0x04 byte. The root has no start tag and no closing byte: the packet boundary marks its end. - The root selects the encoding (RootMessage::ResetForProtoGroup(), PerfettoPbMsgInitWithEncoding()). Each child inherits it. Length-delimited stays the default and its output does not change. - Message stores no root flag. The handle type selects Message::FinalizeRoot() at compile time, and RootMessage::Finalize() hides Message::Finalize(). The bool template stays private to message.cc so that no template crosses the exported class. - Wire types 3 and 4 stay out of ProtoWireType and PerfettoPbWireType because protozero decoders do not support groups. They are constants in proto_utils.h and pb_utils.h. - Incremental C string and packed fields need a length before the payload. They abort on a proto group stream before writing a tag.