foundations — Engineering Performance

Add user-facing tracing pipeline Adds a user-facing span pipeline to `foundations`, parallel to and independent of the existing internal tracing pipeline. Application code emits spans into a separate harness that exports OTLP over HTTP on a Unix domain socket, with per-trace routing metadata (an application-defined `RoutingMetadata`) attached to each span and carried to the endpoint in a configurable header, plus W3C `traceparent` continuation in and out. The public API lives in a dedicated `tracing::user_tracing` module that mirrors the top-level internal-tracing functions (`start_trace`, `span`, `add_span_tags!`, ...), plus a top-level `dual_span` that records into both pipelines at once; `#[span_fn(user = true)]` emits a dual span. The user span rides on `TelemetryContext`, so it propagates across `.await`, spawns, and hooks without manual threading. Two rules shape the surface: the user and internal pipelines are fully independent (separate harnesses, scope stacks, and exporters), and the public surface speaks W3C. Sampling is not configured inside foundations -- per-request activation is driven by the caller. Everything is gated behind the `user-tracing` cargo feature.

Refactor slog::Drain setup code and fix rate limiting bug - fix: Apply a single global rate limiter, instead of creating a new one each time `set_verbosity` is called. (was in `apply_filters_to_drain`) - perf: Skip `redact_keys` filter evaluation if no redactions are configured. - chore: Introduce `DrainExt` extension trait to make drain layering more readable. - chore: Correctly forward `slog::Drain` helper methods in our drain wrappers.

Gracefully degrade on metrics encoding failures By default, `prometheus-client` aborts metrics collection/encoding on _any_ error. This means one error from the EncodeMetric implementation of a single metric in a Registry makes all the metrics in that Registry unavailable. To avoid this, we introduce an EncodeMetric wrapper that swallows errors (after reporting them via logging/`eprintln!`.) Due to the way `prometheus-client` is designed, this could leave a partially-written metrics line in the output. We fix this by introducing `RewindableWriter`, a `Write` implementation that we can rewind to the last newline in case we do see an error. `prometheus-client` does not give us access to the underlying writer in EncodeMetric impls, so we are forced to use a side channel via thread-local storage to activate the rewind behavior.

Prepare foundations for multi-consumer tracing output - Introduce optional limit for span queue size. - Add metrics for total spans, dropped spans, current span queue size, and maximum configured span queue size. - Add `tokio::sync::mpsc` receiver wrapper to allow multi-consumer semantics. The next commit will introduce the option to start multiple consumer tasks. - Use batching `recv_many` calls in Jaeger UDP tracing output. I reviewed all the well-known async MPMC queue implementations prior to landing on the async mutex wrapper for tokio's own channels. The common problem shared by almost all MPMC implementations is that they do not support batch receive operations (i.e., `recv_many`). This, combined with the bad locality of single-queue MPMC channels, makes me believe a Mutex-wrapped MPSC channel with batch receives will perform better for the tracing use case. There are 2 MPMC implementations that do offer batching (batch-channel and burstq). These don't work for our use case either: - burstq pre-allocates the entire channel size, which is prohibitive since we expect to only use a fraction of it >99% of the time. - batch-channel does batching inside the sender. This means sending requires exclusive ownership over the sender, so we would have to put it inside a Mutex to share it between spans. This moves the locking from the (few) consumer tasks to the many, many spans that may be generated. In contrast, the Mutex-wrapped MPSC receiver will have 1 active receiving task at any time and a FIFO queue of other tasks waiting to become the active receiver next. The active receiver gets a batch of spans, and while the Mutex is passed on to the next task a new batch accumulates in the channel. We can revisit this decision with production metrics later on if needed.

Add Settings/RawSettings wrappers to protect sensitive values

feat(telemetry): add deferred user span activation

Add an owned UserSpan handle for user tracing The user-tracing API is built around an ambient scope stack: `span()` parents under whatever is current and hands back an RAII guard that pops on drop. The scope stack is a `ThreadLocal` and its guard is deliberately `!Send`, so a span can never be held across an await point. That model can't express a per-request span in a service whose request pipeline is a state machine of separate callbacks rather than a call tree. The span outlives every individual callback, so there is no lexical scope for it to live in, and the only `Send` escape hatch available today is `into_context()` — which is meant for propagating into a future, and drags a snapshot of the whole ambient `TelemetryContext` along with it. `UserSpan` is an owned, `Send` handle to a user span, detached from the scope stack. It can live in a struct field, cross threads, and be finished in a different callback from the one that started it. The span is reported when the last reference to it drops. That is normally the handle, though `enter()` takes its own, as does any context built from the resulting scope. Parents are named explicitly rather than inferred from a stack, because spans in a callback-driven pipeline overlap without nesting: an upstream fetch can still be unfinished while response-side work that is not part of it is running, so "innermost active span" is the wrong parent. A handle is always usable — there is no `Option`. When no trace is active, or it wasn't sampled, or no pipeline is configured, the handle is inactive: tagging does nothing, children are inactive in turn, nothing is reported. This keeps `if let Some(span)` out of call sites, and mirrors how cf-rustracing's own inactive spans already behave. `user_tracing::start_trace` and `w3c_traceparent` are re-expressed on top of the new type, and `create_user_span` now shares its child-construction with `UserSpan::child`, so each behaviour has a single implementation. `enter()` bridges back to the ambient API for callers that prefer it. Purely additive: no existing caller changes. `Tag` is now re-exported, since it appears in the `set_tags` bound.

Allow multiple concurrent trace exporter tasks All tasks share a channel (gRPC)/socket (UDP) for output, but can process spans and send batches in parallel. This requires us to spawn the exporter futures rather than add them to TelemetryDriver's `tele_futures`, where they would be polled sequentially. While this changes the behavior of foundations' tracing output, we consider this a bugfix and thus appropriate for a minor release. Previously, trace output stopped as soon as the TelemetryDriver was dropped, but the process-wide tracer was still active.

Add MaybeExternal helper to load config values from external sources

Add `unprefixed` flag to `#[metrics]` macro This change is twofold. First, it refactors the registry selection in `metrics::internal::Registries` to use boolean flags rather than separate methods to select a registry to use. As part of this, we add a combination of flags to return a registry without the global service name prefix that is normally present. Second, we add a new `unprefixed` argument to the `metrics` proc-macro. If present, this argument causes all metrics in the associated module to be registered in the registry mentioned above. This works for both the "main" and the "optional" registry. I've added both a proc-macro expansion test and an integration test to validate the flag's functionality.