David Wei

IoUringProvidedBufferRing: dynamically grow buffer areas on demand Summary: Replace the single fixed-size buffer allocation with a pool of dynamically growable buffer areas. Previously the ring mapped one contiguous block holding exactly ringBufferCount_ buffers; once all of them were handed out to the user and still held, further reads returned -ENOBUFS as backpressure. This change decouples the kernel ring (still ringBufferCount_ entries) from the backing storage, which is now a set of BufferArea blocks, each holding a full ring's worth of buffers with its own BufferState[] and an `outstanding` counter (posted-in-ring plus held-by-user). The ring is refilled from these areas: - mapMemory() is split into mapRing() (the io_uring_buf ring) and allocateMemoryArea()/createArea()/appendAreas() for the buffer areas. - The pool starts with kInitialAreaCount (2) areas and grows by one each time there is no area available. - An area can only be reused once fully drained (outstanding == 0), preventing corruption of buffers still held by the user. - getIoBuf() now tracks the consume-side area (bufferConsumedArea_) so it follows refill's posted area order across area boundaries, including bundles that straddle areas, rather than assuming a naive (area + 1) % areaCount step. - getUtilPct() is reworked to count only user-held buffers across all areas. - returnBuffer() is replaced by ringRefill()/bufferConsumed() and the area `outstanding` accounting. Adds //folly:expected as a dependency (growAreas()/areaGetNextFree() return folly::Expected). Reviewed By: keithbusch Differential Revision: D116323996 fbshipit-source-id: ed6c1417797147935520a3bdd8e001a0be27795c

IoUringProvidedBufferRing: dynamically grow buffer areas on demand Summary: Replace the single fixed-size buffer allocation with a pool of dynamically growable buffer areas. Previously the ring mapped one contiguous block holding exactly ringBufferCount_ buffers; once all of them were handed out to the user and still held, further reads returned -ENOBUFS as backpressure. This change decouples the kernel ring (still ringBufferCount_ entries) from the backing storage, which is now a set of BufferArea blocks, each holding a full ring's worth of buffers with its own BufferState[] and an `outstanding` counter (posted-in-ring plus held-by-user). The ring is refilled from these areas: - mapMemory() is split into mapRing() (the io_uring_buf ring) and allocateMemoryArea()/createArea()/appendAreas() for the buffer areas. - The pool starts with kInitialAreaCount (2) areas and grows by one each time there is no area available. - An area can only be reused once fully drained (outstanding == 0), preventing corruption of buffers still held by the user. - getIoBuf() now tracks the consume-side area (bufferConsumedArea_) so it follows refill's posted area order across area boundaries, including bundles that straddle areas, rather than assuming a naive (area + 1) % areaCount step. - getUtilPct() is reworked to count only user-held buffers across all areas. - returnBuffer() is replaced by ringRefill()/bufferConsumed() and the area `outstanding` accounting. Adds //folly:expected as a dependency (growAreas()/areaGetNextFree() return folly::Expected). Reviewed By: keithbusch Differential Revision: D106526092 fbshipit-source-id: 5e8c8517ee5a37ac13bd94429032ae3285035449

Back out "Revert D106526092: [folly] IoUringProvidedBufferRing: dynamically grow buffer areas on demand" Summary: Original commit changeset: 10772972d73f Original Phabricator Diff: D106526092 Reviewed By: jlhe97 Differential Revision: D115634171 fbshipit-source-id: 284d245d93a67b16aba6541df331b2d182ba12c6

IoUringBufferProvider: add buffer provided dispatch Summary: Add `IoUringBufferProvider` to dispatch to either `IoUringProvidedBufferRing` or `IoUringDynamicProvidedBufferRing`. Add `IoUringProvidedBufferRingMode` to `IoUringOptions`, defaulting to the fixed implementation. Reviewed By: jlhe97 Differential Revision: D116735819 fbshipit-source-id: 72aae4f932aef32162d3f2296634514f73047a78

Pluggable ClientThreadDriver + busy-poll ClientThread Summary: The stress-test client only supported TCP/Rocket over an epoll EventBase (ScopedEventBaseThread running loopForever). Custom datagram transports (AF_XDP/XSK, UDP) have no fd registered in the event loop, so they cannot be epoll-driven -- they must be pumped inline on their own thread every iteration. This adds a seam for that without disturbing any existing stress binary. Two pieces: 1. A ClientThreadDriver interface (createClients / progress / shutdown) plus a createClientThreadDriver pluggable func (default returns nullptr). A downstream binary SETs it to inject a transport-specific driver the framework cannot name; returning nullptr selects the stock path. 2. ClientThread becomes an abstract base holding all transport-agnostic logic (workload scheduling, stats, memory histogram, timeouts) behind two virtuals, getEventBase() and buildClients(), with two subclasses: - RocketClientThread: stock path; owns a folly::ScopedEventBaseThread (epoll loopForever) and builds Rocket clients. - BusyPollClientThread: driver-backed; owns an EventBase + a raw std::thread that interleaves driver_->progress() (transport pump) with a non-blocking loopOnce() (coroutine continuations, timers, cross-thread tasks). ClientRunner picks the subclass per client thread from createClientThreadDriver's return (null -> Rocket, non-null -> busy-poll), before thread construction. BusyPollClientThread solely owns the driver and does createClients / progress / shutdown / driver destruction all on its own poll thread -- the single-thread-free invariant AF_XDP zero-copy frame ownership relies on. buildClients() is non-hopping and runs inside initClients' single runInEventBaseThreadAndWait, so client construction (and its connectSuccess()/ latch firing) runs on the owning EventBase thread in one pass, with no nested waits. Existing stress binaries (ExampleScenarios, SRTestSuite, Example, Suite, conformance ExampleScenarios) are unchanged: they do not SET the pluggable, so createClientThreadDriver returns nullptr and they get RocketClientThread as before. Reviewed By: keithbusch Differential Revision: D111743748 fbshipit-source-id: b34c5980156ab2365a22c9f42fced834ed1c4c73

IoUringZeroCopyBufferPool: track buffer scarcity near exhaustion Summary: Buffers for zcrx are shared between kernel and userspace. Kernel v7.2-fb adds support for multiple buffer areas per queue and signals for buffer exhaustion, but until this is widely available in the fleet, another way to deal with running out of buffers is needed. Change `IoUringZeroCopyBufferPool::getIoBuf()` to return both the received `IOBuf` and an `isScarce` signal, set when no more than `ceil(numBuffers / 8)` buffers remain available to the kernel. Update existing callers. This signal will be propagated up into the Thrift parser ReadCallback, which will latch per Thrift frame. When a complete Thrift frame is ready, it will heap allocate a contiguous buffer and copy the entire Thrift frame in. This will then cause the buffers owned by IoUringZeroCopyBufferPool to be freed and returned to the kernel. Reviewed By: jlhe97 Differential Revision: D117807703 fbshipit-source-id: 5c2e85e90026af517415886adcf011fe8254c0f8

Delay socket close for pending MSG_ZEROCOPY Summary: Separate logical close from physical fd close while the existing zero-copy completion maps contain kernel-accepted sends. Stop normal socket I/O immediately, keep the fd and `AsyncSocket` alive through the EventBase, poll `MSG_ERRQUEUE` asynchronously, and close the fd after the last completion releases its IOBuf. Reject EventBase detachment while this drain is active. Reviewed By: dmm-fb Differential Revision: D114629846 fbshipit-source-id: 9a3e7a60db6aa300d3d44ef58f1f6e49bd07e6e6

AsyncSocket: readBufferAvailable delivers all data to ReadCallbacks Summary: Native `AsyncSocket` io_uring recv delivers owned `IOBuf`s through `readBufferAvailable()`. Previously, `checkForImmediateRead()` split `preReceivedData_` into callback-sized pieces while recv data were delivered whole. That gave the same callback different delivery semantics depending on the source of the bytes and required `AsyncSocket` to retain and write back a remainder after invoking application code. Make native io_uring delivery consistent: - Require native io_uring read callbacks to support movable buffers. - Transfer the complete `preReceivedData_` chain in one `readBufferAvailable()` call. Callbacks that inspect only a prefix now own the complete delivered buffer and must replay any bytes they do not consume. Reviewed By: jlhe97 Differential Revision: D116347830 fbshipit-source-id: 4344ace5e67133a5681d8551f8fe9220efb288e9

IoUringProvidedBufferRing: shrink buffer areas when drained Summary: Add dynamic shrinking of buffer areas to complement the recent growth support. Previously the pool could only grow but never shrank back, leading to persistent memory overhead after transient spikes. We remove at most one area, be sure to always have at least 1 free area and two area in total. Reviewed By: keithbusch Differential Revision: D116324000 fbshipit-source-id: 34e13695969f51dc8b89b184d2087d0a0bd70566

Back out "Revert D113010343: [folly] IoUringProvidedBufferRing: shrink buffer areas when drained" Summary: Original commit changeset: a29c51aeb43c Original Phabricator Diff: D113010343 Reviewed By: jlhe97 Differential Revision: D115634710 fbshipit-source-id: 27d9e3a28d6ec47855f27b24371336bd5561446e