David Wei

move IOBufChain and IOBufChainCursor into apache::thrift Summary: Thrift is open source and xplat, meaning it can't depend on a non-open source fbcode-only `prolly::IOBufChain`. Move `IOBufChain` into the `apache::thrift` namespace in `thrift/lib/cpp2`. Reviewed By: vitaut Differential Revision: D121021753 fbshipit-source-id: 4f849d11716f70aad6902f622563b1fbc0d22d15

IoUringProvidedBufferRing: dynamically grow buffer areas on demand Summary: Replace the single fixed-size buffer allocation with a pool of dynamically growable buffer areas. Previously the ring mapped one contiguous block holding exactly ringBufferCount_ buffers; once all of them were handed out to the user and still held, further reads returned -ENOBUFS as backpressure. This change decouples the kernel ring (still ringBufferCount_ entries) from the backing storage, which is now a set of BufferArea blocks, each holding a full ring's worth of buffers with its own BufferState[] and an `outstanding` counter (posted-in-ring plus held-by-user). The ring is refilled from these areas: - mapMemory() is split into mapRing() (the io_uring_buf ring) and allocateMemoryArea()/createArea()/appendAreas() for the buffer areas. - The pool starts with kInitialAreaCount (2) areas and grows by one each time there is no area available. - An area can only be reused once fully drained (outstanding == 0), preventing corruption of buffers still held by the user. - getIoBuf() now tracks the consume-side area (bufferConsumedArea_) so it follows refill's posted area order across area boundaries, including bundles that straddle areas, rather than assuming a naive (area + 1) % areaCount step. - getUtilPct() is reworked to count only user-held buffers across all areas. - returnBuffer() is replaced by ringRefill()/bufferConsumed() and the area `outstanding` accounting. Adds //folly:expected as a dependency (growAreas()/areaGetNextFree() return folly::Expected). Reviewed By: keithbusch Differential Revision: D116323996 fbshipit-source-id: ed6c1417797147935520a3bdd8e001a0be27795c

IoUringProvidedBufferRing: dynamically grow buffer areas on demand Summary: Replace the single fixed-size buffer allocation with a pool of dynamically growable buffer areas. Previously the ring mapped one contiguous block holding exactly ringBufferCount_ buffers; once all of them were handed out to the user and still held, further reads returned -ENOBUFS as backpressure. This change decouples the kernel ring (still ringBufferCount_ entries) from the backing storage, which is now a set of BufferArea blocks, each holding a full ring's worth of buffers with its own BufferState[] and an `outstanding` counter (posted-in-ring plus held-by-user). The ring is refilled from these areas: - mapMemory() is split into mapRing() (the io_uring_buf ring) and allocateMemoryArea()/createArea()/appendAreas() for the buffer areas. - The pool starts with kInitialAreaCount (2) areas and grows by one each time there is no area available. - An area can only be reused once fully drained (outstanding == 0), preventing corruption of buffers still held by the user. - getIoBuf() now tracks the consume-side area (bufferConsumedArea_) so it follows refill's posted area order across area boundaries, including bundles that straddle areas, rather than assuming a naive (area + 1) % areaCount step. - getUtilPct() is reworked to count only user-held buffers across all areas. - returnBuffer() is replaced by ringRefill()/bufferConsumed() and the area `outstanding` accounting. Adds //folly:expected as a dependency (growAreas()/areaGetNextFree() return folly::Expected). Reviewed By: keithbusch Differential Revision: D106526092 fbshipit-source-id: 5e8c8517ee5a37ac13bd94429032ae3285035449

Back out "Revert D106526092: [folly] IoUringProvidedBufferRing: dynamically grow buffer areas on demand" Summary: Original commit changeset: 10772972d73f Original Phabricator Diff: D106526092 Reviewed By: jlhe97 Differential Revision: D115634171 fbshipit-source-id: 284d245d93a67b16aba6541df331b2d182ba12c6

parser: add IOBufChain specialization of FrameLengthParserStrategy Summary: ## Motivation Storage services receive large binary fields as long scatter/gather lists especially when using io_uring and AF_XDP zero copy that produce fragments as small as 4 KB. The existing Thrift path represents these binary fields as intrusive IOBuf linked lists. Any operation on the chain metadata or data require walking the chain from the start, which is TLB and cache unfriendly. Both Thrift and applications walk these chains regularly. With Fizz, each TLS record is at most 16 KB and Fizz does relatively small 4K allocations, so the overall chain element count for larger Thrift frames in the multi-MB can also be long. IOBufChain is a new type that addresses these problems: * It is a first-class type representing a single logical buffer spread across multiple scatter/gather fragments. * It has cached chain data length and element counts, maintained as the chain changes in size. * It stores IOBuf fragments in an linked array rather than a linked list. ## Proposal Create an IOBufChain at the lowest layers in Thrift, and push it throughout the entire stack, exposing it as a new type to applications. From a high level: * Represent a Thrift frame using a single IOBufChain in the parser layer. * Deserialize using IOBufChainCursor operating on the IOBufChain with a similar API to Cursor. * A parallel BinaryChainProtocolReader that uses IOBufChain and IOBufChainCursor instead of IOBufQueue and Cursor. Binary fields represented as IOBufChain rather than unique_ptr<IOBuf>. This will require changes across the Thrift framework stack in the parser, protocol, generated code, and the service handler API. ## Compatibility No changes to the wire format, and strictly opt-in only. When not used, the standard IOBufQueue, IOBuf, and Cursor types are used. ## This Change Add a new specialization `FrameLengthParserStrategy` with `IOBufChain`. The parser algorithm is the same; the only main difference is a running `frameBufCountEstimate_` that estimates the number of buffers remaining in the current Thrift frame. For zero copy transports where Thrift does not allocate and use the `getReadBuffer`/`readDataAvailable` ReadCallback API, but instead the `readBufferAvailable` API, it has no control over the buffer sizes. `estimatedBufLength_` is a best-effort estimate so that IOBufChain can try to size the IOBuf array so that the entire Thrift frame can be represented using just one array. Reviewed By: vybv Differential Revision: D120581242 fbshipit-source-id: 87e25cf8602e1848ea057bbb02272e745a133191

protocol: add BinaryProtocolChainReader Summary: ## Motivation Storage services receive large binary fields as long scatter/gather lists especially when using io_uring and AF_XDP zero copy that produce fragments as small as 4 KB. The existing Thrift path represents these binary fields as intrusive IOBuf linked lists. Any operation on the chain metadata or data require walking the chain from the start, which is TLB and cache unfriendly. Both Thrift and applications walk these chains regularly. With Fizz, each TLS record is at most 16 KB and Fizz does relatively small 4K allocations, so the overall chain element count for larger Thrift frames in the multi-MB can also be long. IOBufChain is a new type that addresses these problems: * It is a first-class type representing a single logical buffer spread across multiple scatter/gather fragments. * It has cached chain data length and element counts, maintained as the chain changes in size. * It stores IOBuf fragments in an linked array rather than a linked list. ## Proposal Create an IOBufChain at the lowest layers in Thrift, and push it throughout the entire stack, exposing it as a new type to applications. From a high level: * Represent a Thrift frame using a single IOBufChain in the parser layer. * Deserialize using IOBufChainCursor operating on the IOBufChain with a similar API to Cursor. * A parallel BinaryChainProtocolReader that uses IOBufChain and IOBufChainCursor instead of IOBufQueue and Cursor. Binary fields represented as IOBufChain rather than unique_ptr<IOBuf>. This will require changes across the Thrift framework stack in the parser, protocol, generated code, and the service handler API. ## Compatibility No changes to the wire format, and strictly opt-in only. When not used, the standard IOBufQueue, IOBuf, and Cursor types are used. ## This Change Extract the existing `BinaryProtocolReader` into a templated base type `BinaryProtocolReaderBase` on the buffer and cursor type. Then, `BinaryProtocolReader` is an instantiation with `folly::IOBuf` and `folly::io::Cursor`. The new `BinaryProtocolChainReader` is an instantiation with `IOBufChain` and `IOBufChainCursor`. Reviewed By: vitaut Differential Revision: D120398862 fbshipit-source-id: 33a38e2adb26c4343e124b1780da5e27d8fd9451

protocol: support representing binary Thrift type as IOBufChain Summary: ## Motivation Storage services receive large binary fields as long scatter/gather lists especially when using io_uring and AF_XDP zero copy that produce fragments as small as 4 KB. The existing Thrift path represents these binary fields as intrusive IOBuf linked lists. Any operation on the chain metadata or data require walking the chain from the start, which is TLB and cache unfriendly. Both Thrift and applications walk these chains regularly. With Fizz, each TLS record is at most 16 KB and Fizz does relatively small 4K allocations, so the overall chain element count for larger Thrift frames in the multi-MB can also be long. IOBufChain is a new type that addresses these problems: * It is a first-class type representing a single logical buffer spread across multiple scatter/gather fragments. * It has cached chain data length and element counts, maintained as the chain changes in size. * It stores IOBuf fragments in an linked array rather than a linked list. ## Proposal Create an IOBufChain at the lowest layers in Thrift, and push it throughout the entire stack, exposing it as a new type to applications. From a high level: * Represent a Thrift frame using a single IOBufChain in the parser layer. * Deserialize using IOBufChainCursor operating on the IOBufChain with a similar API to Cursor. * A parallel BinaryChainProtocolReader that uses IOBufChain and IOBufChainCursor instead of IOBufQueue and Cursor. Binary fields represented as IOBufChain rather than unique_ptr<IOBuf>. This will require changes across the Thrift framework stack in the parser, protocol, generated code, and the service handler API. ## Compatibility No changes to the wire format, and strictly opt-in only. When not used, the standard IOBufQueue, IOBuf, and Cursor types are used. ## Current Change Make IOBufChain a supported representation for Thrift binary values in binary and compact protocols. Reviewed By: iahs, vitaut Differential Revision: D120390005 fbshipit-source-id: b1c48a1e2aa2bb6d59815648bf36b1c40f4740f9

IoUringBufferProvider: add buffer provided dispatch Summary: Add `IoUringBufferProvider` to dispatch to either `IoUringProvidedBufferRing` or `IoUringDynamicProvidedBufferRing`. Add `IoUringProvidedBufferRingMode` to `IoUringOptions`, defaulting to the fixed implementation. Reviewed By: jlhe97 Differential Revision: D116735819 fbshipit-source-id: 72aae4f932aef32162d3f2296634514f73047a78

Pluggable ClientThreadDriver + busy-poll ClientThread Summary: The stress-test client only supported TCP/Rocket over an epoll EventBase (ScopedEventBaseThread running loopForever). Custom datagram transports (AF_XDP/XSK, UDP) have no fd registered in the event loop, so they cannot be epoll-driven -- they must be pumped inline on their own thread every iteration. This adds a seam for that without disturbing any existing stress binary. Two pieces: 1. A ClientThreadDriver interface (createClients / progress / shutdown) plus a createClientThreadDriver pluggable func (default returns nullptr). A downstream binary SETs it to inject a transport-specific driver the framework cannot name; returning nullptr selects the stock path. 2. ClientThread becomes an abstract base holding all transport-agnostic logic (workload scheduling, stats, memory histogram, timeouts) behind two virtuals, getEventBase() and buildClients(), with two subclasses: - RocketClientThread: stock path; owns a folly::ScopedEventBaseThread (epoll loopForever) and builds Rocket clients. - BusyPollClientThread: driver-backed; owns an EventBase + a raw std::thread that interleaves driver_->progress() (transport pump) with a non-blocking loopOnce() (coroutine continuations, timers, cross-thread tasks). ClientRunner picks the subclass per client thread from createClientThreadDriver's return (null -> Rocket, non-null -> busy-poll), before thread construction. BusyPollClientThread solely owns the driver and does createClients / progress / shutdown / driver destruction all on its own poll thread -- the single-thread-free invariant AF_XDP zero-copy frame ownership relies on. buildClients() is non-hopping and runs inside initClients' single runInEventBaseThreadAndWait, so client construction (and its connectSuccess()/ latch firing) runs on the owning EventBase thread in one pass, with no nested waits. Existing stress binaries (ExampleScenarios, SRTestSuite, Example, Suite, conformance ExampleScenarios) are unchanged: they do not SET the pluggable, so createClientThreadDriver returns nullptr and they get RocketClientThread as before. Reviewed By: keithbusch Differential Revision: D111743748 fbshipit-source-id: b34c5980156ab2365a22c9f42fced834ed1c4c73

IoUringZeroCopyBufferPool: track buffer scarcity near exhaustion Summary: Buffers for zcrx are shared between kernel and userspace. Kernel v7.2-fb adds support for multiple buffer areas per queue and signals for buffer exhaustion, but until this is widely available in the fleet, another way to deal with running out of buffers is needed. Change `IoUringZeroCopyBufferPool::getIoBuf()` to return both the received `IOBuf` and an `isScarce` signal, set when no more than `ceil(numBuffers / 8)` buffers remain available to the kernel. Update existing callers. This signal will be propagated up into the Thrift parser ReadCallback, which will latch per Thrift frame. When a complete Thrift frame is ready, it will heap allocate a contiguous buffer and copy the entire Thrift frame in. This will then cause the buffers owned by IoUringZeroCopyBufferPool to be freed and returned to the kernel. Reviewed By: jlhe97 Differential Revision: D117807703 fbshipit-source-id: 5c2e85e90026af517415886adcf011fe8254c0f8