← All posts

Benchmark · August 2026 · 9 min read

Connections, queue depth, and batch reads: tuning a one-sided RDMA cache

The number on our front page is a 1 MiB read at 56 microseconds. It is real, and it is close to the least useful number in the sweep, because one read at a time leaves most of a 200 Gbps link sitting empty. The question that took actual work was how many reads have to be in flight before the pipe is full, and what it costs to keep them there. Numbers below are from two c8gn.16xlarge nodes in a cluster placement group.

Before any numbers: getting EFA to pass traffic

Before any of the numbers, the setup. None of this is Rasa's code and all of it cost us time, so it is worth writing down.

• EFA cannot be attached to a running instance. Both nodes had to be stopped, an ENI created with --interface-type efa in the same subnet, attached at device index 1, then the instances started again.
• The default outbound security-group rule does not pass EFA traffic. This is the one that hurt. All traffic → 0.0.0.0/0 looks like it allows everything, and for TCP and ICMP it does. SRD is not IP, so only security-group-referencing rules match it. The symptom is nasty because everything looks healthy: both endpoints set up cleanly, queue pairs created, address vectors inserted, and then silence. send_wrs climbs while tx_pkts stays at zero. The fix is all-traffic rules, inbound and outbound, with the security group itself as source and destination.
• The EFA ENI's DHCP default route breaks egress. The second interface adds a second default route and package downloads hang. The EFA data path needs no IP routing, so deleting the ens51 default route is safe.
• fi_pingpong takes a different port flag on each side. The server wants -B <port>, the client wants -P <port>. Pass -B on the client and it connects to the default port instead, which surfaces as connection refused on a completely healthy setup.

If you are standing EFA up for the first time, budget for this part separately from the benchmarking.

One read at a time is a latency test, not a throughput test

A single connection issuing one read at a time gives you the floor. At 1 MiB that is about 50 microseconds p50 and 56 p99, and because the reads go back to back it also gives you 19,985 of them per second, which is 168 Gbps. At a megabyte a single outstanding read already fills 84% of the link. Drop to 4 KiB and the same one-at-a-time pattern gets you 61,403 reads per second and 2 Gbps, which is 1% of the link.

One caveat on that 168, because it is the number we quote on the front page: it is a no-validate run. Turn on client-side validation and the same single connection drops to 9,622 reads per second and 81 Gbps. The wire latency does not change, around 50 microseconds either way. What changes is that the client now spends roughly as long computing a CRC32 over each megabyte as it spent fetching it, and that time lands between reads. If you plan to validate every read, halve the single-connection throughput number in your head.

The reason small objects fall so far short is the bandwidth-delay product. While a read is in flight for its round trip, the link could have carried far more bytes than that read contains. Fill it by having enough reads outstanding that their bytes cover the round trip. The smaller the object, the more you need.

How many connections each size needed

Scaling connections, each issuing one read at a time, until throughput stopped climbing:

size	conns	reads/s	bandwidth	p99	p99.99
4 KiB	32	1.53M	50 Gbps	31 µs	77 µs
32 KiB	16	607k	159 Gbps	37 µs	80 µs
128 KiB	6	184k	193 Gbps	44 µs	52 µs
1 MiB	2	24k	201 Gbps	100 µs	125 µs
4 MiB	1	6k	201 Gbps	206 µs	230 µs
16 MiB	1	1.5k	201 Gbps	753 µs	812 µs

Thirty-two connections for 4 KiB, one from 4 MiB up. Large objects carry enough bytes per read that a single one keeps the link full; small ones need a crowd.

4 KiB never reaches line rate at all. At 32 connections and 1.53 million reads per second it tops out near 50 Gbps, and adding connections past that does nothing, because the limit has stopped being the NIC. It is how many operations the client can issue and poll for. That is a client CPU problem and no amount of bandwidth fixes it.

Concurrency is paid for in tail latency

Those p99s are the loaded tail, not the floor, and the gap is large. A 4 KiB read alone is about 19 microseconds p99; at the 32-connection throughput point it is 31, and p99.99 goes from 21 to 77. At 1 MiB the floor is 56 and the two-connection line-rate point is 100.

From 4 MiB up there is no gap, because one connection is both the floor and the ceiling. So pick an operating point and say which one you picked. A benchmark that quotes the floor's latency next to the ceiling's throughput is describing two different runs.

Queue depth is cheaper than connections

Connections are not the only way to get reads in flight. Each connection has a queue depth, and total concurrency is connections times queue depth. Throughput tracks the product, not how you split it: two connections at queue depth 16 and thirty-two connections at queue depth 1 both put 32 reads in flight, and in our runs both delivered about the same throughput at the same size.

The bill is different. Rasa's reader busy-polls its completion queue, so a connection costs roughly a core whether its queue depth is 1 or 16. The 32-connection version burns about 32 client cores for throughput the 2-connection version gets for about 2. So raise queue depth first and add connections only when one core cannot keep a queue fed.

The tail is flat, and that part is not our doing

At the floor, latency barely spreads. A 1 MiB read is 56 microseconds at p99, 58.5 at p99.9, 61 at p99.99. 4 KiB goes 19, 20, 21. Even 16 MiB only moves from 753 to 812. The one-in-ten-thousand read is within about ten percent of the typical one at every size.

There is not much for a one-sided read to jitter on. No server thread to queue behind, no garbage collector, no lock, no scheduler decision, no per-request CPU work. The NIC moves bytes over a path SRD already chose, and SRD's hardware retransmission absorbs a dropped packet in microseconds, before it can become an outlier. What is left is fabric and PCIe scheduling.

Under load that changes, and the variance is the variance you added. 1 MiB at two connections is 100 then 125; 4 KiB at 32 connections is 31 then 77. That widening is queueing you introduced on purpose, not the network degrading.

About that "one core"

Through all of this the data node sat at about one core, roughly 1.6% of its 64, and it is worth being precise about what that core is doing, because it is not serving reads.

libfabric 2.4's efa provider reports FI_PROGRESS_MANUAL, so kv-server's progress loop busy-polls. That is a policy, not load. The reads themselves are served by the NIC out of registered memory and never enter the process. A production server would sleep or use a wait set and hand the core back. The other 63 stay idle either way.

The claim we are comfortable making is that server CPU does not track load. Not that the server uses no CPU.

The write path fooled us

Reads were the easy direction. Writes go the other way: request a slab over the control plane, RDMA-write each filled block into the server's remote-writable arena at some queue depth, then commit the batch so the server publishes descriptors.

The first cut sustained about 5 to 6 GiB/s and would not move. At 1 MiB it went 5.58, then 5.26, then 5.16 GiB/s as queue depth went 16, 32, 64. Throughput drifting slightly down as you add concurrency is a useful signal: whatever is limiting you is not the fabric.

It was per-block CPU work on a single thread. The writer does things the reader does not. It fills each value slot and computes a CRC32 over the whole payload before issuing the write, and the source ring is only queue_depth slots wide. The CRC32 already uses Graviton4's hardware crc32 instructions through the crc32fast crate, and one thread doing it was still the ceiling. Reads never hit this because a read does no per-block work at all.

For small objects, the metadata is the tax

There is a second cost with nothing to do with the NIC. Every read needs a descriptor: which node holds the value, at what address, under what key. Resolving one is a Valkey lookup. Do that per key and you have put a control-plane round trip in front of every RDMA read, and for a 4 KiB value that lookup can cost more than the read it enables.

Batching fixes it. get_many resolves every descriptor in a single Valkey MGET, inserts each node's fabric address once, then builds a read plan and pipelines the reads rather than waiting for each. One metadata round trip for the batch, then the NIC stays busy. For a KV cache manifest it goes further: one manifest object names every block, so the client lands the whole thing with one lookup into a contiguous buffer.

What we learned

Get enough reads in flight to cover the round trip, which is a handful for large objects and dozens for small ones. Reach that with queue depth before connections, because connections cost client cores. Batch the descriptor lookups. And below a few hundred kilobytes you are bounded by operations and client CPU rather than bandwidth, which is the honest reason Rasa is aimed at large objects: that is where a single connection already runs at line rate and the tuning stops mattering.

Every dial we turned was on the client. That is the odd part of tuning a one-sided cache, and it took us a while to stop looking at the server.

Single-node EFA sweep on c8gn.16xlarge, no client-side validation unless noted. Concurrency is connections times queue depth; wire-rate figures assume a 200 Gbps link. Latency figures are from the run at the stated operating point, not mixed across runs.