When the cache server writes the answer into you
A one-sided RDMA read is a pull. The client holds a descriptor, points the NIC at remote memory, and pulls the bytes itself, with no CPU on the far end. That works because the value lives in registered DRAM. The moment it does not, the pull breaks, and the fix is to flip the direction of the transfer.
Why pull breaks below DRAM
We are adding an NVMe tier under the DRAM arenas, so a node can hold far more than its pinned
memory. But the NIC can only fi_read memory it has registered: DRAM, or GPU HBM.
It cannot read an NVMe block by rkey. A value sitting on NVMe is, by construction, not
reachable one-sided. The client can resolve where it is and still have no way to pull it.
The obvious fix, and the better one
The obvious fix is to stage: the node reads the block into a registered DRAM slot, publishes a new descriptor, and the client re-resolves and reads it one-sided. It works, but it is two or three round trips, and it spends a DRAM slot on a value that may never be read again.
The better fix is to stop making the client pull at all. The client already has a registered
buffer waiting for the answer. So let it hand that buffer to the node: its address, the rkey
that authorizes a write, and the length. The NVMe tier is a memory-mapped file, so the node
fi_writes the value straight from those file pages into the client's buffer, with
no copy into a holding buffer first. This is the classic RDMA rendezvous run for a cache read,
and it is the same move sendfile makes: stream a file out without staging it
through userspace.
One round trip, and the reply is the receipt
The whole exchange is a single request and response. The client sends one message asking for the key and carrying the write-capability. The node does its NVMe read and its RDMA write, and only then replies. That reply is the completion signal: when it lands, the bytes are already in the client's buffer, whole. No second descriptor, no re-resolve, no separate notification. The client validates the CRC it always validates, and a torn or failed write fails that check and becomes a clean miss.
One detail makes the reverse path free. If the request travels as a two-sided message over the fabric rather than a side channel, the node learns the client's address from receiving it, so it can write back over the same connection without any extra setup.
What you trade for it
Push asks the client to grant the node a write into its own memory, which sounds alarming until you place it. Rasa is single-tenant and self-hosted: the client and the data node are the same operator's processes, not two parties across a tenant boundary. The capability is not crossing a trust line it would cross in a shared service.
The cost that does need watching is on the node. A pulled read costs the node nothing on the read path; a pushed read makes it post a write, on the same transmit context its real writes use. Cold reads now compete with writes for that one resource. The answer is to give promote-writes their own context and to keep push for what it is good at: the cold path. Hot values stay in DRAM and stay a pull. We only flip the direction when the value fell to NVMe and the client could not have pulled it anyway.
We ended up treating the direction of the transfer as a knob rather than a property of the system. Pull is right when the data is one-sided-reachable and re-read often. Push is right when it is not, and a single round trip that ends with the bytes already in your buffer beats staging them somewhere first. Same fabric, opposite direction, chosen per tier.