How Rasa writes: allocate, RDMA-write, commit
We expected the write path to be the mirror image of reads. It is not. Reads can be served by memory that already exists. Writes have to create a new truth without letting any reader see a half-truth.
The write is easy. Publishing is hard.
A writer does not ask the data node to receive a value over TCP. It asks the control
plane for space, writes the framed value directly into that remote slot with
fi_write, waits for completion, then commits. Only after commit does a
descriptor become visible to readers.
Getting that ordering wrong is how a reader sees a half-written value. The writer may have completed the RDMA write, but readers still cannot discover the value until the data node commits it and publishes a signed descriptor. A racing reader either sees the old version, a miss, or the new complete value. It should never see a slot while it is being filled.
What is in the remote slot
Rasa writes one contiguous frame: a small value header followed by the payload. The header carries the key, version, payload length, and checksum. On the read side, the client validates that header and CRC after the RDMA read. The data node does not need to inspect every byte on the data path, but the value is still self-checking.
Why not update in place?
In-place mutation is the tempting shortcut. It is also how readers get torn values: half old payload, half new payload, or a new header pointing at bytes that have not arrived yet. Rasa treats writes as publishing a new version into a new slot. The old slot can be reclaimed later, after descriptors and leases that might reference it have expired.
Reads want stable addresses. Writes want fresh space. The control plane exists to keep those two wishes from fighting in the same memory slot.
Bulk writes are mostly allocation math
The same pattern works in batches. Instead of asking for one slot per value, a writer
can ask for a slab: N fixed-size slots with deterministic addresses. The remote address
of slot i is base_addr + i * slot_size. The writer fills many
slots with RDMA writes, then commits the completed keys. That amortizes control-plane
calls while preserving the important rule: no descriptor is visible until its frame is
complete.
Why the writer talks to the control plane, not the node
The first shape we considered was the obvious one: ask the data node for space, write, tell it you are done. That dies immediately on the numbers. Prefill produces thousands of blocks per second. A manager round trip per block does not make a hot path, it makes a queue.
So the writer leases a slab from the control plane instead, and allocates blocks out of it locally until it runs low. The round trips are amortised over roughly a hundred blocks each on both ends, allocation and commit, which puts them off the per-block cost entirely. It is the same reason the read path batches descriptor lookups. On a fabric this fast, any per-operation round trip to anything is the bottleneck by default.
The real write path is a lifecycle
The bandwidth part is straightforward. The lifecycle is where production systems are won or lost: allocation grants need TTLs, commits need to be idempotent, old descriptors need leases, and freed slots need a quarantine period before reuse. If a region is re-registered, stale rkeys must stop working. If metadata is untrusted, descriptors need signatures. The RDMA write moves bytes. The system around it decides when those bytes become a value.
Implementation notes: Rasa's SDK calls the control plane for an allocation grant, frames the value locally, writes it to the granted remote address with libfabric, polls the write completion queue, and sends a commit to the data node. The data node publishes the signed descriptor that future readers resolve.