← All posts

Insight · August 2026 · 6 min read

How a one-sided RDMA cache stays correct without trusting the reader

No server runs on the read path. That is why a one-sided read is fast, and it is also why we spent more design time here than on the data path itself. The client computes a remote address and pulls the bytes with nothing in the loop to vet the result. One decision makes that safe rather than reckless: a value, once written, is never changed. Most of what follows is a consequence of that rule.

The referee is gone

On a normal key/value store the server is the referee. It owns the data, it decides what a GET returns, and it can refuse a read that should not happen. A one-sided RDMA read deletes the referee. The data node registers a region of memory with the NIC and then steps aside; the client issues an fi_read and the network card moves the bytes with no software in the loop. That is where the throughput comes from, and it means every guarantee a server would normally enforce has to be enforced somewhere else.

The instinct is to add that enforcement back as a pile of checks. The better move is to remove most of the failure modes outright, so there is less to check. That is the payoff of immutability.

The principle: write once, never mutate

A Rasa value is immutable. A writer allocates a fresh slot, RDMA-writes the full value into it, and only then publishes the descriptor that points at it. Updating a key does not edit those bytes; it writes a brand-new slot for the new value and swaps the descriptor to point there. The original bytes are never written over again for as long as they live. There is no in-place update anywhere in the system.

That one rule removes the two worst failure modes of a server-less read before any validation code runs:

• Torn reads vanish. A reader can only catch a half-written value if something is overwriting a value it might read. Nothing ever overwrites a live value, so there is no torn state to catch. And because the descriptor is published only after the slot is fully written, a key never even resolves to memory that is still being filled.
• Stale-in-place vanishes. Updating a key never touches the old value, so a reader already working from an earlier descriptor keeps reading a complete, consistent earlier value. The two versions live in different memory; readers and writers never contend for the same bytes.

This is the familiar payoff of immutable data, applied to remote memory. If bytes never change underneath you, a reader that bypasses the server can trust what it lands, because the one thing that could have made it wrong, a concurrent in-place write, was designed out rather than guarded against. (It is also why mutability is the hard case; we wrote about why separately.)

What the principle does not cover

Immutability removes tearing and staleness. Two risks are left, and notice that neither is about the bytes changing. They are about the address:

• The address could be forged: a tampered descriptor aiming the NIC at memory it should not read.
• The memory could be reclaimed: immutable values accumulate, so eventually a slot has to be reused, and that reuse is the one moment the never-changes contract is broken.

Add one belt-and-suspenders check (a reader should confirm it landed the value it asked for, immutable or not) and the whole correctness story is three guards: authenticate the address, validate the bytes, and fence reclamation.

The rule we wrote down

Written down plainly, the invariant everything else serves is one sentence:

Everything below exists to make that true without ever tracking individual in-flight reads, which is impossible here because the server CPU never sees one. We substitute two things for read-tracking: time, in the form of leases plus a grace period, and hardware fencing, in the form of rkey rotation. Neither is as precise as a refcount. Together they are sufficient, which is a different and lower bar, and it is the bar we are working to.

Guard 1: the address is a signed capability

A client cannot read what it cannot resolve. To read a key it first fetches that key's descriptor from the control plane: a small record holding the remote address, the rkey that authorizes the read, the length, the version, and a checksum. The descriptor is the only thing that turns a key into an fi_read, so it is the natural place to put authentication.

Each descriptor is stored next to an HMAC tag, {key}:sig = HMAC(tenant_key, descriptor), computed with a per-tenant secret that is passed out of band and never written into the data path. Before the client trusts the address and rkey, it verifies that tag. A descriptor that was forged or tampered with does not verify, and the read never leaves the client. The downgrade move, deleting the signature so the reader falls back to an unsigned descriptor, is closed too: when a tenant key is configured, a missing signature is a reject, not a shrug.

Guard 2: every read validates the bytes it lands

Authenticating the address proves where to read, not what came back. So the bytes are checked too. A value is framed as a 32-byte header plus payload, and the header carries the magic, the key, the version, the payload length, and a CRC32 over the payload. After the read lands, the client validates in order: magic, key, version, length, and finally the checksum over the actual bytes. A mismatch on any of them is a failed read, not a returned value.

The fast path for LMCache skips the inline header to read straight into a pre-registered arena, but it does not skip the check. It lands only the payload and validates its CRC against the checksum carried in the authenticated descriptor. Same CRC routine, same guarantee: the bytes you got are the bytes the writer committed, or you get nothing.

The version field keeps "stale" safe. The descriptor names a version and the bytes carry it; if a slot has moved on, the version in memory no longer matches the version the descriptor promised, and validation fails. A reader chasing an out-of-date descriptor misses. It never gets a confidently wrong answer.

Guard 3: reclamation is fenced in hardware

This is the one place immutability gets hard. Because values are never mutated, the only way to free memory is to stop honoring a whole region of old values at once and reuse it. Reuse is the one moment the never-changes contract is broken, and it is the case we kept coming back to: a client still holding an old descriptor would read the right address with a valid rkey and land whatever new value now occupies that memory. The version check would likely catch it, but "likely" is not the bar for memory safety. So reclamation is fenced in hardware.

Memory is reclaimed by rotating an arena: the node closes the NIC's registration of that region and re-registers the same memory, which mints a new rkey. Every descriptor published with the old rkey is now pointing at a key the NIC no longer honors, so any late read against it is rejected by the card before it touches a byte. The node then deletes those descriptors from the control plane. The single moment immutability is broken and memory becomes reusable is exactly the moment every capability into that memory is revoked. There is no overlap to get wrong.

What happens if we get the window wrong

Worth being concrete about the failure mode, because "it is safe" is easy to assert.

Between unpublishing a block and rotating its region, a client on an old manifest reads the still-correct pre-move copy. After rotation its descriptor carries the old rkey and the read hard-fails. So the version field is a second line of defence during that window, and the ordering of the two mechanisms means the worst case is a still-correct stale copy followed by a clean failure and a refetch. You never read wrong bytes. A reader that crashes mid-lease leaks nothing either; the lease is a Valkey EXPIRE and it lapses.

Why delete can be lazy

These guards are what let delete be cheap. Deleting a key does not scrub memory or rotate an arena. It unpublishes the descriptor so new lookups miss, and it marks the slot quarantined so it drops out of the live set; the bytes and the rkey stay valid until the next rotation reclaims them. That sounds unsafe, and on a store you had to trust it would be. Here it is fine, because nothing trusts a slot's continued existence: a new lookup misses, a held descriptor still validates against bytes that have not changed, and the only thing that frees and re-keys the memory is the rotation that fences it. A delete races nothing.

The same reasoning is why the client's local address cache is safe to keep. It is never a source of truth; it can only ever turn into a miss. Delete and put drop the key from it so the next read re-resolves, and even a stale entry that slips through still has to clear all three guards before its bytes are returned.

Immutability is the load-bearing decision. Because a value is never changed in place, tearing and staleness do not exist, and what is left is small: authenticate the address, validate the bytes, and fence the one moment memory is reused. Correctness lives in those three moments instead of in a server, and the reader can stay untrusted by construction. The property that falls out is the blast radius: when something is wrong, whether a forged pointer, a stale cache entry, or a read that races a rotation, the worst case is a failed read and a recompute. Never a wrong value.