Why mutability is hard with RDMA
A one-sided RDMA read is fast because the remote server is not involved: the NIC moves bytes from registered memory while the remote CPU does nothing. That same property, nobody home on the read path, also makes changing a value in place hard.
The root cause: no one is home
In a normal server, mutation is easy because the server mediates every access. It can take a lock, swap the value, release the lock, and the next reader sees a clean result. A one-sided read has none of that. The reader holds an address and a remote key and asks the NIC to copy whatever is there, right now. There is no server thread to serialize against, no lock the reader and a writer share, and no code on the remote side to say "wait, this is mid-update." Three concrete problems follow.
1. Torn reads
If a writer updates a value in place while a reader is pulling it, the read can return a mix of old and new bytes. The two transfers are independent DMA operations racing over the same memory, and nothing orders them. For a small scalar you might reach for an atomic, but RDMA atomic operations, where a fabric offers them at all, act on a single 64-bit word, nowhere near a multi-kilobyte value, and EFA's operation set is deliberately small. So there is no primitive that atomically swaps a whole value, and no shared lock to fall back on. A reader cannot even tell a torn read happened without help.
2. No validation at access time
With a two-sided request the server validates on every call: it can check that the key still exists, that the caller may read it, and that it is returning the current version. A one-sided read skips all of that. The reader trusts the address and key it was handed earlier, and the NIC honors them whether or not the data behind them is still what the reader expects. If the value was replaced, moved, or freed, the read still returns bytes; they are just the wrong bytes. The check that a normal server does for free has to be reconstructed somewhere else.
3. Reclaiming memory you changed
Suppose you avoid in-place writes by putting the new value somewhere else and repointing future lookups to it. Now you have to reclaim the old copy, and that is its own hard problem. A one-sided read is invisible to the server, so the server cannot know whether a reader is still pulling the old version at this instant. You cannot reference-count readers you cannot see. Free or overwrite the old memory too early and an in-flight reader silently gets corruption. The only safe signal is time and a hardware fence: wait out a lease during which a reader could still be mid-read, then rotate the memory region's remote key so any straggler using the old key gets a protection error and a clean miss instead of bad data.
How systems cope
The usual answers trade in-place mutation away to differing degrees:
- Self-validating values. Stamp each value with a version and a checksum so a reader can detect a torn or stale read and retry. This makes reads safe but not writes; it only tells the reader something went wrong.
- Seqlock-style versioning. The writer bumps a version counter before and after the payload; the reader reads the version, the value, then the version again, and retries if they differ. It works for small, hot fields but the retry window grows with value size and it leans on careful ordering that out-of-order transports complicate.
- Two-sided writes. Keep reads one-sided but route writes through the server CPU, so the mutation path has someone to serialize and validate while the fast path stays CPU-free.
- Immutability. Never mutate in place at all. A change becomes a new, immutable value published under a new identity, with old copies reclaimed later.
Why immutability usually wins
Immutability removes the race instead of managing it. If the bytes at an address never change under a reader, a one-sided read is automatically safe: no lock, no torn read, no version check on the hot path. A write produces a new immutable value, the control plane repoints lookups to it, and the old value is reclaimed only after its lease expires and its remote key is rotated, which is the same fence that makes deletes safe. The cost is explicit and bounded: updates and deletes are lazy rather than instant, you store versions until they are reclaimed, and durability is a cache contract rather than a database guarantee.
Rasa takes that trade. Values are write-once and validated on every read, the control plane resolves each key to a current descriptor, and memory is reclaimed by lease and remote-key rotation. The reward is what made one-sided RDMA attractive in the first place: reads that the server never has to wake up for. For more background, see the RDMA primer.