KV cache in 56µs. TTFT at NIC speed.
Rasa keeps KV cache blocks in registered memory on EFA data nodes. Clients read them directly with one-sided RDMA: a full megabyte chunk in 56µs at p99. Prefixes you have already served come back over the wire, so your GPUs spend their time on new requests instead of repeating old ones.
Rasa makes RDMA simple.
p99s, by chunk size
Remote reads from a c8gn.16xl. Each row is one operating point: the connection count that reaches that size’s ceiling, and the tail you get there.
p99 by chunk size
Your VPC. Your nodes.
Clients and a fleet of data nodes share an EFA placement group per AZ. Valkey is the control plane, not the data plane: it maps keys to descriptors, and the values never flow through it. The data path is libfabric RMA behind a small transport trait, so a second fabric is a provider swap rather than a rewrite.
A Valkey-powered control plane.
Valkey runs the exchange. Every read, write, and delete starts there: it maps each key to its Rasa descriptor and hands the client the address, rkey, and lease it needs, then coordinates slot allocation, node membership, and memory reclamation across the data nodes. Nothing on the read path talks to it except the client, once, before the read.
Read: one descriptor lookup in Valkey, then a one-sided RDMA pull straight from the data node's registered memory. The node's CPU is off the read path.
You never touch the fabric.
Doing RDMA by hand means libfabric, queue pairs, memory registration, remote keys, completion-queue polling, and getting EFA's SRD transport and security groups exactly right. Rasa keeps all of it behind a key/value API.
Rasa owns the fabric code
Memory registration, rkeys, descriptor refresh, leases, and completion polling all stay behind the client API. You never touch a fabric primitive.
EFA set up at deploy
Provider discovery, SRD, address exchange, and the security-group gotchas are configured when you stand it up.
Just three calls
The whole client API is get, put, delete. Addressing, leasing and fencing live behind it.
Throughput saturates the NIC.
The data node serves the full 200 Gbps line rate, at any value size, while sitting at 1.6% CPU utilization. To be precise about that 1.6%: it is a busy-polling progress loop. The NIC serves the reads out of registered memory and they never enter the process.
What it is good at.
Large values, small latency
Built for 128 KiB to 16 MiB chunks, read in tens of microseconds to under a millisecond at p99.
Bandwidth, not cores
Reads bypass the server CPU entirely, so throughput scales with the NIC. One data node serves 200 Gbps of remote reads.
Simple and safe
Three calls: get, put, delete. Values are immutable and validated on every read; memory is reclaimed safely by lease and rkey rotation.
Where Rasa plugs in.
An early preview built in collaboration with AWS. Here is what is in flight and what is coming soon.
LMCache connector
Offload vLLM’s KV cache to Rasa as a remote backend. The existence check and the read address resolve together, so a prefix lookup and its load are one round trip. How it works →
In progressvLLM connector
A direct connector for vLLM’s KV transfer interface, for deployments that do not run LMCache in front of it.
In progressNVMe tiering
Capacity well beyond registered DRAM. The NIC cannot pull a block off NVMe, so the direction flips: the node RDMA-writes the value into your buffer and the completion is the reply. Still one round trip. The write-key path →
Shipping soon, with AWSNIXL backend
Speak NIXL, so any inference engine that already does can reach Rasa without adopting an SDK. Integration should be a config change.
On the roadmapGPUDirect RDMA
Land KV blocks straight in GPU HBM, skipping the hop through host memory and the copy across PCIe. The NIC can already register HBM; this wires it to the read path.
On the roadmapWhat it gives up to go this fast.
One-sided reads are fast because the server is not involved, and that same fact decides what the system can promise.
Write-once, immutable
Values do not change in place; deletes are lazy. Durability is a cache contract: lose a node and you recompute. That immutability lets a one-sided read be served safely without a lock.
Reclaim by time, not locks
A one-sided read is invisible to the server, so in-flight readers cannot be counted. Memory is reclaimed only after leases expire and the remote key is rotated: a hardware fence, not a mutex.
One fabric, done well
Single-tenant, AWS with EFA on Linux, built in collaboration with AWS. We went deep on one fabric to get to production customers sooner. The technique is not EFA-specific: the same read path runs over RoCEv2 or InfiniBand once it is pointed at a different provider.
From the blog.
Try a Tbps+ remote-memory cache in your own AWS account.
Single-tenant, self-hosted, EFA-native. An early preview you benchmark in your own VPC.