Momento Rasa
← All posts

Primer · June 2026 · 8 min read

RDMA on AWS: what it is, how it works, and what makes EFA different

Remote Direct Memory Access lets one machine read or write another machine's memory without involving the remote CPU. Most primers stop at the concept. The parts that cost us time were further down: which instances support a one-sided read, why you cannot test any of this on a laptop, and what the remote key does and does not protect.

Quick primer on RDMA

A network card on one host transfers data directly to or from memory on another host, bypassing the operating system kernel on the data path and, for one-sided operations, the remote CPU entirely. Two things do the work. The first is kernel bypass: applications post operations straight to the NIC from user space, with no system calls, no kernel network stack, and no per-packet interrupt on the hot path. The second is zero copy: the NIC moves bytes between the wire and application memory by DMA, with no intermediate buffer copies. The result is single-digit-microsecond latency and line-rate throughput at low CPU cost, because the work a TCP/IP stack normally spends on copies, syscalls, interrupts, and protocol processing is offloaded to the adapter.

How it works

The mechanics are consistent across RDMA fabrics. An application first registers a region of its memory with the NIC, which pins those pages and returns two keys: a local key for its own use and a remote key (rkey) that a peer must present to touch the region. That rkey is the basic protection boundary. Communication then flows through a queue pair, a send queue and a receive queue, alongside a completion queue: the application posts a work request describing an operation, the NIC carries it out, and the application polls a completion to learn it finished.

Two-sided and one-sided operations are different animals. A two-sided operation (SEND/RECV) is a message: both CPUs participate, and the receiver must have posted a buffer in advance. A one-sided operation (RDMA READ or WRITE) names a remote address and rkey, and the NIC performs the transfer on its own; the remote CPU is not involved and need not even know it happened. One-sided reads are what let a server answer a request without spending a core. Applications reach all of this through the verbs API (libibverbs) or, more commonly today, the higher-level libfabric (OFI) library, often with MPI or NCCL above it.

The classic fabrics

Before the cloud, RDMA meant one of three transports:

  • InfiniBand, a purpose-built lossless fabric that dominates HPC.
  • RoCE (RDMA over Converged Ethernet), which runs RDMA over Ethernet and, in v2, is routable.
  • iWARP, which layers RDMA over TCP/IP.

They share an assumption: an ordered, near-lossless network. InfiniBand provides it by design; RoCE typically needs priority flow control to approximate it. That assumption is hard to maintain across a large, multi-path, multi-tenant cloud network.

How AWS does it: EFA and SRD

AWS exposes RDMA through the Elastic Fabric Adapter (EFA), a network device you enable on supported instances. EFA keeps the OS-bypass data path of classic RDMA, but it replaces the transport. Instead of an ordered reliable connection, EFA uses SRD (Scalable Reliable Datagram), and that choice drives most of the differences. SRD guarantees delivery but not ordering, leaving reassembly to the layer above. Giving up in-order delivery is what lets it spray packets across many network paths at once, so a single flow can use the full bisection bandwidth and route around congested or failed links, which is how cloud networks behave. Because it expects loss and reordering, EFA needs no specially tuned lossless fabric or flow control to perform well. It is driven through libfabric's efa provider (with a verbs-compatible path as well), so most software runs libfabric, MPI, or aws-ofi-nccl on top.

A few practical constraints fall out of this design:

  • EFA traffic is not IP-routed. Peers communicate within a subnet and Availability Zone, ideally inside a cluster placement group for the lowest latency.
  • The EFA device must be enabled at launch. You cannot add it to a running plain network interface.
  • The security group must allow all traffic to and from itself, inbound and outbound. A plain allow-all egress to the internet is not sufficient for EFA SRD.

One-sided RDMA read is what lets a service answer straight from registered memory without waking a remote CPU, and it is the capability you have to check for. It arrived with EFA protocol v4 (libfabric 1.10), gated on firmware, the EFA kernel module, and rdma-core, which the EFA installer handles. What it is not gated on is anything you can infer from the instance name.

Which EC2 instances support EFA

EFA is available on a broad and growing set of instance types, generally the larger sizes of the network-optimized, HPC, and accelerated families. Representative examples:

  • Network-optimized (the n / gn suffix): c5n, c6in, c6gn, c7gn, c8gn, m5n, m6in, r6in and similar, usually only on the larger sizes and .metal.
  • HPC: hpc6a, hpc6id, hpc7g, hpc7a.
  • Accelerated / ML: p3dn.24xlarge, p4d and p4de, p5 and p5e, trn1 and trn1n.

The exact list changes by region and over time, so AWS's EFA documentation is the source of truth. But "supports EFA" and "supports a one-sided read" are different questions, and the second does not follow the naming:

  • Nitro v3 / EFA v1 parts (c5n, i3en, m5n) have EFA and no RDMA read at all.
  • p4d and p4de do read but not write.
  • Some Graviton HPC parts (c7gn, hpc7g) are read-only too.
  • Most Nitro v4 and later have both.

So probe at runtime. Ask for FI_READ and FI_REMOTE_READ in your fi_info hints and confirm fi_getinfo hands them back. Do not branch on the instance type string. Otherwise: EFA is usually limited to the larger sizes of a family and bare metal, all nodes belong to one cluster placement group in a single Availability Zone, and the device is opted in at launch.

Why RDMA is hard

The speed is not free; RDMA pushes work the operating system used to hide back onto the application. Memory must be registered and pinned before the NIC will touch it, so you manage buffer pools and their lifetime by hand rather than handing the NIC an arbitrary pointer. And because a one-sided transfer never wakes the remote CPU, there is nothing on the far side to validate a request, take a lock, or choose the right version at access time; all of that coordination has to happen out of band, before the transfer.

Memory safety has no backstop either. A one-sided read can land on memory the remote has freed or reused, and the remote CPU cannot refuse a transfer already in flight, so correctness depends on explicit fencing such as rotating remote keys, leases, and immutable buffers. The NIC's own resources (queue pairs, completion queues, registered regions) are finite, and reliable-connected models scale poorly to many peers, which is one reason AWS chose the connectionless SRD. It is low-level to operate, too: peers exchange addresses and keys out of band (there is no DNS for memory regions), errors surface as completion status codes, and because the kernel is bypassed, tools like tcpdump never see the traffic.

Four things that surprised us

These are not in the concept explanations, and all four cost us time.

  • There is no emulator. EFA is OS-bypass over real Nitro hardware, so you cannot develop against it on a laptop or a non-EFA instance. We made our benchmark abort loudly when fi_info -p efa returns no provider rather than fall back to a TCP provider. A TCP number in an RDMA evaluation is worse than no number.
  • You do not choose your remote keys. The EFA provider sets FI_MR_PROV_KEY, so rkeys are provider-chosen. Read one back with fi_mr_key() after every registration and re-exchange it. You cannot pin a stable key and cache it forever.
  • Remote addressing is by virtual address. FI_MR_VIRT_ADDR means the client holds the server's actual VA for the arena. Re-map it, or let ASLR move it, and every descriptor a client is holding points at nothing.
  • Max transfer size is device-dependent. Messages cap out near the MTU, roughly 8 KiB, and RMA operations have their own device limit. Query fi_getopt(FI_OPT_MAX_RMA_SIZE) at runtime and segment large values yourself. A megabyte is not one operation on the wire.

RDMA versus a normal key/value GET

Put a one-sided RDMA read next to an ordinary networked GET (TCP/HTTP, or a cache like Redis or Valkey over the network). With a normal GET the client sends a request and the server does the thinking: its CPU wakes, parses the request, looks up the value, copies it into a socket buffer, the kernel runs TCP and copies again, and the NIC sends. The server guides every call, which is what lets it locate the data, validate the request, and authorize it; the price is that both kernels and a server core are on the path for every request, latency is dominated by software, and throughput is capped by cores.

A one-sided RDMA read inverts that. There is no server to guide it. The client must already know the remote host's memory layout, the exact address and rkey where the value lives, and it goes and reads that spot in the box. The remote NIC DMAs the bytes from registered memory straight back while the remote CPU does nothing, so latency is mostly wire time and throughput scales with the NIC rather than with cores. The catch is everything the server used to do for free: a GET is self-describing and safe, while a one-sided read is just "move bytes from this address." Those responsibilities move off the data path instead, into a control plane that tells the client where the value is, immutability and fencing so the bytes cannot change or vanish mid-read, and keys that authorize access.

Why it matters

RDMA collapses the cost of moving bytes between machines: microsecond latency, line-rate throughput, and, with one-sided operations, a remote server that spends no CPU answering reads. EFA brings that to commodity cloud hardware using a transport designed for the cloud's network rather than a tuned data-center fabric. That combination, one-sided reads over EFA, is the foundation Rasa is built on.

Sources to go deeper: the AWS EFA user guide, the libfabric documentation, and the SRD paper "A Cloud-Optimized Transport Protocol for Elastic and Scalable HPC."