How Does Remote Direct Memory Access Work?

How Does Remote Direct Memory Access Work? Explainer

Remote Direct Memory Access, or RDMA, is a way for one computer to move data directly to or from memory on another computer with very little involvement from either CPU. In a conventional network path, an application hands data to the operating system, the networking stack packages it, and the receiving system may copy the payload through several buffers before the destination application can use it. RDMA reduces that copying and lets an RDMA-capable network adapter place the payload into an application-approved memory region. The goal is not to make distance disappear; the bits still cross a network. The advantage is that less CPU time and memory bandwidth are spent shuffling the same bytes inside each machine.

Before a transfer can happen, an application registers a memory region with its RDMA adapter. Registration pins the relevant pages so they will not move while hardware is using them and establishes permissions for the operation. The application exchanges connection information and a limited access key with its peer through a trusted control path. It then posts work requests to queues managed by the adapter. An RDMA write sends local data into an advertised remote buffer. An RDMA read fetches data from a remote buffer into local memory. A send operation delivers a message to a receive buffer prepared by the other side. When hardware finishes, it records a completion so the application knows the buffer is ready.

That queue-based design allows an application to submit work and continue instead of waiting for the CPU to process every packet. The network adapter handles segmentation, transport details, placement, and completion tracking according to the RDMA technology in use. InfiniBand was designed around these ideas, while RoCE carries RDMA over Ethernet and iWARP builds it over TCP. The options differ in their network requirements and congestion behavior, but they share the principle of direct placement. Storage systems, distributed databases, high-performance computing, machine-learning clusters, and Windows SMB Direct can use RDMA when predictable low latency and high throughput justify specialized hardware and tuning.

Direct placement does not mean unrestricted memory access. A remote peer can reach only buffers that have been explicitly registered and exposed with the correct key, bounds, operation type, and protection domain. Keys can be invalidated, and completed buffers can be deregistered. These controls are essential because a mistake in a traditional socket program may corrupt an application buffer, while a mistake in an RDMA program can also involve device-managed access. Implementations must validate sizes, protect connection setup, rotate or revoke access, and keep completion handling synchronized with buffer ownership.

RDMA also shifts work rather than eliminating it. Memory registration has a cost, pinned pages reduce operating-system flexibility, and queue pairs and completion queues consume resources. Ethernet deployments may need careful congestion control and loss management. Small messages may not benefit enough to repay the setup overhead, and badly designed applications can still bottleneck on locks, memory, or storage. The practical payoff appears when large or frequent transfers can remain in registered buffers and the software can keep many operations in flight. In that setting, RDMA lets the network adapter move data where the application needs it while the CPU spends more time on the application itself.