VALKEY-RDMA(7)

💖 VALKEY-RDMA 💖

okay bestie, sit down, hydrate, because we're about to talk about the VALKEY-RDMA(7) page — that's the one in the Valkey Manual — and it's literally so good. RDMA experimental support?? in my Valkey?? it's giving speed, it's giving ✨hardware magic✨, and yes I'm gonna explain every single bit of it to you, no cap.

---

✨ NAME ✨

valkey-RDMA — RDMA experimental support 🦋

---

🌸 DESCRIPTION 🌸

So, real talk: Valkey supports the Remote Direct Memory Access (RDMA) connection type through a Valkey module that you can dynamically load whenever you feel like it. On demand, like ordering fries. 🍟

🎀 Getting Started 🎀

RDMA lets networked computers' main memory just… talk directly to each other?? Bypassing the processors and the operating systems entirely. Like going straight to your bestie's house instead of mailing a letter through the whole postal system. 💅

And the result? RDMA gives you better performance than TCP/IP. Test results indicate that Valkey Over RDMA achieves approximately 2 times higher QPS and lower latency. TWO TIMES. higher. QPS. I'm obsessed. 🔥

but bestie, listen — Valkey Over RDMA is currently supported only on Linux. And before you even think about using it, you must understand how to configure RDMA on your server and client machines. Do your homework first, periodt. 📚

🧚 Running manually 🧚

To run a Valkey server with RDMA mode:

./src/valkey-server --protected-mode no \
     --loadmodule src/valkey-rdma.so bind=192.168.122.100 port=6379

The bind address/port of RDMA can be modified at runtime using this command:

192.168.122.100:6379> CONFIG SET rdma-port 6380

And here's the iconic part — Valkey can run both RDMA and TCP/IP concurrently on the same port:

./src/valkey-server --protected-mode no \
     --loadmodule src/valkey-rdma.so bind=192.168.122.100 port=6379 \
     --port 6379

Or you can just append loadmodule src/valkey-rdma.so bind=192.168.122.100 port=6379 in valkey.conf, and then:

./src/valkey-server valkey.conf

Now. To test whether the RDMA server is working properly, do not use valkey-cli, which (as of Valkey 8.1) does not yet support RDMA. She's not ready yet, don't set her up like that. 😭 Instead, there's a shell script ./runtest-rdma, which simply wraps a Python script ./tests/rdma/run.py. That Python script would start a server locally, start a client locally, and test for ~6000 key-value pairs using RDMA write/read operations. To test remote RDMA connections, specify ./tests/rdma/rdma-test -h $remote_rdma_ip.

🫶 Prerequisites 🫶

Note that the network interface (192.168.122.100 in this example) should support RDMA. To test whether a server supports RDMA or not:

rdma dev show (a new version iproute2 package)

Or:

ibv_devices (ibverbs-utils package of Debian/Ubuntu)

💜 Performance tuning 💜

Okay so the RDMA completion queue will use the completion vector to signal completion events via hardware interrupts. A large number of hardware interrupts can affect CPU performance — she gets overwhelmed, honestly same. It's possible to tune the performance using rdma-comp-vector.

See ibv_create_cq(3) 📖

⭐ Example 1 ⭐

• Pin hardware interrupt vectors [0, 3] to CPU [0, 3].
• Set CPU affinity for valkey to CPU [4, X].
• Any valkey server uses a random RDMA completion vector [-1].

All valkey servers will not affect each other and will be isolated from kernel interrupts. Look at them, coexisting, main character energy each, no drama. 👑

     SYS    SYS    SYS    SYS  VALKEY VALKEY     VALKEY
      |      |      |      |      |      |          |
    CPU0   CPU1   CPU2   CPU3   CPU4   CPU5   ... CPUX
      |      |      |      |
    INTR0  INTR1  INTR2  INTR3

🌈 Example 2 🌈

• Pin hardware interrupt vectors [0, X] to CPU [0, X].
• Set CPU affinity for valkey [M] to CPU [M].
• Valkey server [M] uses RDMA completion vector [M].

A single CPU [M] handles hardware interrupts, the RDMA completion vector [M], and the valkey server [M] within its context only. This avoids overhead and function calls across multiple CPUs, fully isolating each valkey server from one another. Every CPU is in her own lane, thriving, unbothered. 🦋

   VALKEY VALKEY VALKEY VALKEY VALKEY VALKEY     VALKEY
      |      |      |      |      |      |          |
    CPU0   CPU1   CPU2   CPU3   CPU4   CPU5  ...  CPUX
      |      |      |      |      |      |          |
    INTR0  INTR1  INTR2  INTR3  INTR4  INTR5      INTRX

Use 0 and positive numbers to specify the RDMA completion vector, or specify -1 to allow the server to use a random vector for a new connection. The default vector is -1.

---

🦋 Protocol 🦋

The protocol defines the Queue Pairs (QP) type reliable connection (RC), like TCP, communication commands, and payload exchange mechanism. This dependency is based solely on the RDMA (aka Infiniband) specification and is independent of both software (including the OS and user libraries) and hardware (including vendors and low-level transports). Independent!! She doesn't need anybody. 💅

Valkey Over RDMA has control-plane (control messages) and data-plane (payload transfer). Two planes, both slaying. ✈️

🎀 Control message 🎀

Control messages use fixed 32-byte big-endian message structures:

typedef struct ValkeyRdmaFeature {
    /* defined as following Opcodes */
    uint16_t opcode;
    /* select features */
    uint16_t select;
    uint8_t reserved[20];
    /* feature bits */
    uint64_t features;
} ValkeyRdmaFeature;

typedef struct ValkeyRdmaKeepalive {
    /* defined as following Opcodes */
    uint16_t opcode;
    uint8_t reserved[30];
} ValkeyRdmaKeepalive;

typedef struct ValkeyRdmaMemory {
    /* defined as following Opcodes */
    uint16_t opcode;
    uint8_t reserved[14];
    /* address of a transfer buffer which is used to receive remote streaming data,
     * aka 'RX buffer address'. The remote side should use this as 'TX buffer address' */
    uint64_t addr;
    /* length of the 'RX buffer' */
    uint32_t length;
    /* the RDMA remote key of 'RX buffer' */
    uint32_t key;
} ValkeyRdmaMemory;

typedef union ValkeyRdmaCmd {
    ValkeyRdmaFeature feature;
    ValkeyRdmaKeepalive keepalive;
    ValkeyRdmaMemory memory;
} ValkeyRdmaCmd;

⚡ Opcodes ⚡

• GetServerFeature — value 0 — required, get the features offered by Valkey server
• SetClientFeature — value 1 — required, negotiate features and set it to Valkey server
• Keepalive — value 2 — required, detect unexpected orphan connection
• RegisterXferMemory — value 3 — required, tell the RX transfer buffer information to the remote side, and the remote side uses this as TX transfer buffer

Once any new feature and command are introduced into Valkey Over RDMA, the client should detect the new feature VALKEY_RDMA_FEATURE_FOO through the GetServerFeature command, and then use the SetClientFeature command to enable the feature VALKEY_RDMA_FEATURE_FOO. Once VALKEY_RDMA_FEATURE_FOO is negotiated successfully, the optional ValkeyRdmaFoo command will be supported within the connection. Negotiation!! Communication!! We love a healthy relationship. 🫶

🔥 RDMA Operations 🔥

• Send a control message by RDMA ibv_post_send with opcode IBV_WR_SEND with structure ValkeyRdmaCmd.
• Receive a control message by RDMA ibv_post_recv, and the received buffer size should be size of ValkeyRdmaCmd.
• Transfer stream data by RDMA ibv_post_send with opcode IBV_WR_RDMA_WRITE (optional) and IBV_WR_RDMA_WRITE_WITH_IMM (required), to write data segments into a connection by RDMA [WRITE][WRITE][WRITE]...[WRITE WITH IMM], the length of total buffer is described by immediate data (unsigned int 32). For example: a, [WRITE 128 bytes][WRITE 256 bytes][WRITE 128 bytes WITH IMM 512] writes 512 bytes to the remote side, the remote side is notified only once. b, [WRITE 128 bytes WITH IMM 128][WRITE 256 bytes WITH IMM 256][WRITE 128 bytes WITH IMM 128] writes 512 bytes to the remote side, the remote side is notified three times. Both example a and b write the same 512 bytes, example a has better performance, however b is easier to implement.

💅 Maximum WQEs of RDMA 💅

No specific limit, 1024 recommended for WQEs. Flow control for WQE MAY be defined/implemented in the future. May. 👀

🌟 The workflow of this protocol 🌟

                                                                              valkey-server
                                                                              listen RDMA port
             valkey-client
                          -------------------RDMA connect-------------------->
                                                                              accept connection
                          <--------------- Establish RDMA --------------------

                          --------Get server feature [@IBV_WR_SEND] --------->

                          --------Set client feature [@IBV_WR_SEND] --------->
                                                                              setup RX buffer
                          <---- Register transfer memory [@IBV_WR_SEND] ------
          [@ibv_post_recv]
          setup TX buffer
                          ----- Register transfer memory [@IBV_WR_SEND] ----->
                                                                              [@ibv_post_recv]
                                                                              setup TX buffer
                          -- Valkey commands [@IBV_WR_RDMA_WRITE_WITH_IMM] -->
                          <- Valkey response [@IBV_WR_RDMA_WRITE_WITH_IMM] ---
                                            .......
                          -- Valkey commands [@IBV_WR_RDMA_WRITE_WITH_IMM] -->
                          <- Valkey response [@IBV_WR_RDMA_WRITE_WITH_IMM] ---
                                            .......

          RX is full
                          ----- Register transfer memory [@IBV_WR_SEND] ----->
                                                                              [@ibv_post_recv]
                                                                              setup TX buffer
                          <- Valkey response [@IBV_WR_RDMA_WRITE_WITH_IMM] ---
                                            .......

                                                                                  RX is full
                              <---- Register transfer memory [@IBV_WR_SEND] ------
              [@ibv_post_recv]
              setup TX buffer
                              -- Valkey commands [@IBV_WR_RDMA_WRITE_WITH_IMM] -->
                              <- Valkey response [@IBV_WR_RDMA_WRITE_WITH_IMM] ---
                                                .......

                              -------------------RDMA disconnect----------------->
                              <------------------RDMA disconnect------------------

okay bestie, so the Valkey Over RDMA protocol is literally built to move stream data around efficiently, and the way it does that has major overlap with a bunch of mechanisms that showed up in academic papers first 🌸 — though not a one-to-one copy, there are some differences, and we're gonna name-drop the papers like the icons they are:

• Socksdirect: datacenter sockets can be fast and compatible
• LITE Kernel RDMA Support for Datacenter Applications
• FaRM: Fast Remote Memory

---

🦋 How does Valkey use RDMA 🦋

So Valkey supports a connection abstraction framework — think of it as the group chat that hides all the messy stuff: listen/connect/accept/shutdown/read/write, and so on. it's giving "don't worry about it, babe, I got it." 💅

The way this works: connection types get to register themselves into the Valkey core during startup time. and bestie, a connection type can be either Valkey built-in (Ex, TCP/IP and Unix domain socket) or Valkey module (Ex, TLS). both valid, both iconic.

Now, turning on RDMA support means you need to link additional libraries. and here's the key thing — instead of making valkey-server take on an extra dependency on those shared libraries, you build Valkey Over RDMA into a Valkey module. then a user starts valkey-server with that RDMA module, and valkey-server loads the additional shared libraries on demand. chef's kiss. no bloat, all vibes. 🎀

---

💀 Limitations 💀

ngl bestie, read these before you commit, fr fr:

• Valkey Over RDMA is experimental — it may be changed or be removed in any minor or major version. like, no promises, it's giving situationship 😭
• TLS is not supported by Valkey Over RDMA. But it is workable in theory by a certain amount of work. (theoretically slay, practically not today)
• Valkey Over RDMA is supported on Linux only. periodt.
• Not compatible with replication currently — TCP/TLS is needed for replication. so if you want replication, keep that TCP/TLS around, it's not going anywhere 💅
• Depending on different hardware, too many active queue pairs may lead performance drop. so don't be greedy with the queue pairs, bestie, it's a whole vibe check 🙈

---

and that's the page, babes 🫶 this was VALKEY-RDMA(7), page 7, straight from version 9.0.0 and dated 2025-10-21 — mwah, go forth and RDMA responsibly 💖✨