VALKEY-CLUSTER-TUTORIAL(7)

📦
version 9.0.6 · 2 versions
, in fedora 44

VALKEY-CLUSTER-TUTORIAL 💖

okay bestie, buckle UP, this is the valkey-cluster-tutorial(7) page and it's about to be your whole personality ✨

✨ NAME ✨

valkey-cluster-tutorial — Cluster tutorial 🎀

🌸 DESCRIPTION 🌸

So here's the tea: Valkey scales horizontally with a deployment topology called Valkey Cluster. Horizontal scaling, we love to see it. This topic is gonna teach you how to set up, test, and operate Valkey Cluster in production. Like, actually in production, not just vibes. You're gonna learn all about the availability and consistency characteristics of Valkey Cluster from the end user's point of view 👑

If you're planning to run a production Valkey Cluster deployment, or you just wanna understand better how Valkey Cluster works internally, go read valkey-cluster-spec(7) — the Valkey Cluster specification. That's the deep lore fr fr 📚

🦋 Valkey Cluster 101 🦋

Valkey Cluster gives you a way to run a Valkey installation where data is automatically sharded across multiple Valkey nodes. Not me finally understanding sharding. Iconic. Valkey Cluster also provides some degree of availability during partitions — in practical terms, the ability to keep on keeping on when some nodes fail or can't communicate. BUT, and this is a big but, the cluster will become unavailable in the event of larger failures. Like, for example, when the majority of primaries are unavailable. 💀

So, with Valkey Cluster, you get the ability to:

  • Automatically split your dataset among multiple nodes. ✨
  • Continue operations when a subset of the nodes are experiencing failures or are unable to communicate with the rest of the cluster. 🫶

🌈 Valkey Cluster TCP ports 🌈

Every single Valkey Cluster node requires two open TCP connections:

  1. a Valkey TCP port used to serve clients, e.g., 6379
  2. a second port known as the cluster bus port

By default, the cluster bus port is set by adding 10000 to the data port (e.g., 16379). However — and this is the fun part — you can override this in the cluster-port configuration. 💅

The cluster bus is a node-to-node communication channel that uses a binary protocol, which is more suited to exchanging information between nodes because it means little bandwidth and processing time. Nodes use the cluster bus for failure detection, configuration updates, failover authorization, and so forth. She's doing so much, honestly.

Clients should never try to communicate with the cluster bus port, bestie. Use the Valkey command port instead. But like, make sure you open both ports in your firewall, otherwise Valkey cluster nodes won't be able to communicate and everything falls apart 😭

For a Valkey Cluster to work properly you need, for each node:

  1. The client communication port (usually 6379) used to communicate with clients and be open to all the clients that need to reach the cluster, plus all the other cluster nodes that use the client port for key migrations.
  2. The cluster bus port must be reachable from all the other cluster nodes.

If you don't open both TCP ports, your cluster will not work as expected. No cap. 🚨

🐳 Valkey Cluster and Docker 🐳

Currently, Valkey Cluster does not support NATted environments, and in general environments where IP addresses or TCP ports are remapped. Sorry girly, that's just how it is 🙈

Docker uses a technique called port mapping: programs running inside Docker containers may be exposed with a different port compared to the one the program believes to be using. This is useful for running multiple containers using the same ports, at the same time, in the same server. Very slay of Docker, honestly.

To make Docker compatible with Valkey Cluster, you need to use Docker's host networking mode. Please see the --net=host option in the Docker documentation for more information. 👑

🎀 Valkey Cluster data sharding 🎀

Valkey Cluster does not use consistent hashing, but a different form of sharding where every key is conceptually part of what we call a hash slot.

There are 16384 hash slots in Valkey Cluster, and to compute the hash slot for a given key, we simply take the CRC16 of the key modulo 16384. Math girlies rise up 🧚

Every node in a Valkey Cluster is responsible for a subset of the hash slots, so, for example, you may have a cluster with 3 nodes, where:

  • Node A contains hash slots from 0 to 5500.
  • Node B contains hash slots from 5501 to 11000.
  • Node C contains hash slots from 11001 to 16383.

This makes it easy to add and remove cluster nodes. For example, if I want to add a new node D, I need to move some hash slots from nodes A, B, C to D. Similarly, if I want to remove node A from the cluster, I can just move the hash slots served by A to B and C. Once node A is empty, I can remove it from the cluster completely. Spring cleaning but make it distributed systems.

Moving hash slots from a node to another does not require stopping any operations. Therefore, adding and removing nodes, or changing the percentage of hash slots held by a node, requires no downtime. Zero. Zilch. Periodt 💅

Valkey Cluster supports multiple key operations as long as all of the keys involved in a single command execution (or whole transaction, or Lua script execution) belong to the same hash slot. The user can force multiple keys to be part of the same hash slot by using a feature called hash tags.

Hash tags are documented in the Valkey Cluster specification, but the gist is: if there is a substring between {} brackets in a key, only what is inside the string is hashed. For example, the keys user:{123}:profile and user:{123}:account are guaranteed to be in the same hash slot because they share the same hash tag. As a result, you can operate on these two keys in the same multi-key operation. 🫶

👑 Valkey Cluster primary-replica model 👑

To remain available when a subset of primary nodes are failing or are not able to communicate with the majority of nodes, Valkey Cluster uses a primary-replica model where every hash slot has from 1 (the primary itself) to N replicas (N-1 additional replica nodes).

In our example cluster with nodes A, B, C, if node B fails the cluster is not able to continue, since we no longer have a way to serve hash slots in the range 5501-11000. 😭

However, when the cluster is created (or at a later time), we add a replica node to every primary, so that the final cluster is composed of A, B, C that are primary nodes, and A1, B1, C1 that are replica nodes. This way, the system can continue if node B fails. Backup besties!!

Node B1 replicates B, and B fails, the cluster will promote node B1 as the new primary and will continue to operate correctly. ✨

However, note that if nodes B and B1 fail at the same time, Valkey Cluster will not be able to continue to operate. 💀

🌸 Valkey Cluster consistency guarantees 🌸

Valkey Cluster does not guarantee strong consistency. In practical terms this means that under certain conditions it is possible that Valkey Cluster will lose writes that were acknowledged by the system to the client. Yes, really. She said what she said.

The first reason why Valkey Cluster can lose writes is because it uses asynchronous replication. This means that during writes the following happens:

  • Your client writes to the primary B.
  • The primary B replies OK to your client.
  • The primary B propagates the write to its replicas B1, B2 and B3.

As you can see, B does not wait for an acknowledgement from B1, B2, B3 before replying to the client, since this would be a prohibitive latency penalty for Valkey. So if your client writes something, B acknowledges the write, but crashes before being able to send the write to its replicas, one of the replicas (that did not receive the write) can be promoted to primary, losing the write forever. 😭

This is very similar to what happens with most databases that are configured to flush data to disk every second, so it is a scenario you are already able to reason about because of past experiences with traditional database systems not involving distributed systems. Similarly you can improve consistency by forcing the database to flush data to disk before replying to the client, but this usually results in prohibitively low performance. That would be the equivalent of synchronous replication in the case of Valkey Cluster.

Basically, there is a trade-off to be made between performance and consistency. Pick your fighter 🥊

Valkey Cluster has support for synchronous writes when absolutely needed, implemented via the WAIT command. This makes losing writes a lot less likely. However, note that Valkey Cluster does not implement strong consistency even when synchronous replication is used: it is always possible, under more complex failure scenarios, that a replica that was not able to receive the write will be elected as primary.

There is another notable scenario where Valkey Cluster will lose writes, that happens during a network partition where a client is isolated with a minority of instances including at least a primary.

Take as an example our 6 nodes cluster composed of A, B, C, A1, B1, C1, with 3 primaries and 3 replicas. There is also a client, that we will call Z1.

After a partition occurs, it is possible that in one side of the partition we have A, C, A1, B1, C1, and in the other side we have B and Z1.

Z1 is still able to write to B, which will accept its writes. If the partition heals in a very short time, the cluster will continue normally. However, if the partition lasts enough time for B1 to be promoted to primary on the majority side of the partition, the writes that Z1 has sent to B in the meantime will be lost. 🙈

Note: There is a maximum window to the amount of writes Z1 will be able to send to B: if enough time has elapsed for the majority side of the partition to elect a replica as primary, every primary node in the minority side will have stopped accepting writes.

This amount of time is a very important configuration directive of Valkey Cluster, and is called the node timeout. ⏰

After node timeout has elapsed, a primary node is considered to be failing, and can be replaced by one of its replicas. Similarly, after node timeout has elapsed without a primary node to be able to sense the majority of the other primary nodes, it enters an error state and stops accepting writes. 🚨

⭐ Valkey Cluster configuration parameters ⭐

We are about to create an example cluster deployment. Before we continue, let's introduce the configuration parameters that Valkey Cluster introduces in the valkey.conf file. 🎀

  • cluster-enabled <yes/no> — If yes, enables Valkey Cluster support in a specific Valkey instance. Otherwise the instance starts as a standalone instance as usual.
  • cluster-config-file <filename> — Note that despite the name of this option, this is not a user editable configuration file, but the file where a Valkey Cluster node automatically persists the cluster configuration (the state, basically) every time there is a change, in order to be able to re-read it at startup. The file lists things like the other nodes in the cluster, their state, persistent variables, and so forth. Often this file is rewritten and flushed on disk as a result of some message reception.

more parts incoming, don't go anywhere bestie 💜

  • cluster-node-timeout <milliseconds> — bestie this is the max amount of time a Valkey Cluster node can be unreachable before everyone's like "okay she's gone" and it counts as failing 😭. If a primary isn't reachable for longer than this, its replicas will fail it over. This knob also runs a bunch of other important stuff in Valkey Cluster, no cap — notably, any node that can't reach the majority of primary nodes for this long will straight up stop accepting queries. 🛑
  • cluster-replica-validity-factor <factor> — set this to zero and a replica will always think it's valid, so it'll always try to failover a primary, no matter how long the link between primary and replica was disconnected. 🙈 If the value is positive, a max disconnection time gets calculated as the node timeout value multiplied by the factor you gave, and if the node is a replica, it won't try to start a failover if the primary link was disconnected for longer than that. Example time: node timeout set to 5 seconds and validity factor set to 10 means a replica disconnected from the primary for more than 50 seconds won't try to failover its primary. ✨ But heads up — any value other than zero may result in Valkey Cluster being unavailable after a primary failure if there's no replica able to failover it. In that case the cluster only comes back to being available when the original primary rejoins the cluster. 💀
  • cluster-migration-barrier <count> — the minimum number of replicas a primary stays connected with, so that another replica can migrate to a primary that's no longer covered by any replica. Check the replica migration section in this tutorial for the tea. 🫖
  • cluster-require-full-coverage <yes/no> — if this is yes (which it is by default), the cluster stops accepting writes if some percentage of the key space isn't covered by any node. Set it to no and the cluster will still serve queries even if only requests about a subset of keys can be processed. 💅
  • cluster-allow-reads-when-down <yes/no> — if this is no (default, bestie), a node in a Valkey Cluster will stop serving all traffic when the cluster is marked as failed — either when a node can't reach a quorum of primaries or when full coverage isn't met. This prevents reading potentially inconsistent data from a node that's unaware of changes in the cluster. 🧠 You can set it to yes to allow reads from a node during the fail state, which is super useful for apps that want to prioritize read availability but still want to prevent inconsistent writes. It can also be used when you're running Valkey Cluster with only one or two shards, since it lets the nodes keep serving writes when a primary fails but automatic failover is impossible. 🔥

🌈 Create and use a Valkey Cluster 🌈

Okay so to create and use a Valkey Cluster, follow these steps:

  • Create a Valkey Cluster
  • Interact with the cluster
  • Write an example app with redis-rb-cluster
  • Reshard the cluster
  • A more interesting example application
  • Test the failover
  • Manual failover
  • Add a new node
  • Remove a node
  • Replica migration
  • Upgrade nodes in a Valkey Cluster
  • Migrate to Valkey Cluster

But first, get familiar with the requirements for creating a cluster. 👑

🎀 Requirements to create a Valkey Cluster 🎀

To create a cluster, the first thing you need is a few empty Valkey instances running in cluster mode. ✨

At minimum, set the following directives in the valkey.conf file:

port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

To enable cluster mode, set the cluster-enabled directive to yes. Every instance also contains the path of a file where the configuration for this node is stored, which by default is nodes.conf. This file is never touched by humans 🙅 — it's simply generated at startup by the Valkey Cluster instances, and updated every time it's needed.

Note that the minimal cluster that works as expected must contain at least three primary nodes. For deployment, we strongly recommend a six-node cluster, with three primaries and three replicas. 💖

You can test this locally by creating the following directories named after the port number of the instance you'll run inside any given directory.

For example:

mkdir cluster-test
cd cluster-test
mkdir 7000 7001 7002 7003 7004 7005

Create a valkey.conf file inside each of the directories, from 7000 to 7005. As a template for your configuration file just use the small example above, but make sure to replace the port number 7000 with the right port number according to the directory name. 📝

You can start each instance as follows, each running in a separate terminal tab:

cd 7000
valkey-server ./valkey.conf

You'll see from the logs that every node assigns itself a new ID:

[82462] 26 Nov 11:56:55.329 * No cluster configuration found, I'm 97a3a64667477371c4479320d683e4c8db5858b1

This ID will be used forever by this specific instance, so the instance has a unique name in the context of the cluster. Every node remembers every other node using these IDs, and not by IP or port. IP addresses and ports may change, but the unique node identifier will never change for all the life of the node. We call this identifier simply Node ID. 🌟

💅 Create a Valkey Cluster 💅

Now that we have a number of instances running, you need to create your cluster by writing some meaningful configuration to the nodes.

You can configure and execute individual instances manually or use the create-cluster script. Let's go over how you do it manually. 🛠️

To create the cluster, run:

valkey-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 \
127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \
--cluster-replicas 1

The command used here is create, since we want to create a new cluster. The option --cluster-replicas 1 means that we want a replica for every primary created. 💖

The other arguments are the list of addresses of the instances I want to use to create the new cluster.

valkey-cli will propose a configuration. Accept the proposed configuration by typing yes. The cluster will be configured and joined, which means that instances will be bootstrapped into talking with each other. Finally, if everything has gone well, you'll see a message like this:

[OK] All 16384 slots covered

This means that there is at least one primary instance serving each of the 16384 available slots. 🔥

If you don't want to create a Valkey Cluster by configuring and executing individual instances manually as explained above, there is a much simpler system (but you'll not learn the same amount of operational details). 😌

Find the utils/create-cluster directory in the Valkey distribution. There is a script called create-cluster inside (same name as the directory it is contained into), it's a simple bash script. In order to start a 6 nodes cluster with 3 primaries and 3 replicas just type the following commands:

  1. create-cluster start
  2. create-cluster create

Reply to yes in step 2 when the valkey-cli utility wants you to accept the cluster layout. ✅

You can now interact with the cluster, the first node will start at port 30001 by default. When you are done, stop the cluster with:

  1. create-cluster stop

Please read the README inside this directory for more information on how to run the script. 📖

🦋 Interact with the cluster 🦋

To connect to Valkey Cluster, you'll need a cluster-aware Valkey client. See the documentation for your client of choice to determine its cluster support. 💜

You can also test your Valkey Cluster using the valkey-cli command line utility:

$ valkey-cli -c -p 7000
127.0.0.1:7000> set foo bar
-> Redirected to slot [12182] located at 127.0.0.1:7002
OK
127.0.0.1:7002> set hello world
-> Redirected to slot [866] located at 127.0.0.1:7000
OK
127.0.0.1:7000> get foo
-> Redirected to slot [12182] located at 127.0.0.1:7002
"bar"
127.0.0.1:7002> get hello
-> Redirected to slot [866] located at 127.0.0.1:7000
"world"
Note: if you created the cluster using the script, your nodes may listen on different ports, starting from 30001 by default. 🙈

The valkey-cli cluster support is very basic, so it always uses the fact that Valkey Cluster nodes are able to redirect a client to the right node. A serious client is able to do better than that, and cache the map between hash slots and nodes addresses, to directly use the right connection to the right node. 🧠 The map is refreshed only when something changed in the cluster configuration — for example after a failover, or after the system administrator changed the cluster layout by adding or removing nodes. Slay. ✨

🌸 Write an example app with redis-rb-cluster 🌸

Before going forward showing how to operate the Valkey Cluster — doing things like a failover, or a resharding — we need to create some example application, or at least be able to understand the semantics of a simple Valkey Cluster client interaction. 💅

In this way we can run an example and at the same time try to make nodes failing, or start a resharding, to see how Valkey Cluster behaves under real world conditions. It is not very helpful to see what happens while nobody is writing to the cluster. 😭

This section explains some basic usage of redis-rb-cluster showing two examples. The first is the following, and is the example.rb file inside the redis-rb-cluster distribution:

require './cluster'

if ARGV.length != 2
    startup_nodes = [
        {:host => "127.0.0.1", :port => 7000},
        {:host => "127.0.0.1", :port => 7001}
    ]
else
    startup_nodes = [
        {:host => ARGV[0], :port => ARGV[1].to_i}
    ]
end

rc = RedisCluster.new(startup_nodes,32,:timeout => 0.1)

last = false

while not last
    begin
        last = rc.get("__last__")
        last = 0 if !last
    rescue => e
        puts "error #{e.to_s}"
        sleep 1
    end
end
((last.to_i+1)..1000000000).each{|x|
    begin
        rc.set("foo#{x}",x)
        puts rc.get("foo#{x}")
        rc.set("__last__",x)
    rescue => e
        puts "error #{e.to_s}"
    end
    sleep 0.1
}

okay so this app does the simplest thing imaginable: it sets keys shaped like foo<number> to <number>, one after the other, no thoughts head empty. so when you run the program you get this adorable lil stream of commands:

  • SET foo0 0
  • SET foo1 1
  • SET foo2 2
  • And so forth...

The program looks more complicated than it usually should, and that's on purpose: it's built to show errors on your screen instead of just dipping out with an exception, so every single operation it does against the cluster is wrapped in begin rescue blocks. safety first bestie 🫶

Line 14 is the first genuinely interesting line in the program. it creates the Valkey Cluster object, and the arguments it takes are: a list of startup nodes, the maximum number of connections the object is allowed to take against different nodes, and finally the timeout after which a given operation counts as failed. 💅

The startup nodes do not have to be all the nodes in the cluster, okay? the only thing that matters is that at least one node is reachable. also note that redis-rb-cluster updates that list of startup nodes the moment it manages to connect with the first node. expect that same behavior from any other serious client, fr fr.

Now that we've got the Valkey Cluster object instance sitting in the rc variable, we can use the object exactly like a normal Valkey object instance. 👑

That's literally what happens on lines 18 to 26: when we restart the example we don't wanna start all over again from foo0, so we store the counter inside Valkey itself. the code above is built to read that counter, or, if the counter doesn't exist, to assign it the value of zero.

But notice how it's a while loop — we wanna try again and again even if the cluster is down and returning errors. normal applications do not need to be this careful, i promise you.

Lines 28 to 37 kick off the main loop, where the keys get set or an error gets displayed.

Note the sleep call at the end of the loop. in your own tests you can totally remove the sleep if you wanna write to the cluster as fast as possible (relatively speaking, of course — it's still a busy loop with no real parallelism, so you'll get the usual 10k ops/second in the best of conditions).

Normally writes get slowed down just so the example application is easier for humans to follow. considerate king behavior ngl.

Starting the application gives you this:

ruby ./example.rb
1
2
3
4
5
6
7
8
9
^C (I stopped the program here)

Not the most interesting program ever, and we'll use a better one in a sec, but we can already peep what happens during a resharding while the program is running. 👀

🎀 Reshard the cluster 🎀

Now we're ready to try a cluster resharding. 🎉 to do this, keep the example.rb program running, so you can see if there's any impact on the program while it's going. also, you might wanna comment out that sleep call so there's a more serious write load happening during the resharding.

Resharding basically means moving hash slots from one set of nodes to another set of nodes. like cluster creation, it's accomplished with the valkey-cli utility.

To start a resharding, just type:

valkey-cli --cluster reshard 127.0.0.1:7000

You only need to specify a single node — valkey-cli will find all the other nodes automatically. 💅

Currently valkey-cli is only able to reshard with the administrator support, so you can't just say "move 5% of slots from this node to that other one" (even though that's pretty trivial to implement). so it starts with questions. the first one is how much of a resharding you wanna do:

How many slots do you want to move (from 1 to 16384)?

We can try to reshard 1000 hash slots — that should already contain a non-trivial amount of keys if the example is still running without the sleep call.

Then valkey-cli needs to know the target of the resharding, aka the node that's gonna receive the hash slots. i'll use the first primary node, 127.0.0.1:7000, but i need to specify the Node ID of the instance. this was already printed in a list by valkey-cli, but i can always find the ID of a node with the following command if i need to:

$ valkey-cli -p 7000 cluster nodes | grep myself
97a3a64667477371c4479320d683e4c8db5858b1 :0 myself,master - 0 0 0 connected 0-5460

Ok so my target node is 97a3a64667477371c4479320d683e4c8db5858b1. 📌

Now you'll get asked which nodes you wanna take those keys from. i'll just type all so it takes a bit of hash slots from all the other primary nodes.

After the final confirmation you'll see a message for every slot that valkey-cli is about to move from one node to another, and a dot gets printed for every actual key that gets moved from one side to the other. cute lil progress bar energy 🫶

While the resharding is in progress you should be able to see your example program running unaffected. you can stop and restart it multiple times during the resharding if that's your vibe.

At the end of the resharding, you can test the health of the cluster with this command:

valkey-cli --cluster check 127.0.0.1:7000

All the slots will be covered as usual, but this time the primary at 127.0.0.1:7000 will have more hash slots — something around 6461.

Resharding can be performed automatically without needing to manually enter the parameters in an interactive way. that's possible with a command line like the following:

valkey-cli --cluster reshard <host>:<port> --cluster-from <node-id> --cluster-to <node-id> --cluster-slots <number of slots> --cluster-yes

This lets you build some automatism if you're likely to reshard often, however currently there is no way for valkey-cli to automatically rebalance the cluster by checking the distribution of keys across the cluster nodes and intelligently moving slots around as needed. that feature will be added in the future. 🔮

The --cluster-yes option tells the cluster manager to automatically answer "yes" to the command's prompts, which lets it run in a non-interactive mode. note that this option can also be activated by setting the REDISCLI_CLUSTER_YES environment variable. 🌈

🦋 A more interesting example application 🦋

The example application we wrote earlier is not very good, bestie. 😭 it writes to the cluster in a super simple way without even checking whether what got written is the right thing.

From our point of view, the cluster receiving the writes could literally just always write the key foo to 42 on every operation and we would not notice at all. 💀

So in the redis-rb-cluster repository there's a more interesting application called consistency-test.rb. it uses a set of counters, 1000 by default, and sends INCR commands to increment them.

However, instead of just writing, the application does two extra things:

  • When a counter is updated using INCR, the application remembers the write.
  • It also reads a random counter before every write, and checks if the value is what we expected it to be, comparing it against the value it has in memory.

What this means is that this application is a simple consistency checker, and it's able to tell you if the cluster lost some write, or if it accepted a write that we never received an acknowledgment for. in the first case we'll see a counter with a value smaller than the one we remember, while in the second case the value will be greater. iykyk 🔍

Running the consistency-test application produces a line of output every second:

$ ruby consistency-test.rb
925 R (0 err) | 925 W (0 err) |
5030 R (0 err) | 5030 W (0 err) |
9261 R (0 err) | 9261 W (0 err) |
13517 R (0 err) | 13517 W (0 err) |
17780 R (0 err) | 17780 W (0 err) |
22025 R (0 err) | 22025 W (0 err) |
25818 R (0 err) | 25818 W (0 err) |

The line shows the number of Reads and Writes performed, plus the number of errors (queries that weren't accepted because of errors, since the system was not available).

If some inconsistency is found, new lines get added to the output. this is what happens, for example, if i reset a counter manually while the program is running:

$ valkey-cli -h 127.0.0.1 -p 7000 set key_217 0
OK

(in the other tab I see...)

94774 R (0 err) | 94774 W (0 err) |
98821 R (0 err) | 98821 W (0 err) |
102886 R (0 err) | 102886 W (0 err) | 114 lost |
107046 R (0 err) | 107046 W (0 err) | 114 lost |

When i set the counter to 0 the real value was 114, so the program reports 114 lost writes (INCR commands that are not remembered by the cluster).

This program is way more interesting as a test case, so we'll use it to test the Valkey Cluster failover. 🔥

💖 Test the failover 💖

To trigger the failover, the simplest thing we can do (which is also the semantically simplest failure that can occur in a distributed system) is to crash a single process — in our case, a single primary.

Note: during this test, you should have a tab open with the consistency test application running.

We can identify a primary and crash it with the following command:

$ valkey-cli -p 7000 cluster nodes | grep master
3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 127.0.0.1:7001 master - 0 1385482984082 0 connected 5960-10921
2938205e12de373867bf38f1ca29d31d0ddb3e46 127.0.0.1:7002 master - 0 1385482983582 0 connected 11423-16383
97a3a64667477371c4479320d683e4c8db5858b1 :0 myself,master - 0 0 0 connected 0-5959 10922-11422

Ok, so 7000, 7001, and 7002 are primaries. let's crash node 7002 with the DEBUG SEGFAULT command:

$ valkey-cli -p 7002 debug segfault
Error: Server closed the connection

Now we can look at the output of the consistency test to see what it reported.

18849 R (0 err) | 18849 W (0 err) |
23151 R (0 err) | 23151 W (0 err) |
27302 R (0 err) | 27302 W (0 err) |

... many error warnings here ...

29659 R (578 err) | 29660 W (577 err) |
33749 R (578 err) | 33750 W (577 err) |
37918 R (578 err) | 37919 W (577 err) |
42077 R (578 err) | 42078 W (577 err) |

...okay so picture this 💅 during that failover the system straight up could not accept 578 reads and 577 writes, and yet — and this is the part that makes me feral — no inconsistency was created in the database. ✨

If that sounds sus to you, bestie, same, because in the first part of this tutorial we literally said Valkey Cluster can lose writes during the failover, on account of it using asynchronous replication. But what we did not say is that it's not very likely to happen — because Valkey sends the reply to the client and the commands to replicate to the replicas about at the same time, so there's a very small window to lose data. Hard to trigger ≠ impossible though. So this does not change the consistency guarantees Valkey Cluster provides. periodt. 💀

Now let's check what the cluster setup looks like after the failover (little side note: in the meantime I restarted the crashed instance so it rejoins the cluster as a replica):

$ valkey-cli -p 7000 cluster nodes
3fc783611028b1707fd65345e763befb36454d73 127.0.0.1:7004 slave 3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 0 1385503418521 0 connected
a211e242fc6b22a9427fed61285e85892fa04e08 127.0.0.1:7003 slave 97a3a64667477371c4479320d683e4c8db5858b1 0 1385503419023 0 connected
97a3a64667477371c4479320d683e4c8db5858b1 :0 myself,master - 0 0 0 connected 0-5959 10922-11422
3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 127.0.0.1:7005 master - 0 1385503419023 3 connected 11423-16383
3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 127.0.0.1:7001 master - 0 1385503417005 0 connected 5960-10921
2938205e12de373867bf38f1ca29d31d0ddb3e46 127.0.0.1:7002 slave 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 0 1385503418016 3 connected

So rn the primaries are running on ports 7000, 7001 and 7005. And the one that used to be a primary — that's the Valkey instance running on port 7002 — is now a replica of 7005. glow up? glow down? you decide 🦋

The output of CLUSTER NODES may look intimidating ngl, but it's actually pretty simple, and it's composed of the following tokens:

  • Node ID
  • ip:port
  • flags: master, replica, myself, fail, ...
  • if it is a replica, the Node ID of the master
  • Time of the last pending PING still waiting for a reply.
  • Time of the last PONG received.
  • Configuration epoch for this node (see the Cluster specification).
  • Status of the link to this node.
  • Slots served...

🌸 Manual failover 🌸

Sometimes it's genuinely useful to force a failover without actually causing any problem on a primary. Like, say you wanna upgrade the Valkey process of one of the primary nodes — bestie, failing it over first so it turns into a replica is such a good idea, minimal impact on availability, chef's kiss 🎀

Manual failovers are supported by Valkey Cluster using the CLUSTER FAILOVER command, and — important — it must be executed in one of the replicas of the primary you want to failover. 👑

Manual failovers are special, and they're safer compared to failovers resulting from actual primary failures. The way they work avoids data loss in the process, by switching clients from the original primary to the new primary only when the system is sure that the new primary processed all the replication stream from the old one. no cap.

This is what you see in the replica log when you perform a manual failover:

# Manual failover user request accepted.
# Received replication offset for paused primary manual failover: 347540
# All primary replication stream processed, manual failover can start.
# Start of election delayed for 0 milliseconds (rank #0, offset 347540).
# Starting a failover election for epoch 7545.
# Failover election won: I'm the new primary.

Clients sending write commands to the primary get blocked during the failover. When the primary sends its replication offset to the replica, the replica waits to reach the offset on its side. Once the replication offset is reached, the failover starts, and the old primary is informed about the configuration switch. And when the switch is complete? Clients get unblocked on the old primary and redirected to the new primary. slay. ✨

Note: To promote a replica to primary, it must first be known as a replica by a majority of the primaries in the cluster. Otherwise, it cannot win the failover election. If the replica has just been added to the cluster (see Add a new node as a replica), you may need to wait a while before sending the CLUSTER FAILOVER command, to make sure the primaries in the cluster are aware of the new replica. ⏳

🎀 Add a new node 🎀

Adding a new node is basically the process of adding an empty node and then moving some data into it (if it's a new primary), or telling it to set up as a replica of a known node (if it's a replica).

We'll show both, starting with the addition of a new primary instance. 💅

In both cases, the first step you gotta do is add an empty node.

This is as simple as starting a new node on port 7006 (we already used 7000 to 7005 for our existing 6 nodes) with the same configuration used for the other nodes, except for the port number. So here's what you should do to conform with the setup we used for the previous nodes:

  • Create a new tab in your terminal application.
  • Enter the cluster-test directory.
  • Create a directory named 7006.
  • Create a valkey.conf file inside, similar to the one used for the other nodes but using 7006 as port number.
  • Finally start the server with ../valkey-server ./valkey.conf

At this point the server should be running. 🫶

Now we can use valkey-cli as usual to add the node to the existing cluster.

valkey-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000

See how I used the add-node command specifying the address of the new node as the first argument, and the address of a random existing node in the cluster as the second argument? 👀

In practical terms, valkey-cli here did very little to help us — it just sent a CLUSTER MEET message to the node, something you can also accomplish manually. However, valkey-cli also checks the state of the cluster before it operates, so it's a good idea to perform cluster operations always via valkey-cli even when you know how the internals work. iykyk.

Now we can connect to the new node to see if it really joined the cluster:

valkey 127.0.0.1:7006> cluster nodes
3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 127.0.0.1:7001 master - 0 1385543178575 0 connected 5960-10921
3fc783611028b1707fd65345e763befb36454d73 127.0.0.1:7004 slave 3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 0 1385543179583 0 connected
f093c80dde814da99c5cf72a7dd01590792b783b :0 myself,master - 0 0 0 connected
2938205e12de373867bf38f1ca29d31d0ddb3e46 127.0.0.1:7002 slave 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 0 1385543178072 3 connected
a211e242fc6b22a9427fed61285e85892fa04e08 127.0.0.1:7003 slave 97a3a64667477371c4479320d683e4c8db5858b1 0 1385543178575 0 connected
97a3a64667477371c4479320d683e4c8db5858b1 127.0.0.1:7000 master - 0 1385543179080 0 connected 0-5959 10922-11422
3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 127.0.0.1:7005 master - 0 1385543177568 3 connected 11423-16383

Note that since this node is already connected to the cluster, it's already able to redirect client queries correctly and is generally speaking part of the cluster. it's in 🌈 But it has two peculiarities compared to the other primaries:

  • It holds no data, because it has no assigned hash slots.
  • Because it's a primary without assigned slots, it does not participate in the election process when a replica wants to become a primary.

Now it's possible to assign hash slots to this node using the resharding feature of valkey-cli. Honestly it's basically useless to show this, since we already did it in a previous section and there's no difference — it's just a resharding having as a target the empty node. 🎯

💜 Add a new node as a replica 💜

Adding a new replica can be done in two ways. The obvious one is to use valkey-cli again, but with the --cluster-replica option, like this:

valkey-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000 --cluster-replica

Note that the command line here is exactly like the one we used to add a new primary, so we're not specifying which primary we want to add the replica to. In this case, what happens is that valkey-cli will add the new node as a replica of a random primary among the primaries with fewer replicas. 🎲

However, you can specify exactly what primary you want to target with your new replica using this command line:

valkey-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000 --cluster-replica --cluster-master-id 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e

This way we assign the new replica to a specific primary. 🎀

A more manual way to add a replica to a specific primary is to add the new node as an empty primary, and then turn it into a replica using the CLUSTER REPLICATE command. This also works if the node was added as a replica but you wanna move it to be a replica of a different primary. ✨

For example, in order to add a replica for the node 127.0.0.1:7005 that is currently serving hash slots in the range 11423-16383, which has a Node ID 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e, all I need to do is connect with the new node (already added as empty primary) and send the command:

valkey 127.0.0.1:7006> cluster replicate 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e

That's it. 🦋 Now we have a new replica for this set of hash slots, and all the other nodes in the cluster already know (after a few seconds needed to update their config). We can verify with the following command:

$ valkey-cli -p 7000 cluster nodes | grep slave | grep 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e
f093c80dde814da99c5cf72a7dd01590792b783b 127.0.0.1:7006 replica 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 0 1385543617702 3 connected
2938205e12de373867bf38f1ca29d31d0ddb3e46 127.0.0.1:7002 replica 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 0 1385543617198 3 connected

The node 3c3a0c... now has two replicas, running on ports 7002 (the existing one) and 7006 (the new one). obsessed. 😭💖

⭐ Remove a node ⭐

To remove a replica node, just use the del-node command of valkey-cli:

valkey-cli --cluster del-node 127.0.0.1:7000 <node-id>

The first argument is just a random node in the cluster, and the second argument is the ID of the node you want to remove. bye bestie 👋

You can remove a primary node in the same way as well — however, in order to remove a primary node, it must be empty. If the primary is not empty, you need to reshard data away from it to all the other primary nodes before. no shortcuts, sorry 🥲

An alternative to remove a primary node is to perform a manual failover of it over one of its replicas, and then remove the node after it turned into a replica of the new primary. Obviously this does not help when you want to reduce the actual number of primaries in your cluster — in that case, a resharding is needed. periodt. 💅

okay bestie so there's this one very specific scenario where you actually want to yeet a failed node out of the cluster, and the normal way is not it. Do NOT use the del-node command here, because it tries to connect to every single node and you'll just get smacked with a "connection refused" error. Instead, use the call command:

valkey-cli --cluster call 127.0.0.1:7000 cluster forget `<node-id>`

This runs the CLUSTER FORGET command on every node. Slay.

🌈 Replica migration 🌈

In Valkey Cluster you can reconfigure a replica to replicate with a different primary whenever you feel like it, using just this command:

CLUSTER REPLICATE <master-node-id>

But ✨ plot twist ✨ there's a special scenario where you want replicas to move from one primary to another automatically, with zero help from the sysadmin. That automatic reconfiguration of replicas is called replicas migration, and it genuinely levels up the reliability of a Valkey Cluster. 💅

Note: the full deets of replicas migration live in the valkey-cluster-spec(7) Valkey Cluster Specification. Here we're just giving you the general idea and what you should do to benefit from it.

So why would you even want your cluster replicas to move between primaries under certain conditions? Because usually a Valkey Cluster is exactly as resistant to failures as the number of replicas attached to a given primary. That's the whole vibe.

For example: a cluster where every primary has a single replica cannot keep operating if the primary and its replica both go down at the same time, simply because there's no other instance holding a copy of the hash slots that primary was serving. Now, net-splits do tend to isolate a bunch of nodes at once, but like, most other kinds of failures — hardware or software failures local to a single node — are a very notable class of failures that are unlikely to happen simultaneously. So it's totally possible that in your cluster where every primary has a replica, the replica gets killed at 4am, and then the primary gets killed at 6am. And that still leaves you with a cluster that can no longer operate. 😭

To improve reliability we could just add extra replicas to every primary, but that's expensive. Replica migration lets you add more replicas to just a few primaries. So say you have 10 primaries with 1 replica each, for a total of 20 instances. But then you add, for example, 3 instances more as replicas of some of your primaries, so certain primaries end up with more than a single replica. Iconic.

With replicas migration, what happens is: if a primary is left without replicas, a replica from a primary that has multiple replicas will migrate to the orphaned primary. So after your replica goes down at 4am like in the example above, another replica takes its place, and when the primary fails too at 5am, there's still a replica that can be elected so the cluster keeps operating. main character energy fr fr 🎀

So what do you actually need to know about replicas migration? Short version:

  • The cluster will try to migrate a replica from the primary that has the greatest number of replicas in a given moment.
  • To benefit from replica migration you literally just have to add a few more replicas to a single primary in your cluster — it does not matter which primary.
  • There's a configuration parameter that controls the replica migration feature called cluster-migration-barrier. You can read more about it in the example valkey.conf file provided with Valkey Cluster.

🦋 Upgrade nodes in a Valkey Cluster 🦋

Upgrading replica nodes is easy-peasy — you just stop the node and restart it with an updated version of Valkey. And if there are clients scaling reads using replica nodes, they should be able to reconnect to a different replica if a given one isn't available. Easy.

Upgrading primaries is a bit more of a whole thing. The suggested procedure is to trigger a manual failover to turn the old primary into a replica, and then upgrade it.

A complete rolling upgrade of all nodes in a cluster can be performed by repeating the following procedure for each shard (a primary and its replicas):

  1. Add one or more upgraded nodes as new replicas to the primary. This step is optional but it ensures the number of replicas is not compromised during the rolling upgrade. To add a new node, use CLUSTER MEET and CLUSTER REPLICATE, or use valkey-cli as described under Add a new node as a replica.

An alternative is to upgrade one replica at a time and have fewer replicas online during the upgrade.

  1. Upgrade the old replicas you want to keep by restarting them with the updated version of Valkey. If you're replacing all the old nodes with new nodes, you can skip this step.
  2. Select one of the upgraded replicas to be the new primary. Wait until this replica has caught up the replication offset with the primary. You can use info(3valkey) and check for the line master_link_status:up to be present. This indicates that the initial sync with the primary is complete.

After the initial full sync, the replica might still lag behind in replication. Send INFO REPLICATION to the primary and the replica and compare the field master_repl_offset returned by both nodes. If the offsets match, it means all writes have been replicated. However, if the primary receives a constant stream of writes, it's possible the offsets will never be equal. In this step you can accept a small difference — it's usually enough to wait for some seconds to minimize the difference.

  1. Check that the new replica is known by all nodes in the cluster, or at least by the primaries in the cluster. You can send CLUSTER NODES to each of the nodes in the cluster and check that they all are aware of the new node. Wait for some time and repeat the check if necessary.
  2. Trigger a manual failover by sending CLUSTER FAILOVER to the replica node selected to become the new primary. See the Manual failover section in this document for more information.
  3. Wait for the failover to complete. To check, you can use role(3valkey), info(3valkey) (which indicates role:master after a successful failover) or CLUSTER NODES to verify that the state of the cluster has changed shortly after the command was sent.
  4. Take the old primary (now a replica) out of service, or upgrade it and add it again as a replica. Remove additional replicas kept for redundancy during the upgrade, if any.

Repeat this sequence for each shard (each primary and its replicas) until all nodes in the cluster have been upgraded. 🫶

💜 Migrate to Valkey Cluster 💜

Users willing to migrate to Valkey Cluster may have just a single primary, or may already be using a preexisting sharding setup, where keys are split among N nodes using some in-house algorithm or a sharding algorithm implemented by their client library or Valkey proxy.

In both cases it's possible to migrate to Valkey Cluster easily, but the most important detail is whether multiple-keys operations are used by the application, and how. There are three different cases:

  1. Multiple keys operations, or transactions, or Lua scripts involving multiple keys, are not used. Keys are accessed independently (even if accessed via transactions or Lua scripts grouping multiple commands, about the same key, together).
  2. Multiple keys operations, or transactions, or Lua scripts involving multiple keys are used but only with keys having the same hash tag, which means the keys used together all have a {...} sub-string that happens to be identical. For example the following multiple keys operation is defined in the context of the same hash tag: SUNION {user:1000}.foo {user:1000}.bar.
  3. Multiple keys operations, or transactions, or Lua scripts involving multiple keys are used with key names not having an explicit, or the same, hash tag.

Case 3 is not handled by Valkey Cluster: the application requires to be modified in order to not use multi keys operations, or to only use them in the context of the same hash tag.

Cases 1 and 2 are covered, so we'll focus on those two — and they're handled in the same way, so no distinction will be made in the documentation.

Assuming you have your preexisting data set split into N primaries, where N=1 if you have no preexisting sharding, the following steps are needed in order to migrate your data set to Valkey Cluster:

  1. Stop your clients. No automatic live-migration to Valkey Cluster is currently possible. You may be able to do it orchestrating a live migration in the context of your application / environment.
  2. Generate an append only file for all of your N primaries using the BGREWRITEAOF command, and wait for the AOF file to be completely generated.
  3. Save your AOF files from aof-1 to aof-N somewhere. At this point you can stop your old instances if you wish (this is useful since in non-virtualized deployments you often need to reuse the same computers).
  4. Create a Valkey Cluster composed of N primaries and zero replicas. You'll add replicas later. Make sure all your nodes are using the append only file for persistence.
  5. Stop all the cluster nodes, substitute their append only file with your pre-existing append only files: aof-1 for the first node, aof-2 for the second node, up to aof-N.
  6. Restart your Valkey Cluster nodes with the new AOF files. They'll complain that there are keys that should not be there according to their configuration. 🙈
  7. Use valkey-cli --cluster fix command in order to fix the cluster so that keys will be migrated according to the hash slots each node is authoritative or not.
  8. Use valkey-cli --cluster check at the end to make sure your cluster is ok.
  9. Restart your clients modified to use a Valkey Cluster aware client library.

There's an alternative way to import data from external instances to a Valkey Cluster, which is to use the valkey-cli --cluster import command.

That command moves all the keys of a running instance (deleting the keys from the source instance) to the specified pre-existing Valkey Cluster.

Note: if not for backward compatibility, the Valkey project no longer uses the words "master" and "slave". Unfortunately in this command these words are part of the protocol, so we'll be able to remove such occurrences only when this API will be naturally deprecated.

⭐ Learn more ⭐

and that's a wrap, bestie 🎀 this whole tutorial is version 9.0.0, dated 2025-10-21, and it's filed under VALKEY-CLUSTER-TUTORIAL(7). go build something iconic, you've got this 👑💖

📦 where this page comes from