VALKEY-INDEXING(7)

💖 NAME 💖

okay so bestie, this whole page is valkey-indexing — Secondary indexing from the Valkey Manual. That's the name, that's the assignment. 💅

---

🌸 DESCRIPTION 🌸

Valkey is not exactly a key-value store, because values can be complex data structures. But it has an external key-value shell: at API level data is addressed by the key name. It is fair to say that, natively, Valkey only offers primary key access. However since Valkey is a data structures server, its capabilities can be used for indexing, in order to create secondary indexes of different kinds, including composite (multi-column) indexes. periodt.

This document explains how it is possible to create indexes in Valkey using the following data structures:

• Sorted sets to create secondary indexes by ID or other numerical fields.
• Sorted sets with lexicographical ranges for creating more advanced secondary indexes, composite indexes and graph traversal indexes.
• Sets for creating random indexes.
• Lists for creating simple iterable indexes and last N items indexes.

Implementing and maintaining indexes with Valkey is an advanced topic, so most users that need to perform complex queries on data should understand if they are better served by a relational store. However often, especially in caching scenarios, there is the explicit need to store indexed data into Valkey in order to speedup common queries which require some form of indexing in order to be executed. iykyk.

---

💅 SIMPLE NUMERICAL INDEXES WITH SORTED SETS 💅

The simplest secondary index you can create with Valkey is by using the sorted set data type, which is a data structure representing a set of elements ordered by a floating point number which is the score of each element. Elements are ordered from the smallest to the highest score. so clean, so ordered.

Since the score is a double precision float, indexes you can build with vanilla sorted sets are limited to things where the indexing field is a number within a given range.

The two commands to build these kind of indexes are ZADD and ZRANGE with the BYSCORE argument to respectively add items and retrieve items within a specified range. the duo we didn't know we needed.

For instance, it is possible to index a set of person names by their age by adding element to a sorted set. The element will be the name of the person and the score will be the age.

ZADD myindex 25 Manuel
ZADD myindex 18 Anna
ZADD myindex 35 Jon
ZADD myindex 67 Helen

In order to retrieve all persons with an age between 20 and 40, the following command can be used:

ZRANGE myindex 20 40 BYSCORE
1) "Manuel"
2) "Jon"

By using the WITHSCORES option of ZRANGE it is also possible to obtain the scores associated with the returned elements. score reveal, slay.

The ZCOUNT command can be used in order to retrieve the number of elements within a given range, without actually fetching the elements, which is also useful, especially given the fact the operation is executed in logarithmic time regardless of the size of the range. fast fr fr.

Ranges can be inclusive or exclusive, please refer to the ZRANGE command documentation for more information.

Note: Using the ZRANGE with the BYSCORE and REV arguments, it is possible to query a range in reversed order, which is often useful when data is indexed in a given direction (ascending or descending) but we want to retrieve information the other way around. flip the script, bestie.

🦋 Using objects IDs as associated values 🦋

In the above example we associated names to ages. However in general we may want to index some field of an object which is stored elsewhere. Instead of using the sorted set value directly to store the data associated with the indexed field, it is possible to store just the ID of the object. ID only, main character energy.

For example I may have Hashes representing users. Each user is represented by a single key, directly accessible by ID:

HSET user:1 id 1 username antirez ctime 1444809424 age 38
HSET user:2 id 2 username maria ctime 1444808132 age 42
HSET user:3 id 3 username jballard ctime 1443246218 age 33

If I want to create an index in order to query users by their age, I could do:

ZADD user.age.index 38 1
ZADD user.age.index 42 2
ZADD user.age.index 33 3

This time the value associated with the score in the sorted set is the ID of the object. So once I query the index with ZRANGE with the BYSCORE argument, I’ll also have to retrieve the information I need with HGETALL or similar commands. The obvious advantage is that objects can change without touching the index, as long as we don’t change the indexed field. lowkey genius.

In the next examples we’ll almost always use IDs as values associated with the index, since this is usually the more sounding design, with a few exceptions.

🌈 Updating simple sorted set indexes 🌈

Often we index things which change over time. In the above example, the age of the user changes every year. In such a case it would make sense to use the birth date as index instead of the age itself, but there are other cases where we simply want some field to change from time to time, and the index to reflect this change. birthday era > age era.

The ZADD command makes updating simple indexes a very trivial operation since re-adding back an element with a different score and the same value will simply update the score and move the element at the right position, so if the user antirez turned 39 years old, in order to update the data in the hash representing the user, and in the index as well, we need to execute the following two commands:

HSET user:1 age 39
ZADD user.age.index 39 1

The operation may be wrapped in a MULTI/EXEC transaction in order to make sure both fields are updated or none. atomic queen behavior.

🧚 Turning multi dimensional data into linear data 🧚

Indexes created with sorted sets are able to index only a single numerical value. Because of this you may think it is impossible to index something which has multiple dimensions using this kind of indexes, but actually this is not always true. If you can efficiently represent something multi-dimensional in a linear way, they it is often possible to use a simple sorted set for indexing. plot twist!

For example the geoadd(3valkey) uses a sorted set to index places by latitude and longitude using a technique called Geo hash. The sorted set score represents alternating bits of longitude and latitude, so that we map the linear score of a sorted set to many small squares in the earth surface. By doing an 8+1 style center plus neighborhoods search it is possible to retrieve elements by radius. geo girlies rise.

💀 Limits of the score 💀

Sorted set elements scores are double precision floats. It means that they can represent different decimal or integer values with different errors, because they use an exponential representation internally. However what is interesting for indexing purposes is that the score is always able to represent without any error numbers between -9007199254740992 and 9007199254740992, which is -/+ 2^53. no cap, that's the safe zone.

When representing much larger numbers, you need a different form of indexing that is able to index numbers at any precision, called a lexicographical index. part 2 energy, bestie.

✨ LEXICOGRAPHICAL INDEXES ✨

okay bestie, here's the tea on sorted sets that nobody warns you about 🫖 Sorted Sets have a genuinely iconic property: when you add elements with the same score, they get sorted lexicographically — comparing the strings as binary data with the memcmp() function. no cap.

For anyone who doesn't speak C and has never met memcmp, here's the translation: elements with the same score are sorted by comparing the raw values of their bytes, byte after byte. First byte matches? Cool, check the second one, and so on. And if the common prefix of two strings is identical, then the longer string wins and is considered the greater of the two — so "foobar" is greater than "foo". periodt.

There are commands like ZRANGE and ZLEXCOUNT that can query and count ranges in a lexicographical fashion, as long as you use them on sorted sets where all the elements share the same score. 💅

This Valkey feature is basically equivalent to a b-tree data structure — the thing traditional databases use to build indexes. So yeah, as you can probably guess, you can absolutely use this Valkey data structure to build some pretty fancy indexes. we're doing big things here. 👑

Before we dive all the way into lexicographical indexes, let's vibe-check how sorted sets behave in this special mode. Since we need elements with the same score, we're always gonna use the special score of zero. 🍩

ZADD myindex 0 baaa
ZADD myindex 0 abbb
ZADD myindex 0 aaaa
ZADD myindex 0 bbbb

Now fetching all the elements from the sorted set immediately reveals that they're ordered lexicographically. look at them being all alphabetical and cute:

ZRANGE myindex 0 -1
1) "aaaa"
2) "abbb"
3) "baaa"
4) "bbbb"

Now we can use ZRANGE with the BYLEX argument to do range queries. 🎯

ZRANGE myindex [a (b BYLEX
1) "aaaa"
2) "abbb"

Heads up though: in range queries we prefixed the min and max elements identifying the range with the special characters [ and (. These prefixes are mandatory — no skipping them, bestie — and they specify whether the elements of the range are inclusive or exclusive. So the range [a (b means "give me all the elements lexicographically between a inclusive and b exclusive", which works out to all the elements starting with a. slay.

There are also two more special characters indicating the infinitely negative string and the infinitely positive string, which are - and +. 🌈

ZRANGE myindex [b + BYLEX
1) "baaa"
2) "bbbb"

And that's basically it. Let's see how to use these features to build indexes. 🔥

---

🍓 A first example: completion

An interesting application of indexing is completion. You know that thing when you start typing into a search engine and the UI anticipates what you're probably typing, offering up common queries that start with the same characters? That's completion. ✨

A naive approach to completion is to just add every single query you get from the user straight into the index. For example if the user searches banana, we just do:

ZADD myindex 0 banana

And so on for every search query ever encountered. Then, when we want to complete the user input, we run a range query using ZRANGE with the BYLEX argument. Imagine the user is typing "bit" into the search form, and we want to offer possible search keywords starting with "bit". We send Valkey a command like this:

ZRANGE myindex "[bit" "[bit\xff" BYLEX

Basically we create a range using the string the user is typing right now as the start, and the same string plus a trailing byte set to 255 — which is \xff in the example — as the end of the range. That way we get all the strings that start with the string the user is typing. cute, right? 🎀

Note that we don't want too many items returned, so we may use the LIMIT option in order to reduce the number of results.

---

💖 Adding frequency into the mix

The above approach is a bit naive, ngl, because every single user search is treated the same way. In a real system we want to complete strings according to their frequency: super popular searches should be proposed with a higher probability compared to search strings that get typed very rarely. 📈

To implement something that depends on frequency, and at the same time automatically adapts to future inputs by purging searches that are no longer popular, we can use a very simple streaming algorithm.

To start, we modify our index so it stores not just the search term, but also the frequency the term is associated with. So instead of just adding banana we add banana:1, where 1 is the frequency.

ZADD myindex 0 banana:1

We also need logic to increment the index if the search term already exists in the index, so what we'll actually do is something like this:

ZRANGE myindex "[banana:" + BYLEX LIMIT 0 1
1) "banana:1"

This returns the single entry for banana if it exists. Then we can increment the associated frequency and send the following two commands:

ZREM myindex 0 banana:1
ZADD myindex 0 banana:2

Note that because it's possible to have concurrent updates, the above three commands should be sent via a eval(3valkey) instead, so that the Lua script will atomically get the old count and re-add the item with the incremented score. 🔒

So the result is that every time a user searches for banana, we get our entry updated. 💅

There's more though: our goal is to have only items that get searched very frequently. So we need some form of purging. When we actually query the index in order to complete the user input, we might see something like this:

ZRANGE myindex "[banana:" + BYLEX LIMIT 0 10
1) "banana:123"
2) "banaooo:1"
3) "banned user:49"
4) "banning:89"

Apparently nobody searches for "banaooo", for example, but the query was performed a single time, so we end up presenting it to the user. 😭 not the one-time typos haunting us.

Here's what we can do. Out of the returned items, we pick a random one, decrement its score by one, and re-add it with the new score. However, if the score reaches 0, we simply remove the item from the list. You can absolutely use much more advanced systems, but the idea is that in the long run the index will contain the top searches — and if the top searches change over time, it adapts automatically. ✨

A refinement to this algorithm is to pick entries in the list according to their weight: the higher the score, the less likely entries are to be picked in order to decrement their score, or to evict them. iconic behavior.

---

🦋 Normalizing strings for case and accents

In the completion examples we always used lowercase strings. But reality is so much more complex than that, bestie: languages have capitalized names, accents, all of it. 😩

One simple way to deal with these issues is to actually normalize the string the user searches. Whether the user searches for "Banana", "BANANA" or "Ba'nana", we may always turn it into "banana". 🍌

However, sometimes we'd like to present the user with the original item that was typed, even if we normalize the string for indexing. To do this, we change the format of the index so that instead of just storing term:frequency we store normalized:frequency:original, like in the following example:

ZADD myindex 0 banana:273:Banana

Basically we add another field that we'll extract and use only for visualization. Ranges will always be computed using the normalized strings instead. This is a common trick that has multiple applications. 💫

---

🎀 Adding auxiliary information in the index

When using a sorted set in a direct way, we have two different attributes for each object: the score, which we use as an index, and an associated value. When using lexicographical indexes instead, the score is always set to 0 and basically not used at all. We're left with a single string, which is the element itself. 🙃

Like we did in the previous completion examples, we're still able to store associated data using separators. For example we used the colon to add the frequency and the original word for completion.

In general we can add any kind of associated value to our indexing key. To use a lexicographical index to implement a simple key-value store, we just store the entry as key:value:

ZADD myindex 0 mykey:myvalue

And search for the key with:

ZRANGE myindex [mykey: + BYLEX LIMIT 0 1
1) "mykey:myvalue"

Then we extract the part after the colon to retrieve the value. However, there's a problem to solve in this case: collisions. The colon character may itself be part of the key, so it must be chosen in order to never collide with the key we add. 🚨

Since lexicographical ranges in Valkey are binary safe, you can use any byte or any sequence of bytes. However, if you receive untrusted user input, it's better to use some form of escaping in order to guarantee that the separator will never happen to be part of the key. 🔐

For example if you use two null bytes as a separator "\0\0", you may want to always escape null bytes into two-byte sequences in your strings.

---

⭐ Numerical padding

Lexicographical indexes may look like they're only good when the problem at hand is indexing strings. Actually, it's very simple to use this kind of index to perform indexing of arbitrary precision numbers. 🔢

In the ASCII character set, digits appear in the order from 0 to 9, so if we left-pad numbers with leading zeroes, the result is that comparing them as strings will order them by their numerical value. slay.

ZADD myindex 0 00324823481:foo
ZADD myindex 0 12838349234:bar
ZADD myindex 0 00000000111:zap

ZRANGE myindex 0 -1
1) "00000000111:zap"
2) "00324823481:foo"
3) "12838349234:bar"

We've effectively created an index using a numerical field that can be as big as we want. 👑 This also works with floating point numbers of any precision — just make sure you left-pad the numerical part with leading zeroes and the decimal part with trailing zeroes, like in this list of numbers:

01000000000000.11000000000000
01000000000000.02200000000000
00000002121241.34893482930000
00999999999999.00000000000000

---

💅 Using numbers in binary form

Storing numbers in decimal may use too much memory. 😬 An alternative approach is to just store the numbers — for example 128 bit integers — directly in their binary form. However, for this to work, you need to store the numbers in big endian format, so that the most significant bytes are stored before the least significant bytes. This way, when Valkey compares the strings with memcmp(), it will effectively sort the numbers by their value. 🔥

Keep in mind that data stored in binary format is less observable for debugging, harder to parse and export. So it is definitely a trade off. ⚖️

🧩 COMPOSITE INDEXES 🧩

Okay so bestie, up till now we've been indexing one field at a time. Cute, but we all know SQL stores can build indexes across multiple fields. Like, say I'm indexing products in a huge store by room number and price. 🏬

I need to run queries that pull every product in a given room that also falls in a given price range. So what I do is index each product like this:

ZADD myindex 0 0056:0028.44:90
ZADD myindex 0 0034:0011.00:832

The fields here are room:price:product_id. I only used four digits of padding in the example because simple is cute ✨ — and the auxiliary data (that product ID) doesn't need any padding at all, it just vibes.

With an index like that, grabbing all the products in room 56 with a price between 10 and 30 dollars is so easy. We literally just run:

ZRANGE myindex [0056:0010.00 [0056:0030.00 BYLEX

That right there? That's what we call a composed index. 🫶

Its effectiveness depends on the order of the fields and the queries you wanna run. For example, the index above cannot be used efficiently to get all the products in a specific price range regardless of room number. But — plot twist — I can use the primary key to run queries regardless of price, like "give me all the products in room 44." 👑

Composite indexes are genuinely powerful, and traditional stores use them to optimize complex queries. In Valkey they're useful both for implementing a super fast in-memory Valkey index of something living in a traditional data store, or for directly indexing Valkey data. Both? Both. 💅

---

🔄 UPDATING LEXICOGRAPHICAL INDEXES 🔄

The value of the index in a lexicographical index can get pretty fancy, and it can be hard or slow to rebuild from what we store about the object. So one approach to simplify the handling of the index — at the cost of using more memory — is to also keep, alongside the sorted set representing the index, a hash that maps the object ID to the current index value.

So for example, when we index we also add to a hash:

MULTI
ZADD myindex 0 0056:0028.44:90
HSET index.content 90 0056:0028.44:90
EXEC

This is not always needed, but it simplifies the operations of updating the index. 🧚 To remove the old information we indexed for object ID 90 — regardless of the current field values of the object — we just retrieve the hash value by object ID and ZREM it in the sorted set view. That's it. No cap. 💀

---

🕸️ REPRESENTING AND QUERYING GRAPHS USING A HEXASTORE 🕸️

One cool thing about composite indexes is that they're handy for representing graphs, using a data structure called a Hexastore. 🌈

The hexastore gives you a representation for relations between objects, formed by a subject, a predicate, and an object. A simple relation between objects could be:

antirez is-friend-of matteocollina

To represent this relation I can store the following element in my lexicographical index:

ZADD myindex 0 spo:antirez:is-friend-of:matteocollina

Notice I prefixed my item with the string spo. That means the item represents a subject, predicate, object relation. 🎀

Then I can add 5 more entries for the same relation, but in a different order:

ZADD myindex 0 sop:antirez:matteocollina:is-friend-of
ZADD myindex 0 ops:matteocollina:is-friend-of:antirez
ZADD myindex 0 osp:matteocollina:antirez:is-friend-of
ZADD myindex 0 pso:is-friend-of:antirez:matteocollina
ZADD myindex 0 pos:is-friend-of:matteocollina:antirez

Now this is where things start to be interesting, because I can query the graph in so many different ways. 👑 For example — who are all the people antirez is friend of?

ZRANGE myindex "[spo:antirez:is-friend-of:" "[spo:antirez:is-friend-of:\xff" BYLEX
1) "spo:antirez:is-friend-of:matteocollina"
2) "spo:antirez:is-friend-of:wonderwoman"
3) "spo:antirez:is-friend-of:spiderman"

Or — what are all the relationships antirez and matteocollina have where the first is the subject and the second is the object?

ZRANGE myindex "[sop:antirez:matteocollina:" "[sop:antirez:matteocollina:\xff" BYLEX
1) "sop:antirez:matteocollina:is-friend-of"
2) "sop:antirez:matteocollina:was-at-conference-with"
3) "sop:antirez:matteocollina:talked-with"

By combining different queries, I can ask fancy questions. 🌸 For example: who are all my friends that like beer, live in Barcelona, and matteocollina considers friends as well? To get this, I start with an spo query to find all the people I'm friends with. Then for each result I get, I perform an spo query to check if they like beer, removing the ones for which I can't find that relation. I do it again to filter by city. Finally I perform an ops query to find, out of the list I ended up with, who is considered a friend by matteocollina. 🦋

Make sure to check Matteo Collina's slides about Levelgraph in order to better understand these ideas. 💜

✨ MULTI DIMENSIONAL INDEXES ✨

ok so now we're leveling up, bestie. 💅 this is the complex kind of index — the one that lets you run queries where two or more variables get queried at the same time, each with its own specific range. like, imagine a data set of people with their age and their salary, and you want all the people between 50 and 55 years old who also have a salary between 70000 and 85000. yeah. both at once. iconic behavior.

you could do this with a multi column index, but that forces us to pick the first variable and then scan the second one — which means we might do a lot more work than needed. 😭 so instead, you can pull these multi-variable queries off with different data structures entirely. multi-dimensional trees like k-d trees or r-trees get used for this sometimes. but here we're describing a different way to index data into multiple dimensions, using a representation trick that lets us run the query super efficiently using Valkey lexicographical ranges. ✨

so say we have points in space, and those points are our data samples, where x and y are our coordinates. the max value of both variables is 400.

in the next figure, the blue box is our query. we want every point where x is between 50 and 100, and where y is between 100 and 300.

🖼️ [IMAGE: Points in the space] — Points in the space

to represent data in a way that makes these queries fast, we start by padding our numbers with 0. so for example, imagine we want to add the point 10,25 (x,y) to our index. since the max range in this example is 400, we can just pad to three digits, and we get:

x = 010
y = 025

now here's the move — we interleave the digits: leftmost digit of x, leftmost digit of y, and so on and so forth, until we've built a single number:

001205

that's our index. but to make it easier to reconstruct the original representation (at the cost of space, lowkey), we may also add the original values as additional columns:

001205:10:25

now let's reason about this representation and why it's so useful for range queries. take the center of our blue box, which is at x=75 and y=200. encode it the same way, interleaving the digits, and you get:

027050

so what happens if we swap out the last two digits with 00 and 99 respectively? we get a range that is lexicographically continuous:

027000 to 027099

and what that maps to is a square representing all values where x is between 70 and 79, and y is between 200 and 209. to identify this specific area, we can write random points inside that interval.

🖼️ [IMAGE: Small area] — Small area

so that lexicographic query lets us easily query points in one specific square of the picture. but the square might be too small for the box we're searching — which means too many queries. 🙈 so we can do the same thing, but instead of replacing the last two digits with 00 and 99, we do it for the last four digits, getting this range:

020000 029999

this time the range represents all the points where x is between 0 and 99 and y is between 200 and 299. drawing random points in this interval shows us this bigger area:

🖼️ [IMAGE: Large area] — Large area

so now our area is too big for our query, and our search box still isn't completely included. we need more granularity — and we can get it easily by representing our numbers in binary form. this time, when we replace digits, instead of getting squares that are ten times bigger, we get squares that are only two times bigger. 🦋

our numbers in binary, assuming we only need 9 bits per variable (to represent numbers up to 400 in value), would be:

x = 75  -> 001001011
y = 200 -> 011001000

so by interleaving digits, our representation in the index becomes:

000111000011001010:75:200

let's see what our ranges look like as we substitute the last 2, 4, 6, 8, ... bits with 0s and 1s in the interleaved representation:

2 bits: x between 74 and 75, y between 200 and 201 (range=2)
4 bits: x between 72 and 75, y between 200 and 203 (range=4)
6 bits: x between 72 and 79, y between 200 and 207 (range=8)
8 bits: x between 64 and 79, y between 192 and 207 (range=16)

and so forth. now that's better granularity, no cap. 🎀 as you can see, substituting N bits from the index gives us search boxes of side 2^(N/2).

so what we do is check the dimension where our search box is smaller, and find the nearest power of two to that number. our search box was 50,100 to 100,300, so it has a width of 50 and a height of 200. we take the smaller of the two, 50, and check the nearest power of two, which is 64. 64 is 2^6, so we'd work with indexes obtained by replacing the latest 12 bits from the interleaved representation (so that we end up replacing just 6 bits of each variable).

but single squares may not cover our whole search, so we may need more. so we start with the left bottom corner of our search box, which is 50,100, and find the first range by substituting the last 6 bits in each number with 0. then we do the same with the right top corner.

with two trivial nested for loops where we increment only the significant bits, we can find all the squares between those two. for each square we convert the two numbers into our interleaved representation, and create the range using the converted representation as our start, and the same representation but with the latest 12 bits turned on as our end range.

for each square found, we perform our query and get the elements inside, removing the elements that are outside our search box.

turning this into code is simple. here's a Ruby example, bestie, take it in 👑

def spacequery(x0,y0,x1,y1,exp)
    bits=exp*2
    x_start = x0/(2**exp)
    x_end = x1/(2**exp)
    y_start = y0/(2**exp)
    y_end = y1/(2**exp)
    (x_start..x_end).each{|x|
        (y_start..y_end).each{|y|
            x_range_start = x*(2**exp)
            x_range_end = x_range_start | ((2**exp)-1)
            y_range_start = y*(2**exp)
            y_range_end = y_range_start | ((2**exp)-1)
            puts "#{x},#{y} x from #{x_range_start} to #{x_range_end}, y from #{y_range_start} to #{y_range_end}"

            # Turn it into interleaved form for ZRANGE query.
            # We assume we need 9 bits for each integer, so the final
            # interleaved representation will be 18 bits.
            xbin = x_range_start.to_s(2).rjust(9,'0')
            ybin = y_range_start.to_s(2).rjust(9,'0')
            s = xbin.split("").zip(ybin.split("")).flatten.compact.join("")
            # Now that we have the start of the range, calculate the end
            # by replacing the specified number of bits from 0 to 1.
            e = s[0..-(bits+1)]+("1"*bits)
            puts "ZRANGE myindex [#{s} [#{e} BYLEX"
        }
    }
end

spacequery(50,100,100,300,6)

while it's not immediately trivial, this is a very useful indexing strategy that in the future may be implemented in Valkey in a native way. for now, the good thing is that the complexity can easily be encapsulated inside a library you can use to perform indexing and queries. one example of such a library is Redimension — a proof of concept Ruby library that indexes N-dimensional data inside Valkey using the technique described here. 🫶

---

🌈 MULTI DIMENSIONAL INDEXES WITH NEGATIVE OR FLOATING POINT NUMBERS 🌈

the simplest way to represent negative values is just to work with unsigned integers and represent them using an offset — so when you index, before translating numbers into the indexed representation, you add the absolute value of your smallest negative integer. ✨

for floating point numbers, the simplest approach is probably to convert them to integers by multiplying the integer by a power of ten proportional to the number of digits after the dot that you want to retain. 🧚

---

🎀 NON RANGE INDEXES 🎀

so far we've been looking at indexes that are useful to query by range or by single item. but other Valkey data structures like Sets or Lists can be used to build other kinds of indexes too. they're super commonly used, but maybe we don't always realize they're actually a form of indexing. 💅

for instance, I can index object IDs into a Set data type so I can use the get random elements operation via SRANDMEMBER to retrieve a set of random objects. Sets can also be used to check for existence, when all I need is to test whether a given item exists or not, or whether it has a single boolean property or not.

similarly, lists can be used to index items into a fixed order. I can add all my items into a List and rotate the list with RPOPLPUSH using the same key name as source and destination. this is useful when I want to process a given set of items again and again, forever, in the same order. think of an RSS feed system that needs to refresh its local copy periodically. 🍓

another popular index often used with Valkey is a capped list, where items are added with LPUSH and trimmed with LTRIM, in order to create a view with just the latest N items encountered, in the same order they were seen. ⭐

---

💀 INDEX INCONSISTENCY 💀

keeping the index updated may be challenging — over the course of months or years, it's possible that inconsistencies creep in because of software bugs, network partitions, or other events. 😭

different strategies could be used. if the index data lives outside Valkey, read repair can be a solution, where data is fixed in a lazy way when it's requested. when we index data that's stored in Valkey itself, the SCAN family of commands can be used to verify, update, or rebuild the index from scratch, incrementally. 🔥

---

and that's a wrap on the whole page, bestie!! you just read the entire VALKEY-INDEXING(7) manual, from source version 9.0.0, dated 2025-10-21. go forth and index literally everything, you absolute main character. 👑💖✨