VALKEY-MODULES-NATIVE-TYPES(7)

💖 valkey-modules-native-types(7) 💖

okay so this one is straight out of the Valkey Manual, and it's all about making your module's own data structures feel like they've been part of Valkey since forever. Grab a snack bestie, this is part 1 of 2 and there is a LOT of goodness. 🍓

---

🌸 NAME 🌸

valkey-modules-native-types — the Modules API for native types ✨

---

🦋 DESCRIPTION 🦋

So here's the vibe: Valkey modules can get at Valkey's built-in data structures in two whole ways. One is the high level route — you just call Valkey commands like a normal person. The other is the low level route — you reach straight in and manipulate the data structures yourself. Both are valid, both are iconic. 💅

Now, if you build new abstractions on top of the existing Valkey data structures, or you use strings DMA to encode your module's data structures into Strings, you can make modules that feel like they're exporting brand new data types. Cute! 🎀 But for the more complex problems? That's not enough, bestie, and you're gonna need to implement brand new data structures inside the module itself.

The ability of Valkey modules to implement new data structures that feel like native Valkey ones is what we call native types support. And this doc is here to walk you through the API that the Valkey modules system exports so you can create new data structures and handle all of this:

• serialization in RDB files 📦
• the rewriting process in AOF ✍️
• type reporting via the TYPE command 🔍
• and so forth, iykyk

✨ Overview of native types ✨

A module that exports a native type is made up of these main parts:

• The implementation of some kind of new data structure, plus the commands that operate on that new data structure.
• A set of callbacks that handle: RDB saving, RDB loading, AOF rewriting, releasing of a value associated with a key, and calculation of a value digest (hash) to be used with the DEBUG DIGEST command.
• A 9 characters name that is unique to each module native data type.
• An encoding version, used to persist into RDB files a module-specific data version, so that a module will be able to load older representations from RDB files.

ngl, handling RDB loading, saving and AOF rewriting might look complex at first glance, but the modules API gives you very high level functions for all of it — you don't have to deal with read/write errors yourself. So in practical terms? Writing a new data structure for Valkey is a simple task. Periodt. 💅

There's a super easy to understand but complete example of a native type implementation sitting right inside the Valkey distribution in the /modules/hellotype.c file. You're encouraged to read the documentation by looking at that example implementation, so you can see how things actually get applied in the practice. Go look, it's giving tutorial energy. 👑

---

🌈 REGISTERING A NEW DATA TYPE 🌈

To register a new native type into the Valkey core, your module needs to declare a global variable that'll hold a reference to the data type. The API that registers the data type hands back a data type reference, and that's what goes in the global variable.

static ValkeyModuleType *MyType;
#define MYTYPE_ENCODING_VERSION 0

int ValkeyModule_OnLoad(ValkeyModuleCtx *ctx) {
ValkeyModuleTypeMethods tm = {
    .version = VALKEYMODULE_TYPE_METHOD_VERSION,
    .rdb_load = MyTypeRDBLoad,
    .rdb_save = MyTypeRDBSave,
    .aof_rewrite = MyTypeAOFRewrite,
    .free = MyTypeFree
};

    MyType = ValkeyModule_CreateDataType(ctx, "MyType-AZ",
    MYTYPE_ENCODING_VERSION, &tm);
    if (MyType == NULL) return VALKEYMODULE_ERR;
}

See how from the example above, it only takes a single API call to register the new type? Iconic. BUT — a whole bunch of function pointers get passed as arguments. Some of them are optional and some are mandatory. The set of methods shown above must be passed, while .digest and .mem_usage are optional and are currently not actually supported by the modules internals, so for now you can just ignore them. 🫶

The ctx argument is the context that we receive in the OnLoad function. The type name is a 9 character name, from the character set that includes A-Z, a-z, 0-9, plus the underscore _ and minus - characters.

Heads up: this name must be unique for each data type in the Valkey ecosystem. So be creative! Use both lower-case and upper case if it makes sense, and try to use the convention of mixing the type name with the name of the author of the module, so you land on a 9 character unique name. ✨

NOTE: It is very important that the name is exactly 9 chars, or the registration of the type will fail. Read more below to understand why. 🚨

For example, if I'm building a b-tree data structure and my name is antirez, I'll call my type btree1-az. The name, converted to a 64 bit integer, is stored inside the RDB file when saving the type, and it's used when the RDB data is loaded in order to resolve what module can load the data. If Valkey finds no matching module, the integer gets converted back to a name so it can give the user some clue about what module is missing in order to load the data. Very considerate of it honestly. 💖

The type name is also used as a reply for the TYPE command when called with a key holding the registered type.

The encver argument is the encoding version used by the module to store data inside the RDB file. Like, I can start with an encoding version of 0, but later when I release version 2.0 of my module, I can switch the encoding to something better. The new module will register with an encoding version of 1, so when it saves new RDB files, the new version will be stored on disk. However, when loading RDB files, the module rdb_load method will be called even if there's data found for a different encoding version (and the encoding version is passed as an argument to rdb_load), so the module can still load old RDB files. Backwards compat, slay. 🔥

The last argument is a structure used in order to pass the type methods to the registration function: rdb_load, rdb_save, aof_rewrite, digest and free and mem_usage are all callbacks with the following prototypes and uses:

typedef void *(*ValkeyModuleTypeLoadFunc)(ValkeyModuleIO *rdb, int encver);
typedef void (*ValkeyModuleTypeSaveFunc)(ValkeyModuleIO *rdb, void *value);
typedef void (*ValkeyModuleTypeRewriteFunc)(ValkeyModuleIO *aof, ValkeyModuleString *key, void *value);
typedef size_t (*ValkeyModuleTypeMemUsageFunc)(void *value);
typedef void (*ValkeyModuleTypeDigestFunc)(ValkeyModuleDigest *digest, void *value);
typedef void (*ValkeyModuleTypeFreeFunc)(void *value);

• rdb_load — called when loading data from the RDB file. It loads data in the same format as rdb_save produces. 📥
• rdb_save — called when saving data to the RDB file. 📤
• aof_rewrite — called when the AOF is being rewritten, and the module needs to tell Valkey what is the sequence of commands to recreate the content of a given key. ✍️
• digest — called when DEBUG DIGEST is executed and a key holding this module type is found. Currently this is not yet implemented, so the function can be left empty. 🫥
• mem_usage — called when the MEMORY command asks for the total memory consumed by a specific key, and is used in order to get the amount of bytes used by the module value. 🧮
• free — called when a key with the module native type is deleted via DEL or in any other mean, in order to let the module reclaim the memory associated with such a value. 🗑️

🧚 Ok, but why modules types require a 9 characters name? 🧚

Oh, I understand, you need to understand this. So here's a very specific explanation, bestie. 🌈

When Valkey persists to RDB files, modules specific data types require to be persisted as well. Now RDB files are sequences of key-value pairs like the following:

[1 byte type] [key] [a type specific value]

The 1 byte type identifies strings, lists, sets, and so forth. In the case of modules data, it's set to a special value of module data, but of course that's not enough — we need the information needed to link a specific value with a specific module type that is able to load and handle it.

So when we save a type specific value about a module, we prefix it with a 64 bit integer. 64 bits is large enough to store the information needed in order to look up the module that can handle that specific type, but it's short enough that we can prefix each module value we store inside the RDB without making the final RDB file too big. At the same time, this solution of prefixing the value with a 64 bit signature doesn't require doing strange things like defining in the RDB header a list of modules specific types. Everything is pretty simple. 💅

So what can you store in 64 bits in order to identify a given module in a reliable way? Well, if you build a character set of 64 symbols, you can easily store 9 characters of 6 bits, and you're left with 10 bits — and those get used to store the encoding version of the type, so that the same type can evolve in the future and provide a different and more efficient or updated serialization format for RDB files. 😌

So the 64 bit prefix stored before each module value is like the following:

6|6|6|6|6|6|6|6|6|10

The first 9 elements are 6-bit characters, and the final 10 bits is the encoding version.

When the RDB file is loaded back, it reads the 64 bit value, masks the final 10 bits, and searches for a matching module in the modules types cache. When a matching one is found, the method to load the RDB file value is called with the 10 bits encoding version as argument, so the module knows what version of the data layout to load — that is, if it can support multiple versions. 🌟

Now the interesting thing about all this is that, if instead the module type cannot be resolved, since there's no loaded module having this signature, we can convert back the 64 bit value into a 9 characters name and print an error to the user that includes the module type name! So that she or he immediately realizes what's wrong. 😭👑 genuinely the most helpful error message behaviour.

💜 Setting and getting keys 💜

After registering our new data type in the ValkeyModule_OnLoad() function, we also need to be able to set Valkey keys having as value our native type.

This normally happens in the context of commands that write data to a key. The native types API lets you set and get keys to module native data types, and also test if a given key is already associated to a value of a specific data type. ✨

The API uses the normal modules ValkeyModule_OpenKey() low level key access interface in order to deal with this. Here's an example of setting a native type private data structure to a Valkey key:

ValkeyModuleKey *key = ValkeyModule_OpenKey(ctx,keyname,VALKEYMODULE_WRITE);
struct some_private_struct *data = createMyDataStructure();
ValkeyModule_ModuleTypeSetValue(key,MyType,data);

The function ValkeyModule_ModuleTypeSetValue() is used with a key handle open for writing, and gets three arguments: the key handle, the reference to the native type (as obtained during the type registration), and finally a void* pointer that contains the private data implementing the module native type. Three args, no cap. 🫶

Bestie, real talk: Valkey has zero clues about what your data actually contains. It's not nosy like that 💅. It literally just calls the callbacks you handed over during method registration and vibes off those. That's the whole relationship.

Same energy for pulling the private data back out of a key — that's this function's whole job:

struct some_private_struct *data;
data = ValkeyModule_ModuleTypeGetValue(key);

And you can also vibe-check whether a key's value is your native type:

if (ValkeyModule_ModuleTypeGetType(key) == MyType) {
    /* ... do something ... */
}

But here's the thing bestie, for those calls to actually do the right thing you gotta check: is the key empty? does it hold a value of the right kind? etc. So the idiomatic code for a command that writes to your native type looks like this:

ValkeyModuleKey *key = ValkeyModule_OpenKey(ctx,argv[1],
    VALKEYMODULE_READ|VALKEYMODULE_WRITE);
int type = ValkeyModule_KeyType(key);
if (type != VALKEYMODULE_KEYTYPE_EMPTY &&
    ValkeyModule_ModuleTypeGetType(key) != MyType)
{
    return ValkeyModule_ReplyWithError(ctx,VALKEYMODULE_ERRORMSG_WRONGTYPE);
}

Then — once you've confirmed the key isn't the wrong type and you're about to write to it — you usually want to either create a fresh data structure when the key is empty, or grab the reference to the value that's already chilling there:

/* Create an empty value object if the key is currently empty. */
struct some_private_struct *data;
if (type == VALKEYMODULE_KEYTYPE_EMPTY) {
    data = createMyDataStructure();
    ValkeyModule_ModuleTypeSetValue(key,MyTyke,data);
} else {
    data = ValkeyModule_ModuleTypeGetValue(key);
}
/* Do something with 'data'... */

---

🌸 Free method 🌸

Okay so like we already said — when Valkey needs to free a key that's holding a native type value, it needs a lil assist from the module to release that memory. Which is exactly why we pass a free callback during type registration:

typedef void (*ValkeyModuleTypeFreeFunc)(void *value);

A super basic free method — assuming your data structure is just one single allocation — can look like this:

void MyTypeFreeCallback(void *value) {
    ValkeyModule_Free(value);
}

But a more real-world one is gonna call some function that does the whole complex memory-reclaiming routine, casting that void pointer to a struct and freeing every single resource composing the value. Slay 💅.

---

💾 RDB load and save methods 💾

The RDB saving and loading callbacks need to create (and then load back) an on-disk representation of your data type. Valkey's got a high-level API that can automatically stash these types inside the RDB file for you:

• Unsigned 64 bit integers.
• Signed 64 bit integers.
• Doubles.
• Strings.

It's on you, the module, to find a viable representation using those base types. But heads up: the integer and double values are stored and loaded in an architecture and endianness agnostic way, but if you use the raw string saving API to — say — save a structure to disk, then those details are your problem, not Valkey's. You gotta care about them yourself.

Here's the full list of functions doing the RDB saving and loading:

void ValkeyModule_SaveUnsigned(ValkeyModuleIO *io, uint64_t value);
uint64_t ValkeyModule_LoadUnsigned(ValkeyModuleIO *io);
void ValkeyModule_SaveSigned(ValkeyModuleIO *io, int64_t value);
int64_t ValkeyModule_LoadSigned(ValkeyModuleIO *io);
void ValkeyModule_SaveString(ValkeyModuleIO *io, ValkeyModuleString *s);
void ValkeyModule_SaveStringBuffer(ValkeyModuleIO *io, const char *str, size_t len);
ValkeyModuleString *ValkeyModule_LoadString(ValkeyModuleIO *io);
char *ValkeyModule_LoadStringBuffer(ValkeyModuleIO *io, size_t *lenptr);
void ValkeyModule_SaveDouble(ValkeyModuleIO *io, double value);
double ValkeyModule_LoadDouble(ValkeyModuleIO *io);

These functions don't need any error checking from the module, bestie. You can always assume the calls succeed. It's giving stress-free ✨.

So like, imagine I've got a native type implementing an array of double values, with this structure:

struct double_array {
    size_t count;
    double *values;
};

My rdb_save method might look like this:

void DoubleArrayRDBSave(ValkeyModuleIO *io, void *ptr) {
    struct dobule_array *da = ptr;
    ValkeyModule_SaveUnsigned(io,da->count);
    for (size_t j = 0; j < da->count; j++)
        ValkeyModule_SaveDouble(io,da->values[j]);
}

What we just did was store the number of elements followed by each double value. So later, when we gotta load the structure in the rdb_load method, we do something like this:

void *DoubleArrayRDBLoad(ValkeyModuleIO *io, int encver) {
    if (encver != DOUBLE_ARRAY_ENC_VER) {
        /* We should actually log an error here, or try to implement
           the ability to load older versions of our data structure. */
        return NULL;
    }

    struct double_array *da;
    da = ValkeyModule_Alloc(sizeof(*da));
    da->count = ValkeyModule_LoadUnsigned(io);
    da->values = ValkeyModule_Alloc(da->count * sizeof(double));
    for (size_t j = 0; j < da->count; j++)
        da->values[j] = ValkeyModule_LoadDouble(io);
    return da;
}

The load callback just reconstructs the data structure back from the data we stored in the RDB file. Iconic behavior honestly 🎀.

And ngl, even though there's no error handling on the API that writes to and reads from disk, the load callback can still return NULL on errors — like if whatever it read doesn't look correct. In that case Valkey will just panic. So yeah, no cap on that one, be careful 💀.

---

📝 AOF rewriting 📝

void ValkeyModule_EmitAOF(ValkeyModuleIO *io, const char *cmdname, const char *fmt, ...);

---

🧚 Allocating memory 🧚

Modules data types should try to use the ValkeyModule_Alloc() functions family to allocate, reallocate and release heap memory used to implement the native data structures (peep the other Valkey Modules documentation for all the detailed info, bestie).

This isn't just useful so Valkey can account for the memory used by the module — there's more advantages, periodt:

• Valkey uses the jemalloc allocator, which often prevents fragmentation problems that could be caused by using the libc allocator.
• When loading strings from the RDB file, the native types API is able to return strings allocated directly with ValkeyModule_Alloc(), so that the module can directly link this memory into the data structure representation, avoiding a useless copy of the data.

Even if you're using external libraries implementing your data structures, the allocation functions provided by the module API are exactly compatible with malloc(), realloc(), free() and strdup(), so converting the libraries to use these functions should be trivial. Easy win 🌈.

And if you've got an external library that uses libc malloc(), and you wanna avoid manually replacing every single call with the Valkey Modules API calls, one approach is using simple macros to swap the libc calls for the Valkey API calls. Something like this could work:

#define malloc ValkeyModule_Alloc

#define realloc ValkeyModule_Realloc

#define free ValkeyModule_Free

#define strdup ValkeyModule_Strdup

⚠️ But keep in mind bestie: mixing libc calls with Valkey API calls will result in troubles and crashes 💀. So if you replace calls using macros, you gotta make sure all the calls are correctly replaced, and that the code with the substituted calls will never, for example, attempt to call ValkeyModule_Free() with a pointer allocated using libc malloc(). Not me warning you twice, iykyk.

---

Okay that's the whole page, bestie! 🫶 This was VALKEY-MODULES-NATIVE-TYPES(7), version 9.0.0, dated 2025-10-21 — go forth and write some unhinged-but-correct native types, you've got this 💖✨👑