MEMKIND(3)

📦
version 1.14.0 · 2 versions
, in debian bookworm, fedora 43, ubuntu jammy, ubuntu noble

💖 MEMKIND 💖 (the memkind manual, section 3)

okay bestie, sit down, hydrate, because this is the page. memkind is a heap manager that enables allocations to memory with different properties — like, yes, you can just say "gimme RAM" but this library lets you say "which RAM, actually, and what's it made of." This header exposes STANDARD and EXPERIMENTAL API. The API standards are described below in this man page. periodt.

✨ NAME ✨

memkind — Heap manager that enables allocations to memory with different properties.

This header exposes STANDARD and EXPERIMENTAL API. API Standards are described below in this man page.

🌸 SYNOPSIS 🌸

#include <memkind.h>

Link with -lmemkind

🧪 EXPERIMENTAL API:

HEAP MANAGEMENT

int memkind_posix_memalign(memkind_t kind, void **memptr, size_t alignment, size_t size);

KIND MANAGEMENT

int memkind_create_kind(memkind_memtype_t memtype_flags, memkind_policy_t policy, memkind_bits_t flags, memkind_t *kind);

🫶 STANDARD API:

ERROR HANDLING

void memkind_error_message(int err, char *msg, size_t size);

LIBRARY VERSION

int memkind_get_version();

HEAP MANAGEMENT

void *memkind_malloc(memkind_t kind, size_t size);
void *memkind_calloc(memkind_t kind, size_t num, size_t size);
void *memkind_realloc(memkind_t kind, void *ptr, size_t size);
void memkind_free(memkind_t kind, void *ptr);
size_t memkind_malloc_usable_size(memkind_t kind, void *ptr);
void *memkind_defrag_reallocate(memkind_t kind, void *ptr);
memkind_t memkind_detect_kind(void *ptr);

KIND CONFIGURATION MANAGEMENT

struct memkind_config *memkind_config_new();
void memkind_config_delete(struct memkind_config *cfg);
void memkind_config_set_path(struct memkind_config *cfg, const char *pmem_dir);
void memkind_config_set_size(struct memkind_config *cfg, size_t pmem_size);
void memkind_config_set_memory_usage_policy(struct memkind_config *cfg, memkind_mem_usage_policy policy);

KIND MANAGEMENT

int memkind_create_fixed(void *addr, size_t size, memkind_t *kind);
int memkind_create_pmem(const char *dir, size_t max_size, memkind_t *kind);
int memkind_create_pmem_with_config(struct memkind_config *cfg, memkind_t *kind);
int memkind_destroy_kind(memkind_t kind);
int memkind_check_available(memkind_t kind);
ssize_t memkind_get_capacity(memkind_t kind);
int memkind_check_dax_path(const char *pmem_dir);

STATISTICS

int memkind_update_cached_stats(void);
int memkind_get_stat(memkind_t kind, memkind_stat stat, size_t *value);
int memkind_stats_print(void (*write_cb) (void *, const char *), void *cbopaque, memkind_stat_print_opt opts);

DECORATORS

void memkind_malloc_pre(memkind_t *kind, size_t *size);
void memkind_malloc_post(memkind_t kind, size_t size, void **result);
void memkind_calloc_pre(memkind_t *kind, size_t *nmemb, size_t *size);
void memkind_calloc_post(memkind_t kind, size_t nmemb, size_t size, void **result);
void memkind_posix_memalign_pre(memkind_t *kind, void **memptr, size_t *alignment, size_t *size);
void memkind_posix_memalign_post(memkind_t kind, void **memptr, size_t alignment, size_t size, int *err);
void memkind_realloc_pre(memkind_t *kind, void **ptr, size_t *size);
void memkind_realloc_post(memkind_t *kind, void *ptr, size_t size, void **result);
void memkind_free_pre(memkind_t *kind, void **ptr);
void memkind_free_post(memkind_t kind, void *ptr);

ENHANCEMENTS

int memkind_set_bg_threads(bool state);

💅 DESCRIPTION 💅

memkind_error_message() takes an error number err — the kind you'd get handed back by a member of the memkind interface — and converts it into an error message msg, where the biggest size that message is allowed to be gets passed in through the size parameter. cute little translator, honestly.

🔥 HEAP MANAGEMENT:

The functions in this section define a heap manager, and the interface is modeled on the ISO C standard API's — with one iconic twist: you, the user, must specify the kind of memory as the first argument to each function. Wanna know what kinds are even implemented? Scroll to the KINDS section below, bestie, full description's there. For file-backed kind of memory, go look at memkind_create_pmem() or memkind_create_pmem_with_config(). For a memory kind created on a user-specified area, please check memkind_create_fixed().

memkind_malloc() allocates size bytes of uninitialized memory of the specified kind. The allocated space is suitably aligned (after possible pointer coercion) for storage of any type of object. If size is 0, then memkind_malloc() returns NULL. no free real estate, no nothing 💀

memkind_calloc() allocates space for num objects, each size bytes in length, in memory of the specified kind. The result is identical to calling memkind_malloc() with an argument of num * size — except the allocated memory is explicitly initialized to zero bytes. If num or size is 0, then memkind_calloc() returns NULL.

memkind_realloc() changes the size of the previously allocated memory referenced by ptr to size bytes of the specified kind. The contents of the memory remain unchanged up to the lesser of the new and old sizes. If the new size is larger, the contents of the newly allocated portion of the memory are undefined (oop). Upon success, the memory referenced by ptr is freed and a pointer to the newly allocated memory is returned.

Note: memkind_realloc() may move the memory allocation, resulting in a different return value than ptr. so don't go holding onto that old pointer like it's your ex, okay?

If ptr is NULL, the memkind_realloc() function behaves identically to memkind_malloc() for the specified size. If size is equal to zero, and ptr is not NULL, then the call is equivalent to memkind_free(kind, ptr) and NULL is returned. The address ptr, if not NULL, must have been returned by a previous call to memkind_malloc(), memkind_calloc(), memkind_realloc(), memkind_defrag_reallocate() or memkind_posix_memalign() with the same kind as specified to the call to memkind_realloc(). Otherwise, if memkind_free(kind, ptr) was called before, undefined behavior occurs. In cases where the kind is unknown in the context of the call to memkind_realloc(), NULL can be given as the kind specified to memkind_realloc(), but this will require an internal look up for correct kind.

Note: The look up for kind could result in serious performance penalty, which can be avoided by specifying a correct kind. fr fr, just tell it the kind.

If kind is NULL and ptr is NULL, then memkind_realloc() returns NULL and sets errno to EINVAL.

memkind_posix_memalign() allocates size bytes of memory of a specified kind such that the allocation's base address is an even multiple of alignment, and returns the allocation in the value pointed to by memptr. The requested alignment must be a power of 2 at least as large as sizeof(void*). If size is 0, then memkind_posix_memalign() returns 0, with a NULL returned in memptr.

memkind_malloc_usable_size() function provides the same semantics as malloc_usable_size(3), but operates on specified kind.

Note: In cases where the kind is unknown in the context of the call to memkind_malloc_usable_size(), NULL can be given as the kind specified to memkind_malloc_usable_size(), but this could require a internal look up for correct kind. memkind_malloc_usable_size() is supported by TBB heap manager described in ENVIRONMENT section since Intel TBB 2019 Update 4.

memkind_defrag_reallocate() reallocates the object conditionally inside specific kind. Function determines if it's worthwhile to move allocation to reduce degree of external fragmentation of the heap. In case of failure the function returns NULL; otherwise the function returns a pointer to reallocated memory, and the memory referenced by ptr was released and should not be accessed. If ptr is NULL, then memkind_defrag_reallocate() returns NULL. In cases where the kind is unknown in the context of the call to memkind_defrag_reallocate(), NULL can be given as the kind specified to memkind_defrag_reallocate(), but this will require an internal look up for correct kind.

Note: The look up for kind could result in serious performance penalty, which can be avoided by specifying a correct kind.

memkind_detect_kind() returns the kind associated with allocated memory referenced by ptr. This pointer must have been returned by a previous call to memkind_malloc(), memkind_calloc(), memkind_realloc(), memkind_defrag_reallocate() or memkind_posix_memalign(). If ptr is NULL, then memkind_detect_kind() returns NULL.

Note: This function has non-trivial performance overhead. it's giving detective but make it expensive 🔍

memkind_free() causes the allocated memory referenced by ptr to be made available for future allocations. This pointer must have been returned by a previous call to memkind_malloc(), memkind_calloc(), memkind_realloc(), memkind_defrag_reallocate() or memkind_posix_memalign(). Otherwise, if memkind_free(kind, ptr) was already called before, undefined behavior occurs. If ptr is NULL, no operation is performed. In cases where the kind is unknown in the context of the call to memkind_free(), NULL can be given as the kind specified to memkind_free(), but this will require an internal look up for correct kind.

Note: The look up for kind could result in serious performance penalty, which can be avoided by specifying a correct kind.

🎀 KIND CONFIGURATION MANAGEMENT:

The functions in this section define a way to create, delete and update kind specific configuration. Except for memkind_config_new(), the user must specify the memkind configuration with the first argument to each function. The API described here is most useful with file-backed kind of memory, e.g. memkind_create_pmem_with_config() method.

memkind_config_new() creates the memkind configuration. baby config, brand new ✨

memkind_config_delete() deletes previously created memkind configuration, which must have been returned by a previous call to memkind_config_new().

memkind_config_set_path() updates the memkind pmem_dir configuration parameter, which specifies the directory path where file-backed kind of memory will be created.

Note: This function does not validate that pmem_dir specifies a valid path. she trusts you. maybe too much.

memkind_config_set_size() updates the memkind pmem_size configuration parameter, which allows to limit the file-backed kind memory partition.

Note: This function does not validate that pmem_size is in valid range.

memkind_config_set_memory_usage_policy() updates the memkind policy configuration parameter, which allows to tune up memory utilization. The user should set the value based on the characteristics of the application that is using the library (e.g. prioritize memory usage, CPU utilization). For more details about policy, see the MEMORY USAGE POLICY section below.

Note: This function does not validate that policy is in valid range.

👑 KIND MANAGEMENT:

There are built-in kinds that are always available, and these are enumerated in the KINDS section. The user can also create their own kinds of memory. This section describes the API's that enable the tracking of the different kinds of memory and determining their properties. main character energy for real.

memkind_create_fixed() is the "bestie I brought my own memory" function 💅 — it creates a kind sitting on top of a chunk of memory you pick. That memory can be allocated in any possible way, fr: it might be a static array, or an mmaped area, whatever. You can specify any properties you want using functions such as mbind. And you are also the one responsible for de-allocating that memory after the kind gets destroyed — the library is NOT doing it for you, no cap. That memory area must remain valid until fixed_kind is destroyed. The area starts at address addr and has size size. When the heap manager runs out of memory (the memory located under your user-specified area), a call to memkind_malloc() returns NULL and errno is set to ENOMEM. 🫠

memkind_create_pmem() is the convenient one ✨ — it creates a file-backed kind of memory. It allocates a temporary file in the given directory dir. That file is created in a fashion similar to tmpfile(3), so the file name does not appear when the directory is listed, and the space is automatically freed when the program terminates. The file is truncated to a size of max_size bytes and the resulting space is memory-mapped. 🌈

Heads up bestie: the actual file system space is not allocated immediately — only on a call to memkind_pmem_mmap() (see memkind_pmem(3)). This is what lets you create a pmem memkind of a pretty large size without needing to reserve in advance the corresponding file system space for the entire heap. 👑

If the value of max_size equals 0, the pmem memkind is only limited by the capacity of the file system mounted under the dir argument. The minimum max_size value which allows limiting the size of the kind by the library is defined as MEMKIND_PMEM_MIN_SIZE. Calling memkind_create_pmem() with a size smaller than that and different than 0 will return an error. The maximum allowed size is not limited by memkind, but by the file system specified by the dir argument. The max_size you pass in is the raw size of the memory pool, and jemalloc will use some of that space for its own metadata. Returns zero if the pmem memkind is created successfully, or an error code from the ERRORS section if not. 💖

memkind_create_pmem_with_config() is the second function used to create a file-backed kind of memory. It behaves simillar to memkind_create_pmem() but instead of passing dir and max_size arguments, it uses the config param to specify characteristics of the created file-backed kind of memory (see KIND CONFIGURATION MANAGEMENT section). 🎀

memkind_create_kind() creates a kind that allocates memory with a specific memory type, memory binding policy and flags (see MEMORY FLAGS section). The memtype_flags (see MEMORY TYPES section) determine the memory types to allocate, and the policy argument is the policy for specifying page binding to memory types selected by memtype_flags. Returns zero if the specified kind is created successfully, or an error code from the ERRORS section if not. ✨

memkind_destroy_kind() destroys a previously created kind object, which must have been returned by a previous call to memkind_create_pmem(), memkind_create_pmem_with_config() or memkind_create_kind(). Otherwise, or if memkind_destroy_kind(kind) was already called before, undefined behavior occurs — like, don't do that, seriously 💀. Note that when the kind was returned by memkind_create_kind(), all allocated memory must be freed before the kind is destroyed, otherwise this will cause a memory leak. When the kind was returned by memkind_create_pmem() or memkind_create_pmem_with_config(), all allocated memory will be freed after the kind will be destroyed. 🧚

memkind_check_available() returns zero if the specified kind is available, or an error code from the ERRORS section if it is not. 🦋

memkind_get_capacity() returns the memory capacity of nodes available to a given kind (file size or filesystem capacity in case of a file-backed PMEM kind; total area size in the case of a fixed-kind), or -1 in case of an error. The supported kinds are: MEMKIND_DEFAULT, MEMKIND_HIGHEST_CAPACITY, MEMKIND_HIGHEST_CAPACITY_LOCAL, MEMKIND_LOWEST_LATENCY_LOCAL, MEMKIND_HIGHEST_BANDWIDTH_LOCAL, MEMKIND_HUGETLB, MEMKIND_INTERLEAVE, MEMKIND_HBW, MEMKIND_HBW_ALL, MEMKIND_HBW_INTERLEAVE, MEMKIND_DAX_KMEM, MEMKIND_DAX_KMEM_ALL, MEMKIND_DAX_KMEM_INTERLEAVE, MEMKIND_REGULAR, file-backed PMEM and fixed-kind. kind. 😭 For MEMKIND_HUGETLB only pages with a default size of 2MB are supported. ⭐

memkind_check_dax_path() returns zero if the file-backed kind memory in the specified directory path pmem_dir can be created with the DAX attribute, or an error code from the ERRORS section if it is not. 🔥

MEMKIND_PMEM_MIN_SIZE — the minimum size which allows to limit the file-backed memory partition. 💜

📊 STATISTICS

The functions described in this section define a way to get specific memory allocation statistics. Bestie, we love data. 🍓

memkind_update_cached_stats() is used to force an update of cached dynamic allocator statistics. Statistics are not updated real-time by the memkind library, and this method allows you to force its update. 👑

memkind_get_stat() retrieves a statistic of the specified type and returns it in value. For more details about stat see the MEMORY STATISTICS TYPE section below. The measured statistic applies to a specific kind — when NULL is given as kind, then the statistic applies to memory used by the whole memkind library.

Note: You need to call memkind_update_cached_stats() before calling memkind_get_stat() because statistics are cached by the memkind library. Not me reminding you twice, iykyk 😭

memkind_stats_print() prints summary statistics. This function wraps jemalloc's function je_malloc_stats_print(). It uses the write_cb function to print the output. While providing a custom writer function, use syscall(2) rather than write(2). Pass NULL in order to use the default write_cb function which prints the output to the stderr. Use the cbopaque parameter in order to pass some data to your write_cb function. Pass additional options using opts. For more details on opts see the MEMORY STATISTICS PRINT OPTIONS section below. Returns MEMKIND_ERROR_INVALID when it failed to parse the options string, MEMKIND_SUCCESS on success. ✨

🎀 DECORATORS

The memkind library enables the user to define decorator functions that can be called before and after each memkind heap management API. 🌸

The decorators that are called at the beginning of the function end are named after that function with _pre appended to the name, and those that are called at the end of the function are named after that function with _post appended to the name. These are weak symbols, and if they are not present at link time they are not called. The memkind library does not define these symbols — they are reserved for user definition. 💅

These decorators can be used to track calls to the heap management interface or to modify parameters. The decorators that are called at the beginning of the allocator pass all inputs by reference, and the decorators that are called at the end of the allocator pass the output by reference. This enables the modification of the input and output of each heap management function by the decorators. 🦋

🚀 ENHANCEMENTS

memkind_set_bg_threads() enables/disables internal background worker threads in jemalloc. That's it, that's the whole enhancement, slay. 🌈

📚 LIBRARY VERSION

The memkind library version scheme consist of major, minor and patch numbers separated by dot. Combining those numbers, we got the following representation:

major.minor.patch

where:

  • -major — number is incremented whenever the API is changed (loss of backward compatibility) 💀
  • -minor — number is incremented whenever additional extensions are introduced or behavior has been changed
  • -patch — number is incremented whenever small bug fixes are added

The memkind library provides a numeric representation of the version by exposing the following API:

memkind_get_version() returns the version number represented by a single integer number, obtained from the formula:

major * 1000000 + minor * 1000 + patch
Note: major < 1 means unstable API. 👀

API standards:

  • -STANDARD — API, API is considered as stable
  • -NON-STANDARD — API, API is considered as stable, however this is not a standard way to use memkind
  • -EXPERIMENTAL — API, API is considered as unstable and the subject to change

💖 RETURN VALUE

memkind_calloc(), memkind_malloc(), memkind_realloc() and memkind_defrag_reallocate() returns the pointer to the allocated memory or NULL if the request fails. 🫶

memkind_malloc_usable_size() returns the number of usable bytes in the block of allocated memory pointed to by ptr, a pointer to a block of memory allocated by memkind_malloc() or a related function. If ptr is NULL, 0 is returned.

memkind_free() and memkind_error_message() do not have return values. 🎀

All other memkind API's return 0 upon success and an error code defined in the ERRORS section upon failure. The memkind library avoids setting errno directly, but calls to underlying libraries and system calls may set errno (e.g. memkind_create_pmem()). ⭐

KINDS 🌈💖

okay bestie, here are all the memory kinds you can choose from. each one is a whole vibe, no cap. 💅

  • MEMKIND_DEFAULT — default allocation using standard memory and default page size. literally the basic one. ✨
  • MEMKIND_HIGHEST_CAPACITY — allocate from a NUMA node(s) that has the highest capacity among all nodes in the system. going for the biggest, periodt. 👑
  • MEMKIND_HIGHEST_CAPACITY_PREFERRED — same as MEMKIND_HIGHEST_CAPACITY except that if there is not enough memory in the NUMA node that has the highest capacity in the local domain to satisfy the request, the allocation will fall back on other memory NUMA nodes. 💅
Note: for this kind, the allocation will not succeed if there are two or more NUMA nodes that have the highest capacity. 😭
  • MEMKIND_HIGHEST_CAPACITY_LOCAL — allocate from a NUMA node that has the highest capacity among all NUMA Nodes from the local domain. NUMA Nodes have the same local domain for a set of CPUs associated with them, e.g. socket or sub-NUMA cluster. 🦋
Note: if there are multiple NUMA nodes in the same local domain that have the highest capacity - allocation will be done from NUMA node with worse latency attribute. 💀

This kind requires locality information described in SYSTEM CONFIGURATION section. 📍

  • MEMKIND_HIGHEST_CAPACITY_LOCAL_PREFERRED — same as MEMKIND_HIGHEST_CAPACITY_LOCAL except that if there is not enough memory in the NUMA node that has the highest capacity to satisfy the request, the allocation will fall back on other memory NUMA nodes. 🫶
  • MEMKIND_LOWEST_LATENCY_LOCAL — allocate from a NUMA node that has the lowest latency among all NUMA Nodes from the local domain. NUMA Nodes have the same local domain for a set of CPUs associated with them, e.g. socket or sub-NUMA cluster. ⚡
Note: if there are multiple NUMA nodes in the same local domain that have the lowest latency - allocation will be done from NUMA node with smaller memory capacity. 🙈

This kind requires locality and memory performance characteristics information described in SYSTEM CONFIGURATION section. 📊

  • MEMKIND_LOWEST_LATENCY_LOCAL_PREFERRED — same as MEMKIND_LOWEST_LATENCY_LOCAL except that if there is not enough memory in the NUMA node that has the lowest latency to satisfy the request, the allocation will fall back on other memory NUMA nodes. 🍓
  • MEMKIND_HIGHEST_BANDWIDTH_LOCAL — allocate from a NUMA node that has the highest bandwidth among all NUMA Nodes from the local domain. NUMA Nodes have the same local domain for a set of CPUs associated with them, e.g. socket or sub-NUMA cluster. 🌈
Note: if there are multiple NUMA nodes in the same local domain that have the highest bandwidth - allocation will be done from NUMA node with smaller memory capacity. 💅

This kind requires locality and memory performance characteristics information described in SYSTEM CONFIGURATION section. 📈

  • MEMKIND_HIGHEST_BANDWIDTH_LOCAL_PREFERRED — same as MEMKIND_HIGHEST_BANDWIDTH_LOCAL except that if there is not enough memory in the NUMA node that has the highest bandwidth to satisfy the request, the allocation will fall back on other memory NUMA nodes. 🧚
  • MEMKIND_HUGETLB — allocate from standard memory using huge pages. 🐘
Note: this kind requires huge pages configuration described in SYSTEM CONFIGURATION section. ⚙️
  • MEMKIND_GBTLB (DEPRECATED) — allocate from standard memory using 1GB chunks backed by huge pages. 🚫
Note: this kind requires huge pages configuration described in SYSTEM CONFIGURATION section. ⚙️
  • MEMKIND_INTERLEAVE — allocate pages interleaved across all NUMA nodes with transparent huge pages disabled. 🎀
  • MEMKIND_HBW — allocate from the closest high bandwidth memory NUMA node(s) at the time of allocation. if there is not enough high bandwidth memory to satisfy the request errno is set to ENOMEM and the allocated pointer is set to NULL. 😭
Note: this kind requires memory performance characteristics information described in SYSTEM CONFIGURATION section. 📉
  • MEMKIND_HBW_ALL — same as MEMKIND_HBW except decision regarding closest NUMA node(s) is postponed until the time of first write. ⏳
  • MEMKIND_HBW_HUGETLB — same as MEMKIND_HBW except the allocation is backed by huge pages. 🐘
Note: this kind requires huge pages configuration described in SYSTEM CONFIGURATION section. ⚙️
  • MEMKIND_HBW_ALL_HUGETLB — combination of MEMKIND_HBW_ALL and MEMKIND_HBW_HUGETLB properties. 🧬
Note: this kind requires huge pages configuration described in SYSTEM CONFIGURATION section. ⚙️
  • MEMKIND_HBW_PREFERRED — same as MEMKIND_HBW except that if there is not enough high bandwidth memory to satisfy the request, the allocation will fall back on standard memory. 💖
Note: for this kind, the allocation will not succeed if two or more high bandwidth memory NUMA nodes are in the same shortest distance to the same CPU on which process is eligible to run. check on that eligibility is done upon starting the application. 👀
  • MEMKIND_HBW_PREFERRED_HUGETLB — same as MEMKIND_HBW_PREFERRED except the allocation is backed by huge pages. 🐘
Note: this kind requires huge pages configuration described in SYSTEM CONFIGURATION section. ⚙️
  • MEMKIND_HBW_GBTLB (DEPRECATED) — same as MEMKIND_HBW except the allocation is backed by 1GB chunks of huge pages. note that size can take on any value, but full gigabyte pages will allocated for each request, so remainder of the last page will be wasted. 💀
This kind requires huge pages configuration described in SYSTEM CONFIGURATION section. ⚙️
  • MEMKIND_HBW_PREFERRED_GBTLB (DEPRECATED) — same as MEMKIND_HBW_GBTLB except that if there is not enough high bandwidth memory to satisfy the request, the allocation will fall back on standard memory. 🫶
Note: this kind requires huge pages configuration described in SYSTEM CONFIGURATION section. ⚙️ for this kind, the allocation will not succeed if two or more high bandwidth memory NUMA nodes are in the same shortest distance to the same CPU on which process is eligible to run. check on that eligibility is done upon starting the application. 👀
  • MEMKIND_HBW_INTERLEAVE — same as MEMKIND_HBW except that the pages that support the allocation are interleaved across all high bandwidth nodes and transparent huge pages are disabled. 🎀
  • MEMKIND_DAX_KMEM — allocate from the closest persistent memory NUMA node at the time of allocation. if there is not enough memory in the closest persistent memory NUMA node to satisfy the request errno is set to ENOMEM and the allocated pointer is set to NULL. 😭
  • MEMKIND_DAX_KMEM_ALL — allocate from the closest persistent memory NUMA node available at the time of allocation. if there is not enough memory on any of persistent memory NUMA nodes to satisfy the request errno is set to ENOMEM and the allocated pointer is set to NULL. 💀
  • MEMKIND_DAX_KMEM_PREFERRED — same as MEMKIND_DAX_KMEM except that if there is not enough memory in the closest persistent memory NUMA node to satisfy the request, the allocation will fall back on other memory NUMA nodes. ✨
Note: for this kind, the allocation will not succeed if two or more persistent memory NUMA nodes are in the same shortest distance to the same CPU on which process is eligible to run. check on that eligibility is done upon starting the application. 👀
  • MEMKIND_DAX_KMEM_INTERLEAVE — same as MEMKIND_DAX_KMEM except that the pages that support the allocation are interleaved across all persistent memory NUMA nodes. 🌀
  • MEMKIND_REGULAR — allocate from regular memory using the default page size. regular means general purpose memory from the NUMA nodes containing CPUs. 🧠

MEMORY TYPES 🧠✨

the available types of memory:

  • MEMKIND_MEMTYPE_DEFAULT — standard memory, the same as process uses. 💙
  • MEMKIND_MEMTYPE_HIGH_BANDWIDTH — high bandwidth memory (HBM). there must be at least two memory types with different bandwidth to determine which is the HBM. ⚡

MEMORY BINDING POLICY 🎀💅

the available types of memory binding policy:

  • MEMKIND_POLICY_BIND_LOCAL — allocate local memory. if there is not enough memory to satisfy the request errno is set to ENOMEM and the allocated pointer is set to NULL. 😭
  • MEMKIND_POLICY_BIND_ALL — memory locality is ignored. if there is not enough memory to satisfy the request errno is set to ENOMEM and the allocated pointer is set to NULL. 💀
  • MEMKIND_POLICY_PREFERRED_LOCAL — allocate preferred memory that is local. if there is not enough preferred memory to satisfy the request or preferred memory is not available, the allocation will fall back on any other memory. 🫶
  • MEMKIND_POLICY_INTERLEAVE_LOCAL — interleave allocation across local memory. for n memory types the allocation will be interleaved across all of them. 🌈
  • MEMKIND_POLICY_INTERLEAVE_ALL — interleave allocation. locality is ignored. for n memory types the allocation will be interleaved across all of them. 🌀
  • MEMKIND_POLICY_MAX_VALUE — max policy value. 🔝

MEMORY FLAGS 🚩✨

the available types of memory flags:

  • MEMKIND_MASK_PAGE_SIZE_2MB — allocation backed by 2MB page size. 📏

MEMORY USAGE POLICY 📊💖

the available types of memory usage policy:

  • MEMKIND_MEM_USAGE_POLICY_DEFAULT — default memory usage policy. ✨
  • MEMKIND_MEM_USAGE_POLICY_CONSERVATIVE — conservative memory usage policy - prioritize memory usage at cost of performance. 🐢
Note: memory usage policies have no effect for TBB heap manager described in ENVIRONMENT section. 🚫

MEMORY STATISTICS TYPE 📈💜

the available types of memory statistics:

  • MEMKIND_STAT_TYPE_RESIDENT — maximum number of bytes in physically resident data pages mapped. 🏠
  • MEMKIND_STAT_TYPE_ACTIVE — total number of bytes in active pages. 🔥
  • MEMKIND_STAT_TYPE_ALLOCATED — total number of allocated bytes. 💾

✨ MEMORY STATISTICS PRINT OPTIONS ✨

Okay bestie, so when you're printing out your memory statistics, you've got options. Like, you can filter what you see. Here's the whole menu, no cap:

  • MEMKIND_STAT_PRINT_ALL — Prints literally every single statistic. All of them. The whole shebang. 💯
  • MEMKIND_STAT_PRINT_JSON_FORMAT — Prints the stats in JSON format. So you can parse it with your fave JSON tools. 🧚
  • MEMKIND_STAT_PRINT_OMIT_GENERAL — Omits general info that never changes during execution. So like, the boring static stuff? Gone. 🙈
  • MEMKIND_STAT_PRINT_OMIT_MERGED_ARENA — Omits merged arena statistics. Bye bye merged arena stats. 👋
  • MEMKIND_STAT_PRINT_OMIT_DESTROYED_MERGED_ARENA — Omits destroyed merged arena statistics. Those are gone too. 💀
  • MEMKIND_STAT_PRINT_OMIT_PER_ARENA — Omits per arena statistics. No per-arena breakdown for you. 🚫
  • MEMKIND_STAT_PRINT_OMIT_PER_SIZE_CLASS_BINS — Omits per size class statistics for bins. So the bins get no spotlight. 🗑️
  • MEMKIND_STAT_PRINT_OMIT_PER_SIZE_CLASS_LARGE — Omits per size class statistics for large objects. Large objects? Not today. 🐘
  • MEMKIND_STAT_PRINT_OMIT_MUTEX — Omits all mutex statistics. Mutex stats? Not invited to the party. 🔒
  • MEMKIND_STAT_PRINT_OMIT_EXTENT — Omits extent statistics. Extent stats? Ghosted. 👻

😱 ERRORS 😱

Okay so, memkind_posix_memalign() is special. It returns one of the POSIX standard error codes EINVAL or ENOMEM as defined in <errno.h> if an error occurs (and these have positive values, fyi). If the alignment parameter is not a power of two or is not a multiple of sizeof(void*), then EINVAL is returned. If there's insufficient memory to satisfy the request, then ENOMEM is returned. Periodt.

All functions other than memkind_posix_memalign() which have an integer return type return one of the negative error codes as defined in <memkind.h> and described below. So like, negative vibes only. 😭

  • MEMKIND_ERROR_UNAVAILABLE — Requested memory kind is not available. Like, it's just not there. 🙅
  • MEMKIND_ERROR_MBIND — Call to mbind(2) failed. Oops. 💥
  • MEMKIND_ERROR_MMAP — Call to mmap(2) failed. That's a no from the kernel. 🚫
  • MEMKIND_ERROR_MALLOC — Call to jemalloc's malloc() failed. Jemalloc said no. 😤
  • MEMKIND_ERROR_ENVIRON — Error parsing environment variable MEMKIND_*. So your env var is messy. 🧹
  • MEMKIND_ERROR_INVALID — Invalid input arguments to memkind routine. You gave it bad input, bestie. 🙈
  • MEMKIND_ERROR_TOOMANY — Error trying to initialize more than maximum MEMKIND_MAX_KIND number of kinds. Too many kinds! Slow down. ✋
  • MEMKIND_ERROR_BADOPS — Error memkind operation structure is missing or invalid. The ops structure is a mess. 📉
  • MEMKIND_ERROR_HUGETLB — Unable to allocate huge pages. Huge pages said no. 🐘
  • MEMKIND_ERROR_MEMTYPE_NOT_AVAILABLE — Error requested memory type is not available. That memory type? Not here. 🚫
  • MEMKIND_ERROR_OPERATION_FAILED — Error memkind operation failed. The operation just... failed. 💀
  • MEMKIND_ERROR_ARENAS_CREATE — Call to jemalloc's arenas.create() failed. Arena creation went wrong. 🏟️
  • MEMKIND_ERROR_RUNTIME — Unspecified run-time error. Something went wrong but we don't know what. 🤷

📁 FILES 📁

  • /usr/bin/memkind-hbw-nodes — Prints a comma-separated list of high bandwidth nodes. So you get a cute list like 1,2,3. 🎀
  • /usr/bin/memkind-auto-dax-kmem-nodes — Prints a comma-separated list of persistent memory NUMA nodes, which are automatically detected. Auto-magic! ✨

🌿 ENVIRONMENT 🌿

  • MEMKIND_HBW_NODES — This env var is a comma-separated list of NUMA nodes that are treated as high bandwidth. It uses the libnuma routine numa_parse_nodestring() for parsing, so the syntax described in the numa(3) man page for this routine applies: e.g. 1-3,5 is a valid setting. So like, 1-3,5 is totally valid. 🔥
  • MEMKIND_HBW_THRESHOLD — This env var is bandwidth in MB/s that is the threshold for identifying high bandwidth memory. The default threshold is 204800 (200 GB/s), which is used if this variable is not set. When set, it must be greater than or equal to 0. So no negative numbers, okay? 🚫
  • MEMKIND_DAX_KMEM_NODES — This env var is a comma-separated list of NUMA nodes that are treated as PMEM memory. Uses the libnuma routine numa_parse_nodestring() for parsing, so the syntax described in the numa(3) man page for this routine applies: e.g. 1-3,5 is a valid setting. Same vibe as above. 💜
  • MEMKIND_ARENA_NUM_PER_KIND — This env var allows leveraging internal mechanism of the library for setting number of arenas per kind. Value should be a positive integer (not greater than INT_MAX defined in <limits.h>). The user should set the value based on the characteristics of application that is using the library. Higher value can provide better performance in extremely multithreaded applications at the cost of memory overhead. See section IMPLEMENTATION NOTES of jemalloc(3) for more details about arenas. So basically, more arenas = more perf but more memory. Your call. 🤔
  • MEMKIND_HOG_MEMORY — Controls behavior of memkind with regards to returning memory to underlying OS. Setting MEMKIND_HOG_MEMORY to 1 causes memkind to not release memory to OS in anticipation of memory reuse soon. This will improve latency of 'free' operations but increase memory usage. Note: For file-backed kind memory will be released to OS only after calling memkind_destroy_kind(), not after 'free' operations. In context of MEMKIND_MEM_USAGE_POLICY_CONSERVATIVE memory usage policy - it will also impact memory coalescing and results that blocks pages will be often reused (better memory usage at cost of performance). So like, it hogs memory but makes free faster. 🐷
  • MEMKIND_DEBUG — Controls logging mechanism in memkind. Setting MEMKIND_DEBUG to 1 enables printing messages like errors and general information about environment to stderr. So you get debug tea. ☕
  • MEMKIND_BACKGROUND_THREAD_LIMIT — Enable background worker threads. Value should be from range 0 to maximum number of cpus. Setting MEMKIND_BACKGROUND_THREAD_LIMIT to specific value will limit maximum number of background worker threads to this value. 0 means maximum number of background worker threads will be limited to maximum number of cpus. So set it to limit your threads. 🧵
  • MEMKIND_HEAP_MANAGER — Controls heap management behavior in memkind library by switching to one of the available heap managers. Values:
    • JEMALLOC — sets the jemalloc heap manager
    • TBB — sets the Intel Threading Building Blocks heap manager. This option requires installed Intel Threading Building Blocks library.

If the MEMKIND_HEAP_MANAGER is not set then the jemalloc heap manager will be used by default. So jemalloc is the default bestie. 💅

🛠️ SYSTEM CONFIGURATION 🛠️

Interfaces for obtaining 2MB (HUGETLB) memory need allocated huge pages in the kernel's huge page pool.

  • HUGETLB (huge pages) — Current number of "persistent" huge pages can be read from /proc/sys/vm/nr_hugepages file. Proposed way of setting hugepages is: sudo sysctl vm.nr_hugepages=<number_of_hugepages>. More information can be found here: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt

Interfaces for obtaining locality information are provided by libhwloc dependency. Functionality based on locality requires that memkind library is configured and built with the support of libhwloc (./configure --enable-hwloc).

Interfaces for obtaining memory performance characteristics information are based on HMAT (Heterogeneous Memory Attribute Table) https://uefi.org/sites/default/files/resources/ACPI_6_3_final_Jan30.pdf Functionality based on memory performance characteristics requires that platform configuration fully supports HMAT and memkind library is configured and built with the support of libhwloc (./configure --enable-hwloc).

Note: For a given target NUMA Node, the OS exposes only the performance characteristics of the best performing NUMA node.

libhwloc can be reached on: https://www.open-mpi.org/projects/hwloc

🔗 STATIC LINKING 🔗

When linking statically against memkind, libmemkind.a should be used together with its dependencies libnuma and pthread. Pthread can be linked by adding /usr/lib64/libpthread.a as a dependency (exact path may vary). Typically libnuma will need to be compiled from sources to use it as a static dependency. libnuma can be reached on GitHub: https://github.com/numactl/numactl

⚠️ KNOWN ISSUES ⚠️

  • HUGETLB (huge pages) — There might be some overhead in huge pages consumption caused by heap management. If your allocation fails because of OOM, please try to allocate extra huge pages (e.g. 8 huge pages).

Copyright (C) 2014 - 2021 Intel Corporation. All rights reserved.

So like, all rights reserved by Intel. Don't copy without permission, bestie. 💖

👀 SEE ALSO 👀

This page was brought to you by Intel Corporation, dated 2015-03-31, and it's MEMKIND(3) in the building! Stay slay, bestie. 💅✨

📦 where this page comes from