VMALLOC(3)

💖 VMALLOC(3) 💖 — from the Library Functions Manual, bestie 🫶

✨ NAME ✨

vmalloc — virtual memory allocation. That's the name, and honestly it's giving main-character memory energy 💅.

🎀 SYNOPSIS 🎀

#include <vmalloc.h>

💅 Regions 💅

Vmalloc_t* vmopen(Vmdisc_t* disc, Vmethod_t* meth, int flags);
int vmclose(Vmalloc_t*);
int vmclear(Vmalloc_t*);
int vmcompact(Vmalloc_t* region);
int vmset(Vmalloc_t* region, int flags, int type);
Vmalloc_t* Vmheap;
Vmdisc_t* vmdisc(Vmalloc_t* region, Vmdisc_t* disc);
Vmalloc_t* vmmopen(char* file, int project, ssize_t size);
Void_t* vmmvalue(Vmalloc_t* vm, int key, Void_t* value, int op);
Void_t* vmmcleanup(Vmalloc_t* vm);
Void_t* vmmaddress(size_t size);

🌸 Allocation functions 🌸

Void_t* vmalloc(Vmalloc_t* region, size_t size);
Void_t* vmalign(Vmalloc_t* region, size_t size, size_t align);
Void_t* vmresize(Vmalloc_t* region, Void_t* addr, size_t size, int type);
int vmfree(Vmalloc_t* region, Void_t* addr);
Void_t* vmnewof(Vmalloc_t* region, Void_t* addr, type, size_t n, size_t x);
Void_t* vmoldof(Vmalloc_t* region, Void_t* addr, type, size_t n, size_t x);
Void_t* vmgetmem(Vmalloc_t* region, Void_t* addr, size_t size);

🦋 Debugging 🦋

int vmdebug(int);
int vmdbcheck(Vmalloc_t* vm);
int vmdbwatch(Void_t* addr);
static void vmdbwarn(Vmalloc_t*, char* mesg, int n);

⭐ Profiling ⭐

void vmprofile(Vmalloc_t* vm, int fd);

🔥 Information and statistics 🔥

int vmbusy(Vmalloc_t* region);
Vmalloc_t* vmregion(Void_t* addr);
Void_t* vmsegment(Vmalloc_t* region, Void_t* addr);
int vmwalk(Vmalloc_t* region, int(*walkf)(Vmalloc_t*, Void_t*, size_t, Vmdisc_t*);
long vmaddr(Vmalloc_t* region, Void_t* addr);
long vmsize(Vmalloc_t* region, Void_t* addr);
int vmstat(Vmalloc_t* vm, Vmstat_t* statb);
int vmtrace(int fd);
int vmtrbusy(Vmalloc_t* vm);
Void_t* vmdata(Vmalloc_t* vm);

🍓 Malloc-compatible functions 🍓

Void_t* malloc(size_t size);
Void_t* realloc(Void_t* addr, size_t size);
Void_t* calloc(size_t n_obj, size_t s_obj);
int cfree(Void_t* addr);
void free(Void_t* addr);
Void_t* memalign(size_t align, size_t size);
Void_t* valloc(size_t size);
int setregmax(int regmax);

🌈 DESCRIPTION 🌈

Okay so these functions for dynamic storage allocation work in regions of memory. Each region has an allocation method for parceling out blocks of storage and a memory discipline for obtaining raw space. Automatic locking prevents interference by reentrant access to a region.

Pointers to space have type Void_t* where Void_t is #defined as void if possible, otherwise char. Space is counted in type size_t. That's the setup, and it's lowkey iconic ✨.

🎀 Regions 🎀

Regions have type Vmalloc_t. Two predefined regions are pointed to by:

• Vmheap — a general-purpose region, with best-fit allocation, and system memory discipline Vmdcsystem.

These functions manipulate regions:

• vmopen — creates a region with memory discipline disc, allocation method meth, and a setting for control flags. It returns a pointer to the region on success and NULL on failure. The flags, represented by bit values or-ed together, are:
  • VM_SHARE — this region may be accessed concurrently by multiple threads or processes.
  • VM_TRACE — place tracing messages for each allocation event on the tracing file established by vmtrace.
  • VM_DBCHECK, VM_DBABORT — see Debugging below.
• vmclose — closes a region and releases all associated memory according to the region's discipline. The first segment obtained from the discipline's memoryf function (see Disciplines below) will be the last released. vmclose returns -1 on failure and a non-negative value otherwise.
• vmclear — frees all allocated blocks in region regardless of methods. It returns -1 on failure and a non-negative value otherwise.
• vmcompact — releases as much of a region's free space to its discipline's memoryf function as possible. It returns a nonnegative value on success and -1 on failure.
• vmset — adjusts and queries a region's flags. The indicated flags are turned on if type is nonzero, off if zero. vmset returns the previous value of all flags. Thus, vmset(region,0,0) queries the flags without changing them. In addition to the settable flags, one of VM_MTBEST, VM_MTDEBUG, VM_MTPROFILE, VM_MTPOOL, or VM_MTLAST is returned to indicate the method used in creating the region.
• vmdisc — changes the discipline of region to the given new discipline disc if disc is not NULL and its memoryf function is the same as the current discipline. If the current discipline has an exceptf function, it will be called with event VM_DISC. This function always returns the current discipline.
• vmmopen — creates a region to allocate memory obtained via either mmap(2) when project < 0 or shmget(2) when project >= 0. The region is built from a single memory segment guaranteed to be at least as large as size. When project >= 0, file and project are used in a call to ftok(3) to get a key suitable for getting a shared memory segment via shmget(2). Otherwise, file is the backing store for the mapped data. In this case, not only the region may be used concurrently by different processes, it is also persistent. That is, process could even exit, move the file to a different but similar machine then restart and open the same region to continue working.

Note that Vmalloc can protect concurrent accesses only on region entry and exit for memory allocation operations. This means that at the time when regions are being opened or closed, there will be no protection for the memory segments being attached into or detached from process memory space. This limitation has a special impact on vmmopen() as follows.

A shared memory segment opened via vmmopen() corresponds uniquely to a combination of the file and project parameters. Thus, if multiple vmmopen() calls are done in the same process using a same combination of file and project, the joined behavior of such regions will be unpredictable when opening and closing are done concurrently with other memory allocation operations. Beware that this effect can be subtle with library functions that may attempt to create their own memory allocation regions.

• vmmvalue — manages pairs of key and value in a region opened via vmopen(). If op is VM_MMGET, the value associated with key is returned. If op is VM_MMSET, the value associated with key will be set to value. If op is VM_MMADD, the value associated with key will be treated as a signed long value to which val (also treated as a signed long value) will be added. The call always returns the updated data value associated with key.
• vmmcleanup — sets region up to remove backing store or shmid on closing.
• vmmaddress — computes an address suitable for attaching a shared memory segment or memory mapping a segment of file data of the given size. The address is chosen with hope to minimize collision with other activities related to memory such as growth of stack space or space used for dynamically linked libraries, etc.

🧚 Allocation functions 🧚

• vmalloc — returns a pointer to a block of the requested size in a region, aligned to the strictest alignment that is suitable for the needs of any basic data type. It returns NULL on failure.
• vmalign — works like vmalloc, but returns a block aligned to a common multiple of align and the strictest alignment.
• vmresize — attempts to change the length of the block pointed to by addr to the specified size. If that is impossible and type has at least one of VM_RSMOVE and VM_RSCOPY, a new block is allocated and the old block is freed. The bit VM_RSCOPY also causes the new block to be initialized with as much of the old contents as will fit. When a resized block gets larger, the new space will be cleared if type has the bit VM_RSZERO. vmresize returns a pointer to the final block, or NULL on failure. If addr is NULL, vmresize behaves like vmalloc; otherwise, if size is 0, it behaves like vmfree.
• vmfree — makes the currently allocated block pointed to by addr available for future allocations in its region. If addr is NULL, vmfree does nothing. It returns -1 on error, and nonnegative otherwise.
• vmnewof — is a macro function that attempts to change the length of the block pointed to by addr to the size n*sizeof(type)+x. If the block is moved, new space will be initialized with as much of the old content as will fit. Additional space will be set to zero.
• vmoldof — is similar to vmnewof but it neither copies data nor clears space.
• vmgetmem — provides a handy function to creat/close regions and allocate/free memory based on chunks of memory obtained from the heap region Vmheap.
  • vmgetmem(0,0,0) — this call opens a new region.
  • vmgetmem(region, 0, 0) — this call closes the given region.
  • vmgetmem(region,0,n) — this call allocates a block of length n and clears it to zeros.
  • vmgetmem(region,p,0) — this call frees the block p.
  • vmgetmem(region,p,n) — this call resizes the block p to length n and clears the new memory to zeros if the block grows. The block may be moved as deemed necessary by the allocator.

🦋 Memory disciplines 🦋

Memory disciplines have type Vmdisc_t, a structure with these members:

     Void_t* (*memoryf)(Vmalloc_t *region, Void_t* obj,
          size_t csz, size_t nsz, Vmdisc_t *disc);
     int (*exceptf)(Vmalloc_t *region, int type, Void_t* obj, Vmdisc_t *disc);
     int round;

• round — if this value is positive, all size arguments to the memoryf function will be multiples of it.
• memoryf — points to a function to get or release segments of space for the region.
• exceptf — if this pointer is not NULL, the function it points to is called to announce events in a region.

There are two standard disciplines, both with round being 0 and exceptf being NULL.

• Vmdcsystem — a discipline whose memoryf function gets space from the operation system via different available methods which include mmap(2), sbrk(2) and functions from the WIN32 API. For historical reason, Vmdcsbrk is also available and functions like Vmdcsystem.
• Vmdcheap — a discipline whose memoryf function gets space from the region Vmheap. A region with Vmdcheap discipline and Vmlast allocation is good for building throwaway data structures.

A memoryf function returns a pointer to a memory segment on success, and NULL on failure. When nsz >= 0 and csz > 0, the function first attempts to change the current segment addr to fit nsz (for example, nsz == 0 means deleting the segment addr). If this attempt is successful, it should return addr. Otherwise, if nsz > csz, the function may try to allocate a new segment of size nsz-csz. If successful, it should return the address of the new segment. In all other cases, it should return NULL.

Okay so, the exceptf function is called for events identified by type, and the coding is literally this:

• VM_OPEN — this event is raised at the start of the process to open a new region. Argument obj will be a pointer to an object of type Void_t* initialized to NULL before the call. The return value of exceptf is significant as follows:
  • On a negative return value, vmopen will terminate with failure.
  • On a zero return value, exceptf may set *((Void_t**)obj) to some non-NULL value to tell vmopen to allocate the region handle itself via memoryf. Otherwise, the region handle will be allocated from the Vmheap region.
  • On a positive return value, the new region is being reconstructed based on existing states of some previous region. In this case, exceptf should set *(Void_t**)obj to point to the field Vmalloc_t.data of the corresponding previous region (see VM_CLOSE below). If the handle of the previous region was allocated via memoryf as discussed above in the case of the zero return value, then it will be exactly restored. Otherwise, a new handle will be allocated from Vmheap. The ability to create regions sharing the same states allows for managing shared and/or persistent memory.
• VM_ENDOPEN — this event is raised at the end of the process to open a new region. The return value of exceptf will be ignored.
• VM_CLOSE — this event is raised at the start of the process to close a region. The return value of exceptf is significant as follows:
  • On a negative return value, vmclose immediately returns with failure.
  • On a zero return value, vmclose proceeds normally by calling memoryf to free all allocated memory segments and also freeing the region itself.
  • On a positive return value, vmclose will only free the region without deallocating the associated memory segments. That is, the field Vmalloc_t.data of the region handle remains intact. This is useful for managing shared and/or persistent memory (see VM_OPEN above).
• VM_ENDCLOSE — this event is raised at the end of the process to close a region. The return value of exceptf will be ignored.
• VM_NOMEM — an attempt to extend the region by the amount (size_t)obj failed. The region is unlocked, so the exceptf function may free blocks. If the function returns a positive value the memory request will be repeated.
• VM_DISC — the discipline structure is being changed.

🌸 ALLOCATION METHODS 🌸

Methods are of type Vmethod_t*.

• Vmbest — an approximately best-fit allocation strategy.
• Vmlast — a strategy for building structures that are only deleted in whole. Only the latest allocated block can be freed. This means that as soon as a block a is allocated, vmfree calls on blocks other than

bestie, that Vmlast sentence really does stop right there — no cap, I’m not hiding the rest from you 😭

• Vmpool — a strategy for blocks of one size, set by the first vmalloc call after vmopen or vmclear.
• Vmdebug — an allocation strategy with extra-stringent checking and locking. It is useful for finding misuses of dynamically allocated memory, such as writing beyond the boundary of a block, or freeing a block twice.
• Vmprofile — an allocation method that records and prints summaries of memory usage.

🐛 DEBUGGING 🐛

The method Vmdebug is used to debug common memory violation problems. When a problem is found, a warning message is written to file descriptor 2 (standard error). In addition, if flag VM_DBABORT is on, the program is terminated by calling abort(2). Each message is a line of self-explanatory fields separated by colons. The optional flag -DVMFL, if used during compilation, enables recording of file names and line numbers. The following functions work with method Vmdebug.

• vmdebug — resets the file descriptor to write out warnings to the given argument. By default, this file descriptor is 2, the standard error. vmdebug returns the previous file descriptor.
• vmdbcheck — checks a region using Vmdebug or Vmbest for integrity. If Vmdebug, this also checks for block overwriting errors. On errors, vmdbwarn is called. If flag VM_DBCHECK is on, vmdbcheck is called at each invocation of vmalloc, vmfree, or vmresize.
• vmdbwatch — causes address addr to be watched, and reported whenever met in vmalloc, vmresize or vmfree. The watch list has finite size and if it becomes full, watches will be removed in a first-in-first-out fashion. If addr is NULL, all current watches are canceled. vmdbwatch returns the watch bumped out due to an insertion into a full list or NULL otherwise.
• vmdbwarn — is an internal function that processes warning messages for discovered errors. It can’t be called from outside the vmalloc package, but is a good place to plant debugger traps because control goes there at every trouble.

📊 PROFILING 📊

The method Vmprofile is used to profile memory usage. Profiling data are maintained in private memory of a process so Vmprofile should be avoided from regions manipulating persistent or shared memory. The optional flag -DVMFL, if used during compilation, enables recording of file names and line numbers.

vmprofile prints memory usage summary. The summary is restricted to region vm if vm is not NULL; otherwise, it is for all regions created with Vmprofile. Summary records are written to file descriptor fd as lines with colon-separated fields. Here are some of the fields:

• n_alloc,n_free: number of allocation and free calls respectively. Note that a resize operation is coded as a free and an allocation.
• s_alloc,s_free: total amounts allocated and freed. The difference between these numbers is the amount of space not yet freed.
• max_busy, extent: these fields are only with the summary record for region. They show the maximum busy space at any time and the extent of the region.

🔎 INFORMATION AND STATISTICS 🔎

• vmbusy — returns the busy status of a region. A region is busy if some allocation operation is accessing it.
• vmregion — returns the region to which the block pointed to by addr belongs. This works only in regions that allocate with Vmbest, Vmdebug or Vmprofile.
• vmsegment — finds if some segment of memory in region contains the address addr. It returns the address of a found segment or NULL if none found.
• vmwalk — walks all segments in region or if region is NULL, all segments in all regions. At each segment, (*walkf)(vm,addr,size,disc) is called where vm is the region, addr is the segment, size is the size of the segment, and disc is the region’s discipline. If walkf returns a negative value, the walk stops and returns the same value. On success, vmwalk returns 0; otherwise, it returns -1.
• vmaddr — checks whether addr points to an address within some allocated block of the given region. If not, it returns -1. If so, it returns the offset from the beginning of the block. The function does not work for a Vmlast region except on the latest allocated block.
• vmsize — returns the size of the allocated block pointed to by addr. It returns -1 if addr does not point to a valid block in the region. Sizes may be padded beyond that requested; in particular no block has size 0. The function does not work for a Vmlast region except on the latest allocated block.
• vmstat — gathers statistics on the given region. If region is NULL, it computes statistics for the Malloc calls. This may include summing statistics from more than one regions constructed to avoid blocking due to parallel or asynchronous operations. If statb is not NULL, vmstat computes and stores the statistics in statb then returns 0. If statb is NULL, no statistics will be computed and the returned value is either 1 if the region is busy, i.e., being accessed by some allocation call or 0 otherwise.

A Vmstat_t structure has at least these members:

     int     n_busy;   /* # of busy blocks */
     int     n_free;   /* # of free blocks */
     size_t  s_busy;   /* total busy space */
     size_t  s_free;   /* total free space */
     size_t  m_busy;   /* maximum busy block size */
     size_t  m_free;   /* maximum free block size */
     int     n_seg;    /* count of segments */
     size_t  extent;   /* memory extent of region */
     int     n_region; /* total Malloc regions  */
     int     n_open; /* non-blocked operations */
     int     n_lock; /* blocked operations */
     int     n_probe; /* region searches */

Bookkeeping overhead is counted in extent, but not in s_busy or s_free.

vmtrace establishes file descriptor fd as the trace file and returns the previous value of the trace file descriptor. The trace descriptor is initially invalid. Output is sent to the trace file by successful allocation events when flag VM_TRACE is on.

Tools for analyzing traces are described in mtreplay(1). The trace record for an allocation event is a line with colon-separated fields, four numbers and one string.

• old — zero for a fresh allocation; the address argument for freeing and resizing.
• new — zero for freeing; the address returned by allocation or resizing.
• size — the size argument for allocation or resizing; the size freed by freeing. Sizes may differ due to padding for alignment.
• region — the address of the affected region.
• method — a string that tells the region’s method: best, last, pool, profile, or debug.

vmtrbusy outputs a trace of all currently busy blocks in region vm. This only works with the Vmbest, Vmdebug and Vmprofile methods.

vmdata returns the core data of the given region. The core data hold the data structures for allocated and free blocks. Depending on the region discipline, the core data of a region may be in shared or persistent memory even if the region pointer created with vmopen is always in private process memory.

🛍️ MALLOC-COMPATIBLE FUNCTIONS 🛍️

This set of functions implement malloc(3). They allocate via the Vmregion region which is initially set to be Vmheap.

Concurrent accesses are supported unless an application change Vmregion to something other than Vmheap. New regions may be created on the fly to avoid blocking. The maximum number of regions that can be created this way is set to 64 by default. An application could reduce this number by calling setregmax(regmax) to set the maximum number of these extra regions to regmax. setregmax() always returns the previous value.

These functions are instrumented for run-time debugging, profiling and tracing — it's giving built-in detective mode 🔍✨. For accurate reporting of files and line numbers, application code should include vmalloc.h and compile with -DVMFL. The VMALLOC_OPTIONS environment variable is checked once before the first memory allocation and controls the memory allocation method, debugging and tracing; its value is a comma or space separated list of [no]name[=value] options. The options are:

• abort — if Vmregion==Vmdebug then VM_DBABORT is set; otherwise _BLD_DEBUG enabled assertions abort() on failure. 💀
• break — try sbrk() block allocator first.
• check — if Vmregion==Vmbest then the region is checked every op. 👀
• free — disable addfreelist().
• keep — disable free — if code works with this enabled then it probably accesses freed data. bestie, that's a red flag 🚩
• method=method — sets Vmregion=method if not defined; method (Vm prefix optional) may be one of { best debug last profile }.
• mmap — try mmap() block allocator first if break is not set.
• period=n — sets Vmregion=Vmdebug if not defined; if Vmregion==Vmdebug the region is checked every n ops.
• profile=file — sets Vmregion=Vmprofile if not set; if Vmregion==Vmprofile then profile info printed to file file.
• start=n — sets Vmregion=Vmdebug if not defined; if Vmregion==Vmdebug region checking starts after n ops.
• trace=file — enables tracing to file file.
• warn=file — sets Vmregion=Vmdebug if not defined; if Vmregion==Vmdebug then warnings printed to file file.
• watch=address — sets Vmregion=Vmdebug if not defined; if Vmregion==Vmdebug then address is watched.

Output files are created if they don't exist. &n and /dev/fd/n name the file descriptor n which must be open for writing. The pattern %p in a file name is replaced by the process ID. 🫶

VMALLOC_OPTIONS combines the features of these obsolete environment variables: { VMCHECK VMDEBUG VMETHOD VMPROFILE VMTRACE }. the oldies but goodies 🧓💾

---

🦋 RECENT CHANGES 🦋

Vmlast: allocated blocks are now allowed to be resized (09/1998). slay, resizing rights ✅

👀 SEE ALSO 👀

• mtreplay(1)
• malloc(3)

🌸 AUTHOR 🌸

Kiem-Phong Vo, kpv@research.att.com 💌

Okay that's the whole page, bestie 💖 — source/version VMALLOC(3), dated 1 May 1998, signing off ✨🎀