RUMPUSER(3)

🦋 RUMPUSER(3) — rump kernel hypercall interface, bestie 🦋

🎀 NAME 🎀

rumpuser — the rump kernel hypercall interface. It's literally the bridge between a rump kernel and host resources, and it's giving "backstage pass to the host" energy. 💖

📚 LIBRARY 📚

<紫色?>rump User Library</紫色?> (librumpuser, -lrumpuser).

✨ SYNOPSIS ✨

#include <rump/rumpuser.h>

💫 DESCRIPTION 💫

Okay so the rumpuser hypercall interfaces are how a rump kernel reaches host resources. A hypervisor implementation must implement the routines in this page if it wants a rump kernel to run on the host. The implementation included in NetBSD is for POSIX-like hosts (*BSD, Linux, etc.). This document is split into sections based on the functionality group of each hypercall.

Since the hypercall interface is a C function interface, both the rump kernel and the hypervisor have to conform to the same ABI. The interface itself tries to assume as little as possible from the type systems — for example, off_t is passed as int64_t and enums are passed as ints. It's recommended that the hypervisor converts these to the native types before it starts processing the hypercall, for example by assigning the ints back to enums. Small conversion moment, big slay. ✨

🧵 UPCALLS AND RUMP KERNEL CONTEXT 🧵

A hypercall is always entered with the calling thread scheduled in the rump kernel. If the hypercall intends to block while waiting for an event, the hypervisor must first release the rump kernel scheduling context. In other words, the rump kernel context is a resource, and holding on to it while waiting for a rump kernel event/resource may lead to a deadlock. 😭

Even when there's no possibility of deadlock in the strict sense of the term, holding on to the rump kernel context while performing a slow hypercall — like reading a device — will prevent other threads (including the clock interrupt) from using that rump kernel context.

Releasing the context is done by calling the hyp_backend_unschedule() upcall which the hypervisor received from rump kernel as a parameter for rumpuser_init(). Before a hypercall returns back to the rump kernel, the returning thread must carry a rump kernel context. If the hypercall unscheduled itself, it must reschedule itself by calling hyp_backend_schedule().

🛠️ HYPERCALL INTERFACES 🛠️

🌱 Initialization 🌱

int rumpuser_init(int version, struct rump_hyperup *hyp)

Initialize the hypervisor. Yes bestie, this is the setup era. 💅

• version — hypercall interface version number that the kernel expects to be used. In case the hypervisor cannot provide an exact match, this routine must return a non-zero value.
• hyp — pointer to a set of upcalls the hypervisor can make into the rump kernel.

🧠 Memory allocation 🧠

int rumpuser_malloc(size_t len, int alignment, void **memp)

• len — amount of memory to allocate.
• alignment — size the returned memory must be aligned to. For example, if the value passed is 4096, the returned memory must be aligned to a 4k boundary.
• memp — return pointer for allocated memory.

void rumpuser_free(void *mem, size_t len)

• mem — memory to free.
• len — length of allocation. This is always equal to the amount the caller requested from the rumpuser_malloc() which returned mem.

📁 Files and I/O 📁

int rumpuser_open(const char *name, int mode, int *fdp)

Open name for I/O and associate a file descriptor with it. Notably, there needs to be no mapping between name and the host's file system namespace. For example, it's possible to associate the file descriptor with device I/O registers for special values of name. Iconic flexibility, honestly. 🎀

• name — the identifier of the file to open for I/O.
• mode — combination of the following:
  • RUMPUSER_OPEN_RDONLY — open only for reading.
  • RUMPUSER_OPEN_WRONLY — open only for writing.
  • RUMPUSER_OPEN_RDWR — open for reading and writing.
  • RUMPUSER_OPEN_CREATE — do not treat missing name as an error.
  • RUMPUSER_OPEN_EXCL — combined with RUMPUSER_OPEN_CREATE, flag an error if name already exists.
  • RUMPUSER_OPEN_BIO — the caller will use this file for block I/O, usually used in conjunction with accessing file system media. The hypervisor should treat this flag as advisory and possibly enable some optimizations for *fdp based on it.

Notably, the permissions of the created file are left up to the hypervisor implementation.

• fdp — an integer value denoting the open file is returned here.

int rumpuser_close(int fd)

Close a previously opened file descriptor.

• fd — the previously opened file descriptor to close.

int rumpuser_getfileinfo(const char *name, uint64_t *size, int *type)

• name — file for which information is returned. The namespace is equal to that of rumpuser_open().
• size — if non-NULL, size of the file is returned here.
• type — if non-NULL, type of the file is returned here. The options are RUMPUSER_FT_DIR, RUMPUSER_FT_REG, RUMPUSER_FT_BLK, RUMPUSER_FT_CHR, or RUMPUSER_FT_OTHER for directory, regular file, block device, character device or unknown, respectively.

void rumpuser_bio(int fd, int op, void *data, size_t dlen, int64_t off,
    rump_biodone_fn biodone, void *donearg)

Initiate block I/O and return immediately. Not me starting I/O and dipping. 💀

• fd — perform I/O on this file descriptor. The file descriptor must have been opened with RUMPUSER_OPEN_BIO.
• op — transfer data from the file descriptor with RUMPUSER_BIO_READ and transfer data to the file descriptor with RUMPUSER_BIO_WRITE. Unless RUMPUSER_BIO_SYNC is specified, the hypervisor may cache a write instead of committing it to permanent storage.
• data — memory address to transfer data to/from.
• dlen — length of I/O. The length is guaranteed to be a multiple of 512.
• off — offset into fd where I/O is performed.
• biodone — to be called when the I/O is complete. Accessing data is not legal after the call is made.
• donearg — opaque arg that must be passed to biodone.

int rumpuser_iovread(int fd, struct rumpuser_iovec *ruiov,
    size_t iovlen, int64_t off, size_t *retv)

int rumpuser_iovwrite(int fd, struct rumpuser_iovec *ruiov,
    size_t iovlen, int64_t off, size_t *retv)

These routines perform scatter-gather I/O which is not block I/O by nature and therefore cannot be handled by rumpuser_bio().

• fd — file descriptor to perform I/O on.
• ruiov — an array of I/O descriptors. It is defined as follows:

struct rumpuser_iovec {
        void *iov_base;
        size_t iov_len;
};

• iovlen — number of elements in ruiov.
• off — offset of fd to perform I/O on. This can either be a non-negative value or RUMPUSER_IOV_NOSEEK. The latter denotes that no attempt to change the underlying object's offset should be made. Using both types of offsets on a single instance of fd results in undefined behavior. 🚨 So maybe don't mix them, bestie.
• retv — number of bytes successfully transferred is returned here.

int rumpuser_syncfd(int fd, int flags, uint64_t start, uint64_t len)

Synchronizes fd with respect to backing storage. The other arguments are:

• flags — controls how synchronization happens. It must contain one of the following:
  • RUMPUSER_SYNCFD_READ — make sure that the next read sees writes from all other parties. This is useful for example in the case that fd represents memory to write and a DMA read is being performed.
  • RUMPUSER_SYNCFD_WRITE — flush cached writes.

The following additional parameters may be passed in flags:

• RUMPUSER_SYNCFD_BARRIER — issue a barrier. Outstanding I/O operations which were started before the barrier complete before any operations after the barrier are performed.
• RUMPUSER_SYNCFD_SYNC — wait for the synchronization operation to fully complete before returning. For example, this could mean that the data to be written to a disk has hit either the disk or non-volatile memory.

• start — offset into the object.
• len — the number of bytes to synchronize. The value 0 denotes until the end of the object.

⏰ Clocks ⏰

The hypervisor should support two clocks, one for wall time and one for monotonically increasing time, the latter of which may be based on some arbitrary time (e.g. system boot time). If this is not possible, the hypervisor must make a reasonable effort to retain semantics. Clock girlies, rise. 🕰️

int rumpuser_clock_gettime(int enum_rumpclock, int64_t *sec, long *nsec)

• enum_rumpclock — specifies the clock type. In case of RUMPUSER_CLOCK_RELWALL the wall time should be returned. In case of RUMPUSER_CLOCK_ABSMONO the time of a monotonic clock should be returned.
• sec — return value for seconds.
• nsec — return value for nanoseconds.

int rumpuser_clock_sleep(int enum_rumpclock, int64_t sec, long nsec)

so like, continuing with the sleep hypercall, enum_rumpclock is the vibe check for how the sleep works. if it's RUMPUSER_CLOCK_RELWALL, the sleep should last at least as long as specified — like a minimum nap time, no cap. if it's RUMPUSER_CLOCK_ABSMONO, the sleep should last until the hypervisor monotonic clock hits the specified absolute time. ⏰✨

• sec — sleep duration, seconds. exact semantics depend on clk.
• nsec — sleep duration, nanoseconds. exact semantics depend on clk.

💾 Parameter retrieval

int rumpuser_getparam(const char *name, void *buf, size_t buflen)

Retrieve a configuration parameter from the hypervisor. It is up to the hypervisor to decide how the parameters can be set. 🎀

• name — name of the parameter. If the name starts with an underscore, it means a mandatory parameter. The mandatory parameters are RUMPUSER_PARAM_NCPU which specifies the amount of virtual CPUs bootstrapped by the rump kernel and RUMPUSER_PARAM_HOSTNAME which returns a preferably unique instance name for the rump kernel.
• buf — buffer to return the data in as a string
• buflen — length of buffer

💀 Termination

void rumpuser_exit(int value)

Terminate the rump kernel with exit value value. If value is RUMPUSER_PANIC the hypervisor should attempt to provide something akin to a core dump. 💀

🖨️ Console output

Console output is divided into two routines: a per-character one and printf-like one. The former is used e.g. by the rump kernel's internal printf routine. The latter can be used for direct debug prints e.g. very early on in the rump kernel's bootstrap or when using the in-kernel routine causes too much skew in the debug print results (the hypercall runs outside of the rump kernel and therefore does not cause any locking or scheduling events inside the rump kernel). 🖨️

void rumpuser_putchar(int ch)

Output ch on the console.

void rumpuser_dprintf(const char *fmt, ...)

Do output based on printf-like parameters.

📡 Signals

A rump kernel should be able to send signals to client programs due to some standard interfaces including signal delivery in their specifications. Examples of these interfaces include setitimer(2) and write(2). The rumpuser_kill() function advises the hypercall implementation to raise a signal for the process containing the rump kernel. 📡

int rumpuser_kill(int64_t pid, int sig)

• pid — The pid of the rump kernel process that the signal is directed to. This value may be used as the hypervisor as a hint on how to deliver the signal. The value RUMPUSER_PID_SELF may also be specified to indicate no hint. This value will be removed in a future version of the hypercall interface.
• sig — Number of signal to raise. The value is in NetBSD signal number namespace. In case the host has a native representation for signals, the value should be translated before the signal is raised. In case there is no mapping between sig and native signals (if any), the behavior is implementation-defined.

A rump kernel will ignore the return value of this hypercall. The only implication of not implementing rumpuser_kill() is that some application programs may not experience expected behavior for standard interfaces.

As an aside, the rump_sp(7) protocol provides equivalent functionality for remote clients.

🎲 Random pool

int rumpuser_getrandom(void *buf, size_t buflen, int flags, size_t *retp)

• buf — buffer that the randomness is written to
• buflen — number of bytes of randomness requested
• flags — The value 0 or a combination of RUMPUSER_RANDOM_HARD (return true randomness instead of something from a PRNG) and RUMPUSER_RANDOM_NOWAIT (do not block in case the requested amount of bytes is not available).
• retp — The number of random bytes written into buf.

🧵 Threads

int rumpuser_thread_create(void *(*fun)(void *), void *arg, const char *thrname, int mustjoin, int priority, int cpuidx, void **cookie)

Create a schedulable host thread context. The rump kernel will call this interface when it creates a kernel thread. The scheduling policy for the new thread is defined by the hypervisor. In case the hypervisor wants to optimize the scheduling of the threads, it can perform heuristics on the thrname, priority and cpuidx parameters. 🧵

• fun — function that the new thread must call. This call will never return.
• arg — argument to be passed to fun
• thrname — Name of the new thread.
• mustjoin — If 1, the thread will be waited for by rumpuser_thread_join() when the thread exits.
• priority — The priority that the kernel requested the thread to be created at. Higher values mean higher priority. The exact kernel semantics for each value are not available through this interface.
• cpuidx — The index of the virtual CPU that the thread is bound to, or -1 if the thread is not bound. The mapping between the virtual CPUs and physical CPUs, if any, is hypervisor implementation specific.
• cookie — In case mustjoin is set, the value returned in cookie will be passed to rumpuser_thread_join().

void rumpuser_thread_exit(void)

Called when a thread created with rumpuser_thread_create() exits.

int rumpuser_thread_join(void *cookie)

Wait for a joinable thread to exit. The cookie matches the value from rumpuser_thread_create().

void rumpuser_curlwpop(int enum_rumplwpop, struct lwp *l)

Manipulate the hypervisor's thread context database. The possible operations are create, destroy, and set as specified by enum_rumplwpop:

• RUMPUSER_LWP_CREATE — Inform the hypervisor that l is now a valid thread context which may be set. A currently valid value of l may not be specified. This operation is informational and does not mandate any action from the hypervisor.
• RUMPUSER_LWP_DESTROY — Inform the hypervisor that l is no longer a valid thread context. This means that it may no longer be set as the current context. A currently set context or an invalid one may not be destroyed. This operation is informational and does not mandate any action from the hypervisor.
• RUMPUSER_LWP_SET — Set l as the current host thread's rump kernel context. A previous context must not exist.
• RUMPUSER_LWP_CLEAR — Clear the context previous set by RUMPUSER_LWP_SET. The value passed in l is the current thread and is never NULL.

struct lwp * rumpuser_curlwp(void)

Retrieve the rump kernel thread context associated with the current host thread, as set by rumpuser_curlwpop(). This routine may be called when a context is not set and the routine must return NULL in that case. This interface is expected to be called very often. Any optimizations pertaining to the execution speed of this routine should be done in rumpuser_curlwpop().

void rumpuser_seterrno(int errno)

Set an errno value in the calling thread's TLS. Note: this is used only if rump kernel clients make rump system calls.

🔒 Mutexes, rwlocks and condition variables

The locking interfaces have standard semantics, so we will not discuss each one in detail. The data types struct rumpuser_mtx, struct rumpuser_rw and struct rumpuser_cv used by these interfaces are opaque to the rump kernel, i.e. the hypervisor has complete freedom over them. 🔒

Most of these interfaces will (and must) relinquish the rump kernel CPU context in case they block (or intend to block). The exceptions are the "nowrap" variants of the interfaces which may not relinquish rump kernel context.

Mutexes:

void rumpuser_mutex_init(struct rumpuser_mtx **mtxp, int flags)

void rumpuser_mutex_enter(struct rumpuser_mtx *mtx)

void rumpuser_mutex_enter_nowrap(struct rumpuser_mtx *mtx)

int rumpuser_mutex_tryenter(struct rumpuser_mtx *mtx)

void rumpuser_mutex_exit(struct rumpuser_mtx *mtx)

void rumpuser_mutex_destroy(struct rumpuser_mtx *mtx)

void rumpuser_mutex_owner(struct rumpuser_mtx *mtx, struct lwp **lp)

Mutexes provide mutually exclusive locking. The flags, of which at least one must be given, are as follows:

• RUMPUSER_MTX_SPIN — Create a spin mutex. Locking this type of mutex must not relinquish rump kernel context even when rumpuser_mutex_enter() is used.
• RUMPUSER_MTX_KMUTEX — The mutex must track and be able to return the rump kernel thread that owns the mutex (if any). If this flag is not specified, rumpuser_mutex_owner() will never be called for that particular mutex.

Read/write locks:

void rumpuser_rw_init(struct rumpuser_rw **rwp)

void rumpuser_rw_enter(int enum_rumprwlock, struct rumpuser_rw *rw)

int rumpuser_rw_tryenter(int enum_rumprwlock, struct rumpuser_rw *rw)

int rumpuser_rw_tryupgrade(struct rumpuser_rw *rw)

void rumpuser_rw_downgrade(struct rumpuser_rw *rw)

void rumpuser_rw_exit(struct rumpuser_rw *rw)

void rumpuser_rw_destroy(struct rumpuser_rw *rw)

void rumpuser_rw_held(int enum_rumprwlock, struct rumpuser_rw *rw, int *heldp)

Read/write locks provide either shared or exclusive locking. The possible values for lk are RUMPUSER_RW_READER and RUMPUSER_RW_WRITER. Upgrading means trying to migrate from an already owned shared lock to an exclusive lock and downgrading means migrating from an already owned exclusive lock to a shared lock.

Condition variables:

void rumpuser_cv_init(struct rumpuser_cv **cvp)

void rumpuser_cv_destroy(struct rumpuser_cv *cv)

void rumpuser_cv_wait(struct rumpuser_cv *cv, struct rumpuser_mtx *mtx)

void rumpuser_cv_wait_nowrap(struct rumpuser_cv *cv, struct rumpuser_mtx *mtx)

int rumpuser_cv_timedwait(struct rumpuser_cv *cv, struct rumpuser_mtx *mtx, int64_t sec, int64_t nsec)

void rumpuser_cv_signal(struct rumpuser_cv *cv)

void rumpuser_cv_broadcast(struct rumpuser_cv *cv)

void rumpuser_cv_has_waiters(struct rumpuser_cv *cv, int *waitersp)

Condition variables are literally the “wait for the event, bestie” crew 💤. The mtx interlock is what eliminates the race between checking the predicate and sleeping on the condition variable — iconic — and the mutex should be released for the duration of the sleep in the normal atomic manner. The timedwait variant takes a specifier indicating a relative sleep duration; after that, the routine returns with ETIMEDOUT. If a timedwait is signaled before the timeout expires, the routine returns 0. Slay 💅

The order in which the hypervisor reacquires the rump kernel context and interlock mutex before returning into the rump kernel is as follows — pay attention, bestie:

• If the interlock mutex was initialized with both RUMPUSER_MTX_SPIN and RUMPUSER_MTX_KMUTEX, the rump kernel context is scheduled before the mutex is reacquired.
• In the case of a purely RUMPUSER_MTX_SPIN mutex, the mutex is acquired first.
• In the final case, the order is implementation-defined.

✨ RETURN VALUES ✨

All routines which return an integer return an errno value. The hypervisor must translate the value to the native errno namespace used by the rump kernel. Routines which do not return an integer may never fail. Fr fr 🫶

🌸 SEE ALSO 🌸

• rump(3)
• Antti Kantee, “Flexible Operating System Internals: The Design and Implementation of the Anykernel and Rump Kernels”, Aalto University Doctoral Dissertations, 2012, Section 2.3.2: The Hypercall Interface.
• For a list of all known implementations of the rumpuser interface, see https://github.com/rumpkernel/wiki/wiki/Platforms.

🦋 HISTORY 🦋

The rump kernel hypercall API was first introduced in NetBSD 5.0. The API described above first appeared in NetBSD 7.0. Periodt 💜

---

That’s the whole page, bestie 💖. Footer vibe check: source/version Debian, date July 15, 2023, page RUMPUSER(3) 🎀✨