RBD(8)

📦
version 20.2.4 · 20 versions
, in ubuntu stonking

💖 RBD(8) — Ceph manual, but make it cute 💖

okay bestie, sit down, hydrate, because we're about to learn rbd — the Ceph rados block device tool — and honestly? it ATE. no cap. 🫶

✨ NAME ✨

rbd — manage rados block device (RBD) images 🦋

✨ SYNOPSIS ✨

rbd [ -c ceph.conf ] [ -m monaddr ] [--cluster cluster-name]
[ -p | --pool pool ] [ command ... ]

✨ DESCRIPTION ✨

So rbd is the little utility that lets you mess with rados block device (RBD) images. who uses those? the Linux rbd driver, and the rbd storage driver for QEMU/KVM. iconic duo fr. 👑

RBD images are just simple block devices that get striped over objects and stored in a RADOS object store. and here's the vibe check on that: the size of the objects the image is striped over must be a power of two. that's not a suggestion bestie, that's the law. ⭐

✨ OPTIONS ✨

  • -c ceph.conf, --conf ceph.conf — use this ceph.conf config file instead of the default /etc/ceph/ceph.conf when it's figuring out monitor addresses at startup. basically "here's my own map, don't use the default one." 🗺️
  • -m monaddress[:port] — connect straight to the monitor you name (instead of going snooping through ceph.conf). direct message energy. 💌
  • --cluster cluster-name — use a different cluster name than the default one, which is ceph. not me cheating on the default cluster 💀
  • -p pool-name, --pool pool-name — interact with THIS pool. most commands literally require it, it's not optional bestie. 🏊
  • --namespace namespace-name — use a pre-defined image namespace inside a pool. cute little subfolder vibes. 🌸
  • --no-progress — do NOT output progress info. (heads up: for some commands that progress stuff goes to standard error by default.) like, keep me in suspense, I guess 🙈

✨ PARAMETERS ✨

  • --image-format format-id — tells it which object layout to use. default is 2. pick your fighter:
    • format 1 — (deprecated!) the OG format for a new rbd image. every version of librbd and the kernel rbd module understands it, BUT it does not support the newer features like cloning. it's giving legacy 🕸️
    • format 2 — the second rbd format. librbd has supported it since the Bobtail release, and the kernel rbd module has supported it since kernel 3.10 (except for "fancy" striping, which only got supported in kernel 4.17). it adds cloning support and is way more easily extensible so more features can show up in the future. main character format, honestly 👑
  • -s size-in-M/G/T, --size size-in-M/G/T — the size of the new rbd image, or the new size of an existing one, in M/G/T. no suffix? it assumes unit M. 🎀
  • --object-size size-in-B/K/M — object size in B/K/M. it gets rounded up to the nearest power of two; no suffix means unit B is assumed. default object size is 4M, smallest is 4K, maximum is 32M.
psst — you can change that default with the config option rbd_default_order, which takes a power of two (so the default object size is 2 ^ rbd_default_order). ✨
  • --stripe-unit size-in-B/K/M — stripe unit size in B/K/M. no suffix → unit B. see the striping section (it's below, don't worry) for the details. 🌈
  • --stripe-count num — how many objects to stripe over before looping back to the first object. again: striping section below, iykyk. 🔁
  • --snap snap — the snapshot name for the specific operation you're doing. 📸
  • --id username — the username (WITHOUT the client. prefix, okay?) to use with the map command. 🪪
  • --keyring filename — a keyring file holding a secret for the user you specified, used with the map command. if you don't give one, it searches the default keyring locations. 🔑
  • --keyfile filename — a file containing the secret key of the --id user, used with the map command. heads up: this option gets overridden by --keyring if you also specify that one. ⚠️
  • --shared lock-tag — an option for lock add that lets multiple clients lock the same image as long as they use the same tag. the tag is an arbitrary string. super useful when an image has to be open from more than one client at once — like during live migration of a virtual machine, or when it's used underneath a clustered file system. sharing is caring fr fr 🤝
  • --format format — output formatting. default is plain, and you can also do json or xml. 💅
  • --pretty-format — makes json or xml output way more human-readable. glow-up for your output ✨
  • -o krbd-options, --options krbd-options — which options to use when mapping or unmapping an image through the rbd kernel driver. krbd-options is a comma-separated list of options (similar to mount(8) mount options). check the kernel rbd (krbd) options section below for the full tea. ☕
  • --read-only — map the image read-only. equivalent to -o ro. look, don't touch 🔒
  • --image-feature feature-name — which RBD format 2 feature to enable when creating an image. you can enable multiple by repeating this option multiple times. supported features, let's go:
    • layering — layering support 🧅
    • striping — striping v2 support 🦓
    • exclusive-lock — exclusive locking support 🔐
    • object-map — object map support (requires exclusive-lock) 🗺️
    • fast-diff — fast diff calculations (requires object-map) ⚡
    • deep-flatten — snapshot flatten support 🫸
    • journaling — journaled IO support (requires exclusive-lock) 📓
    • data-pool — erasure coded pool support 🧩
  • --image-shared — says the image is gonna be used concurrently by multiple clients. this disables features that depend on exclusive ownership of the image. know the tradeoff bestie 💀
  • --whole-object — says the diff should be limited to the extents of a full object instead of showing intra-object deltas. and here's the slay part: when the object map feature is enabled on an image, limiting the diff to the object extents dramatically improves performance, since the differences can be computed by looking at the in-memory object map instead of querying RADOS for every single object in the image. fast. 🏎️🔥
  • --limit — the limit for the number of snapshots permitted. that's the cap, periodt. ✋

✨ COMMANDS ✨

🛠️ bench

bench --io-type <read | write | readwrite | rw> [--io-size
size-in-B/K/M/G/T] [--io-threads num-ios-in-flight] [--io-total
size-in-B/K/M/G/T] [--io-pattern seq | rand] [--rw-mix-read read
proportion in readwrite] image-spec

Generates a whole series of IOs to the image and measures the IO throughput and latency. no suffix given? unit B is assumed for BOTH --io-size and --io-total. the defaults are: --io-size 4096, --io-threads 16, --io-total 1G, --io-pattern seq, --rw-mix-read 50. 📊

🛠️ children

children snap-spec

Lists the clones of the image at the given snapshot. it checks every pool, and outputs the resulting poolname/imagename.

this one requires image format 2. 🧬

🛠️ clone

clone [--object-size size-in-B/K/M] [--stripe-unit size-in-B/K/M
--stripe-count num] [--image-feature feature-name] [--image-shared]
parent-snap-spec child-image-spec

Creates a clone — that's a copy-on-write child 🐣 — of the parent snapshot. the object size will be identical to the parent image's unless you specify otherwise. the size will be the same as the parent snapshot. the --stripe-unit and --stripe-count arguments are optional, but they have to be used together. they're a package deal bestie.

The parent snapshot must be protected (see rbd snap protect). and yep, this requires image format 2. 🔐

🛠️ config global get

config global get config-entity key

Get a global-level configuration override. 🌍

🛠️ config global list

config global list [--format plain | json | xml] [--pretty-format]
config-entity

List global-level configuration overrides. 📋

🛠️ config global set

config global set config-entity key value

Set a global-level configuration override. ✍️

🛠️ config global remove

config global remove config-entity key

Remove a global-level configuration override. 🗑️

🛠️ config image get

config image get image-spec key

Get an image-level configuration override. 🖼️

🛠️ config image list

config image list [--format plain | json | xml] [--pretty-format]
image-spec

List image-level configuration overrides. 📋

🛠️ config image set

config image set image-spec key value

Set an image-level configuration override. ✍️

🛠️ config image remove

config image remove image-spec key

Remove an image-level configuration override. 🗑️

🛠️ config pool get

config pool get pool-name key

Get a pool-level configuration override. 🏊

🛠️ config pool list

config pool list [--format plain | json | xml] [--pretty-format]
pool-name

List pool-level configuration overrides. 📋

🛠️ config pool set

config pool set pool-name key value

Set a pool-level configuration override. ✍️

🛠️ config pool remove

config pool remove pool-name key

Remove a pool-level configuration override. 🗑️

🛠️ cp

cp (src-image-spec | src-snap-spec) dest-image-spec

Copies the content of a src-image into the newly created dest-image. the dest-image will have the same size, object size, and image format as src-image. note: snapshots are NOT copied — if you want those too, use the deep cp command. 🫶

🛠️ create

create (-s | --size size-in-M/G/T) [--image-format format-id]
[--object-size size-in-B/K/M] [--stripe-unit size-in-B/K/M
--stripe-count num] [--thick-provision] [--no-progress] [--image-feature
feature-name]... [--image-shared] image-spec

Creates a new rbd image. you MUST also give the size via --size. the --stripe-unit and --stripe-count arguments are optional, but again: they must be used together.

if --thick-provision is enabled, it'll fully allocate storage for the image right at creation time — and heads up, that will take a long time to do. also note: thick provisioning requires zeroing the contents of the entire image. 🧱

🛠️ deep cp

deep cp (src-image-spec | src-snap-spec) dest-image-spec

Deep copies the content of a src-image into the newly created dest-image. the Dest-image will have the same size, object size, image format, AND snapshots as src-image. the full package, no crumbs left behind. 🎁

🛠️ device list

device list [-t | --device-type device-type] [--format plain | json |
xml] --pretty-format

Shows the rbd images that are mapped via the rbd kernel module (that's the default) or another supported device. 👀

🛠️ device map

device map [-t | --device-type device-type] [--cookie device-cookie]
[--show-cookie] [--snap-id snap-id] [--read-only] [--exclusive] [-o |
--options device-options] image-spec | snap-spec

Maps the specified image to a block device via the rbd kernel module (default) or another supported device (nbd or ubbd on Linux, or ggate on FreeBSD). 🗺️

the --options argument is a comma separated list of device type specific options, like (opt1,opt2=val,...). 💅

🛠️ device unmap

device unmap [-t | --device-type device-type] [-o | --options
device-options] [--snap-id snap-id] image-spec | snap-spec | device-path

Unmaps the block device that was mapped via the rbd kernel module (default) or another supported device. bye bestie 👋

the --options argument is a comma separated list of device type specific options, like (opt1,opt2=val,...). 💅

🌈 device attach 💅

device attach [-t | --device-type device-type] --device device-path [--cookie device-cookie] [--show-cookie] [--snap-id snap-id] [--read-only] [--exclusive] [--force] [-o | --options device-options] image-spec | snap-spec

Bestie, this one attaches the image you picked to the block device you picked (and rn it's literally only nbd on Linux, that's it, that's the whole list).

⚠️ This operation is unsafe and should not be normally used. And I mean it fr fr — if you name the wrong image or the wrong block device you can get data corruption, because the nbd kernel driver does zero validation for you. No vibe check, no nothing, just vibes. 🙈

The --options argument is a comma separated list of device type specific options (opt1,opt2=val,...).

🦋 device detach ✨

device detach [-t | --device-type device-type] [-o | --options device-options] [--snap-id snap-id] image-spec | snap-spec | device-path

Detaches the block device that was mapped or attached (again: currently only nbd on Linux, we stay consistent).

⚠️ Also unsafe, also should not be normally used. Same energy, same caution, periodt.

The --options argument is a comma separated list of device type specific options (opt1,opt2=val,...).

👑 diff 🔥

diff [--from-snap snap-name] [--whole-object] image-spec | snap-spec

Dumps a list of byte extents in the image that have changed since the start snapshot you named — or, if you didn't name one, since the image was created. 🍼

Every output line has: the starting offset (in bytes), the length of the region (in bytes), and then either 'zero' or 'data' — telling you whether that region is known to be zeros or might have other stuff in it.

🎀 du 🌸

du [-p | --pool pool-name] [image-spec | snap-spec] [--merge-snapshots]

Will calculate the provisioned and actual disk usage of all the images and their associated snapshots inside the pool you named. It can also be pointed at individual images and snapshots if that's more your thing. 💖

Heads up though: if the RBD fast-diff feature is not enabled on those images, this operation has to go query the OSDs for every potential object within the image. So like, it works, it's just doing the most. 😭

The --merge-snapshots will merge snapshots' used space into their parent images.

🔒 encryption format 💜

encryption format image-spec format passphrase-file [--cipher-alg alg]

Formats the image into an encrypted format. Everything previously written to that image becomes unreadable — like, gone-gone, so bestie be sure. 😭

Supported formats: luks1, luks2.

Supported cipher algorithms: aes-128, aes-256 (that one's the default).

💖 export 🎀

export [--export-format format (1 or 2)] (image-spec | snap-spec) [dest-path]

Exports the image to the dest path (or use - for stdout, cute). The --export-format currently accepts '1' or '2'.

Format 2 lets us export not only the content of the image, but also the snapshots and other properties, such as image_order, features. It's giving glow-up. ✨

🧚 export-diff ⭐

export-diff [--from-snap snap-name] [--whole-object] (image-spec | snap-spec) dest-path

Exports an incremental diff for an image to dest path (use - for stdout).

If you specify an initial snapshot, only the changes since that snapshot get included; otherwise, any regions of the image that contain data get included. The end snapshot is specified using the standard --snap option or the @snap syntax (see below). 👇

The image diff format includes metadata about image size changes, and the start and end snapshots. It also efficiently represents discarded or 'zero' regions of the image. Iconic behavior honestly. 👑

🦋 feature disable 🌈

feature disable image-spec feature-name...

Disables the specified feature on the specified image. You can list multiple features, go off. ✨

🦋 feature enable 🌈

feature enable image-spec feature-name...

Enables the specified feature on the specified image. Multiple features can be specified here too, slay. 💅

💅 flatten 🔥

flatten [--encryption-format encryption-format --encryption-passphrase-file passphrase-file]... image-spec

If the image is a clone, this copies all the shared blocks from the parent snapshot and makes the child independent of the parent — severing the link between parent snap and child, no contact, blocked, periodt. 🚫

Once that's done, the parent snapshot can be unprotected and deleted if it has no further dependent clones.

📌 This requires image format 2.

🎀 group create 🌸

group create group-spec

Creates a group. New group, who dis. 💖

🎀 group image add 🌸

group image add group-spec image-spec

Adds an image to a group.

🎀 group image list 🌸

group image list group-spec

Lists the images in a group.

🎀 group image remove 🌸

group image remove group-spec image-spec

Removes an image from a group.

🎀 group ls 🌸

group ls [-p | --pool pool-name]

Lists rbd groups. That's it, that's the whole job. ⭐

🎀 group rename 🌸

group rename src-group-spec dest-group-spec

Renames a group.

📌 Note: rename across pools is not supported. Stay in your lane bestie. 🙈

🎀 group rm 🌸

group rm group-spec

Deletes a group. 💀

🎀 group snap create 🌸

group snap create group-snap-spec

Makes a snapshot of a group.

🎀 group snap list 🌸

group snap list group-spec

Lists the snapshots of a group.

🎀 group snap info 🌸

group snap info group-snap-spec

Gets information about a snapshot of a group.

🎀 group snap rm 🌸

group snap rm group-snap-spec

Removes a snapshot from a group.

🎀 group snap rename 🌸

group snap rename group-snap-spec snap-name

Renames the group's snapshot.

🎀 group snap rollback 🌸

group snap rollback group-snap-spec

Rolls the group back to the snapshot. Rewind time, main character move. ⏪✨

🔍 image-meta get 🍓

image-meta get image-spec key

Gets the metadata value with the key you handed it.

🔍 image-meta list 🍓

image-meta list image-spec

Shows the metadata held on the image. The first column is the key and the second column is the value. Simple as. 💅

🔍 image-meta remove 🍓

image-meta remove image-spec key

Removes the metadata key with the value. Bye bye 👋

🔍 image-meta set 🍓

image-meta set image-spec key value

Sets the metadata key with the value. And guess what — they will be displayed in image-meta list. 🎀

💖 import ✨

import [--export-format format (1 or 2)] [--image-format format-id] [--object-size size-in-B/K/M] [--stripe-unit size-in-B/K/M --stripe-count num] [--image-feature feature-name]... [--image-shared] src-path [image-spec]

Creates a brand new image and imports its data from the path (use - for stdin). The import operation will try to create sparse rbd images if possible. For imports from stdin, the sparsification unit is the data block size of the destination image (object size). 🧚

The --stripe-unit and --stripe-count arguments are optional, but they must be used together — no splitting the duo up, they're a package deal. 🫶

The --export-format currently accepts '1' or '2'. Format 2 lets us import not only the content of the image, but also the snapshots and other properties, such as image_order, features. So much more iconic. 👑

💖 import-diff ✨

import-diff src-path image-spec

Imports an incremental diff of an image and applies it to the current image.

If the diff was generated relative to a start snapshot, we verify that snapshot already exists before continuing. If there was an end snapshot, we verify it does not already exist before applying the changes, and then we create the snapshot when we're done. Responsible queen behavior. 👑

🔎 info 💜

info image-spec | snap-spec

Will dump information (such as size and object size) about a specific rbd image. If the image is a clone, information about its parent is also displayed. And if you specified a snapshot, whether it's protected is shown as well. Full lore drop, no gatekeeping. 📖✨

📓 journal client disconnect 🦋

journal client disconnect journal-spec

Flags the image journal client as disconnected.

📓 journal export 🦋

journal export [--verbose] [--no-error] src-journal-spec path-name

Exports the image journal to path (use - for stdout). You can use this to make a backup of the image journal, especially before attempting dangerous operations. Smart. Safety first bestie. 🛟

📌 Note that this command may not always work if the journal is badly corrupted. 😭

📓 journal import 🦋

journal import [--verbose] [--no-error] path-name dest-journal-spec

Imports the image journal from path (use - for stdin).

📓 journal info 🦋

journal info journal-spec

Shows information about the image journal. Tell me everything. 🍵

📓 journal inspect 🦋

journal inspect [--verbose] journal-spec

Inspects and reports the image journal for structural errors. Detective mode. 🔍

📓 journal reset 🦋

journal reset journal-spec

Resets the image journal.

📓 journal status 🦋

journal status journal-spec

Shows the status of the image journal.

🔐 lock add 💅

lock add [--shared lock-tag] image-spec lock-id

Locks an image. The lock-id is an arbitrary name for the user's convenience — call it whatever, it's yours. 🎀

By default this is an exclusive lock, meaning it will fail if the image is already locked. The --shared option changes this behavior (note that --shared takes a lock-tag here).

⚠️ Note that locking does not affect any operation other than adding a lock. It does not protect an image from being deleted. It's giving security theater, ngl. 💀

🔐 lock ls 💅

lock ls image-spec

Shows the locks held on the image. The first column is the locker to use with the lock remove command. Take notes. 📝

🔐 lock rm 💅

lock rm image-spec lock-id locker

Releases a lock on an image. The lock-id and locker are as output by lock ls. So like, copy them exactly bestie.

📂 ls ⭐

ls [-l | --long] [pool-name]

Will list all rbd images listed in the rbd_directory object. With -l, it also shows snapshots, and uses longer-format output including size, parent (if it's a clone), format, etc. The -l glow-up is real. ✨

🧩 merge-diff 💜

merge-diff first-diff-path second-diff-path merged-diff-path

Merges two continuous incremental diffs of an image into one single diff. The first diff's end snapshot must be equal with the second diff's start snapshot — like puzzle pieces, they gotta click. 🧩

The first diff could be - for stdin, and the merged diff could be - for stdout, which enables multiple diff files to be merged using something like:

rbd merge-diff first second - | rbd merge-diff - third result
📌 Note this command currently only supports the source incremental diff with stripe-count == 1. That's the limit, sorry bestie. 😭

🚚 migration abort 🦋

migration abort image-spec

Cancels the image migration. This step may be run after successful or failed migration prepare or migration execute steps, and it returns the image to its initial (before migration) state.

All modifications to the destination image are lost. Gone. 🫡

🚚 migration commit 🦋

migration commit image-spec

Commits the image migration. This step is run after successful migration prepare and migration execute steps, and it removes the source image data. Sealing the deal — and yeeting the old copy. 💅

🚚 migration execute 🦋

migration execute image-spec

Executes the image migration. This step is run after a successful migration prepare step, and it copies the image data to the destination. The actual carrying-things-over part. 📦

🚚 migration prepare 🌈

migration prepare [--order order] [--object-size object-size] [--image-feature image-feature] [--image-shared] [--stripe-unit stripe-unit] [--stripe-count stripe-count] [--data-pool data-pool] [--import-only] [--source-spec json] [--source-spec-path path] src-image-spec [dest-image-spec]

Prepares the image migration. This is the very first step when migrating an image — aka changing the image location, format, or other parameters that can't be changed dynamically. 👑

The destination can match the source, and in that case dest-image-spec can be omitted entirely. Lazy-girl friendly. 💅

After this step, the source image is set as a parent of the destination image, and the image is accessible in copy-on-write mode by its destination spec. Tee up, ready, go. ✨

🫶 quick lil callback to the migrate glow-up we were just talking about: an image can also be migrated from a read-only import source by adding the --import-only optional and providing a JSON-encoded --source-spec, or a path to a JSON-encoded source-spec file using the --source-spec-path optionals. miss thing came prepared with options 💅
  • mirror image demote image-spec — demotes a primary image down to non-primary for RBD mirroring. it's giving "step back, you're not the main character of this mirror anymore" 😌
  • mirror image disable [--force] image-spec — turns off RBD mirroring for an image. and listen, if the mirroring is configured in image mode for the image's pool? then it has to be disabled for each image individually. no shortcuts bestie 🙈
  • mirror image enable image-spec mode — turns RBD mirroring on for an image. same deal as above: if the mirroring is configured in image mode for the image's pool, it must be enabled for each image individually.

the mirror image mode can either be journal (the default, periodt) or snapshot. and heads up — the journal mode requires the RBD journaling feature. no feature, no journal, no cap 🔥

  • mirror image promote [--force] image-spec — promotes a non-primary image up to primary for RBD mirroring. glow up activated 👑
  • mirror image resync image-spec — forces a resync to the primary image for RBD mirroring. sometimes you just gotta start over fr fr 🔄
  • mirror image status image-spec — shows you the RBD mirroring status for an image. the vibe check of mirroring ✨
  • mirror pool demote [pool-name] — demotes all primary images within a pool or namespace to non-primary. every single mirror-enabled image in the pool or namespace gets demoted, no exceptions 💀
  • mirror pool disable [pool-name] — disables RBD mirroring within a pool or namespace. and when mirroring is disabled on a pool or namespace this way, mirroring will also be disabled on all the images (within the pool or namespace) for which mirroring was enabled — whether that was by default or explicitly. it's a full sweep 🧹
  • mirror pool enable pool-name mode [--remote-namespace remote-namespace-name] — enables RBD mirroring within a pool or namespace. the mirroring mode can either be pool or image.
    • if configured in pool mode → all images in the pool or namespace with the journaling feature enabled are mirrored. automatic, iconic ✨
    • if configured in image mode → mirroring needs to be explicitly enabled (by the mirror image enable command) on each image. you gotta opt in one by one 🙋

and here's the cute part: a namespace can be mirrored to a different namespace on the remote pool using the --remote-namespace option. matchmaker energy 🎀

  • mirror pool info [pool-name] — shows info about the pool or namespace mirroring configuration. for both pools and namespaces, it includes the mirroring mode, mirror UUID and remote namespace. for pools, it additionally includes the site name, peer UUID, remote cluster name, and remote client name. more tea for pools, that's just how it is ☕
  • mirror pool peer add [pool-name] remote-cluster-spec — adds a mirroring peer to a pool. remote-cluster-spec is [remote client name@]remote cluster name.

the default for remote client name is "client.admin".

this requires mirroring to be enabled on the pool. can't make friends if you're not even online 💅

  • mirror pool peer remove [pool-name] uuid — removes a mirroring peer from a pool. the peer UUID is available from the mirror pool info command. bye bestie 👋
  • mirror pool peer set [pool-name] uuid key value — updates mirroring peer settings. the key can be either client or cluster, and the value is corresponding to remote client name or remote cluster name.
  • mirror pool promote [--force] [pool-name] — promotes all non-primary images within a pool or namespace to primary. every mirror-enabled image in the pool or namespace will be promoted. group glow-up 🦋
  • mirror pool status [--verbose] [pool-name] — shows status for all mirrored images in the pool or namespace. with --verbose, it shows additional output status details for every mirror-enabled image in the pool or namespace. the verbose version is the director's cut 🔍
  • mirror snapshot schedule add [-p | --pool pool] [--namespace namespace] [--image image] interval [start-time] — adds a mirror snapshot schedule. the interval can be specified in days, hours, or minutes using the d, h, m suffix respectively. the start-time is a time string in ISO 8601 format.

and here's the plot twist: not providing the --pool, --namespace and --image options creates a global schedule which applies to all mirror-enabled images in the cluster. one schedule to rule them all 💖

  • mirror snapshot schedule list [-R | --recursive] [--format format] [--pretty-format] [-p | --pool pool] [--namespace namespace] [--image image] — lists the mirror snapshot schedule. scrolling the calendar 📅
  • mirror snapshot schedule remove [-p | --pool pool] [--namespace namespace] [--image image] interval [start-time] — removes a mirror snapshot schedule. cancelled ✂️
  • mirror snapshot schedule status [-p | --pool pool] [--format format] [--pretty-format] [--namespace namespace] [--image image] — shows the mirror snapshot schedule status.
  • mv src-image-spec dest-image-spec — renames an image. Note: rename across pools is not supported. stay in your own pool bestie, we don't do long distance 🚫
  • namespace create pool-name/namespace-name — creates a new image namespace within the pool. brand new baby namespace 🍼
  • namespace list pool-name — lists the image namespaces defined within the pool.
  • namespace remove pool-name/namespace-name — removes an empty image namespace from the pool. it has to be empty, so clean it out first 🧺
  • object-map check image-spec | snap-spec — verifies the object map is correct. fact-checking the map 🔎
  • object-map rebuild image-spec | snap-spec — rebuilds an invalid object map for the specified image. an image snapshot can be specified to rebuild an invalid object map for a snapshot. glow-up for a broken map 🛠️
  • pool init [pool-name] [--force] — initializes a pool for use by RBD. newly created pools must be initialized prior to use. no init, no vibes 🔥
  • resize (-s | --size size-in-M/G/T) [--allow-shrink] [--encryption-format encryption-format --encryption-passphrase-file passphrase-file]... image-spec — resizes an rbd image. the size parameter also needs to be specified. the --allow-shrink option lets the size be reduced. growing? free. shrinking? you need permission first 💅
  • rm image-spec — deletes an rbd image (including all data blocks). if the image has snapshots, this fails and nothing is deleted. it's all or nothing, bestie 😭
  • snap create snap-spec — creates a new snapshot. requires the snapshot name parameter to be specified. gotta name your children ✨
  • snap limit clear image-spec — removes any previously set limit on the number of snapshots allowed on an image. unlimited snapshots era 🔓
  • snap limit set [--limit] limit image-spec — sets a limit for the number of snapshots allowed on an image. boundaries, we love them 🚧
  • snap ls image-spec — dumps the list of snapshots of a specific image. the roster 📋
  • snap protect snap-spec — protects a snapshot from deletion, so that clones can be made of it (see rbd clone). snapshots must be protected before clones are made; protection implies that there exist dependent cloned children that refer to this snapshot. rbd clone will fail on a nonprotected snapshot. so protect it or lose it 🛡️

this requires image format 2.

  • snap purge image-spec — removes all unprotected snapshots from an image. protected ones survive, obviously ✨
  • snap rename src-snap-spec dest-snap-spec — renames a snapshot. Note: rename across pools and images is not supported. again with the no long distance rule 🙈
  • snap rm [--force] snap-spec — removes the specified snapshot. bye girl 💨
  • snap rollback snap-spec — rolls the image content back to the snapshot. this will iterate through the entire blocks array and update the data head content to the snapshotted version. full rewind, no skips ⏪
  • snap unprotect snap-spec — unprotects a snapshot from deletion (undoes snap protect). if cloned children remain, snap unprotect fails. (note that clones may exist in different pools than the parent snapshot.) and yes, this requires image format 2.
  • sparsify [--sparse-size sparse-size] image-spec — reclaims space for zeroed image extents. the default sparse size is 4096 bytes and can be changed via the --sparse-size option with the following restrictions: it should be power of two, not less than 4096, and not larger than image object size. rules are rules 📏
  • status image-spec — shows the status of the image, including which clients have it open. who's got it open rn 👀
  • trash ls [pool-name] — lists all entries from the trash. the graveyard directory 💀
  • trash mv [--expires-at <expires-at>] image-spec — moves an image to the trash. images, even ones actively in-use by clones, can be moved to the trash and deleted at a later time. use --expires-at to set the expiration time of an image after which it's allowed to be removed. it's not deleted, it's just... resting 🛋️
  • trash purge [pool-name] — removes all expired images from trash. expired only, don't get messy 🧼
  • trash restore image-id — restores an image from trash. resurrection arc 🦋
  • trash rm [--force] image-id — deletes an image from trash. if the image deferment time has not expired it can be removed using --force. an image that is actively in-use by clones or has snapshots cannot be removed. no negotiating on that one 💅
  • trash purge schedule add [-p | --pool pool] [--namespace namespace] interval [start-time] — adds a trash purge schedule. automation era ⏰
  • trash purge schedule list [-R | --recursive] [--format format] [--pretty-format] [-p | --pool pool] [--namespace namespace] — lists the trash purge schedule.
  • trash purge schedule remove [-p | --pool pool] [--namespace namespace] interval [start-time] — removes a trash purge schedule.
  • trash purge schedule status [-p | --pool pool] [--format format] [--pretty-format] [--namespace namespace] — shows the trash purge schedule status.
  • watch image-spec — watches events on an image. we're literally just observing, no notes 👁️

🦋 IMAGE, SNAP, GROUP AND JOURNAL SPECS 🦋

okay so every command up there was throwing around words like image-spec and snap-spec like you already knew, so here's the cheat sheet bestie:

image-spec      is [pool-name/[namespace-name/]]image-name
snap-spec       is [pool-name/[namespace-name/]]image-name@snap-name
group-spec      is [pool-name/[namespace-name/]]group-name
group-snap-spec is [pool-name/[namespace-name/]]group-name@snap-name
journal-spec    is [pool-name/[namespace-name/]]journal-name

the default for pool-name is "rbd" and namespace-name is "". and if an image name contains a slash character (/), then pool-name is required — you can't just skip it, the parser will be confused 😭

the journal-name is image-id.

you may specify each name individually, using --pool, --namespace, --image, and --snap options — but this is discouraged in favor of the above spec syntax. like sure you can, but the spec syntax is right there and it's so much cuter ✨

✨ STRIPING ✨

okay so bestie, here's the tea on how RBD images actually live in the cluster 💖 RBD images are striped over many objects, which are then stored by the Ceph distributed object store (RADOS). As a result, read and write requests for the image get distributed across many nodes in the cluster, which generally prevents any single node from becoming a bottleneck when individual images get large or busy. it's giving load balancing, it's giving no single point of suffering 👑

The striping is controlled by three parameters, bestie, and they are:

  • object-size — the size of objects we stripe over is a power of two. 🦋 It gets rounded up to the nearest power of two. The default object size is 4 MB, smallest is 4K and maximum is 32M.
  • stripe-unit — each [stripe-unit] contiguous bytes are stored adjacently in the same object, before we move on to the next object. literally one chunk at a time 💅
  • stripe-count — after we write [stripe-unit] bytes to [stripe-count] objects, we loop back to the initial object and write another stripe, until the object reaches its maximum size. At that point, we move on to the next [stripe-count] objects. periodt 🔁

By default, [stripe-unit] is the same as the object size and [stripe-count] is 1. Specifying a different [stripe-unit] and/or [stripe-count] is often referred to as using "fancy" striping and requires format 2. yes queen, fancy ✨

🐧 KERNEL RBD (KRBD) OPTIONS 🐧

Most of these options are useful mainly for debugging and benchmarking, ngl. The default values are set in the kernel and may therefore depend on the version of the running kernel — so like, your mileage may totally vary depending on your kernel, bestie 🙈

🫶 Per client instance rbd device map options

  • fsid=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee — FSID that should be assumed by the client.
  • ip=a.b.c.d[:p] — IP and, optionally, port the client should use.
  • share — enable sharing of client instances with other mappings (default).
  • noshare — disable sharing of client instances with other mappings.
  • crc — enable CRC32C checksumming for msgr1 on-the-wire protocol (default). For msgr2.1 protocol this option is ignored: full checksumming is always on in crc mode and always off in secure mode. 🧐
  • nocrc — disable CRC32C checksumming for msgr1 on-the-wire protocol. Note that only payload checksumming is disabled, header checksumming is always on. For msgr2.1 protocol this option is ignored.
  • cephx_require_signatures — require msgr1 message signing feature (since 3.19, default). ⚠️ This option is deprecated and will be removed in the future as the feature has been supported since the Bobtail release.
  • nocephx_require_signatures — don't require msgr1 message signing feature (since 3.19). ⚠️ This option is deprecated and will be removed in the future.
  • tcp_nodelay — disable Nagle's algorithm on client sockets (since 4.0, default).
  • notcp_nodelay — enable Nagle's algorithm on client sockets (since 4.0).
  • cephx_sign_messages — enable message signing for msgr1 on-the-wire protocol (since 4.4, default). For msgr2.1 protocol this option is ignored: message signing is built into secure mode and not offered in crc mode.
  • nocephx_sign_messages — disable message signing for msgr1 on-the-wire protocol (since 4.4). For msgr2.1 protocol this option is ignored.
  • mount_timeout=x — a timeout on various steps in rbd device map and rbd device unmap sequences (default is 60 seconds). In particular, since 4.2 this can be used to ensure that rbd device unmap eventually times out when there is no network connection to a cluster. 😭 the way this one saves you from hanging forever
  • osdkeepalive=x — OSD keepalive timeout (default is 5 seconds).
  • osd_idle_ttl=x — OSD idle TTL (default is 60 seconds).

💅 Per mapping (block device) rbd device map options

  • rw — map the image read-write (default). Overridden by --read-only.
  • ro — map the image read-only. Equivalent to --read-only.
  • queue_depth=x — queue depth (since 4.2, default is 128 requests).
  • lock_on_read — acquire exclusive lock on reads, in addition to writes and discards (since 4.9).
  • exclusive — disable automatic exclusive lock transitions (since 4.12). Equivalent to --exclusive.
  • lock_timeout=x — a timeout on waiting for the acquisition of exclusive lock (since 4.17, default is 0 seconds, meaning no timeout).
  • notrim — turn off discard and write zeroes offload support to avoid deprovisioning a fully provisioned image (since 4.17). When enabled, discard requests will fail with -EOPNOTSUPP, write zeroes requests will fall back to manually zeroing. 💀 careful bestie
  • abort_on_full — fail write requests with -ENOSPC when the cluster is full or the data pool reaches its quota (since 5.0). The default behaviour is to block until the full condition is cleared.
  • alloc_size — minimum allocation unit of the underlying OSD object store backend (since 5.1, default is 64K bytes). This is used to round off and drop discards that are too small. For bluestore, the recommended setting is bluestore_min_alloc_size (currently set to 4K for all types of drives, previously used to be set to 64K for hard disk drives and 16K for solid-state drives). For filestore with filestore_punch_hole = false, the recommended setting is image object size (typically 4M).
  • crush_location=x — specify the location of the client in terms of CRUSH hierarchy (since 5.8). 🔥 This is a set of key-value pairs separated from each other by '|', with keys separated from values by ':'. Note that '|' may need to be quoted or escaped to avoid it being interpreted as a pipe by the shell. The key is the bucket type name (e.g. rack, datacenter or region with default bucket types) and the value is the bucket name. For example, to indicate that the client is local to rack "myrack", data center "mydc" and region "myregion":
crush_location=rack:myrack|datacenter:mydc|region:myregion

Each key-value pair stands on its own: "myrack" doesn't need to reside in "mydc", which in turn doesn't need to reside in "myregion". The location is not a path to the root of the hierarchy but rather a set of nodes that are matched independently, owning to the fact that bucket names are unique within a CRUSH map. "Multipath" locations are supported, so it is possible to indicate locality for multiple parallel hierarchies:

crush_location=rack:myrack1|rack:myrack2|datacenter:mydc
  • read_from_replica=no — disable replica reads, always pick the primary OSD (since 5.8, default).
  • read_from_replica=balance — when issued a read on a replicated pool, pick a random OSD for serving it (since 5.8).
⚠️ This mode is safe for general use only since Octopus (i.e. after ceph osd require-osd-release octopus). Otherwise it should be limited to read-only workloads such as images mapped read-only everywhere or snapshots.
  • read_from_replica=localize — when issued a read on a replicated pool, pick the most local OSD for serving it (since 5.8). The locality metric is calculated against the location of the client given with crush_location; a match with the lowest-valued bucket type wins. For example, with default bucket types, an OSD in a matching rack is closer than an OSD in a matching data center, which in turn is closer than an OSD in a matching region.
⚠️ This mode is safe for general use only since Octopus (i.e. after ceph osd require-osd-release octopus). Otherwise it should be limited to read-only workloads such as images mapped read-only everywhere or snapshots.
  • compression_hint=none — don't set compression hints (since 5.8, default).
  • compression_hint=compressible — hint to the underlying OSD object store backend that the data is compressible, enabling compression in passive mode (since 5.8).
  • compression_hint=incompressible — hint to the underlying OSD object store backend that the data is incompressible, disabling compression in aggressive mode (since 5.8).
  • ms_mode=legacy — use msgr1 on-the-wire protocol (since 5.11).
  • ms_mode=crc — use msgr2.1 on-the-wire protocol, select crc mode, also referred to as plain mode (since 5.11). If the daemon denies crc mode, fail the connection.
  • ms_mode=secure — use msgr2.1 on-the-wire protocol, select secure mode (since 5.11). secure mode provides full in-transit encryption ensuring both confidentiality and authenticity. 🔐 If the daemon denies secure mode, fail the connection.
  • ms_mode=prefer-crc — use msgr2.1 on-the-wire protocol, select crc mode (since 5.11, default). If the daemon denies crc mode in favor of secure mode, agree to secure mode.
  • ms_mode=prefer-secure — use msgr2.1 on-the-wire protocol, select secure mode (since 5.11). If the daemon denies secure mode in favor of crc mode, agree to crc mode.
  • rxbounce — use a bounce buffer when receiving data (since 5.17). The default behaviour is to read directly into the destination buffer. A bounce buffer is needed if the destination buffer isn't guaranteed to be stable (i.e. remain unchanged while it is being read to). In particular this is the case for Windows where a system-wide "dummy" (throwaway) page may be mapped into the destination buffer in order to generate a single large I/O. Otherwise, "libceph: ... bad crc/signature" or "libceph: ... integrity error, bad crc" errors and associated performance degradation are expected. 😭 not the integrity errors
  • udev — wait for udev device manager to finish executing all matching "add" rules and release the device before exiting (default). This option is not passed to the kernel.
  • noudev — don't wait for udev device manager. When enabled, the device may not be fully usable immediately on exit. 🏃♀️

🌸 rbd device unmap options

  • force — force the unmapping of a block device that is open (since 4.9). The driver will wait for running requests to complete and then unmap; requests sent to the driver after initiating the unmap will be failed. 💪
  • udev — wait for udev device manager to finish executing all matching "remove" rules and clean up after the device before exiting (default). This option is not passed to the kernel.
  • noudev — don't wait for udev device manager.

✨ EXAMPLES ✨

okay bestie this is the part where it all comes together, the little recipe book, the group chat screenshots of rbd being iconic 💅 every single one of these is copy-pasteable, no cap.

To create a new rbd image that is 100 GB:

rbd create mypool/myimage --size 102400

To use a non-default object size (8 MB):

rbd create mypool/myimage --size 102400 --object-size 8M

To delete an rbd image (be careful!):

rbd rm mypool/myimage
bestie be careful is in the original for a REASON, this one does not come with an undo button 😭

To create a new snapshot:

rbd snap create mypool/myimage@mysnap

To create a copy-on-write clone of a protected snapshot:

rbd clone mypool/myimage@mysnap otherpool/cloneimage

To see which clones of a snapshot exist:

rbd children mypool/myimage@mysnap

To delete a snapshot:

rbd snap rm mypool/myimage@mysnap

To map an image via the kernel with CephX enabled:

rbd device map mypool/myimage --id admin --keyfile secretfile

To map an image via the kernel with different cluster name other than default ceph:

rbd device map mypool/myimage --cluster cluster-name

To unmap an image:

rbd device unmap /dev/rbd0

To create an image and a clone from it:

rbd import --image-format 2 image mypool/parent
rbd snap create mypool/parent@snap
rbd snap protect mypool/parent@snap
rbd clone mypool/parent@snap otherpool/child

To create an image with a smaller stripe-unit (to better distribute small writes in some workloads):

rbd create mypool/myimage --size 102400 --stripe-unit 65536B --stripe-count 16

To change an image from one image format to another, export it and then import it as the desired image format:

rbd export mypool/myimage@snap /tmp/img
rbd import --image-format 2 /tmp/img mypool/myimage2

To lock an image for exclusive use:

rbd lock add mypool/myimage mylockid

To release a lock:

rbd lock remove mypool/myimage mylockid client.2485

To list images from trash:

rbd trash ls mypool

To defer delete an image (use --expires-at to set expiration time, default is now):

rbd trash mv mypool/myimage --expires-at "tomorrow"

To delete an image from trash (be careful!):

rbd trash rm mypool/myimage-id

To force delete an image from trash (be careful!):

rbd trash rm mypool/myimage-id  --force

To restore an image from trash:

rbd trash restore mypool/myimage-id

To restore an image from trash and rename it:

rbd trash restore mypool/myimage-id --image mynewimage

To create a mirror snapshot schedule for an image:

rbd mirror snapshot schedule add --pool mypool --image myimage 12h 14:00:00-05:00

🌈 AVAILABILITY 🌈

rbd is part of Ceph, a massively scalable, open-source, distributed storage system. Please refer to the Ceph documentation at <https://docs.ceph.com> for more information.

go read the docs bestie, they're right there, https://docs.ceph.com, it's giving free education ✨

👀 SEE ALSO 👀

  • ceph(8)
  • rados(8)

2010-2014, Inktank Storage, Inc. and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0)

and that's the whole page bestie, you ate that 🫶 this lil masterpiece lives at dev, dated Oct 06, 2026, and it's filed under RBD(8) — go forth and slay your storage, no cap 💖✨👑

📦 where this page comes from