CEPH-BLUESTORE-TOOL(8)

📦
version 20.2.4 · 20 versions
, in ubuntu stonking

💖 NAME 💖

ceph-bluestore-tool — it's the bluestore administrative tool, bestie. That's the whole vibe. It lives in the Ceph manual, section 8 (system admin commands), and it's the girl who does the low-level surgery so your OSDs don't have to. 🩺✨

✨ SYNOPSIS ✨

okay so here's literally every way you can call her:

ceph-bluestore-tool command
[ --dev device ... ]
[ -i osd_id ]
[ --path osd path ]
[ --out-dir dir ]
[ --log-file | -l filename ]
[ --deep ]
ceph-bluestore-tool fsck|repair --path osd path [ --deep ]
ceph-bluestore-tool qfsck       --path osd path
ceph-bluestore-tool allocmap    --path osd path
ceph-bluestore-tool restore_cfb --path osd path
ceph-bluestore-tool show-label --dev device ...
ceph-bluestore-tool show-label-at --dev device --offset lba...
ceph-bluestore-tool prime-osd-dir --dev device --path osd path
ceph-bluestore-tool bluefs-export --path osd path --out-dir dir
ceph-bluestore-tool bluefs-bdev-new-wal --path osd path --dev-target new-device
ceph-bluestore-tool bluefs-bdev-new-db --path osd path --dev-target new-device
ceph-bluestore-tool bluefs-bdev-migrate --path osd path --dev-target new-device --devs-source device1 [--devs-source device2]
ceph-bluestore-tool free-dump|free-score --path osd path [ --allocator block/bluefs-wal/bluefs-db/bluefs-slow ]
ceph-bluestore-tool bluefs-stats --path osd path
ceph-bluestore-tool bluefs-files --path osd path
ceph-bluestore-tool reshard --path osd path --sharding new sharding [ --sharding-ctrl control string ]
ceph-bluestore-tool show-sharding --path osd path
ceph-bluestore-tool trim --path osd path
ceph-bluestore-tool zap-device --dev dev path
ceph-bluestore-tool revert-wal-to-plain --path osd path

🦋 DESCRIPTION 🦋

ceph-bluestore-tool is a utility to perform low-level administrative operations on a BlueStore instance.

that's it. that's the whole girl. she is a utility and she does low-level administrative operations on a BlueStore instance. main character energy, no notes. 💅

👑 COMMANDS 👑

  • help — show help. she's helpful, she's humble, she's giving tutorial. 🫶
  • fsck [ --deep ] (on|off) or (yes|no) or (1|0) or (true|false) — run consistency check on BlueStore metadata. If --deep is specified, also read all object data and verify checksums. like a vibe check but for your metadata, and with --deep she reads everything. everything. 😭
  • repair — Run a consistency check and repair any errors we can. she's not a miracle worker but she will try. 🩹
  • qfsck — run consistency check on BlueStore metadata comparing allocator data (from RocksDB CFB when exists and if not uses allocation-file) with ONodes state. two things, side by side, staring at each other. 👀
  • allocmap — performs the same check done by qfsck and then stores a new allocation-file (command is disabled by default and requires a special build). custom build required, so she's giving limited edition energy. ✨
  • restore_cfb — Reverses changes done by the new NCB code (either through ceph restart or when running allocmap command) and restores RocksDB B Column-Family (allocator-map). it's the undo button, bestie. ↩️
  • bluefs-export — Export the contents of BlueFS (i.e., RocksDB files) to an output directory. packing up her stuff and moving out. 📦
  • bluefs-bdev-sizes --path osd path — Print the device sizes, as understood by BlueFS, to stdout. she tells you how big she thinks things are. 💭
  • bluefs-bdev-expand --path osd path — Instruct BlueFS to check the size of its block devices and, if they have expanded, make use of the additional space. Please note that only the new files created by BlueFS will be allocated on the preferred block device if it has enough free space, and the existing files that have spilled over to the slow device will be gradually removed when RocksDB performs compaction. In other words, if there is any data spilled over to the slow device, it will be moved to the fast device over time. glow-up arc, but make it gradual. 🦋
  • bluefs-bdev-new-wal --path osd path --dev-target new-device — Adds WAL device to BlueFS, fails if WAL device already exists. no duplicates allowed, she's monogamous with her WAL. 💀
  • bluefs-bdev-new-db --path osd path --dev-target new-device — Adds DB device to BlueFS, fails if DB device already exists. same energy, different volume. 🙈
  • bluefs-bdev-migrate --dev-target new-device --devs-source device1 [--devs-source device2] — Moves BlueFS data from source device(s) to the target device. Source devices (except the main one) are removed on success. Expands the target storage (updates the size label), making "bluefs-bdev-expand" unnecessary. The target device can be either a new device or a device that is already attached. If the device is a new device, it is added to the OSD replacing one of the source devices. The following replacement rules apply (in the order of precedence, stop on the first match):
    • if the source list has DB volume - the target device replaces it.
    • if the source list has WAL volume - the target device replaces it.
    • if the source list has slow volume only - the operation isn't permitted and requires explicit allocation via a new-DB/new-WAL command.

the way she just stops on the first match?? decisive. iconic. 🎯

  • show-label --dev device [...] — Show device label(s). The label may be printed while an OSD is running. she can read labels while the OSD is running, multitasking queen. 👑
  • show-label-at --dev device --offset lba[...] — Show device label at specific disk location. Dedicated DB/WAL volumes have a single label at offset 0. Main device could have valid labels at multiple locations: 0/1GiB/10GiB/100GiB/1000GiB. The labels at some locations might not exist though. The label may be printed while an OSD is running. so many locations and possibly some of them are just not there. she doesn't promise anything. 🌸
  • free-dump --path osd path [ --allocator block/bluefs-wal/bluefs-db/bluefs-slow ] — Dump all free regions in allocator. every empty spot, exposed. 🔍
  • free-score --path osd path [ --allocator block/bluefs-wal/bluefs-db/bluefs-slow ] — Give a [0-1] number that represents quality of fragmentation in allocator. 0 represents case when all free space is in one chunk. 1 represents worst possible fragmentation. she will rate your fragmentation out of one and she will be honest. 📊
  • bluefs-stats --path osd path — Shows summary of BlueFS occupied space with split on devices: block/db/wal and roles: wal/log/db. the receipts!! 🧾
  • bluefs-files --path osd path — Lists all BlueFS managed files, printing name, size and space used on devices. name, size, space used. she lists everything. 📝
  • reshard --path osd path --sharding new sharding [ --resharding-ctrl control string ] — Changes sharding of BlueStore's RocksDB. Sharding is build on top of RocksDB column families. This option allows to test performance of new sharding without need to redeploy OSD. Resharding is usually a long process, which involves walking through entire RocksDB key space and moving some of them to different column families. Option --resharding-ctrl provides performance control over resharding process. Interrupted resharding will prevent OSD from running. Interrupted resharding does not corrupt data. It is always possible to continue previous resharding, or select any other sharding scheme, including reverting to original one.
⚠️ not me warning you: Interrupted resharding will prevent OSD from running. good news? Interrupted resharding does not corrupt data, and you can always continue the previous resharding, or pick any other sharding scheme, including reverting to the original one. so breathe. 🫶
  • show-sharding --path osd path — Show sharding that is currently applied to BlueStore's RocksDB. current mood: sharded. 💅
  • command
    • trim --path osd path — An SSD that has been used heavily may experience performance degradation. This operation uses TRIM / discard to free unused blocks from BlueStore and BlueFS block devices, and allows the drive to perform more efficient internal housekeeping. If BlueStore runs with discard enabled, this option may not be useful. self-care for your SSD, literally. 🌿
  • command
    • zap-device --dev dev path — Zeros all device label locations. This effectively makes device appear empty. girl is wiped. gone. empty. no thoughts head empty. 💀
  • command
    • revert-wal-to-plain --path osd path — Changes WAL files from envelope mode to the legacy plain mode. Useful for downgrades, or if you might want to disable this new feature (bluefs_wal_envelope_mode). taking it back to the old ways, and that's valid. 📼

💅 OPTIONS 💅

  • --dev device — Add device to the list of devices to consider.
  • -i osd_id — Operate as OSD osd_id. Connect to monitor for OSD specific options. If monitor is unavailable, add --no-mon-config to read from ceph.conf instead.
  • --devs-source device — Add device to the list of devices to consider as sources for migrate operation.
  • --dev-target device — Specify target device migrate operation or device to add for adding new DB/WAL.
  • --path osd path — Specify an osd path. In most cases, the device list is inferred from the symlinks present in osd path. This is usually simpler than explicitly specifying the device(s) with --dev. Not necessary if -i osd_id is provided.
  • --out-dir dir — Output directory for bluefs-export.
  • -l, --log-file log file — file to log to.
  • --log-level num — debug log level. Default is 30 (extremely verbose), 20 is very verbose, 10 is verbose, and 1 is not very verbose.
  • --deep — deep scrub/repair (read and validate object data, not just metadata).
  • --allocator name — Useful for free-dump and free-score actions. Selects allocator(s).
  • --resharding-ctrl control string — Provides control over resharding process. Specifies how often refresh RocksDB iterator, and how large should commit batch be before committing to RocksDB. Option format is:
<iterator_refresh_bytes>/<iterator_refresh_keys>/<batch_commit_bytes>/<batch_commit_keys>

Default: 10000000/10000/1000000/1000

🌸 ADDITIONAL CEPH.CONF OPTIONS 🌸

Any configuration option that is accepted by OSD can be also passed to ceph-bluestore-tool. Useful to provide necessary configuration options when access to monitor/ceph.conf is impossible and -i option cannot be used.

so when the monitor isn't picking up and ceph.conf isn't answering, you can just hand her the settings directly. she's flexible like that. 💖

🎀 DEVICE LABELS 🎀

Every BlueStore block device has a block label at the beginning of the device. Main device might optionaly have additional labels at different locations for the sake of OSD robustness. You can dump the contents of the label with:

ceph-bluestore-tool show-label --dev device

The main device will have a lot of metadata, including information that used to be stored in small files in the OSD data directory. The auxiliary devices (db and wal) will only have the minimum required fields (OSD UUID, size, device type, birth time). The main device contains additional label copies at offsets: 1GiB, 10GiB, 100GiB and 1000GiB. Corrupted labels are fixed as part of repair:

ceph-bluestore-tool repair --dev device

main device is the one with the lore (OSD UUID, size, device type, birth time are all the db and wal have, and honestly that's enough for them). extra copies of the main label live at those offsets for robustness. corrupted labels? repair handles it at the same time as everything else. 🎀

⭐ OSD DIRECTORY PRIMING ⭐

You can generate the content for an OSD data directory that can start up a BlueStore OSD with the prime-osd-dir command:

ceph-bluestore-tool prime-osd-dir --dev main device --path /var/lib/ceph/osd/ceph-id

making a whole OSD dir from scratch and priming it like it's a canvas. 🎨✨

🌈 BLUEFS LOG RESCUE 🌈

Some versions of BlueStore were susceptible to BlueFS log growing extremely large - beyond the point of making booting OSD impossible. This state is indicated by booting that takes very long and fails in _replay function.

This can be fixed by:

ceph-bluestore-tool fsck --path osd path
--bluefs_replay_recovery=true

It is advised to first check if rescue process would be successful:

ceph-bluestore-tool fsck --path osd path
--bluefs_replay_recovery=true
--bluefs_replay_recovery_disable_compact=true

If above fsck is successful fix procedure can be applied.

🚨 bestie listen to me: the log grew so big the OSD couldn't even boot, which shows up as a boot that takes forever and dies in _replay. do the dry run first with --bluefs_replay_recovery_disable_compact=true, and only if that fsck comes back successful do you apply the actual fix. don't skip the rehearsal. 🌈

🌈 AVAILABILITY 🌈

okay bestie, quick lil housekeeping before we go — ceph-bluestore-tool isn't just floating out here on its own, she's part of Ceph, which is a massively scalable, open-source, distributed storage system. 💖

So if you wanna read more (and honestly, you should, she's fascinating), the docs live at <https://docs.ceph.com>. Go wild. 🦋

👀 SEE ALSO 👀

the girlies that run together, stay together:

  • ceph-osd(8) — her main character sibling, fr fr. If you're deep in BlueStore business, you're probably gonna need her too. 💅

giving credit where credit is due, periodt:

2010-2014, Inktank Storage, Inc. and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0). 🌸

tl;dr: share it, remix it, but keep the credit and pass the same license along. That's the vibe check, iykyk. 💜

and that's a wrap on the whole page, bestie!! 🎀 You made it all the way through — iconic behavior. ✨

This version's marked dev, and it's dated Oct 06, 2026, filed under CEPH-BLUESTORE-TOOL(8) — so if you're reading this in some distant future terminal, hi, love that for you, hope your OSDs are healthy. 🫶💖

now go label a device or rescue a log or whatever, you've got this. no cap. 👑

📦 where this page comes from