CEPH(8)

📦
version 20.2.4 · 20 versions
, in ubuntu stonking

💖 ceph(8) — CEPH 💖

okay bestie, welcome to ceph(8), the absolute icon of Ceph administration tools ✨ she's a control utility, she's manual deployment AND maintenance, she does it all and she does it flawlessly 💅

🌸 SYNOPSIS 🌸

like, just look at her. no notes.

ceph auth [ add | caps | del | export | get | get-key | get-or-create | get-or-create-key | import | list | print-key | print_key ] ...

ceph compact

ceph config [ dump | ls | help | get | show | show-with-defaults | set | rm | log | reset | assimilate-conf | generate-minimal-conf ] ...

ceph config-key [ rm | exists | get | ls | dump | set ] ...

ceph daemon <name> | <path> <command> ...

ceph daemonperf <name> | <path> [ interval [ count ] ]

ceph df {detail}

ceph fs [ add_data_pool | authorize | dump | feature ls | flag set | get | ls | lsflags | new | rename | reset | required_client_features add | required_client_features rm | rm | rm_data_pool | set | swap ] ...

ceph fsid

ceph health {detail}

ceph injectargs <injectedargs> [ <injectedargs>... ]

ceph log <logtext> [ <logtext>... ]

ceph mds [ compat | fail | rm | rmfailed | set_state | stat | repaired ] ...

ceph mon [ add | dump | enable_stretch_mode | getmap | remove | stat ] ...

ceph osd [ blocklist | blocked-by | create | new | deep-scrub | df | down | dump | erasure-code-profile | find | getcrushmap | getmap | getmaxosd | in | ls | lspools | map | metadata | ok-to-stop | ok-to-upgrade | out | pause | perf | pg-temp | force-create-pg | primary-affinity | primary-temp | repair | reweight | reweight-by-pg | rm | destroy | purge | safe-to-destroy | scrub | set | setcrushmap | setmaxosd  | stat | tree | unpause | unset ] ...

ceph osd crush [ add | add-bucket | create-or-move | dump | get-tunable | link | move | remove | rename-bucket | reweight | reweight-all | reweight-subtree | rm | rule | set | set-tunable | show-tunables | tunables | unlink ] ...

ceph osd pool [ create | delete | get | get-quota | ls | mksnap | rename | rmsnap | set | set-quota | stats ] ...

ceph osd pool application [ disable | enable | get | rm | set ] ...

ceph osd tier [ add | add-cache | cache-mode | remove | remove-overlay | set-overlay ] ...

ceph pg [ debug | deep-scrub | dump | dump_json | dump_pools_json | dump_stuck | getmap | ls | ls-by-osd | ls-by-pool | ls-by-primary | map | repair | scrub | stat ] ...

ceph quorum_status

ceph report { <tags> [ <tags>... ] }

ceph status

ceph sync force {--yes-i-really-mean-it} {--i-know-what-i-am-doing}

ceph tell <name (type.id)> <command> [options...]

ceph tell <pgid> <command> [options...]

ceph version

🦋 DESCRIPTION 🦋

so real talk: ceph is a control utility, and she's the one you call when you want to do manual deployment and maintenance of a Ceph cluster 💖 she ships a diverse set of commands — and when I say diverse I mean she will let you deploy monitors, OSDs, placement groups, MDS, plus do all the overall maintenance and administration of the cluster. main character energy, no cap 👑

✨ COMMANDS ✨

🔐 auth

Manage authentication keys, bestie. This is the one that adds, removes, exports, or updates authentication keys for a particular entity — like a monitor 👀 or an OSD. She's got extra subcommands, and yes we're going through every single one.

add

Adds authentication info for a particular entity, pulled from an input file — or a random key if no input is given and/or any caps specified in the command.

Usage:

ceph auth add [--key-type=<cipher>] <entity> {<caps> [<caps>...]}

No key information is output, fyi — she's keeping that to herself 🙈 And if --key-type is omitted, the cluster's preferred cipher is used.

caps

Updates caps for name, from the caps specified in the command.

Usage:

ceph auth caps <entity> <caps> [<caps>...]

del

Deletes all caps for name. all of them. gone. 💀

Usage:

ceph auth del <entity>

export

Writes the keyring for the requested entity — or the master keyring if none was given 👑

Usage:

ceph auth export {<entity>}

get

Writes a keyring file with the requested key.

Usage:

ceph auth get <entity>

get-key

Displays the requested key. simple. iconic. ✨

Usage:

ceph auth get-key <entity>

get-or-create

Adds authentication info for a particular entity from an input file — or a random key if no input is given and/or any caps specified in the command.

Usage:

ceph auth get-or-create [--key-type=<cipher>] <entity> {<caps> [<caps>...]}

The output is the new key in the .ini file format 💅 And again, if --key-type is omitted, the cluster's preferred cipher is used.

get-or-create-key

Gets or adds a key for name from the system/caps pairs specified in the command. Important: if the key already exists, any given caps must match the existing caps for that key — no sneaky mismatches allowed 🚫

Usage:

ceph auth get-or-create-key <entity> {<caps> [<caps>...]}

The output is the new key in plain text. And if --key-type is omitted, the cluster's preferred cipher is used.

import

Reads keyring from an input file.

Usage:

ceph auth import

ls

Lists authentication state.

Usage:

ceph auth ls

Displays the requested key.

Usage:

ceph auth print-key <entity>

Displays the requested key. yes she has two of these and both are valid, don't @ me 🫶

Usage:

ceph auth print_key <entity>

rotate

Rotates the key for the given entity 🔄 so fresh.

Usage:

ceph auth rotate [--key-type=<cipher>] <entity>

The output is the new key in the .ini file format. And yep — if --key-type is omitted, the cluster's preferred cipher is used.

dump-keys

Dumps the entire key database, including rotating service keys. the whole thing, no crumbs left 🌈

Usage:

ceph --format=json auth dump-keys

The output format must be json or json-pretty, periodt.

wipe-rotating-service-keys

Wipes all rotating service keys and forces the Monitors to refresh all keys 😭

bestie please. Do not run this command without understanding its full effects and purpose. Consult the Ceph documentation for more information. I'm being so serious rn.

Usage:

ceph auth wipe-rotating-service-keys

🗜️ compact

Causes compaction of the monitor's RocksDB storage. she's on a lil glow-up arc ✨

Usage:

ceph compact

⚙️ config

Configure the cluster, bestie. By default, Ceph daemons and clients retrieve their configuration options from the monitor when they start — and they get updated if any of the tracked options is changed at run time 👀 She uses the following additional subcommands.

dump

To dump all options for the cluster.

Usage:

ceph config dump

ls

To list all option names for the cluster.

Usage:

ceph config ls

help

To describe the specified configuration option.

Usage:

ceph config help <option>

get

To dump the option(s) for the specified entity.

Usage:

ceph config get <who> {<option>}

show

To display the running configuration of the specified entity.

okay so please note: unlike get, which only shows the options managed by the monitor, show displays all the configurations being actively used. These options are pulled from several sources — for instance, the compiled-in default value, the monitor's configuration database, ceph.conf file on the host. The options can even be overridden at runtime. So there's a chance that the configuration options in the output of show could be different from those in the output of get. it's giving "they're not the same picture" 📸

Usage:

ceph config show {<who>}

show-with-defaults

To display the running configuration along with the compiled-in defaults of the specified entity.

Usage:

ceph config show {<who>}

set

To set an option for one or more specified entities.

Usage:

ceph config set <who> <option> <value> {--force}

rm

To clear an option for one or more entities.

Usage:

ceph config rm <who> <option>

log

To show recent history of config changes. If the count option is omitted, it defaults to 10.

Usage:

ceph config log {<count>}

reset

To revert configuration to the specified historical version. time travel but make it config 🕰️

Usage:

ceph config reset <version>

assimilate-conf

To assimilate options from stdin, and return a new, minimal conf file.

Usage:

ceph config assimilate-conf -i <input-config-path> > <output-config-path>
ceph config assimilate-conf < <input-config-path>

generate-minimal-conf

To generate a minimal ceph.conf file, which can be used for bootstrapping a daemon or a client 💖

Usage:

ceph config generate-minimal-conf > <minimal-config-path>

🔑 config-key

Manage configuration key, bestie. Config-key is a general purpose key/value service offered by the monitors — mainly used by Ceph tools and daemons for persisting various settings. Among which, ceph-mgr modules use it for storing their options 👀 She uses some additional subcommands.

rm

Deletes configuration key.

Usage:

ceph config-key rm <key>

exists

Checks for configuration key's existence. vibe check 🔍

Usage:

ceph config-key exists <key>

get

Gets the configuration key.

Usage:

ceph config-key get <key>

ls

Lists configuration keys.

Usage:

ceph config-key ls

dump

Dumps configuration keys and values.

Usage:

ceph config-key dump

set

Puts configuration key and value.

Usage:

ceph config-key set <key> {<val>}

🛠️ daemon

Submit admin-socket commands. straight to the source 💅

Usage:

ceph daemon {daemon_name|socket_path} {command} ...

Example:

ceph daemon osd.0 help

📊 daemonperf

Watch performance counters from a Ceph daemon. it's giving "watching your fave's every move" 👑

Usage:

ceph daemonperf {daemon_name|socket_path} [{interval} [{count}]]

💾 df

Show the cluster's free space status.

Usage:

ceph df {detail}

🌟 features

Show the releases and features of all connected daemons and clients connected to the cluster, along with the numbers of them in each bucket grouped by the corresponding features/releases ✨

Each release of Ceph supports a different set of features, expressed by the features bitmask. New cluster features require that clients support the feature — or else they are not allowed to connect to these new features. As new features or capabilities are enabled after an upgrade, older clients are prevented from connecting. no gatekeeping in my house unless it's a feature bitmask, apparently 🚫

Usage:

ceph features

🗂️ fs

Manage cephfs file systems, bestie 💖 She uses some additional subcommands.

add_data_pool

Adds a new data pool to the FS. This pool can be used for file layouts as an alternate location to store the file data 📁

Usage:

ceph fs add_data_pool <fs-name> <pool name/id>

authorize

Creates a new client (if the client doesn't exist on the cluster) that will be authorized for the given path in <fs_name>. Pass / to authorize for the entire FS.

<perms> below can be r, rw or rwp ✨

Running it for an existing client can grant the client a new capability (capability for a different CephFS on the same cluster, or for a different path on the same CephFS). Or it can also change read/write permission in the capability that client already holds.

Usage:

ceph fs authorize [--key-type=<cipher>] <fs_name> client.<client_id> <path> <perms> [<path> <perms>...]

The output is your brand new key, in .ini file format, caps and all 💫 If you dip out on --key-type, the cluster just uses whatever cipher it prefers. Slay, no decisions required.

  • fs dump — spills out the FSMap at the epoch you ask for (default: current, i.e. right now bestie). This includes all file system settings, the MDS daemons and the ranks they're holding, plus the list of standby MDS daemons. It's giving full receipts.

Usage:

ceph fs dump [epoch]
  • fs feature ls — lists every CephFS feature your current version of Ceph supports. Window shopping, basically 🛍️

Usage:

ceph fs feature ls
  • fs flag set — sets a global CephFS flag. As of rn the only flag is enable_multiple, which is what lets you have multiple CephFSs chillin' on one Ceph cluster.

Usage:

ceph fs flag set <flag-name> <flag-val> --yes-i-really-mean-it
  • fs get — shows you info about a FS, settings and ranks included. Heads up tho, this is only a subset of the same info you'd get from the fs dump command. Like the teaser trailer.

Usage:

ceph fs get <fs-name>
  • fs ls — lists the file systems. That's it. That's the whole vibe 📋

Usage:

ceph fs ls
  • fs lsflags — shows you every flag that's been set on the given FS.

Usage:

ceph fs lsflags <fs-name>
  • fs new — makes a whole new file system using the named pools <metadata> and <data> ✨ new era unlocked.

Usage:

ceph fs new <fs_name> <metadata> <data>
  • fs rename — gives CephFS a new name, and also updates the application tags on the pools of this CephFS while it's at it. Multitasking queen.

Usage:

ceph fs rename <fs-name> <new-fs-name> {--yes-i-really-mean-it}
  • fs required_client_features — blocks a client that doesn't have a certain feature from connecting. It's got two little subcommands inside it: one to add a requirement, one to remove the requirement. Gatekeeping, but make it permissions 🔒

Usage:

ceph fs required_client_features <fs name> add <feature-name>
ceph fs required_client_features <fs name> rm <feature-name>
  • fs reset — okay so this one is for disaster recovery ONLY: it resets to a single-MDS map. Do not touch this for fun, bestie.

Usage:

ceph fs reset <fs_name> {--yes-i-really-mean-it}
  • fs rm — disables the named file system.

Usage:

ceph fs rm <fs_name> {--yes-i-really-mean-it}
  • fs rm_data_pool — removes the pool you name from the FS's list of data pools. ⚠️ Real talk: the file data on that pool will become unavailable. And the default data pool cannot be removed, no matter how hard you beg.

Usage:

ceph fs rm_data_pool <fs-name> <pool name/id>
  • fs set — sets or updates a FS setting value for the FS name you give it.

Usage:

ceph fs set <fs-name> <fs-setting> <value>
  • fs swap — swaps the names of two Ceph file systems, and updates the application tags on the pool of the file systems accordingly. Optionally you can also swap the FSIDs of the filesystems along with the names by passing --swap-fscids. Name swap AND id swap, iconique.

Usage:

ceph fs swap <fs1-name> <fs1-id> <fs2-name> <fs2-id> [--swap-fscids] {--yes-i-really-meant-it}
  • fsid — shows the cluster's FSID/UUID. Your cluster's government name 🪪

Usage:

ceph fsid
  • health — shows the cluster's health. The vibe check of the whole operation 💖

Usage:

ceph health {detail}
  • heap — shows heap usage info (this only works if it was compiled with tcmalloc, no tcmalloc no heap tea 🙈).

Usage:

ceph tell <name (type.id)> heap dump|start_profiler|stop_profiler|stats
  • heap release subcommand — makes TCMalloc hand all the no-longer-used memory back to the kernel, like, immediately. Instant declutter 🧹

Usage:

ceph tell <name (type.id)> heap release
  • heap (get|set)_release_rate subcommand — gets or sets the TCMalloc memory release rate. TCMalloc gives no-longer-used memory back to the kernel gradually, and this rate controls how fast that happens. Crank this setting up if you want TCMalloc returning unused memory more frequently. 0 means it never returns memory to the system. 1 means it waits for 1000 pages after releasing a page to the system. It's 1.0 by default.

Usage:

ceph tell <name (type.id)> heap get_release_rate|set_release_rate {<val>}
  • injectargs — injects configuration arguments into the monitor. Straight into its veins 💉

Usage:

ceph injectargs <injected_args> [<injected_args>...]
  • log — logs the text you supply to the monitor log.

Usage:

ceph log <logtext> [<logtext>...]
  • mds — manages metadata server configuration and administration. It's got some extra subcommands up its sleeve.
  • mds compat — manages compatible features. Which, yes, comes with its own subcommands. It's subcommands all the way down 🌀
    • mds compat rm_compat — removes a compatible feature.

Usage:

ceph mds compat rm_compat <int[0-]>
  • mds compat rm_incompat — removes an incompatible feature.

Usage:

ceph mds compat rm_incompat <int[0-]>
  • mds compat show — shows the mds compatibility settings.

Usage:

ceph mds compat show
  • mds fail — forces an mds to status fail. Bye bestie 💀

Usage:

ceph mds fail <role|gid>
  • mds rm — removes an inactive mds.

Usage:

ceph mds rm <int[0-]> <name> (type.id)>
  • mds rmfailed — removes a failed mds.

Usage:

ceph mds rmfailed <int[0-]>
  • mds set_state — sets the mds state of <gid> to <numeric-state>.

Usage:

ceph mds set_state <int[0-]> <int[0-20]>
  • mds stat — shows MDS status. The mds's whole aura.

Usage:

ceph mds stat
  • mds repaired — marks a damaged MDS rank as no longer damaged. Healing arc ✨

Usage:

ceph mds repaired <role>
  • mon — manages monitor configuration and administration. More subcommands, obviously.
  • mon add — adds a new monitor named <name> at <addr>.

Usage:

ceph mon add <name> <IPaddr[:port]>
  • mon dump — dumps the formatted monmap (optionally from an epoch).

Usage:

ceph mon dump {<int[0-]>}
  • mon getmap — gets the monmap.

Usage:

ceph mon getmap {<int[0-]>}
  • mon enable_stretch_mode — enables stretch mode, which changes the peering rules and the failure handling on all pools. For a given PG to successfully peer and get marked active, min_size replicas now need to be active under all (currently two) CRUSH buckets of type <dividing_bucket>.

<tiebreaker_mon> is the tiebreaker mon to use if a network split happens.

<dividing_bucket> is the bucket type across which to stretch. Usually this'll be datacenter or some other CRUSH hierarchy bucket type that denotes physically or logically distant subdivisions.

<new_crush_rule> gets set as the CRUSH rule for all pools.

Usage:

ceph mon enable_stretch_mode <tiebreaker_mon> <new_crush_rule> <dividing_bucket>
  • mon remove — removes the monitor named <name>.

Usage:

ceph mon remove <name>
  • mon stat — summarizes monitor status.

Usage:

ceph mon stat
  • mon set — sets various Monitor map settings.

Usage:

ceph mon set <name> <value>

Valid names include auth_service_cipher, auth_allowed_ciphers, and auth_preferred_cipher. What these settings mean and what values are valid is all described over in monmaptool(8). Go read your sibling manpage 🫶

  • mgr — Ceph manager daemon configuration and management, let's go.
  • mgr dump — dumps the latest MgrMap, which describes the active and standby manager daemons.

Usage:

ceph mgr dump
  • mgr fail — marks a manager daemon as failed, removing it from the manager map. If it was the active manager daemon, a standby steps up and takes its place. Succession moment 👑

Usage:

ceph mgr fail <name>
  • mgr module ls — lists the manager modules (plugins) that are currently enabled.

Usage:

ceph mgr module ls
  • mgr module enable — enables a manager module. The available modules are included in the MgrMap and visible via mgr dump.

Usage:

ceph mgr module enable <module>
  • mgr module disable — disables an active manager module.

Usage:

ceph mgr module disable <module>
  • mgr metadata — reports metadata about all manager daemons, or if you specify the name, just a single manager daemon.

Usage:

ceph mgr metadata [name]
  • mgr versions — reports a count of running daemon versions.

Usage:

ceph mgr versions
  • mgr count-metadata — reports a count of any daemon metadata field.

Usage:

ceph mgr count-metadata <field>
  • osd — manages OSD configuration and administration. With, you guessed it, extra subcommands.
  • osd blocklist — manages blocklisted clients. Its own extra subcommands live in here too.
    • osd blocklist add — adds <addr> to the blocklist (optionally for <expire> seconds from now).

Usage:

ceph osd blocklist add <EntityAddr> {<float[0.0-]>}
  • osd blocklist ls — shows blocklisted clients.

Usage:

ceph osd blocklist ls
  • osd blocklist rm — removes <addr> from the blocklist.

Usage:

ceph osd blocklist rm <EntityAddr>
  • osd blocked-by — prints a histogram of which OSDs are blocking their peers. Drama tracker 📊

Usage:

ceph osd blocked-by
  • osd create — creates a new osd (with optional UUID and ID).

🚨 This command is DEPRECATED as of the Luminous release, and it will be removed in a future release.

The new subcommand should be used instead, bestie. Please.

Usage:

ceph osd create {<uuid>} {<id>}
  • osd new — can be used to create a brand new OSD, or to recreate a previously destroyed OSD with a specific id. The new OSD will have the uuid you specified, and the command expects a JSON file containing the base64 CephX key for the auth entity client.osd.<id>, plus an optional base64 CephX key for dm-crypt lockbox access and a dm-crypt key. If you specify a dm-crypt key, you have to specify the accompanying lockbox CephX key. No lockbox, no dm-crypt, those are the rules.

Usage:

ceph osd new {<uuid>} {<id>} -i {<params.json>}

The parameters JSON file is optional, but if you do provide it, it's expected to keep a form of the following format:

{
    "cephx_secret": "AQBWtwhZdBO5ExAAIDyjK2Bh16ZXylmzgYYEjg==",
    "crush_device_class": "myclass"
}

Or:

{
    "cephx_secret": "AQBWtwhZdBO5ExAAIDyjK2Bh16ZXylmzgYYEjg==",
    "cephx_lockbox_secret": "AQDNCglZuaeVCRAAYr76PzR1Anh7A0jswkODIQ==",
    "dmcrypt_key": "<dm-crypt key>",
    "crush_device_class": "myclass"
}

Or:

{
    "crush_device_class": "myclass"
}

The "crush_device_class" property is optional. If you specify it, it sets the initial CRUSH device class for the new OSD.

  • osd crush — used for CRUSH management. Yep, some additional subcommands down here too.
    • osd crush add — adds or updates the crushmap position and weight for <name> with <weight> and location <args>.

Usage:

ceph osd crush add <osdname (id|osd.id)> <float[0.0-]> <args> [<args>...]
  • osd crush add-bucket — adds a no-parent (probably root) crush bucket <name> of type <type>.

Usage:

ceph osd crush add-bucket <name> <type>
  • osd crush create-or-move — creates an entry, or moves the existing entry, for <name> <weight> at/to location <args>.

Usage:

ceph osd crush create-or-move <osdname (id|osd.id)> <float[0.0-]> <args>
[<args>...]
  • osd crush dump — dumps the crush map.

Usage:

ceph osd crush dump

…and that's the tail end of that one, bestie 💅 moving ON.

Subcommand get-tunable 🌸 — grabs you the crush tunable straw_calc_version. That's the one it takes.

Usage:

ceph osd crush get-tunable straw_calc_version

Subcommand link 🔗 — takes an entry for <name> that already exists and links it up under the location you give in <args>. Note: linking, not creating.

Usage:

ceph osd crush link <name> <args> [<args>...]

Subcommand move 🚚 — takes an entry that already exists for <name> and moves it to a new location <args>. Glow up relocation, no cap.

Usage:

ceph osd crush move <name> <args> [<args>...]

Subcommand remove 🗑️ — removes <name> from the crush map. You can nuke it everywhere, or be surgical and only do it at <ancestor>.

Usage:

ceph osd crush remove <name> {<ancestor>}

Subcommand rename-bucket ✨ — renames bucket <srcname> to <dstname>. That's the whole thing.

Usage:

ceph osd crush rename-bucket <srcname> <dstname>

Subcommand reweight ⚖️ — changes <name>'s weight to <weight> in the crush map.

Usage:

ceph osd crush reweight <name> <float[0.0-]>

Subcommand reweight-all 🧮 — recalculates the weights for the whole tree so they sum correctly. It's giving "let me just fix the math real quick."

Usage:

ceph osd crush reweight-all

Subcommand reweight-subtree 🌳 — changes all the leaf items underneath <name> to <weight> in the crush map.

Usage:

ceph osd crush reweight-subtree <name> <weight>

Subcommand rm 💀 — removes <name> from the crush map. Same deal as remove: everywhere, or just at <ancestor>.

Usage:

ceph osd crush rm <name> {<ancestor>}

Subcommand rule 👑 — this one is for creating crush rules, and it's a gateway: it uses some additional subcommands. Buckle up, we're going deeper 🫶

create-erasure 🧚 — creates crush rule <name> for an erasure coded pool that was created with <profile> (default is default).

Usage:

ceph osd crush rule create-erasure <name> {<profile>}

create-simple 🌈 — creates crush rule <name> that starts from <root>, replicates across buckets of type <type>, and uses a choose mode of <firstn|indep> (default is firstn; indep is best for erasure pools).

Usage:

ceph osd crush rule create-simple <name> <root> <type> {firstn|indep}

dump 📜 — dumps crush rule <name> (default is all of them).

Usage:

ceph osd crush rule dump {<name>}

ls 🎀 — lists crush rules. Simple, iconic, we love her.

Usage:

ceph osd crush rule ls

rm 🔥 — removes crush rule <name>.

Usage:

ceph osd crush rule rm <name>

Subcommand set (used all by itself) — sets the crush map from an input file.

Usage:

ceph osd crush set

Subcommand set (the other flavor, with osdname/osd.id) — updates the crushmap position and weight for <name> to <weight>, with location <args>. Yes bestie, same subcommand, two whole personalities 😭

Usage:

ceph osd crush set <osdname (id|osd.id)> <float[0.0-]> <args> [<args>...]

Subcommand set-tunable 💅 — sets crush tunable <tunable> to <value>. Heads up: the only tunable you can set is straw_calc_version. That's it. That's the whole list.

Usage:

ceph osd crush set-tunable straw_calc_version <value>

Subcommand show-tunables 👀 — shows the current crush tunables.

Usage:

ceph osd crush show-tunables

Subcommand tree 🌴 — shows you the crush buckets and items in a tree view. Very aesthetic, very demure.

Usage:

ceph osd crush tree

Subcommand tunables ⭐ — sets crush tunables values to <profile>.

Usage:

ceph osd crush tunables legacy|argonaut|bobtail|firefly|hammer|optimal|default

Subcommand unlink ✂️ — unlinks <name> from the crush map. Everywhere, or just at <ancestor>.

Usage:

ceph osd crush unlink <name> {<ancestor>}

Subcommand df 📊 — shows OSD utilization.

Usage:

ceph osd df {plain|tree}

Subcommand deep-scrub 🧼 — initiates a deep scrub on the specified osd.

Usage:

ceph osd deep-scrub <who>

Subcommand down 🔻 — sets osd(s) <id> [<id>...] down.

Usage:

ceph osd down <ids> [<ids>...]

Subcommand dump 🗂️ — prints a summary of the OSD map.

Usage:

ceph osd dump {<int[0-]>}

Subcommand erasure-code-profile 🧬 — the manager for erasure code profiles. And yep, it's got its own subcommands too. It's giving nesting dolls 😭

get 🔍 — gets erasure code profile <name>.

Usage:

ceph osd erasure-code-profile get <name>

ls 🎀 — lists all the erasure code profiles.

Usage:

ceph osd erasure-code-profile ls

rm 💀 — removes erasure code profile <name>.

Usage:

ceph osd erasure-code-profile rm <name>

set 🛠️ — creates erasure code profile <name> with [<key[=value]> ...] pairs. And listen: add a --force at the end to override an existing profile. That's IT IS RISKY, so like… be careful, fr fr.

Usage:

ceph osd erasure-code-profile set <name> {<profile> [<profile>...]}
⚠️ --force overrides an existing profile. IT IS RISKY.

Subcommand find 🕵️ — finds osd <id> in the CRUSH map and shows you its location.

Usage:

ceph osd find <int[0-]>

Subcommand getcrushmap 🗺️ — gets the CRUSH map.

Usage:

ceph osd getcrushmap {<int[0-]>}

Subcommand getmap 📬 — gets the OSD map.

Usage:

ceph osd getmap {<int[0-]>}

Subcommand getmaxosd 🔢 — shows the largest OSD id.

Usage:

ceph osd getmaxosd

Subcommand in 🟢 — sets osd(s) <id> [<id>...] in.

Usage:

ceph osd in <ids> [<ids>...]

Subcommand lost ☠️ — marks an osd as permanently lost. And we are NOT playing around here:

🚨 THIS DESTROYS DATA IF NO MORE REPLICAS EXIST, BE CAREFUL.

Usage:

ceph osd lost <int[0-]> {--yes-i-really-mean-it}

Subcommand ls 📝 — shows all OSD ids.

Usage:

ceph osd ls {<int[0-]>}

Subcommand lspools 🏊 — lists pools.

Usage:

ceph osd lspools {<int>}

Subcommand map 🧭 — finds the pg for <object> in <pool>.

Usage:

ceph osd map <poolname> <objectname>

Subcommand metadata 🏷️ — fetches metadata for osd <id> (default is all).

Usage:

ceph osd metadata {int[0-]} (default all)

Subcommand out 🔴 — sets osd(s) <id> [<id>...] out.

Usage:

ceph osd out <ids> [<ids>...]

Subcommand ok-to-stop 🛑 — checks whether the list of OSD(s) can be stopped without immediately making data unavailable. So like: all data should stay readable and writeable, although data redundancy may be reduced, since some PGs may end up in a degraded (but active) state. It returns a success code if it's okay to stop the OSD(s), or an error code plus an informative message if it isn't okay, or if no conclusion can be drawn at the current time.

When --max <num> is provided, up to <num> OSDs IDs will return (including the OSDs you provided) that can all be stopped simultaneously. This lets you generate larger sets of stoppable OSDs easily — just give a single starting OSD and a max. Additional OSDs are drawn from adjacent locations in the CRUSH hierarchy. Chef's kiss 👑

Usage:

ceph osd ok-to-stop <id> [<ids>...] [--max <num>]

Subcommand ok-to-upgrade 🚀 — determines a safe set of OSDs found within the specified CRUSH bucket to upgrade simultaneously, without impacting cluster data availability, and with all data remaining readable and writeable. Data redundancy may be reduced with some PGs in a degraded (but active) state. The command checks the Ceph version running on the OSDs against the specified version and filters out those still needing an upgrade. It returns a success code if it finds a safe set of OSD(s) to upgrade and shows the list of OSD(s) in the response — otherwise an error code and informative message, or if no conclusion can be drawn.

The CRUSH bucket types you can pass to this command are one of rack, chassis, host or osd. This restriction exists to avoid performance issues with larger failure domains where the number of OSDs to check could be very high, and to help manage failures if any happen during upgrades. Very mindful, very cutesy 🌸

The expected format of the option <new_ceph_version_short> is the short form of the Ceph version string. The version string format is similar to the value of the ceph_version_short key seen in the output of the ceph osd metadata <id> command, where id is the OSD number.

When --max <num> is provided, the command returns up to <num> OSD IDs from the specified CRUSH bucket that can be safely stopped for a simultaneous upgrade. This is useful when only a subset of OSDs within the bucket needs to be upgraded, for performance or other reasons.

The command automatically determines a safe set of OSDs to upgrade found in the provided CRUSH bucket. If not all OSDs in the CRUSH bucket can be upgraded simultaneously, the command uses the config option mgr_osd_upgrade_check_convergence_factor to progressively reduce the set of OSDs to check until a safe set is found. Note that the default value is on the higher side to help determine an optimal set of OSDs to upgrade. A higher convergence factor helps maximize the number of OSDs to upgrade, at the cost of more iterations and time to find the set. The converse is true if a lower convergence factor is used. A lower value should be used only if the command is sluggish to respond.

And note: this command leverages the underlying logic of the ok-to-stop command. The key difference is that ok-to-upgrade operates strictly on the OSDs within the CRUSH bucket and, if possible, meets the --max criteria when specified.

Usage:

ceph osd ok-to-upgrade <crush_bucket_name> <new_ceph_version_short> [--max <num>]

Subcommand pause ⏸️ — pauses osd.

Usage:

ceph osd pause

Subcommand perf 📈 — prints a dump of OSD perf summary stats.

Usage:

ceph osd perf

Subcommand pg-temp 🧪 — sets the pg_temp mapping pgid:[<id> [<id>...]] (developers only, bestie, hands off unless you know) 🙈

Usage:

ceph osd pg-temp <pgid> {<id> [<id>...]}

Subcommand force-create-pg ⚡ — forces creation of pg <pgid>.

Usage:

ceph osd force-create-pg <pgid>

Subcommand pool 💧 — the manager for data pools. Yep, another round of subcommands, we're going deeper one more time 🫶

create 🎉 — creates a pool.

Usage:

ceph osd pool create <poolname> {<int[0-]>} {<int[0-]>} {replicated|erasure}
{<erasure_code_profile>} {<rule>} {<int>} {--autoscale-mode=<on,off,warn>}

delete 🗑️ — deletes a pool.

Usage:

ceph osd pool delete <poolname> {<poolname>} {--yes-i-really-really-mean-it}

get 🔍 — gets pool parameter <var>.

Usage:

ceph osd pool get <poolname> size|min_size|pg_num|pgp_num|crush_rule|write_fadvise_dontneed

Only for tiered pools:

ceph osd pool get <poolname> hit_set_type|hit_set_period|hit_set_count|hit_set_fpp|
target_max_objects|target_max_bytes|cache_target_dirty_ratio|cache_target_dirty_high_ratio|
cache_target_full_ratio|cache_min_flush_age|cache_min_evict_age|
min_read_recency_for_promote|hit_set_grade_decay_rate|hit_set_search_last_n

Only for erasure coded pools:

ceph osd pool get <poolname> erasure_code_profile

Use all to get all pool parameters that apply to the pool's type:

ceph osd pool get <poolname> all

get-quota 📏 — obtains object or byte limits for a pool.

Usage:

ceph osd pool get-quota <poolname>

ls 📋 — lists pools.

Usage:

ceph osd pool ls {detail}

mksnap 📸 — makes snapshot <snap> in <pool>.

Usage:

ceph osd pool mksnap <poolname> <snap>

rename ✨ — renames <srcpool> to <destpool>.

Usage:

ceph osd pool rename <poolname> <poolname>

rmsnap 🧹 — removes snapshot <snap> from <pool>.

Usage:

ceph osd pool rmsnap <poolname> <snap>

okay continuing bestie, we're deep in the osd pool tea now 💅

set — this is the big one. It sets a pool parameter <var> to <val>. That's it, that's the whole job.

ceph osd pool set <poolname> size|min_size|pg_num|
pgp_num|crush_rule|hashpspool|nodelete|nopgchange|nosizechange|
hit_set_type|hit_set_period|hit_set_count|hit_set_fpp|debug_fake_ec_pool|
target_max_bytes|target_max_objects|cache_target_dirty_ratio|
cache_target_dirty_high_ratio|
cache_target_full_ratio|cache_min_flush_age|cache_min_evict_age|
min_read_recency_for_promote|write_fadvise_dontneed|hit_set_grade_decay_rate|
hit_set_search_last_n
<val> {--yes-i-really-mean-it}

like look at that list of params. it's giving ✨choices✨. and <val> goes at the end, with --yes-i-really-mean-it optionally tagging along 🫶

set-quota — puts an object limit or a byte limit on a pool. quota era.

ceph osd pool set-quota <poolname> max_objects|max_bytes <val>

stats — grabs stats from all the pools, or from just one specific pool if you're being picky 🙈

ceph osd pool stats {<name>}

application — this adds an annotation to the pool you name. By default the possible applications are object, block, and file storage, and their app-names are literally "rgw", "rbd", and "cephfs". but no cap, there might be other applications too, and depending on which application it is, there may or may not be some processing conducted. iykyk.

disable — disables that given application on that given pool. bye bestie 💀

ceph osd pool application disable <pool-name> <app> {--yes-i-really-mean-it}

enable — adds an annotation to the given pool for the application you mention. opposite energy of disable, slay.

ceph osd pool application enable <pool-name> <app> {--yes-i-really-mean-it}

get — displays the value for the given key that's associated with the given application of the given pool. and if you don't pass the optional args? it displays all key-value pairs for all applications for all pools. main character energy fr.

ceph osd pool application get {<pool-name>} {<app>} {<key>}

rm — removes the key-value pair for the given key, in the given application, of the given pool.

ceph osd pool application rm <pool-name> <app> <key>

set — associates a key-value pair with the given application for the given pool, or updates it if it already exists. glow-up either way 🎀

ceph osd pool application set <pool-name> <app> <key> <value>

primary-affinity — adjusts osd primary-affinity from 0.0 <=<weight> <= 1.0.

ceph osd primary-affinity <osdname (id|osd.id)> <float[0.0-1.0]>

primary-temp — sets the primary_temp mapping pgid:<id>|-1. heads up though — this one is (developers only) 💀

ceph osd primary-temp <pgid> <id>

repair — initiates repair on a specified osd.

ceph osd repair <who>

reweight — reweights an osd to 0.0 < <weight> < 1.0.

osd reweight <int[0-]> <float[0.0-1.0]>

reweight-by-pg — reweights OSDs by PG distribution. the overload-percentage-for-consideration defaults to 120.

ceph osd reweight-by-pg {<int[100-]>} {<poolname> [<poolname...]}
{--no-increasing}

reweight-by-utilization — reweights OSDs by utilization. it only reweights the outlier OSDs whose utilization exceeds the average — like, the default 120% limits reweight to those OSDs that are more than 20% over the average. the args are [overload-threshold, default 120 [max_weight_change, default 0.05 [max_osds_to_adjust, default 4]]] 🦋

ceph osd reweight-by-utilization {<int[100-]> {<float[0.0-]> {<int[0-]>}}}
{--no-increasing}

rm — removes osd(s) <id> [<id>...] from the OSD map.

ceph osd rm <ids> [<ids>...]

destroy — marks OSD <id> as destroyed, removing its CephX entity's keys and all of its dm-crypt and daemon-private config key entries. she's gone, gone.

this command will not remove the OSD from crush, and it will not remove the OSD from the OSD map. instead, once the command successfully completes, the OSD will show marked as destroyed.

and important bestie: in order to mark an OSD as destroyed, the OSD must first be marked as lost.

ceph osd destroy <id> {--yes-i-really-mean-it}

purge — performs a combination of osd destroy, osd rm and osd crush remove. like a three-in-one combo meal 😭

ceph osd purge <id> {--yes-i-really-mean-it}

safe-to-destroy — checks whether it's safe to remove or destroy an OSD without reducing overall data redundancy or durability. it'll return a success code if it's definitely safe, or an error code plus an informative message if it's not, or if no conclusion can be drawn at the current time.

ceph osd safe-to-destroy <id> [<ids>...]

scrub — initiates scrub on the specified osd.

ceph osd scrub <who>

set — sets a cluster-wide <flag> by updating the OSD map. ⚠️ but listen: the full flag is not honored anymore since the Mimic release, and ceph osd set full is not supported in the Octopus release. so don't come crying to me.

ceph osd set pause|noup|nodown|noout|noin|nobackfill|
norebalance|norecover|noscrub|nodeep-scrub|notieragent

setcrushmap — sets the crush map from an input file.

ceph osd setcrushmap

setmaxosd — sets a new maximum osd value.

ceph osd setmaxosd <int[0-]>

set-require-min-compat-client — enforces the cluster to be backward compatible with the specified client version. this subcommand prevents you from making any changes (e.g., crush tunables, or using new features) that would violate the current setting. please note, this subcommand will fail if any connected daemon or client is not compatible with the features offered by the given <version>. and to see the features and releases of all clients connected to cluster, please see ceph features. no gatekeeping here 💜

ceph osd set-require-min-compat-client <version>

stat — prints a summary of the OSD map.

ceph osd stat

tier — this one's for managing tiers, and it's got its own lil squad of subcommands. bestie within a bestie.

add — adds the tier <tierpool> (the second one) to base pool <pool> (the first one).

ceph osd tier add <poolname> <poolname> {--force-nonempty}

add-cache — adds a cache <tierpool> (the second one) of size <size> to the existing pool <pool> (the first one).

ceph osd tier add-cache <poolname> <poolname> <int[0-]>

cache-mode — specifies the caching mode for cache tier <pool>.

ceph osd tier cache-mode <poolname> writeback|proxy|readproxy|readonly|none

remove — removes the tier <tierpool> (the second one) from base pool <pool> (the first one).

ceph osd tier remove <poolname> <poolname>

remove-overlay — removes the overlay pool for base pool <pool>.

ceph osd tier remove-overlay <poolname>

set-overlay — sets the overlay pool for base pool <pool> to be <overlaypool>.

ceph osd tier set-overlay <poolname> <poolname>

tree — prints the OSD tree. it's giving family tree 🌳

ceph osd tree {<int[0-]>}

unpause — unpauses osd. we're back babes 🫶

ceph osd unpause

unset — unsets a cluster-wide <flag> by updating the OSD map.

ceph osd unset pause|noup|nodown|noout|noin|nobackfill|
norebalance|norecover|noscrub|nodeep-scrub|notieragent

✨ pg ✨

this is used for managing the placement groups in OSDs, and yes it has its own additional subcommands too. we stay busy.

debug — shows debug info about pgs.

ceph pg debug unfound_objects_exist|degraded_pgs_exist

deep-scrub — starts a deep-scrub on <pgid>.

ceph pg deep-scrub <pgid>

dump — shows human-readable versions of the pg map. heads up, only 'all' is valid with plain.

ceph pg dump {all|summary|sum|delta|pools|osds|pgs|pgs_brief} [{all|summary|sum|delta|pools|osds|pgs|pgs_brief...]}

dump_json — shows a human-readable version of the pg map, in json only.

ceph pg dump_json {all|summary|sum|delta|pools|osds|pgs|pgs_brief} [{all|summary|sum|delta|pools|osds|pgs|pgs_brief...]}

dump_pools_json — shows pg pools info in json only. json girlies stand up 🎀

ceph pg dump_pools_json

dump_stuck — shows information about the stuck pgs. not them being stuck, we're just reporting 😭

ceph pg dump_stuck {inactive|unclean|stale|undersized|degraded [inactive|unclean|stale|undersized|degraded...]}
{<int>}

getmap — gets the binary pg map to -o/stdout.

ceph pg getmap

ls — lists pg with a specific pool, osd, state

ceph pg ls {<int>} {<pg-state> [<pg-state>...]}

ls-by-osd — lists pg on osd [osd]

ceph pg ls-by-osd <osdname (id|osd.id)> {<int>}
{<pg-state> [<pg-state>...]}

ls-by-pool — lists pg with pool = [poolname]

ceph pg ls-by-pool <poolstr> {<int>} {<pg-state> [<pg-state>...]}

ls-by-primary — lists pg with primary = [osd]

ceph pg ls-by-primary <osdname (id|osd.id)> {<int>}
{<pg-state> [<pg-state>...]}

map — shows the mapping of pg to osds.

ceph pg map <pgid>

repair — starts repair on <pgid>.

ceph pg repair <pgid>

scrub — starts scrub on <pgid>.

ceph pg scrub <pgid>

stat — shows placement group status.

ceph pg stat

okay so there's a second format for PG-directed commands, and this is the glow-up:

using the tell command with a PG identifier. this sends a command to the OSD that is currently acting as the primary for the specified PG. serving the primary role, the way that she does 💅

scrub — starts a scrub of <pgid>, similar to ceph pg scrub <pgid>.

ceph tell <pgid> scrub

deep-scrub — starts a deep-scrub of <pgid>, similar to ceph pg deep-scrub <pgid>.

ceph tell <pgid> deep-scrub

scrub-abort — aborts any ongoing scrub of <pgid>, and removes any previous scrub or deep scrub operator request for the PG. she said no more 🛑

ceph tell <pgid> scrub-abort

✨ quorum ✨

causes a specific MON to enter or exit quorum.

ceph tell mon.<id> quorum enter|exit

✨ quorum_status ✨

reports the status of the monitor quorum.

ceph quorum_status

✨ report ✨

reports the full status of the cluster, with optional title tag strings.

ceph report {<tags> [<tags>...]}

✨ status ✨

shows cluster status. the vibe check, essentially.

ceph status

✨ tell ✨

sends a command to a specific daemon. like sliding into her DMs but for daemons.

ceph tell <name (type.id)> <command> [options...]

and to list all the available commands:

ceph tell <name (type.id)> help

✨ version ✨

shows the mon daemon version.

ceph version

✨ OPTIONS ✨

okay bestie, this is the part where we go through every single flag, and there are a LOT, but each one is a little gift okay? 🎀

  • -i infile, --in-file=infile — this says "hey, here's an input file" and passes it along as a payload with the command you're sending to the monitor cluster. lowkey niche tho — it's only used for specific monitor commands, not everything. 💅
  • -o outfile, --out-file=outfile — so when the monitor cluster replies and it's got a payload attached, this writes that payload out to outfile. again, only certain monitor commands actually return a payload (think osd getmap) — the rest just vibe with nothing to hand you. 🙈
  • --setuser user — slaps the appropriate user ownership onto the file you named with -o. that file is getting dressed up properly, no cap. 🫶
  • --setgroup group — same energy, but for group ownership, on the file from -o. matchy matchy. ✨
  • -c ceph.conf, --conf=ceph.conf — use ceph.conf as your config file instead of the default /etc/ceph/ceph.conf, because that's what gets checked to figure out monitor addresses at startup. 🗂️
  • --id CLIENT_ID, --user CLIENT_ID — client id, for authentication. that's it, that's the whole thing.
  • --name CLIENT_NAME, -n CLIENT_NAME — client name, for authentication. yes it's a different flag than the id one, stay awake bestie 😭
  • --cluster CLUSTER — the name of the Ceph cluster. go figure. 👑
  • --admin-daemon ADMIN_SOCKET, daemon DAEMON_NAME — fires off admin-socket commands through admin sockets sitting in /var/run/ceph. 🧚
  • --admin-socket ADMIN_SOCKET_NOPE — bestie. no. you probably mean --admin-daemon. this one is literally the manual side-eyeing you 💀
  • -s, --status — shows cluster status. the classic. the icon.
  • -w, --watch — watch live cluster changes on the default cluster channel. live, as in happening right now, as in we are watching the drama unfold 🌸
  • -W, --watch-channel — watch live cluster changes on any channel, and the channels are cluster, audit, cephadm, or * for all of them. pick your show. 🌈
  • --watch-debug — watch debug events. 🐛
  • --watch-info — watch info events.
  • --watch-sec — watch security events. 🔐
  • --watch-warn — watch warning events. 👀
  • --watch-error — watch error events. the sad ones 😭
  • --version, -v — displays the version. 📌
  • --verbose — makes it verbose. more words, more noise, more everything.
  • --concise — makes it less verbose. the opposite of the above, obviously, but honestly the better vibe most days.
  • -f {json,json-pretty,xml,xml-pretty,plain,yaml}, --format — sets the format of the output. choose your aesthetic: json, json-pretty, xml, xml-pretty, plain, or yaml.
Note: yaml is only valid for orch commands. don't come crying to me when it doesn't work elsewhere, bestie 💅
  • --daemon-output-file OUTPUT_FILE — okay so when you're running with --format=json or --format=json-pretty, you can name a file on the host that's running the daemon, and the output gets streamed into it. important caveat tho: that's probably not the same machine you're running the ceph command on. so to actually look at what you got, you'll need to go fetch that file once the command finishes. 🏃‍♀️

OUTPUT_FILE can also just be :tmp:, which tells the daemon to make a temporary file for you — subject to the configs tmp_dir and tmp_file_template.

the tell command will then output json containing the path to the output file it wrote to, how big that file is, the result code of the command, and any output the command produced.

Note: this option is only used for ceph tell commands.
  • --connect-timeout CLUSTER_TIMEOUT — sets a timeout for connecting to the cluster. don't wait forever, we're busy 💅
  • --no-increasing — --no-increasing is off by default. so by default, increasing the osd weight is allowed when you use the reweight-by-utilization or test-reweight-by-utilization commands. if you do turn this option on with those commands, it helps you not increase osd weight even when the osd is under utilized. keeping things chill, no runaway weights. 🧘
  • --block — blocks until completion. and this one is scrub and deep-scrub only — it does not do anything for other commands. periodt. 🔥

🌟 AVAILABILITY 🌟

ceph is part of Ceph, which is a massively scalable, open-source, distributed storage system. if you want the whole lore, the Ceph documentation lives at <https://docs.ceph.com> 🦋

👀 SEE ALSO 👀

  • ceph-mon(8)
  • ceph-osd(8)
  • ceph-mds(8)

2010-2014, Inktank Storage, Inc. and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0) 💜

and that's the whole page bestie!! 🎀 this part was from dev and dated Oct 06, 2026, page CEPH(8) — go forth and slay your cluster, you absolute main character 👑✨

📦 where this page comes from