OSDMAPTOOL(8)

📦
version 20.2.4 · 20 versions
, in ubuntu stonking

💖 OSDMAPTOOL(8) 💖

ok so bestie, welcome to osdmaptool — it lives in section 8 of the manual (the OSDMAPTOOL(8) of it all) and it's your little Ceph OSD cluster map bestie. sit down, grab a boba, we're doing this together 🧋

✨ NAME ✨

osdmaptool — the ceph osd cluster map manipulation tool. that's it, that's the name, she's simple and she's iconic 💅

🌈 SYNOPSIS 🌈

osdmaptool mapfilename [--print] [--createsimple numosd
[--pgbits bitsperosd ] ] [--clobber]
osdmaptool mapfilename [--import-crush crushmap]
osdmaptool mapfilename [--export-crush crushmap]
osdmaptool mapfilename [--upmap file] [--upmap-max max-optimizations]
[--upmap-deviation max-deviation] [--upmap-pool poolname]
[--save] [--upmap-active]
osdmaptool mapfilename [--upmap-cleanup] [--upmap file]

like, pick your sad little line of flags and go, bestie 🫶

🦋 DESCRIPTION 🦋

so osdmaptool is the utility that lets you create, view, and manipulate OSD cluster maps from the Ceph distributed storage system. Ceph! distributed storage! we love her 💜

and notably — like, the moment you've all been waiting for — it lets you extract the embedded CRUSH map or import a new CRUSH map. both directions, no cap ✨

it can also simulate the upmap balancer mode, so you can get a whole sense of what is needed to balance your PGs. it's giving "preview the fit before you buy it" energy, honestly kinda thoughtful of her 😭💖

🎀 OPTIONS 🎀

okay this is the part where we go one by one. all of them. yes all of them, bestie, we don't skip 💅

  • --print — will simply make the tool print a plaintext dump of the map, after any modifications are made. simple! iconic! the map, printed, in plain text, post-modifications 👑
  • --dump <format> — displays the map in plain text when <format> is 'plain', 'json' if specified format is not supported. this is an alternative to the print option ✨ (so like, 'plain' gives you plain text, anything else unsupported gets you 'json' — that's literally what it says, i don't make the rules)
  • --clobber — will allow osdmaptool to overwrite mapfilename if changes are made. ⚠️ emphasis on overwrite, bestie. she will not ask twice.
  • --import-crush mapfile — will load the CRUSH map from mapfile and embed it in the OSD map. bringing the crush map home 🏠💖
  • --export-crush mapfile — will extract the CRUSH map from the OSD map and write it to mapfile. the opposite! we love a symmetrical queen 🦋
  • --createsimple numosd [--pg-bits bitsperosd] [--pgp-bits bits] — will create a relatively generic OSD map with the numosd devices. if --pg-bits is specified, the initial placement group counts will be set with bitsperosd bits per OSD. that is, the pg_num map attribute will be set to numosd shifted by bitsperosd. and if --pgp-bits is specified, then the pgp_num map attribute will be set to numosd shifted by bits. generics, but make it ✨ math ✨
  • --create-from-conf — creates an osd map with default configurations. no thoughts, just defaults 🌸
  • --test-map-pgs [--pool poolid] [--range-first <first> --range-last <last>] — will print out the mappings from placement groups to OSDs. if range is specified, then it iterates from first to last in the directory specified by argument to osdmaptool. eg: osdmaptool --test-map-pgs --range-first 0 --range-last 2 osdmap_dir. this will iterate through the files named 0,1,2 in osdmap_dir. 🗺️
  • --test-map-pgs-dump [--pool poolid] [--range-first <first> --range-last <last>] — will print out the summary of all placement groups and the mappings from them to the mapped OSDs. if range is specified, then it iterates from first to last in the directory specified by argument to osdmaptool. eg: osdmaptool --test-map-pgs-dump --range-first 0 --range-last 2 osdmap_dir. this will iterate through the files named 0,1,2 in osdmap_dir. summary + mappings, the full meal 🍓
  • --test-map-pgs-dump-all [--pool poolid] [--range-first <first> --range-last <last>] — will print out the summary of all placement groups and the mappings from them to all the OSDs. if range is specified, then it iterates from first to last in the directory specified by argument to osdmaptool. eg: osdmaptool --test-map-pgs-dump-all --range-first 0 --range-last 2 osdmap_dir. this will iterate through the files named 0,1,2 in osdmap_dir. literally the same as above but with ALL the OSDs, she said "no filter" 💀✨
  • --test-random — does a random mapping of placement groups to the OSDs. chaos mode, lowkey 🎲
  • --test-map-pg <pgid> — map a particular placement group (specified by pgid) to the OSDs. one specific girl, one specific mapping 🎯
  • --test-map-object <objectname> [--pool <poolid>] — map a particular placement group (specified by objectname) to the OSDs. yes, by object name. and you can pin the pool with --pool <poolid> 🧚
  • --test-crush [--range-first <first> --range-last <last>] — map placement groups to acting OSDs. if range is specified, then it iterates from first to last in the directory specified by argument to osdmaptool. eg: osdmaptool --test-crush --range-first 0 --range-last 2 osdmap_dir. this will iterate through the files named 0,1,2 in osdmap_dir. acting OSDs specifically, iykyk 🔥
  • --mark-up-in — mark osds up and in (but do not persist). ⚠️ but do not persist, bestie. it's a vibe, not a commitment.
  • --mark-out — mark an osd as out (but do not persist). again. does not persist. 🙈
  • --mark-up <osdid> — mark an osd as up (but do not persist). you know the drill by now 💜
  • --mark-in <osdid> — mark an osd as in (but do not persist). four of four, all non-persistent, all iconic 👑
  • --tree — displays a hierarchical tree of the map. 🌳 hierarchy! structure! mother nature but for OSDs
  • --clear-temp — clears pg_temp and primary_temp variables. swept. gone. fresh start ✨
  • --clean-temps — clean pg_temps. yes it's a different flag from the one above, no i will not be merging them 💅
  • --health — dump health checks. the wellness check, but for your cluster 🩺
  • --with-default-pool — include default pool when creating map. bring the default pool along, she gets lonely 🌸
  • --upmap-cleanup <file> — clean up pg_upmap[_items] entries, writing commands to <file> [default: - for stdout]. default stdout, so if you don't give her a file she just yells into your terminal 📢
  • --upmap <file> — calculate pg upmap entries to balance pg layout, writing commands to <file> [default: - for stdout]. the balancing one! the star of the show ⭐
  • --upmap-max <max-optimizations> — set max upmap entries to calculate [default: 10]. ten. that's the default. don't come for me 🎀
  • --upmap-deviation <max-deviation> — max deviation from target [default: 5]. five! target! don't drift too far bestie 🦋
  • --upmap-pool <poolname> — restrict upmap balancing to 1 pool or the option can be repeated for multiple pools. repeatable! spam it if you have many pools 💖
  • --upmap-active — Act like an active balancer, keep applying changes until balanced. she doesn't stop. she can't stop. balance is the goal and she will get there 🔥
  • --adjust-crush-weight <osdid:weight>[,<osdid:weight>,<...>] — Change CRUSH weight of <osdid>. comma-separated list of osdid:weight pairs, go wild ⚖️
  • --save — write modified osdmap with upmap or crush-adjust changes. 💾 actually persist the thing (unlike those mark flags, who were just messing around)
  • --read <file> — calculate pg upmap entries to balance pg primaries. different balancer, different goal, same energy ✨
  • --read-pool <poolname> — specify which pool the read balancer should adjust. pair it with --read, she doesn't work alone 🫶
  • --vstart — prefix upmap and read output with './bin/'. handy, cute, specific 💜

✨ EXAMPLE ✨

Like okay bestie, let's actually do the thing instead of just reading about it 💖

To create a simple map with 16 devices:

osdmaptool --createsimple 16 osdmap --clobber

To view the result:

osdmaptool --print osdmap

To view the mappings of placement groups for pool 1:

osdmaptool osdmap --test-map-pgs-dump --pool 1

and then it just. spills all this at you:

pool 1 pg_num 8
1.0     [0,2,1] 0
1.1     [2,0,1] 2
1.2     [0,1,2] 0
1.3     [2,0,1] 2
1.4     [0,2,1] 0
1.5     [0,2,1] 0
1.6     [0,1,2] 0
1.7     [1,0,2] 1
#osd    count   first   primary c wt    wt
osd.0   8       5       5       1       1
osd.1   8       1       1       1       1
osd.2   8       2       2       1       1
 in 3
 avg 8 stddev 0 (0x) (expected 2.3094 0.288675x))
 min osd.0 8
 max osd.0 8
size 0  0
size 1  0
size 2  0
size 3  8

In which,

  1. pool 1 has 8 placement groups. And two tables follow:
  2. A table for placement groups. Each row presents a placement group. With columns of:
    • placement group id,
    • acting set, and
    • primary OSD.
  3. A table for all OSDs. Each row presents an OSD. With columns of:
    • count of placement groups being mapped to this OSD,
    • count of placement groups where this OSD is the first one in their acting sets,
    • count of placement groups where this OSD is the primary of them,
    • the CRUSH weight of this OSD, and
    • the weight of this OSD.
  4. Looking at the number of placement groups held by 3 OSDs. We have
    • average, stddev, stddev/average, expected stddev, expected stddev / average
    • min and max
  5. The number of placement groups mapping to n OSDs. In this case, all 8 placement groups are mapping to 3 different OSDs.

Okay now, in a less-balanced cluster (we love a messy queen, but not for our data 😭), we could have following output for the statistics of placement group distribution, whose standard deviation is 1.41421:

#osd    count   first   primary c wt    wt
osd.0   8       5       5       1       1
osd.1   8       1       1       1       1
osd.2   8       2       2       1       1

#osd    count   first    primary c wt    wt
osd.0   33      9        9       0.0145874     1
osd.1   34      14       14      0.0145874     1
osd.2   31      7        7       0.0145874     1
osd.3   31      13       13      0.0145874     1
osd.4   30      14       14      0.0145874     1
osd.5   33      7        7       0.0145874     1
 in 6
 avg 32 stddev 1.41421 (0.0441942x) (expected 5.16398 0.161374x))
 min osd.4 30
 max osd.1 34
size 00
size 10
size 20
size 364

To simulate the active balancer in upmap mode:

        osdmaptool --upmap upmaps.out --upmap-active --upmap-deviation 6 --upmap-max 11 osdmap

and it goes a lil something like:

osdmaptool: osdmap file 'osdmap'
writing upmap command output to: upmaps.out
checking for upmap cleanups
upmap, max-count 11, max deviation 6
pools movies photos metadata data
prepared 11/11 changes
Time elapsed 0.00310404 secs
pools movies photos metadata data
prepared 11/11 changes
Time elapsed 0.00283402 secs
pools data metadata movies photos
prepared 11/11 changes
Time elapsed 0.003122 secs
pools photos metadata data movies
prepared 11/11 changes
Time elapsed 0.00324372 secs
pools movies metadata data photos
prepared 1/11 changes
Time elapsed 0.00222609 secs
pools data movies photos metadata
prepared 0/11 changes
Time elapsed 0.00209916 secs
Unable to find further optimization, or distribution is already perfect
osd.0 pgs 41
osd.1 pgs 42
osd.2 pgs 42
osd.3 pgs 41
osd.4 pgs 46
osd.5 pgs 39
osd.6 pgs 39
osd.7 pgs 43
osd.8 pgs 41
osd.9 pgs 46
osd.10 pgs 46
osd.11 pgs 46
osd.12 pgs 46
osd.13 pgs 41
osd.14 pgs 40
osd.15 pgs 40
osd.16 pgs 39
osd.17 pgs 46
osd.18 pgs 46
osd.19 pgs 39
osd.20 pgs 42
Total time elapsed 0.0167765 secs, 5 rounds

To simulate the active balancer in read mode, first make sure capacity is balanced by running the balancer in upmap mode. Then, balance the reads on a replicated pool with:

        osdmaptool osdmap --read read.out --read-pool <pool name>

and boom, you get the before/after glow-up 💅:

./bin/osdmaptool: osdmap file 'om'
writing upmap command output to: read.out

---------- BEFORE ------------
osd.0 | primary affinity: 1 | number of prims: 3
osd.1 | primary affinity: 1 | number of prims: 10
osd.2 | primary affinity: 1 | number of prims: 3

read_balance_score of 'cephfs.a.meta': 1.88

---------- AFTER ------------
osd.0 | primary affinity: 1 | number of prims: 5
osd.1 | primary affinity: 1 | number of prims: 5
osd.2 | primary affinity: 1 | number of prims: 6

read_balance_score of 'cephfs.a.meta': 1.13

num changes: 5

🌈 AVAILABILITY 🌈

osdmaptool is part of Ceph, a massively scalable, open-source, distributed storage system. Please refer to the Ceph documentation at https://docs.ceph.com for more information. no cap, that's where the ✨lore✨ lives, bestie 🦋

👀 SEE ALSO 👀

  • ceph <>(8)
  • crushtool <>(8)

2010-2014, Inktank Storage, Inc. and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0). so like, share alike bestie, it's the law 💖

And that's a wrap pookie!! 🫶 This lil cutie right here is dev flavored and it was stamped Oct 06, 2026 — OSDMAPTOOL(8), periodt. Go balance some OSDs, you absolute icon 👑✨

📦 where this page comes from