GANESHA-RADOS-GRACE(8)

📦
version 9.16 · 11 versions
, in fedora 44

💖 GANESHA-RADOS-GRACE 💖

okay bestie, so this is the ganesha-rados-grace page from section 8 of the manual (that's the sysadmin-y one, iykyk), and it's part of NFS-Ganesha. it's giving "let me poke the shared grace database directly" energy and honestly? iconic. 🎀

✨ NAME ✨

ganesha-rados-grace — manipulate the shared grace management database 💅

🌸 SYNOPSIS 🌸

ganesha-rados-grace [ --cephconf /path/to/ceph.conf ] [--ns namespace] [ --oid obj_id ] [ --pool pool_id ] [ --userid cephuser ] dump|add|start|join|lift|remove|enforce|noenforce|member [ nodeid ... ]

🦋 DESCRIPTION 🦋

so basically, this tool lets the admin get all up in the database that the rados_cluster recovery backend uses. like, directly. no cap. the cluster nodes use that database to say "hey this is my current state rn" so they can coordinate a whole cluster-wide grace period together. it's a group chat but the messages are flags. 💬

the first argument has to be a command to run against the database. anything left over after that? those are the nodeids of the nodes in the cluster that you want to act on.

most commands will just straight up fail if the grace database isn't there. 💀 the one exception is the add command, which will create the pool, database and namespace if they don't already exist. she's self-sufficient like that.

not me... 🫠 heads up: this program does not consult ganesha.conf. so if you're using non-default values for ceph_conf, userid, grace_oid, namespace or pool in your RADOS_KV config block, you gotta pass them in through the command-line options instead. no shortcuts, bestie.

🎀 OPTIONS 🎀

  • --cephconf — pick which ceph.conf config should be used. default is to just use the normal search path to find one. 🔍
  • --ns — set the RADOS namespace to use within the pool. default is NULL.
  • --oid — set the object id of the grace database RADOS object. default is "grace".
  • --pool — set the RADOS poolid where the grace database object lives. default is "nfs-ganesha".
  • --userid — set the cephx user ID to use when contacting the cluster. default is NULL.

👑 COMMANDS 👑

dump

dump the current status of the grace period database to stdout. this shows you the current and recovery epoch serial numbers, plus a list of hosts currently in the cluster and what flags they've set in their individual records. receipts, basically. 🧾

add

add the specified hosts to the cluster. this must be done before those hosts can take part in the cluster, fr fr. attempts to modify the database by cluster hosts that haven't been added yet will generally fail. 😭 new hosts get added with the enforcing flag already set, since they can't hand out new state until their own grace period has been lifted.

start

start a new grace period. this begins a new grace period in the cluster if one isn't already active, and sets the record for the listed cluster hosts as both needing a grace period and enforcing the grace period. if a grace period is already active, then this is equivalent to join.

join

try to join an existing grace period. works like start, but only if there's already an existing grace period in force. she only pulls up to the party if the party's already happening. 🎉

lift

attempt to lift the current grace period. this clears the need grace flags for the listed hosts. if there are no more hosts in the cluster that require a grace period, then it gets fully lifted and the cluster transitions to normal operations. moving on up. ⬆️

remove

remove one or more existing hosts from the cluster. this removes the listed hosts from the grace database, and it might lift the current grace period if there are no more hosts that need one.

enforce

set the flag for the given hosts that says they're currently enforcing the grace period — meaning they're not allowing clients to acquire new state. 🚫

noenforce

clear the enforcing flag for the given hosts, meaning those hosts are now letting clients acquire new state. 🟢

member

test whether the given hosts are members of the cluster. returns an error if any of the hosts aren't present in the grace db omap.

🌈 FLAGS 🌈

when the dump command is issued, ganesha-rados-grace will display a list of all the nodes in the grace database, plus any flags they've set. the flags are:

  • E (Enforcing) — the node is currently enforcing the grace period by rejecting requests from clients to acquire new state.
  • N (Need Grace) — the node currently requires a grace period. generally this means the node has clients that need to perform recovery.

⭐ NODEID ASSIGNMENT ⭐

every running ganesha daemon needs a nodeid string that's unique within the cluster. it can literally be any value, because ganesha treats it as an opaque string. by default, the ganesha daemon will use the hostname of the node where it's running. 🏠

that might not be suitable when you're running under certain HA clustering infrastructure, so it's generally recommended to manually assign nodeid values to the hosts in the RADOS_KV config block of ganesha.conf. do it yourself, do it right. ✨

💜 GANESHA CONFIGURATION 💜

the ganesha daemon needs to be configured with the RecoveryBackend set to rados_cluster. and if you use a non-default pool, namespace or oid, nodeid then those values need to be set accordingly in the RADOS_KV config block too.

🌸 STARTING A NEW CLUSTER 🌸

first, add the given cluster nodes to the grace database. assuming the nodes in our cluster have nodeids ganesha-1 through ganesha-3:

ganesha-rados-grace add ganesha-1 ganesha-2 ganesha-3

once that's done, you can start the daemons on each host and they'll coordinate to start and lift the grace periods as-needed. teamwork makes the dream work. 🫶

🦋 ADDING NODES TO A RUNNING CLUSTER 🦋

after that, new nodes can be added to the cluster as needed using the add command:

ganesha-rados-grace add ganesha-4

once the node has been added, ganesha.nfsd can be started. it'll then request a new grace period as-needed.

🎀 REMOVING A NODE FROM THE CLUSTER 🎀

to remove a node from the cluster, first unmount any clients that have that node mounted (possibly moving them to other servers). then run the remove command with the nodeids to be removed from the cluster. for example:

ganesha-rados-grace remove ganesha-4

this removes ganesha-4's record from the database, and possibly lifts the current grace period if one is active and it was the last one to need it. bye bestie 👋💖

okay that's the whole page, and that's literally the vibe. 🎀✨ this version comes from the NFS-Ganesha source, dated Aug 11, 2026, and it's the GANESHA-RADOS-GRACE(8) page — go forth and grace-database responsibly, bestie. 💅🦋

📦 where this page comes from