MDMON(8)

💖 MDMON(8) — the manual page, but make it a group chat 💖

okay bestie, so this is mdmon, from the System Manager's Manual, and she's lowkey the most underrated girlboss in your whole RAID setup. let's get into it. 🫶

---

✨ NAME ✨

mdmon — monitor MD external metadata arrays 👀

---

✨ SYNOPSIS ✨

mdmon [--all] [--takeover] [--foreground] CONTAINER

that's it. that's the whole vibe. short, sweet, slays. 💅

---

✨ OVERVIEW ✨

so the 2.6.27 kernel brought us the ability to support external metadata arrays, and honestly? iconic behavior. 🌈

external metadata means user space handles all the updates to the metadata. like, all of them. the kernel's whole job is just to ping user space when a "metadata event" happens — think disk failures, clean-to-dirty transitions, that kind of drama. and in the important cases, the kernel literally waits for user space to take action on those notifications. no cap. it's giving "i'm not moving until you handle this." 😭

---

✨ DESCRIPTION ✨

💫 Metadata updates: 💫

to service metadata update requests, a daemon called mdmon gets introduced to the group chat. her job? polling the sysfs namespace, looking for changes in array_state, sync_action, and per disk state attributes. when she spots a change, she calls a per metadata type handler to go make modifications to the metadata. here's the full rundown of what gets done:

• array_state - inactive — clear the dirty bit for the volume and let the array be stopped 🛑
• array_state - write pending — set the dirty bit for the array and then set array_state to active. writes are blocked until userspace writes active. (patience, bestie 💅)
• array_state - active-idle — the safe mode timer has expired, so set array state to clean to block writes to the array ⏰
• array_state - clean — clear the dirty bit for the volume 🧼
• array_state - read-only — this is the initial state that all arrays start at. mdmon takes one of three actions:
  1. transition the array to read-auto keeping the dirty bit clear if the metadata handler determines that the array does not need resyncing or other modification ✨
  2. transition the array to active if the metadata handler determines a resync or some other manipulation is necessary 🔥
  3. leave the array read-only if the volume is marked to not be monitored; for example, the metadata version has been set to "external:-dev/md127" instead of "external:/dev/md127" 🚫
• sync_action - resync-to-idle — notify the metadata handler that a resync may have completed. if a resync process is idled before it completes, this event allows the metadata handler to checkpoint resync ✅
• sync_action - recover-to-idle — a spare may have completed rebuilding, so tell the metadata handler about the state of each disk. this is the metadata handler's opportunity to clear any "out-of-sync" bits and clear the volume's degraded status. if a recovery process is idled before it completes, this event allows the metadata handler to checkpoint recovery 🩹
• <disk>/state - faulty — a disk failure kicks off a whole series of events. first, notify the metadata handler that a disk has failed, and then notify the kernel that it can unblock writes that were dependent on this disk. after unblocking the kernel, this disk is set to be removed+ from the member array. finally the disk is marked failed in all other member arrays in the container. 💀

+ Note — this behavior differs slightly from native MD arrays where removal is reserved for a mdadm --remove event. in the external metadata case the container holds the final reference on a block device and a mdadm --remove <container> <victim> call is still required. so yeah, you still gotta do the thing. 🙈

💫 Containers: 💫

external metadata formats, like DDF, differ from the native MD metadata formats in that they define a set of disks and a series of sub-arrays within those disks. MD metadata, in comparison, defines a 1:1 relationship between a set of block devices and a RAID array.

for example, to create 2 arrays at different RAID levels on a single set of disks, MD metadata requires the disks be partitioned and then each array can be created with a subset of those partitions. the supported external formats perform this disk carving internally. she does it herself. main character energy. 👑

container devices simply hold references to all member disks and allow tools like mdmon to determine which active arrays belong to which container. some array management commands like disk removal and disk add are now only valid at the container level. attempts to perform these actions on member arrays are blocked with error messages like:

"mdadm: Cannot remove disks from a ´member´ array, perform this
operation on the parent container"

containers are identified in /proc/mdstat with a metadata version string "external:<metadata name>". member devices are identified by "external:/<container device>/<member index>", or "external:-<container device>/<member index>" if the array is to remain readonly. 🎀

---

✨ OPTIONS ✨

• CONTAINER — the container device to monitor. it can be a full path like /dev/md/container, or a simple md device name like md127. 📁
• --foreground — normally, mdmon will fork and continue in the background. adding this option will skip that step and run mdmon in the foreground. so she's right there with you, no disappearing acts. 🫶
• --takeover — this instructs mdmon to replace any active mdmon which is currently monitoring the array. this is primarily used late in the boot process to replace any mdmon which was started from an initramfs before the root filesystem was mounted. this avoids holding a reference on that initramfs indefinitely and ensures that the pid and sock files used to communicate with mdmon are in a standard place. out with the old, in with the new 💅
• --all — this tells mdmon to find any active containers and start monitoring each of them if appropriate. this is normally used with --takeover late in the boot sequence. a separate mdmon process is started for each container as the --all argument is over-written with the name of the container. to allow for containers with names longer than 5 characters, this argument can be arbitrarily extended, e.g. to --all-active-arrays. 🌈

Note that mdmon is automatically started by mdadm when needed and so does not need to be considered when working with RAID arrays. the only times it is run other than by mdadm is when the boot scripts need to restart it after mounting the new root filesystem. so like, she's usually already there. 😌

---

✨ START UP AND SHUTDOWN ✨

as mdmon needs to be running whenever any filesystem on the monitored device is mounted, there are special considerations when the root filesystem is mounted from an mdmon monitored device. note that in general mdmon is needed even if the filesystem is mounted read-only as some filesystems can still write to the device in those circumstances — for example, to replay a journal after an unclean shutdown. 😭

when the array is assembled by the initramfs code, mdadm will automatically start mdmon as required. this means that mdmon must be installed on the initramfs and there must be a writable filesystem (typically tmpfs) in which mdmon can create a .pid and .sock file. the particular filesystem to use is given to mdmon at compile time and defaults to /run/mdadm. 📂

this filesystem must persist through to shutdown time. no cap, it has to stick around. ⏳

after the final root filesystem has be instantiated (usually with pivot_root) mdmon should be run with --all --takeover so that the mdmon running from the initramfs can be replaced with one running in the main root, and so the memory used by the initramfs can be released. bye babe, thanks for the memories 🦋

at shutdown time, mdmon should not be killed along with other processes. also, as it holds a file (socket actually) open in /dev (by default) it will not be possible to unmount /dev if it is a separate filesystem. so like… don't. 🙈

---

✨ EXAMPLES ✨

mdmon --all-active-arrays --takeover

any mdmon which is currently running is killed and a new instance is started. this should be run during in the boot sequence if an initramfs was used, so that any mdmon running from the initramfs will not hold the initramfs active. 🔥

---

✨ SEE ALSO ✨

mdadm(8), md(4).

---

and that's the whole page bestie, you ate that 💅 this was version v4.6 of the manual, served up by the System Manager's Manual. go forth and monitor those external metadata arrays, you absolute icon. 👑✨💖