PNFS(4)

🦋 PNFS(4) 🦋

okay bestie, welcome to the pNFS page — that's the NFS Version 4.1 Parallel NFS Protocol, from the Kernel Interfaces Manual. pull up a chair, this one's a bop 💅

---

✨ NAME ✨

pNFS — NFS Version 4.1 Parallel NFS Protocol. that's the whole vibe. iconic, concise, we love her.

---

🌈 DESCRIPTION 🌈

okay so. the NFSv4.1 client and the NFSv4.1 server both ship support for the pNFS specification — go look at Network File System (NFS) Version 4 Minor Version 1 Protocol RFC 5661 if you want the receipts, no cap.

here's the slay: a pNFS service separates Read/Write operations from, like, every other NFSv4.1 operation. those leftover ones are called Metadata operations. and the Read/Write ops? they get performed directly on the Data Server (DS) where the file's data actually lives — straight up bypassing the NFS server. main character energy, honestly.

meanwhile all the other file operations happen on the NFS server, which we call the Metadata Server (MDS).

and for the NFS clients that don't support pNFS? they just do their Read/Write operations on the MDS, and the MDS acts as a proxy for whichever DS(s) are the right ones. it's giving helpful bestie 🫶

so. the NFSv4.1 protocol hands pNFS-aware clients two pieces of info that let them do Read/Write directly on the DS:

🎀 the first one: DeviceInfo

DeviceInfo is the static info that defines the DS server. the critical piece of it, for the layout types FreeBSD supports, is the IP address used to perform RPCs on the DS. it also tells you which version of NFS the DS supports, the I/O size, and other layout-specific info.

inside the DeviceInfo there's a DeviceID, and for the FreeBSD server that's unique to the DS configuration — it changes whenever the nfsd daemon gets restarted or the server gets rebooted. so don't get attached 😭

🎀 the second one: the layout

the layout is per-file, and it references the DeviceInfo it should use via that DeviceID. it covers a byte range of a file, and it's either Read or Read/Write.

for the FreeBSD server, a layout covers all bytes of a file. periodt.

a layout may be recalled by the MDS using a LayoutRecall callback. and when a client returns a layout via the LayoutReturn operation, it can signal that error(s) were encountered while doing I/O on the DS — at least for certain layout types, like the Flexible File Layout.

---

💖 the two layout types (FreeBSD client + server support both)

⭐ File Layout

described in RFC5661, and it uses the NFSv4.1 protocol to perform I/O on the DS.

it does not support client aware DS mirroring, and because of that, the FreeBSD server only provides File Layout support for non-mirrored configurations.

⭐ Flexible File Layout

this one lets you use NFSv3, NFSv4.0 or NFSv4.1 protocol to perform I/O on the DS, and it does support client aware mirroring. so obviously the FreeBSD server uses Flexible File Layout layouts for the mirrored DS configurations 💅

the FreeBSD server supports the “tightly coupled” variant, and all DSs use the NFSv4.1 protocol for I/O operations.

clients that support the Flexible File Layout will do writes and commits to all DS mirrors in the mirror set. all of them. no favorites. 🌸

---

🧚 setting up the vibe

a FreeBSD pNFS service is made of a single MDS server plus one or more DS servers — and every single one of them is a FreeBSD system.

for a non-mirrored configuration, the FreeBSD server issues File Layout layouts by default. but you can flip that default over to the Flexible File Layout by setting the sysctl(1) sysctl “vfs.nfsd.default_flexfile” to one.

Mirrored server configurations will only issue Flexible File Layouts. that's just how she is.

pNFS clients mount the MDS the exact same way they'd mount a single NFS server. easy.

---

🦋 the client side

a FreeBSD pNFS client must be running the nfscbd(8) daemon, and it must use these mount options:

“nfsv4,minorversion=1,pnfs”

that's the whole spell. don't skip a word of it ✨

---

🍓 what actually shows up on disk

when files get created, the MDS builds a file tree identical to what a single NFS server would create — except that all the regular (VREG) files will be empty.

so if you go peek at the exported tree on the MDS directly on the MDS server (not through an NFS mount), every file will be size zero. not a bug, bestie, that's the design 🫠

each of those files also gets two extended attributes in the system attribute name space:

• pnfsd.dsfile — this extended attribute stores the information that the MDS needs to find the data file on a DS(s) for this file.
• pnfsd.dsattr — this extended attribute stores the Size, AccessTime, ModifyTime and Change attributes for the file.

---

👑 the data files

for each regular (VREG) file, the MDS creates a data file on one of the DS(s) — or on N of them for the mirrored case, where N is the mirror_level — and that's where the file's data lives.

the name of this file? it's the file handle of the file on the MDS, in hexadecimal, at time of file creation. cute, right?

the data file will have the same file ownership, mode, and NFSv4 ACL (if ACLs are enabled for the file system) as the file on the MDS — so that permission checking can be done on the DS. this is what we mean by “tightly coupled” for the Flexible File Layout.

---

💜 who gets what

for pNFS aware clients, the service generates File Layout or Flexible File Layout layouts plus the associated DeviceInfo.

for non-pNFS aware NFS clients, the pNFS service just looks like a normal NFS service. the MDS will perform the I/O operations on the appropriate DS(s), acting as a proxy for that non-pNFS aware client. and that goes for NFSv3 and NFSv4.0 mounts too, since those are always non-pNFS aware. no exceptions.

---

🌸 assigning DSs

you can assign a DS to a specific MDS exported file system, so that it will store data for files on that MDS exported file system.

and if a DS is not assigned to an MDS exported file system, it will store data for files on all exported file systems on the MDS. generous of her 💖

---

🔥 mirroring, and the whole "things break" situation

if mirroring is enabled, the pNFS service will keep functioning even when DS(s) have failed — so long as there is at least one DS still operational that stores data for files on all of the MDS exported file systems.

after a disabled mirrored DS gets repaired, it's possible to recover that DS as a mirror while the pNFS service continues to function. glow-up in real time ✨

see pnfsserver(4) for info on how to set up a FreeBSD pNFS service. go read it, bestie, it's the setup guide.

---

🎀 SEE ALSO 🎀

• nfsv4(4)
• pnfsserver(4)
• exports(5)
• fstab(5)
• rc.conf(5)
• nfscbd(8)
• nfsd(8)
• nfsuserd(8)
• pnfsdscopymr(8)
• pnfsdsfile(8)
• pnfsdskill(8)

---

💀 BUGS 💀

okay so the Linux situation. let's talk about it.

• Linux kernel versions prior to 4.12 only support NFSv3 DSs in their client, and they will do all I/O through the MDS. all of it.
• for Linux 4.12 kernels, support for NFSv4.1 DSs was added — but the author has literally seen Linux client crashes when testing this client. 😭
• for Linux 4.17-rc2 kernels, the author has not seen client crashes during testing, but it only supports the “loosely coupled” variant.

to make it work correctly when mounting the FreeBSD server, you must do one of these two things:

1. patch the Flexible File Layout client driver with a patch like:

http://people.freebsd.org/~rmacklem/flexfile.patch

2. or set the sysctl “vfs.nfsd.flexlinuxhack” to one, so that it works around the Linux client driver's limitations.

and one more, because we keep it real:

since the MDS cannot be mirrored, it is a single point of failure — just as a non pNFS server is. no cap, plan accordingly. 🫶

---

and that's the page!! this one came out of the Debian crew, dated August 5, 2018, for the PNFS(4) manual. ty for reading my little essay, go mount something responsibly 💖✨🎀🦋