ioctl_fideduperange(2)

💖 ioctl_fideduperange — the manual page, but make it a group chat 💖

okay bestie so you found yourself in the System Calls Manual section, which is honestly where all the main character energy lives. this one is ioctl_fideduperange(2) and it's lowkey one of the coolest little tricks in the kernel. sit down, hydrate, let's go 🫶

---

✨ NAME ✨

ioctl_fideduperange — share some the data of one file with another file

(yes the original says "share some the data", we are keeping it, it's iconic and I refuse to edit it 💅)

---

📚 LIBRARY 📚

Standard C library (libc, -lc) — the usual, no surprises, she's standard for a reason 💖

---

🎀 SYNOPSIS 🎀

#include <linux/fs.h>      /* Definition of FIDEDUPERANGE and
                              FILE_DEDUPE_* constants*/
#include <sys/ioctl.h>

int ioctl(int src_fd, FIDEDUPERANGE, struct file_dedupe_range *arg);

---

🌸 DESCRIPTION 🌸

so like. imagine two files. imagine they're besties. imagine they have literally the exact same data. if your filesystem is the kind that can let multiple files share physical storage, this ioctl(2) operation lets you make some of the data in the src_fd file show up in the dest_fd file by sharing the underlying storage — but ONLY if the file data is identical. that's what "deduplication" means, bestie, and it is so slay 🌈

both files have to live inside the same filesystem. no long-distance relationships here, they need to be in the same house 💀

why do we care? because it reduces storage consumption — the filesystem gets to store one shared copy of the data instead of two. that's the whole point, that's the vibe, that's the assignment 👑

and here's the part that makes it safe: if a write ever happens to a shared region, the filesystem must make sure those changes stay private to the file that's being written to. this behaviour is commonly called "copy on write". so everyone gets to share until someone needs to change something, and then they get their own copy. it's giving "I love you but I also need boundaries" 💅

This ioctl performs the "compare and share if identical" operation on up to src_length bytes from file descriptor src_fd at offset src_offset.

This information is conveyed in a structure of the following form:

struct file_dedupe_range {
    __u64 src_offset;
    __u64 src_length;
    __u16 dest_count;
    __u16 reserved1;
    __u32 reserved2;
    struct file_dedupe_range_info info[0];
};

Deduplication is atomic with regards to concurrent writes, so no locks need to be taken to obtain a consistent deduplicated copy. no locks!! she just handles it!! obsessed 😭

The fields reserved1 and reserved2 must be zero.

Destinations for the deduplication operation are conveyed in the array at the end of the structure. The number of destinations is given in dest_count, and the destination information is conveyed in the following form:

struct file_dedupe_range_info {
    __s64 dest_fd;
    __u64 dest_offset;
    __u64 bytes_deduped;
    __s32 status;
    __u32 reserved;
};

Each deduplication operation targets src_length bytes in file descriptor dest_fd at offset dest_offset. The field reserved must be zero.

During the call, src_fd must be open for reading and dest_fd must be open for writing. 📖✍️

The combined size of the struct file_dedupe_range and the struct file_dedupe_range_info array must not exceed the system page size. (that's a hard limit, don't fight it 🫠)

The maximum size of src_length is filesystem dependent and is typically 16 MiB. This limit will be enforced silently by the filesystem — so like, she just won't tell you, she'll just do less than you asked. rude but fine.

By convention, the storage used by src_fd is mapped into dest_fd and the previous contents in dest_fd are freed. bye bye old bytes 👋

Upon successful completion of this ioctl, the number of bytes successfully deduplicated is returned in bytes_deduped and a status code for the deduplication operation is returned in status.

⚠️ okay listen carefully bestie, this part matters: If even a single byte in the range does not match, the deduplication operation request will be ignored and status set to FILE_DEDUPE_RANGE_DIFFERS. all-or-nothing, no partial credit, no crumbs. 💀

The status code is set to:

• FILE_DEDUPE_RANGE_SAME — for success 🎉
• a negative error code — in case of error 😭
• FILE_DEDUPE_RANGE_DIFFERS — if the data did not match 🙈

---

🔥 RETURN VALUE 🔥

On error, -1 is returned, and errno is set to indicate the error.

---

💅 ERRORS 💅

Possible errors include (but are not limited to) the following:

• EBADF — src_fd is not open for reading; dest_fd is not open for writing or is open for append-only writes; or the filesystem which src_fd resides on does not support deduplication. (three different ways to fumble the bag, pick your fighter 🎀)
• EINVAL — The filesystem does not support deduplicating the ranges of the given files. This error can also appear if either file descriptor represents a device, FIFO, or socket. Disk filesystems generally require the offset and length arguments to be aligned to the fundamental block size. Neither Btrfs nor XFS support overlapping deduplication ranges in the same file.
• EISDIR — One of the files is a directory and the filesystem does not support shared regions in directories.
• ENOMEM — The kernel was unable to allocate sufficient memory to perform the operation or dest_count is so large that the input argument description spans more than a single page of memory.
• EOPNOTSUPP — This can appear if the filesystem does not support deduplicating either file descriptor, or if either file descriptor refers to special inodes.
• EPERM — dest_fd is immutable. (she said no changes, and she meant it ✋)
• ETXTBSY — One of the files is a swap file. Swap files cannot share storage. periodt.
• EXDEV — dest_fd and src_fd are not on the same mounted filesystem. long distance didn't work out, sorry 💔

---

🌈 VERSIONS 🌈

Some filesystems may limit the amount of data that can be deduplicated in a single call. (see also: the 16 MiB thing up there, she's real 🫶)

---

⭐ STANDARDS ⭐

Linux. that's it, that's the standard. 🐧

---

🦋 HISTORY 🦋

Linux 4.5.

It was previously known as BTRFS_IOC_FILE_EXTENT_SAME and was private to Btrfs. glow-up era, honestly 👑

---

🍓 NOTES 🍓

Because a copy-on-write operation requires the allocation of new storage, the fallocate(2) operation may unshare shared blocks to guarantee that subsequent writes will not fail because of lack of disk space.

tl;dr: dedupe is cute until you write to it and suddenly the filesystem needs actual room. plan accordingly bestie 🙈

---

👀 SEE ALSO 👀

• ioctl(2)

---

okay that's the whole page, you ate that, go deduplicate something responsibly 💖✨

this was brought to you by Linux man-pages 6.7, dated 2024-03-03 — thanks for coming to my ted talk, mwah 🎀🫶💅