SENDFILE(2)

💅 SENDFILE(2) 💅 — sending a whole file to a socket, the iconic way

okay bestie, sit down, hydrate, because today we're talking about sendfile(2) — the system call that yeets a regular file straight out a socket and it is so much cooler than it sounds. this is the System Calls Manual entry, she's serving Debian energy, and I'm gonna walk you through every single crumb. no skips, no skimming, we are doing this thoroughly. 🎀

---

✨ NAME ✨

sendfile — send a file to a socket 🦋

---

📚 LIBRARY 📚

Standard C Library (libc, -lc) 💖

---

🧚 SYNOPSIS 🧚

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/uio.h>

int
sendfile(int fd, int s, off_t offset, size_t nbytes,
    struct sf_hdtr *hdtr, off_t *sbytes, int flags);

---

🌸 DESCRIPTION 🌸

so the sendfile() system call takes a regular file or a shared memory object — whatever descriptor you handed it in fd — and sends it out a stream socket, which is the descriptor s. that's it. that's the whole vibe. file in, socket out, no cap. ✨

the offset argument is literally just where in the file you wanna start. and here's a lil gotcha bestie: if offset falls past the end of the file, the system does not yell at you — it returns success and reports 0 bytes sent, exactly like the return values section says down below. iconic behavior honestly. 💅

the nbytes argument says how many bytes of the file should go out, and 0 has a special meaning: send until the end of file has been reached. so zero means "the rest of it, bestie, all of it." 🔥

you can also send an optional header and/or trailer before and after the file data, and you do that by passing a pointer to a struct sf_hdtr, which looks like this:

struct sf_hdtr {
        struct iovec *headers;  /* pointer to header iovecs */
        int hdr_cnt;            /* number of header iovecs */
        struct iovec *trailers; /* pointer to trailer iovecs */
        int trl_cnt;            /* number of trailer iovecs */
};

the headers and trailers pointers, if they're non-NULL, point at arrays of struct iovec structures. if you need the lore on what an iovec even is, go read the writev() system call, she explains it. and the number of iovecs in those arrays? that's hdr_cnt and trl_cnt, respectively. 🫶

if sbytes is non-NULL, the system writes the total number of bytes sent on the socket into the variable it points to. cute little receipt, we love that.

now. the least significant 16 bits of the flags argument are a bitmap of these values:

• SF_NODISKIO — this flag makes sendfile return EBUSY instead of blocking when it runs into a busy page. this is a rare situation and it can happen if some other process is currently working with the same region of the file. the advice is to retry the operation after a short period. also important lore: in older FreeBSD versions, SF_NODISKIO had a slightly different notion — the flag prevented sendfile from running I/O operations if an invalid (not cached) page was encountered, so it avoided blocking on I/O. starting with FreeBSD 11, sendfile sending files off the ffs(7) filesystem does not block on I/O (peep “IMPLEMENTATION NOTES”), so that condition no longer applies. however, it is safe if an application utilizes SF_NODISKIO and on EBUSY performs the same action as it did in older FreeBSD versions, e.g., aio_read(2), read(2) or sendfile in a different context. 😌
• SF_NOCACHE — the data sent to socket will not be cached by the virtual memory system, and will be freed directly to the pool of free pages. bye bye data, you served. 👋
• SF_SYNC — sendfile sleeps until the network stack no longer references the VM pages of the file, making subsequent modifications to it safe. please note that this is not a guarantee that the data has actually been sent. do not conflate those two things bestie, I'm begging. 😭
• SF_USER_READAHEAD — sendfile has some internal heuristics to do readahead when sending data. this flag forces sendfile to override any heuristically calculated readahead and use exactly the application specified readahead. see “SETTING READAHEAD” for more details on readahead. 👑

when you're using a socket marked for non-blocking I/O, sendfile() may send fewer bytes than requested. in this case, the number of bytes successfully written is returned in *sbytes (if specified), and the error EAGAIN is returned. so check your sbytes, bestie, don't assume. 💀

---

📖 SETTING READAHEAD 📖

sendfile uses internal heuristics based on request size and file system layout to do readahead. on top of that, an application may request extra readahead. the most significant 16 bits of flags specify the amount of pages that sendfile may read ahead when reading the file. there's a macro, SF_FLAGS(), provided specifically to combine readahead amount and flags. here's an example showing a readahead of 16 pages plus the SF_NOCACHE flag:

SF_FLAGS(16, SF_NOCACHE)

sendfile will use either the application specified readahead or the internally calculated one — whichever is bigger. and if you set the SF_USER_READAHEAD flag, that turns off any heuristics and sets the maximum possible readahead length to the number of pages specified via flags. no guessing, just your number. periodt. 💅

---

🛠️ IMPLEMENTATION NOTES 🛠️

the FreeBSD implementation of sendfile() does not block on disk I/O when it sends a file off the ffs(7) filesystem. the syscall returns success before the actual I/O completes, and the data gets put into the socket later, unattended, living its little unsupervised life. however, the order of data in the socket is preserved, so it is safe to do further writes to the socket. so like, chaotic but trustworthy. 🦋

the FreeBSD implementation of sendfile() is "zero-copy", meaning it's been optimized so that copying of the file data is avoided. she's efficient, she's low-overhead, she's the moment. 👑

---

🎛️ TUNING 🎛️

on some architectures, this system call internally uses a special sendfile() buffer (struct sf_buf) to handle sending file data to the client. so here's the tea:

• if the sending socket is blocking, and there are not enough sendfile() buffers available, sendfile() will block and report a state of "sfbufa".
• if the sending socket is non-blocking and there are not enough sendfile() buffers available, the call will block and wait for the necessary buffers to become available before finishing the call.

the number of sf_buf's allocated should be proportional to the number of nmbclusters used to send data to a client via sendfile(). tune accordingly to avoid blocking! busy installations that make extensive use of sendfile() may want to increase these values to be inline with their kern.ipc.nmbclusters (see tuning(7) for details). 🫶

the number of sendfile() buffers available is determined at boot time by either the kern.ipc.nsfbufs loader.conf(5) variable or the NSFBUFS kernel configuration tunable. the number of sendfile() buffers scales with kern.maxusers. the kern.ipc.nsfbufsused and kern.ipc.nsfbufspeak read-only sysctl(8) variables show current and peak sendfile() buffers usage respectively — and yes, these values may also be viewed through netstat -m. 📊

and if a value of zero is reported for kern.ipc.nsfbufs, that means your architecture does not need to use sendfile() buffers, because their task can be efficiently performed by the generic virtual memory structures. she's built different, she doesn't need the extra furniture. 💅

---

🏆 RETURN VALUES 🏆

the sendfile() function returns the value 0 if successful; otherwise the value -1 is returned and the global variable errno is set to indicate the error. classic. 👑

---

💀 ERRORS 💀

• [EAGAIN] — the socket is marked for non-blocking I/O and not all data was sent due to the socket buffer being filled. if specified, the number of bytes successfully sent will be returned in *sbytes. partial slay. 🙈
• [EBADF] — the fd argument is not a valid file descriptor.
• [EBADF] — the s argument is not a valid socket descriptor.
• [EBUSY] — a busy page was encountered and SF_NODISKIO had been specified. partial data may have been sent.
• [EFAULT] — an invalid address was specified for an argument.
• [EINTR] — a signal interrupted sendfile() before it could be completed. if specified, the number of bytes successfully sent will be returned in *sbytes.
• [EINVAL] — the fd argument is not a regular file.
• [EINVAL] — the s argument is not a SOCK_STREAM type socket.
• [EINVAL] — the offset argument is negative.
• [EIO] — an error occurred while reading from fd.
• [EINTEGRITY] — corrupted data was detected while reading from fd. 😭
• [ENOTCAPABLE] — the fd or the s argument has insufficient rights.
• [ENOBUFS] — the system was unable to allocate an internal buffer.
• [ENOTCONN] — the s argument points to an unconnected socket.
• [ENOTSOCK] — the s argument is not a socket.
• [EOPNOTSUPP] — the file system for descriptor fd does not support sendfile().
• [EPIPE] — the socket peer has closed the connection. 💔

---

👀 SEE ALSO 👀

netstat(1), open(2), send(2), socket(2), writev(2), tuning(7)

K. Elmeleegy, A. Chanda, A. L. Cox, and W. Zwaenepoel, “A Portable Kernel Abstraction for Low-Overhead Ephemeral Mapping Management”, The Proceedings of the 2005 USENIX Annual Technical Conference, pp 223-236, 2005.

---

🕰️ HISTORY 🕰️

the sendfile() system call first appeared in FreeBSD 3.0. this manual page first appeared in FreeBSD 3.1. in FreeBSD 10, support for sending shared memory descriptors had been introduced. in FreeBSD 11, a non-blocking implementation had been introduced. a whole glow-up timeline, honestly. 🌈

---

👑 AUTHORS 👑

the initial implementation of the sendfile() system call and this manual page were written by David G. Lawrence <dg@dglawrence.com>. the FreeBSD 11 implementation was written by Gleb Smirnoff <glebius@FreeBSD.org>. legends, both of them. 🫶

---

🐛 BUGS 🐛

• the sendfile() system call will not fail — i.e., it will not return -1 and set errno to EFAULT — if provided an invalid address for sbytes. so like. she just won't catch that for you. bestie be careful. 💀
• the sendfile() system call does not support SCTP sockets; it will return -1 and set errno to EINVAL.

---

and that's the whole page, bestie!! this was the sendfile(2) manual from the Debian edition, dated March 30, 2020 — so go forth, send those files zero-copy style, and remember: nbytes of 0 means all of it. mwah. 💖✨🎀🦋🌸💅🫶👑