TCP(4)

💅 NAME 💅

tcp — the Internet Transmission Control Protocol. That's the name, that's the whole vibe, periodt. ✨

---

✨ SYNOPSIS ✨

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>

int
socket(AF_INET, SOCK_STREAM, 0);

---

🦋 DESCRIPTION 🦋

Okay so bestie, picture this: TCP is the protocol that gives you reliable, flow-controlled, two-way transmission of data. Like, it's the friend who actually texts back, confirms plans, and makes sure the whole conversation arrives in the right order. It's a byte-stream protocol, and it's what powers the SOCK_STREAM abstraction. 🌊

TCP rocks the standard Internet address format, and on top of that it hands you a per-host collection of "port addresses". So every single address is made of an Internet address (which spells out the host and the network) plus a specific TCP port on that host that identifies the peer entity. Two-part address, chef's kiss. 🎀

Now, TCP sockets come in two flavours: "active" and "passive". Active sockets go out there and initiate connections to passive sockets. By default, TCP sockets are born active — so if you want a passive one, you gotta call listen(2) after you bind the socket with bind(2). And listen up: only passive sockets get to use accept(2) to accept incoming connections, and only active sockets get to use connect(2) to start connections. No switching roles mid-character, that's the rule. 👑

Passive sockets are allowed to "underspecify" their location so they can match incoming connection requests coming from multiple networks. This whole technique is called "wildcard addressing", and it lets one single server serve clients across multiple networks. Main character energy fr fr. 🌈 To make a socket listen on all the networks, you bind the Internet address INADDR_ANY. You can still specify the TCP port at that point — and if you don't, the system will just assign one for you. 🫶

Once a connection is established, the socket's address gets locked in by wherever the peer entity is. The address assigned to the socket is the address tied to the network interface that packets are actually being transmitted and received through. Normally that address lines up with the peer entity's network. 💫

TCP supports a bunch of socket options, and you can set them with setsockopt(2) and test them with getsockopt(2). Here's the whole roster, no skips:

• TCP_INFO — Wanna peek at the tea about a socket's underlying TCP session? Pass the read-only option TCP_INFO to getsockopt(2) and you're in. It takes a single argument: a pointer to an instance of struct tcp_info. ⚠️ Heads up though, this API is subject to change — so go consult the source to figure out which fields are currently filled out by this option. FreeBSD specific additions include send window size, receive window size, and bandwidth-controlled window space.
• TCP_CCALGOOPT — Set or query congestion control algorithm specific parameters. See mod_cc(4) for the details.
• TCP_CONGESTION — Select or query the congestion control algorithm that TCP is gonna use for the connection. See mod_cc(4) for the details.
• TCP_FUNCTION_BLK — Select or query the set of functions that TCP will use for this connection. This lets a user pick an alternate TCP stack. The alternate TCP stack has to already be loaded in the kernel, no exceptions. Wanna see the available TCP stacks? Check functions_available in the "MIB Variables" section further down. Wanna see the default TCP stack? That's functions_default, same section. 🧚
• TCP_KEEPINIT — This setsockopt(2) option takes a per-socket timeout argument of u_int, in seconds, for new, non-established TCP connections. For the global default in milliseconds, see keepinit in the "MIB Variables" section further down.
• TCP_KEEPIDLE — This setsockopt(2) option takes an argument of u_int for the amount of time, in seconds, that the connection has to be idle before keepalive probes (if enabled) get sent for this socket's connection. If you set it on a listening socket, the value is inherited by the newly created socket upon accept(2). For the global default in milliseconds, see keepidle in the "MIB Variables" section further down.
• TCP_KEEPINTVL — This setsockopt(2) option takes an argument of u_int to set the per-socket interval, in seconds, between keepalive probes sent to a peer. If you set it on a listening socket, the value is inherited by the newly created socket upon accept(2). For the global default in milliseconds, see keepintvl in the "MIB Variables" section further down.
• TCP_KEEPCNT — This setsockopt(2) option takes an argument of u_int and lets you tune, per-socket, the number of probes sent with no response before the connection gets dropped. If you set it on a listening socket, the value is inherited by the newly created socket upon accept(2). For the global default, see keepcnt in the "MIB Variables" section further down.
• TCP_NODELAY — Under most circumstances, TCP sends data the moment it's presented. But when outstanding data hasn't been acknowledged yet, it gathers small amounts of output so it can send them in a single packet once an acknowledgement comes in. For a small number of clients — think window systems firing off a stream of mouse events that get no replies — this packetization can cause significant delays. The boolean option TCP_NODELAY defeats this algorithm. 💥
• TCP_MAXSEG — By default, a sender-TCP and a receiver-TCP negotiate among themselves to figure out the maximum segment size for each connection. The TCP_MAXSEG option lets the user determine the result of that negotiation, and reduce it if desired.
• TCP_NOOPT — TCP usually sends a number of options in each packet, matching up with the various TCP extensions provided in this implementation. The boolean option TCP_NOOPT is there to disable TCP option use on a per-connection basis.
• TCP_NOPUSH — By convention, the sender-TCP sets the "push" bit and starts transmission immediately (if permitted) at the end of every user call to write(2) or writev(2). When this option is set to a non-zero value, TCP delays sending any data at all until either the socket is closed, or the internal send buffer gets filled. 😤
• TCP_MD5SIG — This option enables the use of MD5 digests (also known as TCP-MD5) on writes to the specified socket. Outgoing traffic gets digested; digests on incoming traffic get verified. When this option is enabled on a socket, all inbound and outgoing TCP segments must be signed with MD5 digests. 🔐

One common use for this in a FreeBSD router deployment is letting based routers interwork with Cisco equipment at peering points. Support for this feature conforms to RFC 2385.

In order for this option to function correctly, the administrator has to add a tcp-md5 key entry to the system's security associations database (SADB) using the setkey(8) utility. At this time, this entry can only be specified on a per-host basis. 📌

If an SADB entry can't be found for the destination, the system does not send any outgoing segments and drops any inbound segments.

Each dropped segment gets taken into account in the TCP protocol statistics.

The option level for the setsockopt(2) call is the protocol number for TCP, which you can grab from getprotobyname(3), or just use IPPROTO_TCP. All options are declared in <netinet/tcp.h>. 💖

Options at the IP transport level can be used with TCP too; see ip(4). Incoming connection requests that are source-routed get noted, and the reverse source route is used in responding. 🌀

The default congestion control algorithm for TCP is cc_newreno(4). Other congestion control algorithms can be made available using the mod_cc(4) framework. Slay. 🔥

---

🎀 MIB Variables 🎀

The TCP protocol implements a number of variables in the net.inet.tcp branch of the sysctl(3) MIB. Let's go:

• TCPCTL_DO_RFC1323 (rfc1323) — Implement the window scaling and timestamp options of RFC 1323 (default is true).
• TCPCTL_MSSDFLT (mssdflt) — The default value used for the maximum segment size ("MSS") when no advice to the contrary is received from MSS negotiation.
• TCPCTL_SENDSPACE (sendspace) — Maximum TCP send window.
• TCPCTL_RECVSPACE (recvspace) — Maximum TCP receive window.
• log_in_vain — Log any connection attempts to ports where there isn't a socket accepting connections. A value of 1 limits the logging to SYN (connection establishment) packets only. A value of 2 results in any TCP packets to closed ports being logged. Any value not listed above disables the logging (default is 0, i.e., logging is disabled).
• msl — The Maximum Segment Lifetime, in milliseconds, for a packet.
• keepinit — Timeout, in milliseconds, for new, non-established TCP connections. The default is 75000 msec.
• keepidle — Amount of time, in milliseconds, that the connection must be idle before keepalive probes (if enabled) are sent. The default is 7200000 msec (2 hours).
• keepintvl — The interval, in milliseconds, between keepalive probes sent to remote machines, when no response is received on a keepidle probe. The default is 75000 msec.
• keepcnt — Number of probes sent, with no response, before a connection is dropped. The default is 8 packets.
• always_keepalive — okay so this one just assumes SO_KEEPALIVE is set on every TCP connection, no matter what. The kernel will periodically ping the remote host just to check the connection is still alive. vibes check, basically 💅
• icmp_may_rst — certain ICMP unreachable messages are allowed to abort connections that are sitting in SYN-SENT state. like, bye bestie 🫶
• do_tcpdrain — flush the packets chillin' in the TCP reassembly queue if the system is running low on mbufs. cleaning out the group chat when storage gets full 🧹
• blackhole — if you turn this on, it disables sending RST when someone tries to connect to a port where there's no socket accepting connections. see blackhole(4) for the full tea.
• delayed_ack — delay the ACK so it can hopefully piggyback onto a data packet. two birds, one packet, no cap 🦋
• delacktime — the max amount of time, in milliseconds, before a delayed ACK actually gets sent. it can only procrastinate so long 💀
• path_mtu_discovery — enables Path MTU Discovery. slay ✨
• tcbhashsize — the size of the TCP control-block hash table (read-only, so don't even try to change it here 🙈). you can tune it with the kernel option TCBHASHSIZE, or by setting net.inet.tcp.tcbhashsize in loader(8).
• pcbcount — the number of active process control blocks (read-only). just a lil headcount 📋
• syncookies — decides whether SYN cookies get generated for outbound SYN-ACK packets or not. SYN cookies are literally a lifesaver during SYN flood attacks, and they're enabled by default. (peep syncookies(4).)
• isn_reseed_interval — the interval (in seconds) for how often the secret data used in RFC 1948 initial sequence number calculations gets reseeded. by default this variable is set to zero, which means no reseeding happens at all. and honestly? reseeding shouldn't even be necessary, and it will break TIME_WAIT recycling for a few minutes. so like… don't 😭
• reass.cursegments — the current total number of segments hanging out in all the reassembly queues right now.
• reass.maxsegments — the maximum limit on the total number of segments across all reassembly queues. you can adjust this limit as a tunable.
• reass.maxqueuelen — the maximum number of segments allowed in each individual reassembly queue. by default the system picks a limit based on each TCP connection's receive buffer size and maximum segment size (MSS). and the actual limit that gets applied to a session's reassembly queue is the lower of the system-calculated automatic limit and the user-specified reass.maxqueuelen limit. lowest bidder wins, iconic 💅
• rexmit_initial, rexmit_min, rexmit_slop — these three adjust the retransmit timer calculation for TCP. the slop is typically added to the raw calculation to account for occasional variances that the SRTT (smoothed round-trip time) just can't accommodate, while the minimum specifies an absolute minimum. so here's the thing: a bunch of TCP RFCs suggest a 1 second minimum, but those RFCs are kinda focused on streaming behavior and they fail to deal with the fact that a 1 second minimum has severe detrimental effects over lossy interactive connections — like an 802.11b wireless link — and over very fast but lossy connections for those cases the fast retransmit code doesn't cover. so for this reason, we use 200ms of slop and a near-0 minimum, which gives an effective minimum of 200ms (similar to Linux 🐧). the initial value is what gets used before an RTT measurement has been performed.
• initcwnd_segments — enables the ability to specify the initial congestion window in number of segments. the default value is 10, as suggested by RFC 6928. changing the value on the fly would not affect connections using a congestion window from the hostcache, jsyk. > caution: this regulates the burst of packets allowed to be sent in the first RTT. the value should be relative to the link capacity. start with small values for lower-capacity links. large bursts can cause buffer overruns and packet drops if routers have small buffers or the link is experiencing congestion. so like, be chill with it 😭
• rfc6675_pipe — calculate the bytes in flight using the algorithm described in RFC 6675. it's also a prerequisite to enable Proportional Rate Reduction, so it's kinda the main character of that whole situation 👑
• rfc3042 — enable the Limited Transmit algorithm as described in RFC 3042. it helps avoid timeouts on lossy links, and also when the congestion window is small — which is exactly what happens on short transfers. tiny window, big help 🫶
• rfc3390 — enable support for RFC 3390, which allows a variable-sized starting congestion window on new connections, depending on the maximum segment size. this helps throughput in general, but it particularly affects short transfers and high-bandwidth large propagation-delay connections. it's giving range ✨
• sack.enable — enable support for RFC 2018, the TCP Selective Acknowledgment option. it lets the receiver tell the sender about all the segments that successfully arrived, so the sender only has to retransmit the missing ones. efficiency, we love to see it 🌸
• sack.maxholes — maximum number of SACK holes per connection. defaults to 128.
• sack.globalmaxholes — maximum number of SACK holes per system, across all connections. defaults to 65536.
• maxtcptw — okay so when a TCP connection enters the TIME_WAIT state, its associated socket structure gets freed (since it's of negligible size and use), and a brand new structure is allocated to hold the minimal amount of info necessary for sustaining a connection in this state. that's called the compressed TCP TIME_WAIT state. since this structure is smaller than a socket structure, it can save a significant amount of system memory 💖. the net.inet.tcp.maxtcptw MIB variable controls the maximum number of these structures allocated. by default, it's initialized to kern.ipc.maxsockets / 5.
• nolocaltimewait — suppress the creation of compressed TCP TIME_WAIT states for connections where both endpoints are local. if it's all in-house, no need for the compressed era 💅
• fast_finwait2_recycle — recycle TCP FIN_WAIT_2 connections faster when the socket is marked as SBS_CANTRCVMORE (meaning no user process has the socket open, and data received on the socket can't be read). the timeout used here is finwait2_timeout.
• finwait2_timeout — the timeout to use for fast recycling of TCP FIN_WAIT_2 connections. defaults to 60 seconds.
• ecn.enable — enable support for TCP Explicit Congestion Notification (ECN). ECN lets a TCP sender reduce its transmission rate in order to avoid packet drops. settings:
  • 0 — disable ECN.
  • 1 — allow incoming connections to request ECN. outgoing connections will request ECN.
  • 2 — allow incoming connections to request ECN. outgoing connections will not request ECN.
• ecn.maxretries — number of retries (SYN or SYN/ACK retransmits) before disabling ECN on a specific connection. this is needed to help with connection establishment when a broken firewall is sitting in the network path. rude of the firewall honestly 🙄
• pmtud_blackhole_detection — turn on automatic path MTU blackhole detection. in case of retransmits, the OS will lower the MSS to check whether it's an MTU problem. if the current MSS is greater than the configured value to try, it'll be set to that configured value; otherwise, the MSS gets set to the default values (net.inet.tcp.mssdflt and net.inet.tcp.v6mssdflt). detective work 🔍
• pmtud_blackhole_mss — the MSS to try for IPv4 if PMTU blackhole detection is turned on.
• v6pmtud_blackhole_mss — the MSS to try for IPv6 if PMTU blackhole detection is turned on.
• pmtud_blackhole_activated — number of times configured values were used in an attempt to downshift.
• pmtud_blackhole_activated_min_mss — number of times the default MSS was used in an attempt to downshift.
• pmtud_blackhole_failed — number of connections where retransmits kept going even after an MSS downshift. 😭
• functions_available — list of available TCP function blocks (TCP stacks).
• functions_default — the default TCP function block (TCP stack).
• functions_inherit_listen_socket_stack — determines whether to inherit the listen socket's tcp stack, or to use the current system default tcp stack as defined by functions_default. default is true.
• insecure_rst — use the criteria defined in RFC793 instead of RFC5961 for accepting RST segments. default is false. 👀
• insecure_syn — use the criteria defined in RFC793 instead of RFC5961 for accepting SYN segments. default is false. 👀
• ts_offset_per_conn — when initializing the TCP timestamps, use a per connection offset instead of a per host pair offset. the default is to use per connection offsets, as recommended in RFC 7323.

💀 ERRORS 💀

okay bestie, real talk: a socket operation may fail, and when it does, it's gonna return one of the following errors. every single one is a whole mood. let's break them down. 🫶

if you see one of these, that's the socket telling you exactly why it's not slaying today. no cap.

• [EISCONN] — when you're trying to establish a connection on a socket that already has one. bestie, it's taken. 💅
• [ENOBUFS] or [ENOMEM] — when the system runs out of memory for an internal data structure. the vibes are starving. 😭
• [ETIMEDOUT] — when a connection was dropped due to excessive retransmissions. it tried, it retried, it died. 💀
• [ECONNRESET] — when the remote peer forces the connection to be closed. ghosted by the other side, fr fr. 👻
• [ECONNREFUSED] — when the remote peer actively refuses connection establishment (usually because no process is listening to the port). not me getting left on read at the port. 🙈
• [EADDRINUSE] — when an attempt is made to create a socket with a port which has already been allocated. that port is somebody else's main character. 👑
• [EADDRNOTAVAIL] — when an attempt is made to create a socket with a network address for which no network interface exists. the address is giving imaginary friend. 🧚
• [EAFNOSUPPORT] — when an attempt is made to bind or connect a socket to a multicast address. not supported bestie, it's not on the guest list. 🎀
• [EINVAL] — when trying to change TCP function blocks at an invalid point in the session. wrong time, wrong place, wrong vibe. ✨
• [ENOENT] — when trying to use a TCP function block that is not available. it's giving 404, block not found. 🍓

---

👀 SEE ALSO 👀

go read these next, bestie, they're iconic:

• getsockopt(2), socket(2), sysctl(3), blackhole(4), inet(4), intro(4), ip(4), mod_cc(4), siftr(4), syncache(4), setkey(8), tcp_functions(9)

and the reading list / receipts:

• V. Jacobson, R. Braden, and D. Borman, TCP Extensions for High Performance, RFC 1323.
• A. Heffernan, Protection of BGP Sessions via the TCP MD5 Signature Option, RFC 2385.
• K. Ramakrishnan, S. Floyd, and D. Black, The Addition of Explicit Congestion Notification (ECN) to IP, RFC 3168.

---

📜 HISTORY 📜

the lore is lore-ing. the TCP protocol appeared in 4.2BSD. the RFC 1323 extensions for window scaling and timestamps were added in 4.4BSD. and the TCP_INFO option was introduced in Linux 2.6 and is subject to change. so don't get too attached, bestie. 💖

---

and that's the tea: this page is from Debian, dated December 1, 2019, and it's TCP(4). periodt. 🎀✨