MULTICAST(4)

📦 version 12.2, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

💖 NAME 💖

multicast — Multicast Routing — it's the MULTICAST(4) kernel interfaces manual page bestie, and we're gonna get through it together 💅

✨ SYNOPSIS ✨

okay so first things first, the kernel config situation:

options MROUTING

and then all the headers you're gonna need, no cap:

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/ip_mroute.h>
#include <netinet6/ip6_mroute.h>

and here's the function signatures, exactly as they are, don't come for me:

int
getsockopt(int s, IPPROTO_IP, MRT_INIT, void *optval,
      socklen_t *optlen);

int
setsockopt(int s, IPPROTO_IP, MRT_INIT, const void *optval,
      socklen_t optlen);

int
getsockopt(int s, IPPROTO_IPV6, MRT6_INIT, void *optval,
      socklen_t *optlen);

int
setsockopt(int s, IPPROTO_IPV6, MRT6_INIT, const void *optval,
      socklen_t optlen);

🌸 DESCRIPTION 🌸

so multicast routing is literally the girlboss way to efficiently propagate data packets to a whole set of multicast listeners across multipoint networks 🦋. like, imagine if unicast was doing the replication — some of the network links would end up carrying multiple copies of the exact same data packets. embarrassing. with multicast routing, the overhead gets reduced to one copy (at most) per network link. ate. ✨

every multicast-capable router must run a common multicast routing protocol — it's giving mandatory 💀. and the recommendation is that you use either Protocol Independent Multicast - Sparse Mode (PIM-SM), or Protocol Independent Multicast - Dense Mode (PIM-DM), because those are now the generally accepted protocols in the Internet community. periodt. (if you wanna know about the older multicast routing protocols, the “HISTORY” section discusses them — we'll get there.)

to actually start multicast routing, you the user must enable multicast forwarding in the kernel (peep the “SYNOPSIS” for the kernel configuration options 👀), AND you must run a multicast routing capable user-level process. no skipping steps bestie. from the developer's point of view, the programming guide described in the “Programming Guide” section is what you should use to control the multicast forwarding in the kernel.

🎀 Programming Guide 🎀

this section is giving basic multicast routing API energy 💖. the so-called “advanced multicast API”? that's over in the “Advanced Multicast API Programming Guide” section. we'll get to her.

first up: a multicast routing socket must be open. that socket is the one you're gonna use to control the multicast forwarding in the kernel. and ngl, note that most operations below require certain privilege (i.e., root privilege) — main character energy only 👑:

/* IPv4 */
int mrouter_s4;
mrouter_s4 = socket(AF_INET, SOCK_RAW, IPPROTO_IGMP);

int mrouter_s6;
mrouter_s6 = socket(AF_INET6, SOCK_RAW, IPPROTO_ICMPV6);

okay important note bestie 🫶: if the router needs to open an IGMP or ICMPv6 socket (in case of IPv4 and IPv6 respectively) for sending or receiving of IGMP or MLD multicast group membership messages, then the same mrouter_s4 or mrouter_s6 sockets should be used for sending and receiving respectively IGMP or MLD messages. in case of BSD-derived kernel, it may be possible to open separate sockets for IGMP or MLD messages only. HOWEVER 😭 some other kernels (e.g., Linux) require that the multicast routing socket must be used for sending and receiving of IGMP or MLD messages. so for portability reason the multicast routing socket should be reused for IGMP and MLD messages as well. iykyk.

after the multicast routing socket is open, it can be used to enable or disable multicast forwarding in the kernel ✨:

/* IPv4 */
int v = 1;        /* 1 to enable, or 0 to disable */
setsockopt(mrouter_s4, IPPROTO_IP, MRT_INIT, (void *)&v, sizeof(v));

/* IPv6 */
int v = 1;        /* 1 to enable, or 0 to disable */
setsockopt(mrouter_s6, IPPROTO_IPV6, MRT6_INIT, (void *)&v, sizeof(v));
...
/* If necessary, filter all ICMPv6 messages */
struct icmp6_filter filter;
ICMP6_FILTER_SETBLOCKALL(&filter);
setsockopt(mrouter_s6, IPPROTO_ICMPV6, ICMP6_FILTER, (void *)&filter,
           sizeof(filter));

after multicast forwarding is enabled, the multicast routing socket can be used to enable PIM processing in the kernel if we are running PIM-SM or PIM-DM (see pim(4)). slay 🌈

for each network interface (e.g., physical or a virtual tunnel) that would be used for multicast forwarding, a corresponding multicast interface must be added to the kernel:

/* IPv4 */
struct vifctl vc;
memset(&vc, 0, sizeof(vc));
/* Assign all vifctl fields as appropriate */
vc.vifc_vifi = vif_index;
vc.vifc_flags = vif_flags;
vc.vifc_threshold = min_ttl_threshold;
vc.vifc_rate_limit = 0;
memcpy(&vc.vifc_lcl_addr, &vif_local_address, sizeof(vc.vifc_lcl_addr));
setsockopt(mrouter_s4, IPPROTO_IP, MRT_ADD_VIF, (void *)&vc,
           sizeof(vc));

the vif_index must be unique per vif, no duplicates allowed 💅. the vif_flags contains the VIFF_* flags as defined in <netinet/ip_mroute.h>.

⚠️ the VIFF_TUNNEL flag is no longer supported by FreeBSD. if you wish to forward multicast datagrams over a tunnel, you should consider configuring a gif(4) or gre(4) tunnel and using it as a physical interface instead.

the min_ttl_threshold contains the minimum TTL a multicast data packet must have to be forwarded on that vif. typically, it would have value of 1.

the max_rate_limit argument is no longer supported in FreeBSD and should be set to 0 😭. users who wish to rate-limit multicast datagrams should consider the use of dummynet(4) or altq(4).

the vif_local_address contains the local IP address of the corresponding local interface. the vif_remote_address contains the remote IP address in case of DVMRP multicast tunnels.

/* IPv6 */
struct mif6ctl mc;
memset(&mc, 0, sizeof(mc));
/* Assign all mif6ctl fields as appropriate */
mc.mif6c_mifi = mif_index;
mc.mif6c_flags = mif_flags;
mc.mif6c_pifi = pif_index;
setsockopt(mrouter_s6, IPPROTO_IPV6, MRT6_ADD_MIF, (void *)&mc,
           sizeof(mc));

the mif_index must be unique per vif ✨. the mif_flags contains the MIFF_* flags as defined in <netinet6/ip6_mroute.h>. the pif_index is the physical interface index of the corresponding local interface.

a multicast interface is deleted by:

/* IPv4 */
vifi_t vifi = vif_index;
setsockopt(mrouter_s4, IPPROTO_IP, MRT_DEL_VIF, (void *)&vifi,
           sizeof(vifi));

/* IPv6 */
mifi_t mifi = mif_index;
setsockopt(mrouter_s6, IPPROTO_IPV6, MRT6_DEL_MIF, (void *)&mifi,
           sizeof(mifi));

after the multicast forwarding is enabled, and the multicast virtual interfaces are added, the kernel may deliver upcall messages (also called signals later in this text — we're keeping that name, don't @ me) on the multicast routing socket that was open earlier with MRT_INIT or MRT6_INIT. the IPv4 upcalls have struct igmpmsg header (see <netinet/ip_mroute.h>) with field im_mbz set to zero. note that this header follows the structure of struct ip with the protocol field ip_p set to zero. the IPv6 upcalls have struct mrt6msg header (see <netinet6/ip6_mroute.h>) with field im6_mbz set to zero. note that this header follows the structure of struct ip6_hdr with the next header field ip6_nxt set to zero. 🦋

the upcall header contains field im_msgtype and im6_msgtype with the type of the upcall IGMPMSG_* and MRT6MSG_* for IPv4 and IPv6 respectively. the values of the rest of the upcall header fields and the body of the upcall message depend on the particular upcall type.

if the upcall message type is IGMPMSG_NOCACHE or MRT6MSG_NOCACHE, this is an indication that a multicast packet has reached the multicast router, but the router has no forwarding state for that packet 👀. typically, the upcall would be a signal for the multicast routing user-level process to install the appropriate Multicast Forwarding Cache (MFC) entry in the kernel. it's giving "bestie we don't know her, add her to the list" 💅

an MFC entry is added by:

/* IPv4 */
struct mfcctl mc;
memset(&mc, 0, sizeof(mc));
memcpy(&mc.mfcc_origin, &source_addr, sizeof(mc.mfcc_origin));
memcpy(&mc.mfcc_mcastgrp, &group_addr, sizeof(mc.mfcc_mcastgrp));
mc.mfcc_parent = iif_index;
for (i = 0; i < maxvifs; i++)
    mc.mfcc_ttls[i] = oifs_ttl[i];
setsockopt(mrouter_s4, IPPROTO_IP, MRT_ADD_MFC,
           (void *)&mc, sizeof(mc));

/* IPv6 */
struct mf6cctl mc;
memset(&mc, 0, sizeof(mc));
memcpy(&mc.mf6cc_origin, &source_addr, sizeof(mc.mf6cc_origin));
memcpy(&mc.mf6cc_mcastgrp, &group_addr, sizeof(mf6cc_mcastgrp));
mc.mf6cc_parent = iif_index;
for (i = 0; i < maxvifs; i++)
    if (oifs_ttl[i] > 0)
        IF_SET(i, &mc.mf6cc_ifset);
setsockopt(mrouter_s6, IPPROTO_IPV6, MRT6_ADD_MFC,
           (void *)&mc, sizeof(mc));

the source_addr and group_addr are the source and group address of the multicast packet (as set in the upcall message). the iif_index is the virtual interface index of the multicast interface the multicast packets for this specific source and group address should be received on. the oifs_ttl[] array contains the minimum TTL (per interface) a multicast packet should have to be forwarded on an outgoing interface. if the TTL value is zero, the corresponding interface is not included in the set of outgoing interfaces 🙅. and note that in case of IPv6 only the set of outgoing interfaces can be specified.

an MFC entry is deleted by:

/* IPv4 */
struct mfcctl mc;
memset(&mc, 0, sizeof(mc));
memcpy(&mc.mfcc_origin, &source_addr, sizeof(mc.mfcc_origin));
memcpy(&mc.mfcc_mcastgrp, &group_addr, sizeof(mc.mfcc_mcastgrp));
setsockopt(mrouter_s4, IPPROTO_IP, MRT_DEL_MFC,
           (void *)&mc, sizeof(mc));

/* IPv6 */
struct mf6cctl mc;
memset(&mc, 0, sizeof(mc));
memcpy(&mc.mf6cc_origin, &source_addr, sizeof(mc.mf6cc_origin));
memcpy(&mc.mf6cc_mcastgrp, &group_addr, sizeof(mf6cc_mcastgrp));
setsockopt(mrouter_s6, IPPROTO_IPV6, MRT6_DEL_MFC,
           (void *)&mc, sizeof(mc));

the following method can be used to get various statistics per installed MFC entry in the kernel (e.g., the number of forwarded packets per source and group address) 📊:

/* IPv4 */
struct sioc_sg_req sgreq;
memset(&sgreq, 0, sizeof(sgreq));
memcpy(&sgreq.src, &source_addr, sizeof(sgreq.src));
memcpy(&sgreq.grp, &group_addr, sizeof(sgreq.grp));
ioctl(mrouter_s4, SIOCGETSGCNT, &sgreq);

/* IPv6 */
struct sioc_sg_req6 sgreq;
memset(&sgreq, 0, sizeof(sgreq));
memcpy(&sgreq.src, &source_addr, sizeof(sgreq.src));
memcpy(&sgreq.grp, &group_addr, sizeof(sgreq.grp));
ioctl(mrouter_s6, SIOCGETSGCNT_IN6, &sgreq);

and the following method can be used to get various statistics per multicast virtual interface in the kernel (e.g., the number of forwarded packets per interface) ✨:

/* IPv4 */
struct sioc_vif_req vreq;
memset(&vreq, 0, sizeof(vreq));
vreq.vifi = vif_index;
ioctl(mrouter_s4, SIOCGETVIFCNT, &vreq);

/* IPv6 */
struct sioc_mif_req6 mreq;
memset(&mreq, 0, sizeof(mreq));
mreq.mifi = vif_index;
ioctl(mrouter_s6, SIOCGETMIFCNT_IN6, &mreq);

okay that's part one bestie, keep reading, there's more 💖

✨ Advanced Multicast API Programming Guide ✨

ok so picture this: the kernel peeps want to add shiny new features, but they ALSO can't break everything that already exists (binary compatibility AND API compatibility, both, at the same time, iconic behaviour honestly). And they still wanna let user-level processes actually use the new stuff if the kernel has it. the solution? a lil negotiation. a vibe check, if you will, between the user-level process and the kernel:

  1. The user-level process tries to enable in the kernel the set of new features (and the corresponding API) it would like to use.
  2. The kernel returns the (sub)set of features it knows about and is willing to be enabled.
  3. The user-level process uses only that set of features the kernel has agreed on.

to keep that sweet backward compatibility, if the user-level process doesn't ask for any new features, the kernel just defaults to the basic multicast API (peep the “Programming Guide” section bestie). and rn? the advanced multicast API exists only for IPv4 — IPv6 support is coming in the future, watch this space 💅

below is the summary of the expandable API solution. note that all new options and structures are defined in <netinet/ip_mroute.h> and <netinet6/ip6_mroute.h>, unless stated otherwise.

the user-level process uses new getsockopt()/setsockopt() options to perform the API features negotiation with the kernel. this negotiation must be performed right after the multicast routing socket is open. the set of desired/allowed features is stored in a bitset (currently, in uint32_t; i.e., maximum of 32 new features). the new getsockopt()/setsockopt() options are MRT_API_SUPPORT and MRT_API_CONFIG. example:

uint32_t v;
getsockopt(sock, IPPROTO_IP, MRT_API_SUPPORT, (void *)&v, sizeof(v));

would set in v the pre-defined bits that the kernel API supports. the eight least significant bits in uint32_t are same as the eight possible flags MRT_MFC_FLAGS_* that can be used in mfcc_flags as part of the new definition of struct mfcctl (more on those flags below, don't worry bestie), which leaves 24 flags for other new features. the value returned by getsockopt(MRT_API_SUPPORT) is read-only — in other words, setsockopt(MRT_API_SUPPORT) would fail. it's giving "look but don't touch" ✨

to modify the API, and to set some specific feature in the kernel, then:

uint32_t v = MRT_MFC_FLAGS_DISABLE_WRONGVIF;
if (setsockopt(sock, IPPROTO_IP, MRT_API_CONFIG, (void *)&v, sizeof(v))
    != 0) {
    return (ERROR);
}
if (v & MRT_MFC_FLAGS_DISABLE_WRONGVIF)
    return (OK);        /* Success */
else
    return (ERROR);

in other words, when setsockopt(MRT_API_CONFIG) is called, the argument to it specifies the desired set of features to be enabled in the API and the kernel. the return value in v is the actual (sub)set of features that were enabled in the kernel. to obtain later the same set of features that were enabled, then:

getsockopt(sock, IPPROTO_IP, MRT_API_CONFIG, (void *)&v, sizeof(v));

the set of enabled features is global. in other words, setsockopt(MRT_API_CONFIG) should be called right after setsockopt(MRT_INIT). no dilly-dallying, no side quests in between 🫶

currently, the following set of new features is defined:

#define MRT_MFC_FLAGS_DISABLE_WRONGVIF (1 << 0) /* disable WRONGVIF signals */
#define MRT_MFC_FLAGS_BORDER_VIF   (1 << 1)  /* border vif              */
#define MRT_MFC_RP                 (1 << 8)  /* enable RP address       */
#define MRT_MFC_BW_UPCALL          (1 << 9)  /* enable bw upcalls       */

🌸 the new struct, she's here 🌸

the advanced multicast API uses a newly defined struct mfcctl2 instead of the traditional struct mfcctl. the original struct mfcctl is kept as is (no notes, she stays iconic and untouched). the new struct mfcctl2 is:

/*
 * The new argument structure for MRT_ADD_MFC and MRT_DEL_MFC overlays
 * and extends the old struct mfcctl.
 */
struct mfcctl2 {
        /* the mfcctl fields */
        struct in_addr  mfcc_origin;       /* ip origin of mcasts       */
        struct in_addr  mfcc_mcastgrp;     /* multicast group associated*/
        vifi_t          mfcc_parent;       /* incoming vif              */
        u_char          mfcc_ttls[MAXVIFS];/* forwarding ttls on vifs   */

        /* extension fields */
        uint8_t         mfcc_flags[MAXVIFS];/* the MRT_MFC_FLAGS_* flags*/
        struct in_addr  mfcc_rp;            /* the RP address           */
};

the new fields are mfcc_flags[MAXVIFS] and mfcc_rp. note that for compatibility reasons they are added at the end. the way they just... appended them. the restraint. the elegance. 😭💖

the mfcc_flags[MAXVIFS] field is used to set various flags per interface per (S,G) entry. currently, the defined flags are:

#define MRT_MFC_FLAGS_DISABLE_WRONGVIF (1 << 0) /* disable WRONGVIF signals */
#define MRT_MFC_FLAGS_BORDER_VIF       (1 << 1) /* border vif          */

the MRT_MFC_FLAGS_DISABLE_WRONGVIF flag is used to explicitly disable the IGMPMSG_WRONGVIF kernel signal at the (S,G) granularity if a multicast data packet arrives on the wrong interface. usually, this signal is used to complete the shortest-path switch in case of PIM-SM multicast routing, or to trigger a PIM assert message. however, it should not be delivered for interfaces that are not in the outgoing interface set, and that are not expecting to become an incoming interface. hence, if the MRT_MFC_FLAGS_DISABLE_WRONGVIF flag is set for some of the interfaces, then a data packet that arrives on that interface for that MFC entry will NOT trigger a WRONGVIF signal. if that flag is not set, then a signal is triggered (the default action). so like, silence it if you know it's not the one, otherwise it's gonna yell. periodt 🎀

the MRT_MFC_FLAGS_BORDER_VIF flag is used to specify whether the Border-bit in PIM Register messages should be set (in case when the Register encapsulation is performed inside the kernel). if it is set for the special PIM Register kernel virtual interface (see pim(4)), the Border-bit in the Register messages sent to the RP will be set.

the remaining six bits are reserved for future usage. six bits just sitting there, chilling, waiting for their main character moment ⭐

the mfcc_rp field is used to specify the RP address (in case of PIM-SM multicast routing) for a multicast group G if we want to perform kernel-level PIM Register encapsulation. the mfcc_rp field is used only if the MRT_MFC_RP advanced API flag/capability has been successfully set by setsockopt(MRT_API_CONFIG).

if the MRT_MFC_RP flag was successfully set by setsockopt(MRT_API_CONFIG), then the kernel will attempt to perform the PIM Register encapsulation itself instead of sending the multicast data packets to user level (inside IGMPMSG_WHOLEPKT upcalls) for user-level encapsulation. the RP address would be taken from the mfcc_rp field inside the new struct mfcctl2. however, even if the MRT_MFC_RP flag was successfully set, if the mfcc_rp field was set to INADDR_ANY, then the kernel will still deliver an IGMPMSG_WHOLEPKT upcall with the multicast data packet to the user-level process. so like... you said you'd handle it, but if you hand it INADDR_ANY it's still gonna hand it right back to you 🙈

in addition, if the multicast data packet is too large to fit within a single IP packet after the PIM Register encapsulation (e.g., if its size was on the order of 65500 bytes), the data packet will be fragmented, and then each of the fragments will be encapsulated separately. note that typically a multicast data packet can be that large only if it was originated locally from the same hosts that performs the encapsulation; otherwise the transmission of the multicast data packet over Ethernet for example would have fragmented it into much smaller pieces. so yeah, unless it's a local homegrown packet, Ethernet already did the chopping for you ✂️

🌈 bandwidth monitoring, aka the glow-up 🌈

typically, a multicast routing user-level process would need to know the forwarding bandwidth for some data flow. for example, the multicast routing process may want to timeout idle MFC entries, or in case of PIM-SM it can initiate (S,G) shortest-path switch if the bandwidth rate is above a threshold for example.

the original solution for measuring the bandwidth of a dataflow was that a user-level process would periodically query the kernel about the number of forwarded packets/bytes per (S,G), and then based on those numbers it would estimate whether a source has been idle, or whether the source's transmission bandwidth is above a threshold. that solution is far from being scalable, hence the need for a new mechanism for bandwidth monitoring. it's giving "we used to walk uphill both ways" energy fr fr 💀

below is a description of the bandwidth monitoring mechanism.

  • If the bandwidth of a data flow satisfies some pre-defined filter, the kernel delivers an upcall on the multicast routing socket to the multicast routing process that has installed that filter.
  • The bandwidth-upcall filters are installed per (S,G). There can be more than one filter per (S,G).
  • Instead of supporting all possible comparison operations (i.e., < <= == != > >= ), there is support only for the <= and >= operations, because this makes the kernel-level implementation simpler, and because practically we need only those two. Further, the missing operations can be simulated by secondary user-level filtering of those <= and >= filters. For example, to simulate !=, then we need to install filter “bw <= 0xffffffff”, and after an upcall is received, we need to check whether “measured_bw != expected_bw”.
  • The bandwidth-upcall mechanism is enabled by setsockopt(MRT_API_CONFIG) for the MRT_MFC_BW_UPCALL flag.
  • The bandwidth-upcall filters are added/deleted by the new setsockopt(MRT_ADD_BW_UPCALL) and setsockopt(MRT_DEL_BW_UPCALL) respectively (with the appropriate struct bw_upcall argument of course).

from application point of view, a developer needs to know about the following:

okay bestie this next bit is where the multicast API really shows off — it's the bandwidth-monitoring glow-up 💅 the kernel literally hands you a struct with a whole essay in the comments, and honestly? iconic. here it is, every line, no edits:

/*
 * Structure for installing or delivering an upcall if the
 * measured bandwidth is above or below a threshold.
 *
 * User programs (e.g. daemons) may have a need to know when the
 * bandwidth used by some data flow is above or below some threshold.
 * This interface allows the userland to specify the threshold (in
 * bytes and/or packets) and the measurement interval. Flows are
 * all packet with the same source and destination IP address.
 * At the moment the code is only used for multicast destinations
 * but there is nothing that prevents its use for unicast.
 *
 * The measurement interval cannot be shorter than some Tmin (currently, 3s).
 * The threshold is set in packets and/or bytes per_interval.
 *
 * Measurement works as follows:
 *
 * For >= measurements:
 * The first packet marks the start of a measurement interval.
 * During an interval we count packets and bytes, and when we
 * pass the threshold we deliver an upcall and we are done.
 * The first packet after the end of the interval resets the
 * count and restarts the measurement.
 *
 * For <= measurement:
 * We start a timer to fire at the end of the interval, and
 * then for each incoming packet we count packets and bytes.
 * When the timer fires, we compare the value with the threshold,
 * schedule an upcall if we are below, and restart the measurement
 * (reschedule timer and zero counters).
 */

struct bw_data {
        struct timeval  b_time;
        uint64_t        b_packets;
        uint64_t        b_bytes;
};

struct bw_upcall {
        struct in_addr  bu_src;         /* source address            */
        struct in_addr  bu_dst;         /* destination address       */
        uint32_t        bu_flags;       /* misc flags (see below)    */
#define BW_UPCALL_UNIT_PACKETS (1 << 0) /* threshold (in packets)    */
#define BW_UPCALL_UNIT_BYTES   (1 << 1) /* threshold (in bytes)      */
#define BW_UPCALL_GEQ          (1 << 2) /* upcall if bw >= threshold */
#define BW_UPCALL_LEQ          (1 << 3) /* upcall if bw <= threshold */
#define BW_UPCALL_DELETE_ALL   (1 << 4) /* delete all upcalls for s,d*/
        struct bw_data  bu_threshold;   /* the bw threshold          */
        struct bw_data  bu_measured;    /* the measured bw           */
};

/* max. number of upcalls to deliver together */
#define BW_UPCALLS_MAX                          128
/* min. threshold time interval for bandwidth measurement */
#define BW_UPCALL_THRESHOLD_INTERVAL_MIN_SEC    3
#define BW_UPCALL_THRESHOLD_INTERVAL_MIN_USEC   0

so real talk, the tl;dr of that comment: userland gets to say "hey kernel, tell me when this flow's bandwidth goes over or under this line" 🫶 a flow here just means every packet sharing the same source and destination IP address. right now the code only gets used for multicast destinations, but nothing is stopping you from using it for unicast either. the measurement interval can't be shorter than some Tmin (currently 3s), and the threshold is set in packets and/or bytes per_interval. and the two measurement modes work completely differently, so pay attention:

  • for >= measurements — the first packet marks the start of a measurement interval</>. during that interval we count packets and bytes, and the moment we pass the threshold we deliver an upcall and we're done. the first packet after the interval ends resets the count and restarts the measurement. 🔄
  • for <= measurements — we start a timer to fire at the end of the interval, then for each incoming packet we count packets and bytes. when the timer fires, we compare the value with the threshold, schedule an upcall if we're below, and restart the measurement (reschedule timer and zero counters).

the bw_upcall structure is what you hand over as an argument to setsockopt(MRT_ADD_BW_UPCALL) and setsockopt(MRT_DEL_BW_UPCALL). every setsockopt(MRT_ADD_BW_UPCALL) installs a filter in the kernel for the source and destination address in the bw_upcall argument, and that filter triggers an upcall according to this pseudo-algorithm:

if (bw_upcall_oper IS ">=") {
   if (((bw_upcall_unit & PACKETS == PACKETS) &&
        (measured_packets >= threshold_packets)) ||
       ((bw_upcall_unit & BYTES == BYTES) &&
        (measured_bytes >= threshold_bytes)))
      SEND_UPCALL("measured bandwidth is >= threshold");
 }
 if (bw_upcall_oper IS "<=" && measured_interval >= threshold_interval) {
   if (((bw_upcall_unit & PACKETS == PACKETS) &&
        (measured_packets <= threshold_packets)) ||
       ((bw_upcall_unit & BYTES == BYTES) &&
        (measured_bytes <= threshold_bytes)))
      SEND_UPCALL("measured bandwidth is <= threshold");
 }

in the same bw_upcall you can specify the unit in both</> BYTES and PACKETS — that's totally fine, bestie, have both 🎀 but the GEQ and LEQ flags are mutually exclusive. pick a lane. it's giving "one or the other, not both."

basically, an upcall gets delivered if the measured bandwidth is >= or <= the threshold bandwidth (within the specified measurement interval). and for practical reasons the smallest value for the measurement interval is 3 seconds. if smaller values were allowed, the bandwidth estimation could be less accurate, or the potentially very high frequency of the generated upcalls could introduce too much overhead 😭 and here's the cute asymmetry:

  • for the >= operation, the answer may already be known before the end of threshold_interval, so the upcall may be delivered earlier — main character energy, arriving fashionably early ✨
  • for the <= operation though, we simply have to wait until the threshold interval has expired to know the answer. no shortcuts, periodt.

example of usage:

struct bw_upcall bw_upcall;
/* Assign all bw_upcall fields as appropriate */
memset(&bw_upcall, 0, sizeof(bw_upcall));
memcpy(&bw_upcall.bu_src, &source, sizeof(bw_upcall.bu_src));
memcpy(&bw_upcall.bu_dst, &group, sizeof(bw_upcall.bu_dst));
bw_upcall.bu_threshold.b_data = threshold_interval;
bw_upcall.bu_threshold.b_packets = threshold_packets;
bw_upcall.bu_threshold.b_bytes = threshold_bytes;
if (is_threshold_in_packets)
    bw_upcall.bu_flags |= BW_UPCALL_UNIT_PACKETS;
if (is_threshold_in_bytes)
    bw_upcall.bu_flags |= BW_UPCALL_UNIT_BYTES;
do {
    if (is_geq_upcall) {
        bw_upcall.bu_flags |= BW_UPCALL_GEQ;
        break;
    }
    if (is_leq_upcall) {
        bw_upcall.bu_flags |= BW_UPCALL_LEQ;
        break;
    }
    return (ERROR);
} while (0);
setsockopt(mrouter_s4, IPPROTO_IP, MRT_ADD_BW_UPCALL,
          (void *)&bw_upcall, sizeof(bw_upcall));

to delete a single filter, use MRT_DEL_BW_UPCALL, and the fields of bw_upcall must be set exactly</> same as when MRT_ADD_BW_UPCALL was called. same vibes, same struct, zero creative liberties 🙈

to delete all bandwidth filters for a given (S,G), then only the bu_src and bu_dst fields in struct bw_upcall need to be set, and then just set only the BW_UPCALL_DELETE_ALL flag inside field bw_upcall.bu_flags. boom, wiped. 💅

the bandwidth upcalls are received by aggregating them in the new upcall message:

#define IGMPMSG_BW_UPCALL 4 /* BW monitoring upcall */

this message is an array of struct bw_upcall elements (up to BW_UPCALLS_MAX = 128). the upcalls are delivered when there are 128 pending upcalls, or when 1 second has expired since the previous upcall (whichever comes first). in an struct upcall element, the bu_measured field is filled-in to indicate the particular measured values. but heads up — because of the way the particular intervals are measured, the user should be careful how bu_measured.b_time is used. for example, if the filter is installed to trigger an upcall if the number of packets is >= 1, then bu_measured may have a value of zero in the upcalls after the first one, because the measured interval for >= filters is “clocked” by the forwarded packets. hence, this upcall mechanism should not be used for measuring the exact value of the bandwidth of the forwarded data. to measure the exact bandwidth, the user would need to get the forwarded packets statistics with the ioctl(SIOCGETSGCNT) mechanism (see the “Programming Guide” section). 📖

note that the upcalls for a filter are delivered until the specific filter is deleted, but no more frequently than once per bu_threshold.b_time. for example, if the filter is specified to deliver a signal if bw >= 1 packet, the first packet will trigger a signal, but the next upcall will be triggered no earlier than bu_threshold.b_time after the previous upcall.

✨ SEE ALSO ✨

go read these next, bestie, they're the whole extended universe 🦋

🌸 HISTORY 🌸

the Distance Vector Multicast Routing Protocol (DVMRP) was the first</> multicast routing protocol ever developed. later, other protocols such as Multicast Extensions to OSPF (MOSPF) and Core Based Trees (CBT) were developed as well. routers at autonomous system boundaries may now exchange multicast routes with peers via the Border Gateway Protocol (BGP). and many other routing protocols are able to redistribute multicast routes for use with PIM-SM and PIM-DM. the lore is deep 🫶

👑 AUTHORS 👑

giving credit where it's due, because these people built the whole thing:

  • the original multicast code was written by David Waitzman</> (BBN Labs), and later modified by: Steve Deering</> (Stanford), Mark J. Steiglitz</> (Stanford), Van Jacobson</> (LBL), Ajit Thyagarajan</> (PARC), Bill Fenner</> (PARC).
  • the IPv6 multicast support was implemented by the KAME project</> (http://www.kame.net), and was based on the IPv4 multicast code.
  • the advanced multicast API and the multicast bandwidth monitoring were implemented by Pavlin Radoslavov</> (ICSI) in collaboration with Chris Brown</> (NextHop).
  • the IGMPv3 and MLDv2 multicast support was implemented by Bruce Simpson</>.
  • this manual page was written by Pavlin Radoslavov</> (ICSI).

and that's the whole page, bestie, we made it 🎉 this one shipped with Debian</> and the date on it is May 27, 2009</> — it's MULTICAST(4), it's been serving cunt since the aughts, and now you know it top to bottom. go forth and setsockopt responsibly 💖✨

📦 where this page comes from