ovn-architecture(7)

📦

version 26.09.0 · 16 versions

🦋 NAME 🦋

ovn-architecture — the Open Virtual Network architecture, aka the blueprint bestie 💖

🌸 DESCRIPTION 🌸

okay so first things first: OVN, the Open Virtual Network, is a whole system for supporting logical network abstraction in virtual machine and container environments. It complements everything OVS already does by adding native support for logical network abstractions — think logical L2 and L3 overlays, security groups, all of it. And services like DHCP? Also very much desirable. Just like OVS, OVN's design goal is to be a production-quality implementation that can operate at significant scale. no cap. 💅

A physical network is made of physical wires, switches, and routers. A virtual network extends a physical network into a hypervisor or container platform, bridging VMs or containers into the physical network. An OVN logical network is a network implemented in software that is insulated from physical (and therefore also virtual) networks by tunnels or other encapsulations. This is the iconic part: it lets IP and other address spaces used in logical networks overlap with the ones used on physical networks without causing conflicts ✨. Logical network topologies can be arranged however you want, without caring about the topologies of the physical networks they run on. So VMs that are part of a logical network can migrate from one physical machine to another without network disruption. See Logical Networks, below, for more — iykyk.

The encapsulation layer prevents VMs and containers connected to a logical network from communicating with nodes on physical networks. For clustering VMs and containers, that can be acceptable or even desirable, but in many cases VMs and containers do need connectivity to physical networks. OVN provides multiple forms of gateways for exactly this. See Gateways, below, for more. 🫶

An OVN deployment consists of several components:

A Cloud Management System (CMS), which is OVN's ultimate client (via its users and administrators). OVN integration requires installing a CMS-specific plugin and related software (see below). OVN initially targets OpenStack as CMS.

We generally speak of “the” CMS, but like, one can imagine scenarios in which multiple CMSes manage different parts of an OVN deployment 👀.

An OVN Database physical or virtual node (or, eventually, cluster) installed in a central location.
One or more (usually many) hypervisors. Hypervisors must run Open vSwitch and implement the interface described in Documentation/topics/integration.rst in the Open vSwitch source tree. Any hypervisor platform supported by Open vSwitch is acceptable. periodt.
Zero or more gateways. A gateway extends a tunnel-based logical network into a physical network by bidirectionally forwarding packets between tunnels and a physical Ethernet port. This allows non-virtualized machines to participate in logical networks. A gateway may be a physical host, a virtual machine, or an ASIC-based hardware switch that supports the vtep(5) schema. ✨

Hypervisors and gateways are together called transport node or chassis.

The diagram below shows how the major components of OVN and related software interact. Starting at the top of the diagram, we have:

The Cloud Management System, as defined above.
The OVN/CMS Plugin is the component of the CMS that interfaces to OVN. In OpenStack, this is a Neutron plugin. The plugin's main purpose is to translate the CMS's notion of logical network configuration — stored in the CMS's configuration database in a CMS-specific format — into an intermediate representation understood by OVN.

This component is necessarily CMS-specific, so a new plugin needs to be developed for each CMS that is integrated with OVN. All of the components below this one in the diagram are CMS-independent. 🧚

The OVN Northbound Database receives the intermediate representation of logical network configuration passed down by the OVN/CMS Plugin. The database schema is meant to be “impedance matched” with the concepts used in a CMS, so that it directly supports notions of logical switches, routers, ACLs, and so on. See ovn-nb(5) for details.

The OVN Northbound Database has only two clients: the OVN/CMS Plugin above it and ovn-northd below it.

ovn-northd(8) connects to the OVN Northbound Database above it and the OVN Southbound Database below it. It translates the logical network configuration in terms of conventional network concepts, taken from the OVN Northbound Database, into logical datapath flows in the OVN Southbound Database below it. 💫
The OVN Southbound Database is the center of the whole system. Its clients are ovn-northd(8) above it and ovn-controller(8) on every transport node below it.

The OVN Southbound Database contains three kinds of data: Physical Network (PN) tables that specify how to reach hypervisor and other nodes, Logical Network (LN) tables that describe the logical network in terms of “logical datapath flows,” and Binding tables that link logical network components' locations to the physical network. The hypervisors populate the PN and Port_Binding tables, whereas ovn-northd(8) populates the LN tables.

OVN Southbound Database performance must scale with the number of transport nodes. This will likely require some work on ovsdb-server(1) as we encounter bottlenecks. Clustering for availability may be needed. 😭

The remaining components are replicated onto each hypervisor:

ovn-controller(8) is OVN's agent on each hypervisor and software gateway. Northbound, it connects to the OVN Southbound Database to learn about OVN configuration and status and to populate the PN table and the Chassis column in Binding table with the hypervisor's status. Southbound, it connects to ovs-vswitchd(8) as an OpenFlow controller, for control over network traffic, and to the local ovsdb-server(1) to allow it to monitor and control Open vSwitch configuration.
ovs-vswitchd(8) and ovsdb-server(1) are conventional components of Open vSwitch.

                                         CMS
                                          |
                                          |
                              +-----------|-----------+
                              |           |           |
                              |     OVN/CMS Plugin    |
                              |           |           |
                              |           |           |
                              |   OVN Northbound DB   |
                              |           |           |
                              |           |           |
                              |       ovn-northd      |
                              |           |           |
                              +-----------|-----------+
                                          |
                                          |
                                +-------------------+
                                | OVN Southbound DB |
                                +-------------------+
                                          |
                                          |
                       +------------------+------------------+
                       |                  |                  |
         HV 1          |                  |    HV n          |
       +---------------|---------------+  .  +---------------|---------------+
       |               |               |  .  |               |               |
       |        ovn-controller         |  .  |        ovn-controller         |
       |         |          |          |  .  |         |          |          |
       |         |          |          |     |         |          |          |
       |  ovs-vswitchd   ovsdb-server  |     |  ovs-vswitchd   ovsdb-server  |
       |                               |     |                               |
       +-------------------------------+     +-------------------------------+

✨ Information Flow in OVN ✨

Configuration data in OVN flows north to south ⬇️. The CMS, through its OVN/CMS plugin, passes the logical network configuration to ovn-northd via the northbound database. In turn, ovn-northd compiles the configuration into a lower-level form and passes it to all of the chassis via the southbound database.

Status information in OVN flows south to north ⬆️. OVN currently provides only a few forms of status information. First, ovn-northd populates the up column in the northbound Logical_Switch_Port table: if a logical port's chassis column in the southbound Port_Binding table is nonempty, it sets up to true, otherwise to false. This allows the CMS to detect when a VM's networking has come up. 👑

Second, OVN provides feedback to the CMS on the realization of its configuration — that is, whether the configuration provided by the CMS has actually taken effect. This feature requires the CMS to participate in a sequence number protocol, which works the following way:

When the CMS updates the configuration in the northbound database, as part of the same transaction, it increments the value of the nb_cfg column in the NB_Global table. (This is only necessary if the CMS wants to know when the configuration has been realized.)
When ovn-northd updates the southbound database based on a given snapshot of the northbound database, it copies nb_cfg from northbound NB_Global into the southbound database SB_Global table, as part of the same transaction. (Thus, an observer monitoring both databases can determine when the southbound database is caught up with the northbound.) 🌈
Once ovn-northd gets the 💌 confirmation from the southbound database server that its changes have committed, it updates sb_cfg in the northbound NB_Global table to the nb_cfg version that just got pushed down. (So the CMS, or honestly anyone else watching 👀, can figure out when the southbound database has caught up without needing a connection to the southbound database. iconic.)
The ovn-controller process on every chassis picks up the updated southbound database, with the updated nb_cfg in it. That process then goes and updates the physical flows installed in the chassis's Open vSwitch instances. When Open vSwitch confirms back that the physical flows have been updated, ovn-controller updates nb_cfg in its own Chassis_Private record in the southbound database. (OVN versions before v20.09.0 used the nb_cfg column from the Chassis table instead. That column is still there, but bestie, it's deprecated in favor of the same column in Chassis_Private and it is no longer updated.)
ovn-northd monitors the nb_cfg column in all of the Chassis_Private records in the southbound database. It keeps track of the minimum value among all those records and copies it into the hv_cfg column in the northbound NB_Global table. (So again, the CMS or another observer can tell when all the hypervisors have caught up to the northbound configuration. group project but everyone actually did their part 💅)

🔧 Chassis Setup 🔧

Every chassis in an OVN deployment has to be configured with an Open vSwitch bridge dedicated to OVN's use, and that bridge is called the integration bridge. System startup scripts are allowed to create this bridge before starting ovn-controller if they feel like it. But if the bridge doesn't exist yet when ovn-controller starts, no worries — it gets created automatically with the default configuration suggested down below. 🫶

The ports on the integration bridge include:

On any chassis, tunnel ports that OVN uses to keep logical network connectivity alive. ovn-controller adds, updates, and removes these tunnel ports.
On a hypervisor, any VIFs that are to be attached to logical networks. For instances connected through software emulated ports such as TUN/TAP or VETH pairs, the hypervisor itself will normally create the ports and plug them into the integration bridge. For instances connected through representor ports, typically used with hardware offload, the ovn-controller may — on CMS direction — consult a VIF plug provider for representor port lookup and plug them into the integration bridge (please go read Documentation/topics/vif-plug-providers/vif-plug-providers.rst for more). In both cases the conventions described in Documentation/topics/integration.rst in the Open vSwitch source tree are followed, so that the mapping between OVN logical port and VIF stays correct. (This is pre-existing integration work that's already been done on hypervisors that support OVS. not us reinventing anything 🙈)
On a gateway, the physical port used for logical network connectivity. System startup scripts add this port to the bridge before starting ovn-controller. In more sophisticated setups, this can be a patch port to another bridge instead of a physical port.

Other ports should not be attached to the integration bridge. Specifically: physical ports attached to the underlay network (which is not the same thing as gateway ports, which are physical ports attached to logical networks) must not be attached to the integration bridge. Underlay physical ports should instead be attached to a separate Open vSwitch bridge — and honestly they need not be attached to any bridge at all, in fact.

The integration bridge should be configured as described below. The effect of each of these settings is documented in ovs-vswitchd.conf.db(5):

fail-mode=secure — Avoids switching packets between isolated logical networks before ovn-controller starts up. See Controller Failure Settings in ovs-vsctl(8) for more info.
other-config:disable-in-band=true — Suppresses in-band control flows for the integration bridge. It would be unusual for such flows to show up anyway, because OVN uses a local controller (over a Unix domain socket) instead of a remote controller. It's possible, however, for some other bridge in the same system to have an in-band remote controller, and in that case this suppresses the flows that in-band control would ordinarily set up. Refer to the documentation for more.

The customary name for the integration bridge is br-int, but another name may be used. (she answers to br-int but she's flexible 💖)

🌐 Logical Networks 🌐

Logical network concepts in OVN include logical switches and logical routers — the logical versions of Ethernet switches and IP routers, respectively. Just like their physical cousins, logical switches and routers can be connected into sophisticated topologies. And they're ordinarily purely logical entities, meaning they are not associated or bound to any physical location, and they're implemented in a distributed manner at each hypervisor that participates in OVN. no cap, fully distributed. ✨

Logical switch ports (LSPs) are the points of connectivity into and out of logical switches. There are many kinds of logical switch ports. The most ordinary kind represent VIFs, that is, attachment points for VMs or containers. A VIF logical port is associated with the physical location of its VM, which might change as the VM migrates. (And a VIF logical port can be associated with a VM that is powered down or suspended. Such a logical port has no location and no connectivity. it's giving ghost 👻)

Logical router ports (LRPs) are the points of connectivity into and out of logical routers. An LRP connects a logical router either to a logical switch or to another logical router. Logical routers only connect to VMs, containers, and other network nodes indirectly, through logical switches.

Logical switches and logical routers have distinct kinds of logical ports, so properly speaking you should usually say logical switch port or logical router port. However, an unqualified "logical port" usually refers to a logical switch port.

When a VM sends a packet to a VIF logical switch port, the Open vSwitch flow tables simulate the packet's journey through that logical switch and any other logical routers and logical switches that it might encounter. And this happens without transmitting the packet across any physical medium — the flow tables implement all of the switching and routing decisions and behavior. If the flow tables ultimately decide to output the packet at a logical port attached to another hypervisor (or another kind of transport node), then that's the moment the packet gets encapsulated for physical network transmission and sent. periodt. 💫

🎀 Logical Switch Port Types 🎀

OVN supports a number of kinds of logical switch ports. The VIF ports that connect to VMs or containers, described above, are the most ordinary kind of LSP. In the OVN northbound database, VIF ports have an empty string for their type. This section describes some of the additional port types.

A router logical switch port connects a logical switch to a logical router, designating a particular LRP as its peer.

A localnet logical switch port bridges a logical switch to a physical VLAN. A logical switch may have one or more localnet ports. Such a logical switch is used in two scenarios:

With one or more router logical switch ports, to attach L3 gateway routers and distributed gateways to a physical network.
With one or more VIF logical switch ports, to attach VMs or containers directly to a physical network. In this case, the logical switch is not really logical, since it's bridged to the physical network rather than insulated from it, and therefore it cannot have independent but overlapping IP address namespaces, etc. A deployment might nevertheless choose such a configuration to take advantage of the OVN control plane and features such as port security and ACLs. (we love a pragmatic queen 🫶)

When a logical switch contains multiple localnet ports, the following is assumed:

Each chassis has a bridge mapping for one of the localnet physical networks only.
To facilitate interconnectivity between VIF ports of the switch that are located on different chassis with different physical network connectivity, the fabric implements L3 routing between these adjacent physical network segments.

Note: nothing said above implies that a chassis cannot be plugged to multiple physical networks — as long as they belong to different switches. 🌈

A localport logical switch port is a special kind of VIF logical switch port. These ports are present in every chassis, not bound to any particular one. Traffic to such a port will never be forwarded through a tunnel, and traffic from such a port is expected to be destined only to the same chassis, typically in response to a request it received. OpenStack Neutron uses a localport port to serve metadata to VMs. A metadata proxy process is attached to this port on every host, and all VMs within the same network will reach it at the same IP/MAC address without any traffic being sent over a tunnel. For further details, see the OpenStack documentation for networking-ovn. 📚

LSP types vtep and l2gateway are used for gateways. See Gateways, below, for more information.

🔍 Implementation Details 🔍

These concepts are details of how OVN is implemented internally. They might still be of interest to users and administrators, though. iykyk. 👑

Logical datapaths are an implementation detail of logical networks in the OVN southbound database. ovn-northd translates each logical switch or router in the northbound database into a logical datapath in the southbound database Datapath_Binding table.

Okay so for the most part, ovn-northd also translates each logical switch port in the OVN northbound database into a record in the southbound database Port_Binding table. That table is lowkey the mirror-ish cousin of the northbound Logical_Switch_Port table. It’s giving “many types of logical port bindings,” and a bunch of those types line up directly with northbound LSP types. The LSP types handled this way include VIF (which is the empty string), localnet, localport, vtep, and l2gateway. Periodt.

The Port_Binding table has some types of port binding that do not correspond directly to logical switch port types. The common one is patch port bindings, aka logical patch ports. These always come in pairs, and a packet that enters on either side comes out on the other. ovn-northd connects logical switches and logical routers together using logical patch ports. No cap.

Port bindings with types vtep, l2gateway, l3gateway, and chassisredirect are used for gateways. They’re explained in Gateways, below. 🫶

Gateways 🌉

Gateways give you limited connectivity between logical networks and physical ones. They can also connect different OVN deployments. This section is focused on the first thing; the second thing gets described in detail in section OVN Deployments Interconnection. 💅

OVN supports multiple kinds of gateways. Slay.

VTEP Gateways 🛡️

A “VTEP gateway” connects an OVN logical network to a physical (or virtual) switch that implements the OVSDB VTEP schema that comes with Open vSwitch. (The term “VTEP gateway” is a misnomer, because a VTEP is just a VXLAN Tunnel Endpoint, but it’s a well-established name.) See Life Cycle of a VTEP gateway, below, for more info. ✨

The main intended use case for VTEP gateways is to attach physical servers to an OVN logical network using a physical top-of-rack switch that supports the OVSDB VTEP schema. Obsessed.

L2 Gateways 🎀

A L2 gateway just attaches a designated physical L2 segment available on some chassis to a logical network. The physical network effectively becomes part of the logical network. Iconic.

To set up a L2 gateway, the CMS adds an l2gateway LSP to an appropriate logical switch, setting LSP options to name the chassis on which it should be bound. ovn-northd copies this configuration into a southbound Port_Binding record. On the designated chassis, ovn-controller forwards packets appropriately to and from the physical segment. 💫

L2 gateway ports have features in common with localnet ports. But with a localnet port, the physical network becomes the transport between hypervisors. With an L2 gateway, packets are still transported between hypervisors over tunnels, and the l2gateway port is only used for packets that are on the physical network. The application for L2 gateways is similar to that for VTEP gateways, e.g. to add non-virtualized machines to a logical network, but L2 gateways do not require special support from top-of-rack hardware switches. Periodt.

L3 Gateway Routers 👑

As described above under Logical Networks, ordinary OVN logical routers are distributed: they are not implemented in a single place, but rather in every hypervisor chassis. This is a problem for stateful services like SNAT and DNAT, which need to be implemented in a centralized manner. 😭

To allow for this kind of functionality, OVN supports L3 gateway routers, which are OVN logical routers implemented in a designated chassis. Gateway routers are typically used between distributed logical routers and physical networks. The distributed logical router and the logical switches behind it, to which VMs and containers attach, effectively reside on each hypervisor. The distributed router and the gateway router are connected by another logical switch, sometimes referred to as a “join” logical switch. (OVN logical routers may be connected to one another directly, without an intervening switch, but the OVN implementation only supports gateway logical routers that are connected to logical switches. Using a join logical switch also reduces the number of IP addresses needed on the distributed router.) On the other side, the gateway router connects to another logical switch that has a localnet port connecting to the physical network. 💖

The following diagram shows a typical situation. One or more logical switches LS1, ..., LSn connect to distributed logical router LR1, which in turn connects through LSjoin to gateway logical router GLR, which also connects to logical switch LSlocal, which includes a localnet port to attach to the physical network. 🌈

                                       LSlocal
                                          |
                                         GLR
                                          |
                                       LSjoin
                                          |
                                         LR1
                                          |
                                     +----+----+
                                     |    |    |
                                    LS1  ...  LSn

To configure an L3 gateway router, the CMS sets options:chassis in the router’s northbound Logical_Router to the chassis’s name. In response, ovn-northd uses a special l3gateway port binding (instead of a patch binding) in the southbound database to connect the logical router to its neighbors. In turn, ovn-controller tunnels packets to this port binding to the designated L3 gateway chassis, instead of processing them locally. 🔥

DNAT and SNAT rules may be associated with a gateway router, which provides a central location that can handle one-to-many SNAT (aka IP masquerading). Distributed gateway ports, described below, also support NAT. 💅

Distributed Gateway Ports 🦋

A distributed gateway port is a logical router port that is specially configured to designate one distinguished chassis, called the gateway chassis, for centralized processing. A distributed gateway port should connect to a logical switch that has an LSP that connects externally, that is, either a localnet LSP or a connection to another OVN deployment (see OVN Deployments Interconnection). Packets that traverse the distributed gateway port are processed without involving the gateway chassis when they can be, but when needed they do take an extra hop through it. No cap.

The following diagram illustrates the use of a distributed gateway port. A number of logical switches LS1, ..., LSn connect to distributed logical router LR1, which in turn connects through the distributed gateway port to logical switch LSlocal that includes a localnet port to attach to the physical network. 🌸

                                       LSlocal
                                          |
                                         LR1
                                          |
                                     +----+----+
                                     |    |    |
                                    LS1  ...  LSn

ovn-northd creates two southbound Port_Binding records to represent a distributed gateway port, instead of the usual one. One of these is a patch port binding named for the LRP, which is used for as much traffic as it can. The other one is a port binding with type chassisredirect, named cr-port. The chassisredirect port binding has one specialized job: when a packet is output to it, the flow table causes it to be tunneled to the gateway chassis, at which point it is automatically output to the patch port binding. Thus, the flow table can output to this port binding in cases where a particular task has to happen on the gateway chassis. The chassisredirect port binding is not otherwise used (for example, it never receives packets). 💀

The CMS may configure distributed gateway ports three different ways. See Distributed Gateway Ports in the documentation for Logical_Router_Port in ovn-nb(5) for details. 📚

Distributed gateway ports support high availability. When more than one chassis is specified, OVN only uses one at a time as the gateway chassis. OVN uses BFD to monitor gateway connectivity, preferring the highest-priority gateway that is online. 👑

A logical router can have multiple distributed gateway ports, each connecting different external networks. Load balancing is not yet supported for logical routers with more than one distributed gateway port configured. 🚫

Physical VLAN MTU Issues 😵‍💫

Consider the preceding diagram again:

                                       LSlocal
                                          |
                                         LR1
                                          |
                                     +----+----+
                                     |    |    |
                                    LS1  ...  LSn

Suppose that each logical switch LS1, ..., LSn is bridged to a physical VLAN-tagged network attached to a localnet port on LSlocal, over a distributed gateway port on LR1. If a packet originating on LSi is destined to the external network, OVN sends it to the gateway chassis over a tunnel. There, the packet traverses LR1’s logical router pipeline, possibly undergoes NAT, and eventually ends up at LSlocal’s localnet port. If all of the physical links in the network have the same MTU, then the packet’s transit across a tunnel causes an MTU problem: tunnel overhead prevents a packet that uses the full physical MTU from crossing the tunnel to the gateway chassis (without fragmentation). 🫠

OVN offers two solutions to this problem, the reside-on-redirect-chassis and redirect-type options. Both solutions require each logical switch LS1, ..., LSn to include a localnet logical switch port LN1, ..., LNn respectively, that is present on each chassis. Both cause packets to be sent over the localnet ports instead of tunnels. They differ in which packets—some or all—are sent this way. The most prominent tradeoff between these options is that reside-on-redirect-chassis is easier to configure and that redirect-type performs better for east-west traffic. ⚖️

The first solution is the reside-on-redirect-chassis option for logical router ports. Setting this option on a LRP from (e.g.) LS1 to LR1 disables forwarding from LS1 to LR1 except on the gateway chassis. On chassis other than the gateway chassis, this single change means that packets that would otherwise have been forwarded to LR1 are instead forwarded to LN1. The instance of LN1 on the gateway chassis then receives the packet and forwards it to LR1. The packet traverses the LR1 logical router pipeline, possibly undergoes NAT, and eventually ends up at LSlocal’s localnet port. The packet never traverses a tunnel, avoiding the MTU issue. 💅

okay so this option has one more lowkey unhinged side effect: it centralizes the "distributed" logical router LR1, because literally no packets get forwarded from LS1 to LR1 on any chassis other than the gateway chassis. So east-west traffic ends up going through the gateway chassis too, not just north-south. 😭 And bestie, do not try the naive "fix" of letting east-west traffic flow directly between chassis over LN1 — that does not work, because routing sets the Ethernet source address to LR1's source address. Seeing that single Ethernet source address originate from all of the chassis will confuse the physical switch. 💀

Do not set the reside-on-redirect-chassis option on a distributed gateway port. In the diagram above, it would be set on the LRPs connecting LS1, ..., LSn to LR1.

The second solution is the redirect-type option for distributed gateway ports. Setting this option to bridged makes packets that are redirected to the gateway chassis go over the localnet ports instead of being tunneled. Important: this option does not change how OVN treats packets that are not redirected to the gateway chassis.

The redirect-type option requires the administrator or the CMS to configure each participating chassis with a unique Ethernet address for the logical router by setting ovn-chassis-mac-mappings in the Open vSwitch database, for use by ovn-controller. ngl that makes it more difficult to configure than reside-on-redirect-chassis.

Set the redirect-type option on a distributed gateway port. 🫶

✨ Using Distributed Gateway Ports For Scalability ✨

So distributed gateway ports are mostly about reaching external networks, but there's a special use case for scalability too. 👑

In some deployments — like the ones using ovn-kubernetes — logical switches are bound to individual chassises, and they're connected by a distributed logical router. In those deployments, the chassis level logical switches are centralized on the chassis instead of distributed, which means the ovn-controller on each chassis doesn't need to process flows and ports of logical switches on other chassises. iconic, right? But without any specific hint, ovn-controller would still process all the logical switches as if they are fully distributed. 🙈

In this case, distributed gateway port can be very useful. The chassis level logical switches can be connected to the distributed router using distributed gateway ports, by setting the gateway chassis (or HA chassis groups with only a single chassis in it) to the chassis that each logical switch is bound to. Then ovn-controller would skip processing the logical switches on all the other chassises, largely improving the scalability — especially when there are a big number of chassises. 🦋

🌸 Life Cycle of a VIF 🌸

Tables and their schemas presented in isolation are difficult to understand. Here's an example. 💅

A VIF on a hypervisor is a virtual network interface attached either to a VM or a container running directly on that hypervisor. (This is different from the interface of a container running inside a VM.)

heads up: the steps in this example refer often to details of the OVN and OVN Northbound database schemas. Please see ovn-sb(5) and ovn-nb(5), respectively, for the full story on these databases. 📚

A VIF's life cycle begins when a CMS administrator creates a new VIF using the CMS user interface or API and adds it to a switch (one implemented by OVN as a logical switch). The CMS updates its own configuration. This includes associating unique, persistent identifier vif-id and Ethernet address mac with the VIF.
The CMS plugin updates the OVN Northbound database to include the new VIF, by adding a row to the Logical_Switch_Port table. In the new row, name is vif-id, mac is mac, switch points to the OVN logical switch's Logical_Switch record, and other columns are initialized appropriately.
ovn-northd receives the OVN Northbound database update. In turn, it makes the corresponding updates to the OVN Southbound database, by adding rows to the OVN Southbound database Logical_Flow table to reflect the new port, e.g. add a flow to recognize that packets destined to the new port's MAC address should be delivered to it, and update the flow that delivers broadcast and multicast packets to include the new port. It also creates a record in the Binding table and populates all its columns except the column that identifies the chassis.
On every hypervisor, ovn-controller receives the Logical_Flow table updates that ovn-northd made in the previous step. As long as the VM that owns the VIF is powered off, ovn-controller cannot do much; it cannot, for example, arrange to send packets to or receive packets from the VIF, because the VIF does not actually exist anywhere.
Eventually, a user powers on the VM that owns the VIF. On the hypervisor where the VM is powered on, the integration between the hypervisor and Open vSwitch (described in Documentation/topics/integration.rst in the Open vSwitch source tree) adds the VIF to the OVN integration bridge and stores vif-id in external_ids:iface-id to indicate that the interface is an instantiation of the new VIF. (None of this code is new in OVN; this is pre-existing integration work that has already been done on hypervisors that support OVS.)
On the hypervisor where the VM is powered on, ovn-controller notices external_ids:iface-id in the new Interface. In response, in the OVN Southbound DB, it updates the Binding table's chassis column for the row that links the logical port from external_ids: iface-id to the hypervisor. Afterward, ovn-controller updates the local hypervisor's OpenFlow tables so that packets to and from the VIF are properly handled.
Some CMS systems, including OpenStack, fully start a VM only when its networking is ready. To support this, ovn-northd notices the chassis column updated for the row in Binding table and pushes this upward by updating the up column in the OVN Northbound database's Logical_Switch_Port table to indicate that the VIF is now up. The CMS, if it uses this feature, can then react by allowing the VM's execution to proceed.
On every hypervisor but the one where the VIF resides, ovn-controller notices the completely populated row in the Binding table. This provides ovn-controller the physical location of the logical port, so each instance updates the OpenFlow tables of its switch (based on logical datapath flows in the OVN DB Logical_Flow table) so that packets to and from the VIF can be properly handled via tunnels.
Eventually, a user powers off the VM that owns the VIF. On the hypervisor where the VM was powered off, the VIF is deleted from the OVN integration bridge.
On the hypervisor where the VM was powered off, ovn-controller notices that the VIF was deleted. In response, it removes the Chassis column content in the Binding table for the logical port.
On every hypervisor, ovn-controller notices the empty Chassis column in the Binding table's row for the logical port. This means that ovn-controller no longer knows the physical location of the logical port, so each instance updates its OpenFlow table to reflect that.
Eventually, when the VIF (or its entire VM) is no longer needed by anyone, an administrator deletes the VIF using the CMS user interface or API. The CMS updates its own configuration.
The CMS plugin removes the VIF from the OVN Northbound database, by deleting its row in the Logical_Switch_Port table.
ovn-northd receives the OVN Northbound update and in turn updates the OVN Southbound database accordingly, by removing or updating the rows from the OVN Southbound database Logical_Flow table and Binding table that were related to the now-destroyed VIF.
On every hypervisor, ovn-controller receives the Logical_Flow table updates that ovn-northd made in the previous step. ovn-controller updates OpenFlow tables to reflect the update, although there may not be much to do, since the VIF had already become unreachable when it was removed from the Binding table in a previous step.

🎀 Life Cycle of a Container Interface Inside a VM 🎀

OVN provides virtual network abstractions by converting information written in OVN_NB database to OpenFlow flows in each hypervisor. Secure virtual networking for multi-tenants can only be provided if OVN controller is the only entity that can modify flows in Open vSwitch. When the Open vSwitch integration bridge resides in the hypervisor, it is a fair assumption to make that tenant workloads running inside VMs cannot make any changes to Open vSwitch flows. 🔒

If the infrastructure provider trusts the applications inside the containers not to break out and modify the Open vSwitch flows, then containers can be run in hypervisors. This is also the case when containers are run inside the VMs and Open vSwitch integration bridge with flows added by OVN controller resides in the same VM. For both the above cases, the workflow is the same as explained with an example in the previous section ("Life Cycle of a VIF"). ✨

This section talks about the life cycle of a container interface (CIF) when containers are created in the VMs and the Open vSwitch integration bridge resides inside the hypervisor. In this case, even if a container application breaks out, other tenants are not affected, because the containers running inside the VMs cannot modify the flows in the Open vSwitch integration bridge. 🫶

When multiple containers are created inside a VM, there are multiple CIFs associated with them. The network traffic associated with these CIFs need to reach the Open vSwitch integration bridge running in the hypervisor for OVN to support virtual network abstractions. OVN should also be able to distinguish network traffic coming from different CIFs. There are two ways to distinguish network traffic of CIFs.

Okay bestie, so we gotta pick how we're gonna do the container-to-network thing, and there are literally two options:

One way is to give every single CIF its own VIF (the 1:1 model). Sounds cute, but bestie, that means your hypervisor could end up with a ton of network devices. That slows OVS <em>all</em> the way down because managing all those VIFs eats CPU cycles like it's nothing. Also 💀 it means whoever's out there spawning containers inside a VM also has to be able to create the matching VIFs up in the hypervisor. Rude.

The second way is one single VIF for all the CIFs (the 1:many model). OVN can then tell traffic from different CIFs apart by a tag written into every packet. OVN uses this mechanism and uses VLAN as the tagging mechanism. slay ✨

A CIF's life cycle begins when a container is spawned inside a VM by either the same CMS that created the VM, or a tenant that owns that VM, or even a container Orchestration System that's different from the CMS that initially created the VM. Whoever the entity is, it needs to know the vif-id that's associated with the network interface of the VM through which the container interface's network traffic is expected to go. The entity that creates the container interface also needs to pick an unused VLAN inside that VM. 💅
The container spawning entity (either directly or through the CMS that manages the underlying infrastructure) updates the OVN Northbound database to include the new CIF, by adding a row to the Logical_Switch_Port table. In that new row, name is any unique identifier, parent_name is the vif-id of the VM through which the CIF's network traffic is expected to go, and tag is the VLAN tag that identifies the network traffic of that CIF. no cap.
ovn-northd receives the OVN Northbound database update. In turn, it makes the corresponding updates to the OVN Southbound database, by adding rows to the OVN Southbound database's Logical_Flow table to reflect the new port, and also by creating a new row in the Binding table and populating <em>all</em> its columns except the column that identifies the chassis. 👑
On every hypervisor, ovn-controller subscribes to the changes in the Binding table. When a new row is created by ovn-northd that includes a value in the parent_port column of the Binding table, the ovn-controller in the hypervisor whose OVN integration bridge has that same value in vif-id in external_ids:iface-id updates the local hypervisor's OpenFlow tables so that packets to and from the VIF with the particular VLAN tag are properly handled. Afterward it updates the chassis column of the Binding to reflect the physical location. 🦋
You can only start the application inside the container after the underlying network is ready. To support this, ovn-northd notices the updated chassis column in the Binding table and updates the up column in the OVN Northbound database's Logical_Switch_Port table to indicate that the CIF is now up. The entity responsible to start the container application queries this value and starts the application. ✨
Eventually the entity that created and started the container stops it. The entity, through the CMS (or directly) deletes its row in the Logical_Switch_Port table. bye bestie 👋
ovn-northd receives the OVN Northbound update and in turn updates the OVN Southbound database accordingly, by removing or updating the rows from the OVN Southbound database Logical_Flow table that were related to the now-destroyed CIF. It also deletes the row in the Binding table for that CIF.
On every hypervisor, ovn-controller receives the Logical_Flow table updates that ovn-northd made in the previous step. ovn-controller updates OpenFlow tables to reflect the update. 💫

🌈 Architectural Physical Life Cycle of a Packet 🌈

This section describes how a packet travels from one virtual machine or container to another through OVN. This description focuses on the physical treatment of a packet; for a description of the logical life cycle of a packet, please refer to the Logical_Flow table in ovn-sb(5).

This section mentions several data and metadata fields, and for clarity they're all summarized right here, bestie:

tunnel key — When OVN encapsulates a packet in Geneve or another tunnel, it attaches extra data to it to allow the receiving OVN instance to process it correctly. This takes different forms depending on the particular encapsulation, but in each case we refer to it here as the "tunnel key." See Tunnel Encapsulations, below, for details.
logical datapath field — A field that denotes the logical datapath through which a packet is being processed. OVN uses the field that OpenFlow 1.1+ simply (and confusingly 😭) calls "metadata" to store the logical datapath. (This field is passed across tunnels as part of the tunnel key.)
logical input port field — A field that denotes the logical port from which the packet entered the logical datapath. OVN stores this in Open vSwitch extension register number 14.

Geneve tunnels pass this field as part of the tunnel key. Ramp switch VXLAN tunnels do not explicitly carry a logical input port, but since they are used to communicate with gateways that from OVN's perspective consist of only a single logical port, so that OVN can set the logical input port field to this one on ingress to the OVN logical pipeline. As for regular VXLAN tunnels, they don't carry input port field at all. This puts additional limitations on cluster capabilities that are described in the Tunnel Encapsulations section.

logical output port field — A field that denotes the logical port from which the packet will leave the logical datapath. This is initialized to 0 at the beginning of the logical ingress pipeline. OVN stores this in Open vSwitch extension register number 15.

Geneve and regular VXLAN tunnels pass this field as part of the tunnel key. Ramp switch VXLAN tunnels do not transmit the logical output port field, and since they do not carry a logical output port field in the tunnel key, when a packet is received from a ramp switch VXLAN tunnel by an OVN hypervisor, the packet is resubmitted to table 8 to determine the output port(s); when the packet reaches table 54, these packets are resubmitted to table 56 for local delivery by checking a MLF_RCV_FROM_RAMP flag, which is set when the packet arrives from a ramp tunnel.

conntrack zone field for logical ports — A field that denotes the connection tracking zone for logical ports. The value only has local significance and is not meaningful between chassis. This is initialized to 0 at the beginning of the logical ingress pipeline. OVN stores this in the lower 16 bits of the Open vSwitch extension register number 13.
conntrack zone fields for routers — Fields that denote the connection tracking zones for routers. These values only have local significance and are not meaningful between chassis. OVN stores the zone information for north to south traffic (for DNATting or ECMP symmetric replies) in Open vSwitch extension register number 11 and zone information for south to north traffic (for SNATing) in Open vSwitch extension register number 12.
Encap ID for logical ports — A field that records an ID that indicates which encapsulation IP should be used when sending packets to a remote chassis, according to the original input logical port. This is useful when there are multiple IPs available for encapsulation. The value only has local significance and is not meaningful between chassis. This is initialized to 0 at the beginning of the logical ingress pipeline. OVN stores this in the higher 16 bits of the Open vSwitch extension register number 13.
logical flow flags — The logical flags are intended to handle keeping context between tables in order to decide which rules in subsequent tables are matched. These values only have local significance and are not meaningful between chassis. OVN stores the logical flags in Open vSwitch extension register number 10.
VLAN ID — The VLAN ID is used as an interface between OVN and containers nested inside a VM (see Life Cycle of a container interface inside a VM, above, for more information).

Initially, a VM or container on the ingress hypervisor sends a packet on a port attached to the OVN integration bridge. Then: 🚀

OpenFlow table 0 performs physical-to-logical translation. It matches the packet's ingress port. Its actions annotate the packet with logical metadata, by setting the logical datapath field to identify the logical datapath that the packet is traversing and the logical input port field to identify the ingress port. Then it resubmits to table 8 to enter the logical ingress pipeline.

Packets that originate from a container nested within a VM are treated in a slightly different way. The originating container can be distinguished based on the VIF-specific VLAN ID, so the physical-to-logical translation flows additionally match on VLAN ID and the actions strip the VLAN header. Following this step, OVN treats packets from containers just like any other packets. ✨

Table 0 also handles packets that roll in from other chassis, bestie — and the way it tells them apart is by their ingress port, which is a tunnel. Same energy as packets just entering the OVN pipeline: the actions stamp them with logical datapath metadata. And for the tunnel types that support it, they also get annotated with logical ingress port metadata 🎀. On top of that, the actions set the logical output port field — which is available here literally because in OVN, tunneling happens after the logical output port is already known. All of that info comes out of the tunnel encapsulation metadata (peep Tunnel Encapsulations for the encoding deets). Then the actions resubmit to table 62 to enter the logical egress pipeline. Periodt. ✨

OpenFlow tables 8 through 41 run the logical ingress pipeline from the Logical_Flow table in the OVN Southbound database. These tables are expressed entirely in terms of logical concepts like logical ports and logical datapaths. A huge chunk of ovn-controller's whole job is translating them into equivalent OpenFlow — and yeah, that includes translating the table numbers: Logical_Flow tables 0 through 33 become OpenFlow tables 8 through 41 💅.

Each logical flow maps to one or more OpenFlow flows. A real packet ordinarily matches only one of these, though sometimes it can match more than one — which is not a problem, no cap, because all of them have the same actions. ovn-controller uses the first 32 bits of the logical flow's UUID as the cookie for its OpenFlow flow or flows. (This is not necessarily unique, bestie, since the first 32 bits of a logical flow's UUID is not necessarily unique. 🙈)

Some logical flows can map to the Open vSwitch conjunctive match extension (see ovs-fields(7)). Flows with a conjunction action use an OpenFlow cookie of 0, because they can correspond to multiple logical flows. The OpenFlow flow for a conjunctive match includes a match on conj_id.

Some logical flows may not be represented in the OpenFlow tables on a given hypervisor, if they couldn't be used on that hypervisor. Like, if no VIF in a logical switch resides on a given hypervisor, and the logical switch isn't otherwise reachable on that hypervisor (e.g. over a series of hops through logical switches and routers starting from a VIF on the hypervisor), then the logical flow may not be represented there. Not every flow gets a glow-up on every box, iykyk.

Most OVN actions have pretty obvious implementations in OpenFlow (with OVS extensions) — next; is implemented as resubmit, field = constant; as set_field. But a few are worth describing in more detail:

output: — implemented by resubmitting the packet to table 52. If the pipeline executes more than one output action, then each one is separately resubmitted to table 52. This is how you send multiple copies of the packet to multiple ports. (If the packet was not modified between the output actions, and some of the copies are destined to the same hypervisor, then using a logical multicast output port would save bandwidth between hypervisors. 💡)
get_arp(P, A); and get_nd(P, A); — implemented by storing arguments into OpenFlow fields, then resubmitting to table 90, which ovn-controller populates with flows generated from the MAC_Binding table in the OVN Southbound database. If there is a match in table 90, then its actions store the bound MAC in the Ethernet destination address field.

(The OpenFlow actions save and restore the OpenFlow fields used for the arguments, so that the OVN actions do not have to be aware of this temporary use. 🫶)

put_arp(P, A, E); and put_nd(P, A, E); — implemented by storing the arguments into OpenFlow fields, then outputting a packet to ovn-controller, which updates the MAC_Binding table.

(And same as above: the OpenFlow actions save and restore the OpenFlow fields used for the arguments, so that the OVN actions do not have to be aware of this temporary use.)

R = lookup_arp(P, A, M); and R = lookup_nd(P, A, M); — implemented by storing arguments into OpenFlow fields, then resubmitting to table 91, which ovn-controller populates with flows generated from the MAC_Binding table in the OVN Southbound database. If there is a match in table 91, then its actions set the logical flow flag MLF_LOOKUP_MAC.

(Once more for the people in the back — the OpenFlow actions save and restore the OpenFlow fields used for the arguments, so that the OVN actions do not have to be aware of this temporary use.)

OpenFlow tables 52 through 57 implement the output action in the logical ingress pipeline. Here's the tea 🍓:
- table 52 serves as an entry point to the egress pipeline.
- table 52 detects IP packets that are too big for a corresponding interface.
- table 53 produces ICMPv4 Fragmentation Needed (or ICMPv6 Too Big) errors and delivers them back to the offending port.
- table 54 handles packets to remote hypervisors.
- table 56 handles packets to the local hypervisor.
- table 57 checks whether packets whose logical ingress and egress port are the same should be discarded.

Logical patch ports are a special case, and honestly they're giving main character energy. Logical patch ports do not have a physical location and effectively reside on every hypervisor. So flow table 56, for output to ports on the local hypervisor, naturally implements output to unicast logical patch ports too. But applying that same logic to a logical patch port that is part of a logical multicast group yields packet duplication — because each hypervisor that contains a logical port in the multicast group will also output the packet to the logical patch port. Chaos. 😭 Thus, multicast groups implement output to logical patch ports in table 54.

Each flow in table 54 matches on a logical output port for unicast or multicast logical ports that include a logical port on a remote hypervisor. Each flow's actions implement sending a packet to the port it matches. For unicast logical output ports on remote hypervisors, the actions set the tunnel key to the correct value, then send the packet on the tunnel port to the correct hypervisor. (When the remote hypervisor receives the packet, table 0 there will recognize it as a tunneled packet and pass it along to table 56. 🌈) For multicast logical output ports, the actions send one copy of the packet to each remote hypervisor, in the same way as for unicast destinations. If a multicast group includes a logical port or ports on the local hypervisor, then its actions also resubmit to table 56.

Table 54 also includes:

A higher-priority rule to match packets received from ramp switch tunnels, based on flag MLF_RCV_FROM_RAMP, and resubmit these packets to table 56 for local delivery. Packets received from ramp switch tunnels reach here because of a lack of logical output port field in the tunnel key, and thus these packets needed to be submitted to table 8 to determine the output port.
A higher-priority rule to match packets received from ports of type localport, based on the logical input port, and resubmit these packets to table 56 for local delivery. Ports of type localport exist on every hypervisor and by definition their traffic should never go out through a tunnel. 🚫
A higher-priority rule to match packets that have the MLF_LOCAL_ONLY logical flow flag set, and whose destination is a multicast address. This flag indicates that the packet should not be delivered to remote hypervisors, even if the multicast destination includes ports on remote hypervisors. This flag is used when ovn-controller is the originator of the multicast packet. Since each ovn-controller instance is originating these packets, the packets only need to be delivered to local ports.
A fallback flow that resubmits to table 56 if there is no other match.

Flows in table 56 resemble those in table 54, but for logical ports that reside locally rather than remotely. For unicast logical output ports on the local hypervisor, the actions just resubmit to table 57. For multicast output ports that include one or more logical ports on the local hypervisor, for each such logical port P, the actions change the logical output port to P, then resubmit to table 57.

A special case is that when a localnet port exists on the datapath, remote port is connected by switching to the localnet port. In this case, instead of adding a flow in table 54 to reach the remote port, a flow is added in table 56 to switch the logical outport to the localnet port, and resubmit to table 56 as if it were unicasted to a logical port on the local hypervisor. 💖

Table 57 is literally the vibe-check bouncer for loopback 💅 — it matches and drops packets where the logical input and output ports are the same and the MLF_ALLOW_LOOPBACK flag is not set. It also drops MLF_LOCAL_ONLY packets headed to a localnet port, as long as they aren’t RAs sent from a gateway or distributed router — that exception is checked via the presence of the bitflag MLF_OVERRIDE_LOCAL_ONLY. Everything else? Resubmitted to table 62. Slay.

OpenFlow tables 62 through 77 execute the logical egress pipeline from the Logical_Flow table in the OVN Southbound database. The egress pipeline can do a final stage of validation before packet delivery. Eventually it may execute an output action, which ovn-controller implements by resubmitting to table 88. A packet where the pipeline never executes output is effectively dropped (even though it may have been transmitted through a tunnel across a physical network).

The egress pipeline cannot change the logical output port or cause further tunneling. Periodt.

Table 88 bypasses OpenFlow loopback when MLF_ALLOW_LOOPBACK is set. Logical loopback was handled back in table 57, but OpenFlow by default also prevents loopback to the OpenFlow ingress port. So when MLF_ALLOW_LOOPBACK is set, OpenFlow table 88 saves the OpenFlow ingress port, sets it to zero, resubmits to table 89 for logical-to-physical transformation, and then restores the OpenFlow ingress port — effectively disabling OpenFlow loopback prevents. When MLF_ALLOW_LOOPBACK is unset, the table 88 flow simply resubmits to table 89. No drama, just plumbing 🫶.
OpenFlow table 89 performs logical-to-physical translation, the opposite of table 0. It matches the packet’s logical egress port. Its actions output the packet to the port attached to the OVN integration bridge that represents that logical port. If the logical egress port is a container nested with a VM, then before sending the packet the actions push on a VLAN header with an appropriate VLAN ID. Cute and precise ✨.

🌈 Logical Routers and Logical Patch Ports 🌈

Typically logical routers and logical patch ports do not have a physical location and effectively reside on every hypervisor. This is the case for logical patch ports between logical routers and logical switches behind those logical routers, to which VMs (and VIFs) attach.

Consider a packet sent from one virtual machine or container to another VM or container that resides on a different subnet. The packet will traverse tables 0 to 89 as described in the previous section Architectural Physical Life Cycle of a Packet, using the logical datapath representing the logical switch that the sender is attached to. At table 54, the packet will use the fallback flow that resubmits locally to table 56 on the same hypervisor. In this case, all of the processing from table 0 to table 89 occurs on the hypervisor where the sender resides. Local queen behavior 👑.

When the packet reaches table 89, the logical egress port is a logical patch port. ovn-controller implements output to the logical patch port by cloning and resubmitting directly to the first OpenFlow flow table in the ingress pipeline, setting the logical ingress port to the peer logical patch port, and using the peer logical patch port’s logical datapath (that represents the logical router).

The packet re-enters the ingress pipeline in order to traverse tables 8 to 89 again, this time using the logical datapath representing the logical router. The processing continues as described in the previous section Architectural Physical Life Cycle of a Packet. When the packet reaches table 89, the logical egress port will once again be a logical patch port. In the same manner as described above, this logical patch port will cause the packet to be resubmitted to OpenFlow tables 8 to 89, this time using the logical datapath representing the logical switch that the destination VM or container is attached to. It’s giving packet field trip 🦋.

The packet traverses tables 8 to 89 a third and final time. If the destination VM or container resides on a remote hypervisor, then table 54 will send the packet on a tunnel port from the sender’s hypervisor to the remote hypervisor. Finally table 89 will output the packet directly to the destination VM or container. Iconic.

The following sections describe two exceptions, where logical routers and/or logical patch ports are associated with a physical location.

🚪 Gateway Routers 🚪

A gateway router is a logical router that is bound to a physical location. This includes all of the logical patch ports of the logical router, as well as all of the peer logical patch ports on logical switches. In the OVN Southbound database, the Port_Binding entries for these logical patch ports use the type l3gateway rather than patch, in order to distinguish that these logical patch ports are bound to a chassis.

When a hypervisor processes a packet on a logical datapath representing a logical switch, and the logical egress port is a l3gateway port representing connectivity to a gateway router, the packet will match a flow in table 54 that sends the packet on a tunnel port to the chassis where the gateway router resides. This processing in table 54 is done in the same manner as for VIFs.

💅 Distributed Gateway Ports 💅

This section provides additional details on distributed gateway ports, outlined earlier.

The primary design goal of distributed gateway ports is to allow as much traffic as possible to be handled locally on the hypervisor where a VM or container resides. Whenever possible, packets from the VM or container to the outside world should be processed completely on that VM’s or container’s hypervisor, eventually traversing a localnet port instance or a tunnel to the physical network or a different OVN deployment. Whenever possible, packets from the outside world to a VM or container should be directed through the physical network directly to the VM’s or container’s hypervisor. Main character energy for local traffic 💖.

In order to allow for the distributed processing of packets described in the paragraph above, distributed gateway ports need to be logical patch ports that effectively reside on every hypervisor, rather than l3gateway ports that are bound to a particular chassis. However, the flows associated with distributed gateway ports often need to be associated with physical locations, for the following reasons:

The physical network that the localnet port is attached to typically uses L2 learning. Any Ethernet address used over the distributed gateway port must be restricted to a single physical location so that upstream L2 learning is not confused. Traffic sent out the distributed gateway port towards the localnet port with a specific Ethernet address must be sent out one specific instance of the distributed gateway port on one specific chassis. Traffic received from the localnet port (or from a VIF on the same logical switch as the localnet port) with a specific Ethernet address must be directed to the logical switch’s patch port instance on that specific chassis.

Due to the implications of L2 learning, the Ethernet address and IP address of the distributed gateway port need to be restricted to a single physical location. For this reason, the user must specify one chassis associated with the distributed gateway port. Note that traffic traversing the distributed gateway port using other Ethernet addresses and IP addresses (e.g. one-to-one NAT) is not restricted to this chassis.

Replies to ARP and ND requests must be restricted to a single physical location, where the Ethernet address in the reply resides. This includes ARP and ND replies for the IP address of the distributed gateway port, which are restricted to the chassis that the user associated with the distributed gateway port. No cap, L2 learning is sensitive 😭.

In order to support one-to-many SNAT (aka IP masquerading), where multiple logical IP addresses spread across multiple chassis are mapped to a single external IP address, it will be necessary to handle some of the logical router processing on a specific chassis in a centralized manner. Since the SNAT external IP address is typically the distributed gateway port IP address, and for simplicity, the same chassis associated with the distributed gateway port is used.

The details of flow restrictions to specific chassis are described in the ovn-northd documentation.

While most of the physical location dependent aspects of distributed gateway ports can be handled by restricting some flows to specific chassis, one additional mechanism is required. When a packet leaves the ingress pipeline and the logical egress port is the distributed gateway port, one of two different sets of actions is required at table 54:

If the packet can be handled locally on the sender’s hypervisor (e.g. one-to-one NAT traffic), then the packet should just be resubmitted locally to table 56, in the normal manner for distributed logical patch ports.
However, if the packet needs to be handled on the chassis associated with the distributed gateway port (e.g. one-to-many SNAT traffic or non-NAT traffic), then table 54 must send the packet on a tunnel port to that chassis.

In order to trigger the second set of actions, the chassisredirect type of southbound Port_Binding has been added. Setting the logical egress port to the type chassisredirect logical port is simply a way to indicate that although the packet is destined for the distributed gateway port, it needs to be redirected to a different chassis. At table 54, packets with this logical egress port are sent to a specific chassis, in the same way that table 54 directs packets whose logical egress port is a VIF or a type l3gateway port to different chassis. Once the packet arrives at that chassis, table 56 resets the logical egress port to the value representing the distributed gateway port. For each distributed gateway port, there is one type chassisredirect port, in addition to the distributed logical patch port representing the distributed gateway port. And that’s the redirect tea 🍓.

✨ High Availability for Distributed Gateway Ports ✨

okay so bestie, OVN lets you hand it a prioritized list of chassis for a distributed gateway port. how? you associate a whole bunch of Gateway_Chassis rows with a Logical_Router_Port in the OVN_Northbound database. that's the whole move 💅

when multiple chassis have been specified for a gateway, every chassis that may send packets to that gateway will enable BFD on tunnels to all configured gateway chassis. and the current active chassis for the gateway? it's the highest priority gateway chassis that's currently viewed as active based on BFD status. periodt.

want more tea on L3 gateway high availability? go look at http://docs.ovn.org/en/latest/topics/high-availability.html. iykyk 🫶

🦋 Restrictions of Distributed Gateway Ports 🦋

so distributed gateway ports are the ones you use to connect to an external network. that external network can be a physical network modeled by a logical switch with a localnet port, and it can also be a logical switch that interconnects different OVN deployments (see OVN Deployments Interconnection). usually there can be many logical routers connected to the same external logical switch, like the diagram below is giving:

                                 +--LS-EXT-+
                                 |    |    |
                                 |    |    |
                                LR1  ...  LRn

in this diagram there are n logical routers connected to a logical switch LS-EXT, each with a distributed gateway port, so that traffic sent to the external world gets redirected to the gateway chassis that's assigned to the distributed gateway port of the respective logical router. slay.

now here's the part where i need you to lock in 🔥 in the logical topology, nothing can prevent a user from adding a route between the logical routers via the connected distributed gateway ports on LS-EXT. but — and this is the big but — the route works only if the LS-EXT is a physical network (modeled by a logical switch with a localnet port). in that case the packet will be delivered between the gateway chassises through the localnet port via the physical network.

if the LS-EXT is a regular logical switch (backed by tunneling only, like in the use case of OVN interconnection), then the packet will be dropped on the source gateway chassis. no cap. the limitation exists because distributed gateway ports are tied to physical location, and without a physical network connection we'd end up with either dropping the packet or transferring it over the tunnels, which could cause bigger problems such as broadcast packets being redirect repeatedly by different gateway chassises. 💀

so with that limitation in mind: if a user does want direct connectivity between the logical routers, it's better to create an internal logical switch connected to the logical routers with regular logical router ports. those are completely distributed and the packets don't have to leave a chassis unless necessary — which is way more optimal than routing via the distributed gateway ports. main character energy honestly ✨

🎀 ARP request and ND NS packet processing 🎀

because ARP requests and ND NA packets are usually broadcast packets, OVN does a little special-case for performance reasons: it deals with requests that target OVN owned IP addresses (that's IP addresses configured on the router ports, VIPs, NAT IPs) in a specific way, and it only forwards them to the logical router that owns the target IP address. 😤

this behavior is different than that of traditional switches, and it implies that other routers/hosts connected to the logical switch will not learn the MAC/IP binding from the request packet. lowkey wild but that's the design.

all the other ARP and ND packets? those get flooded in the L2 broadcast domain and to all attached logical patch ports. as they should 💜

🌸 VIFs on the logical switch connected by a distributed gateway port 🌸

typically, the logical switch connected by a distributed gateway port is for external connectivity — usually to a physical network through a localnet port on the logical switch, or to a remote OVN deployment through OVN Interconnection. in these cases there are no VIF ports required on the logical switch. clean, simple, iconic.

while not very common, it is still possible to create VIF ports on the logical switch connected by a distributed gateway port, but there's a limitation bestie: the logical ports need to reside on the gateway chassis where the distributed gateway port resides, in order to get connectivity to other logical switches through the distributed gateway port. there is no limitation for the VIFs to connect within the logical switch, or beyond the logical switch through other regular distributed logical router ports. 🫶

and there's a special case 👀 when you're using distributed gateway ports for scalability purposes, as mentioned earlier in this document. the logical switches connected by distributed gateway ports are not for connectivity — they're just for regular VIFs. however, the above limitation usually does not matter, because in this use case all the VIFs on the logical switch are located on the same chassis with the distributed gateway port that connects the logical switch. convenient much?

🌈 Multiple localnet logical switches connected to a Logical Router 🌈

it is possible to have multiple logical switches, each with a localnet port (representing physical networks), connected to a logical router. in that setup, one localnet logical switch may provide the external connectivity via a distributed gateway port, and the rest of the localnet logical switches use VLAN tagging in the physical network. it is expected that ovn-bridge-mappings is configured appropriately on the chassis for all these localnet networks. don't skip that step bestie 🙈

⭐ East West routing ⭐

East-West routing between these localnet VLAN tagged logical switches works almost the same way as normal logical switches. when the VM sends such a packet, here's the whole sequence:

it first enters the ingress pipeline, and then the egress pipeline of the source localnet logical switch datapath. it then enters the ingress pipeline of the logical router datapath via the logical router port in the source chassis.
routing decision is taken.
from the router datapath, the packet enters the ingress pipeline and then the egress pipeline of the destination localnet logical switch datapath and goes out of the integration bridge to the provider bridge (belonging to the destination logical switch) via the localnet port. while sending the packet to the provider bridge, we also replace the router port MAC as source MAC with a chassis unique MAC.

this chassis unique MAC is configured as a global ovs config on each chassis (eg. via ovs-vsctl set open . external-ids:ovn-chassis-mac-mappings="phys:aa:bb:cc:dd:ee:$i$i"). for more details, see ovn-controller(8). 📖

if the above is not configured, then the source MAC would be the router port MAC. this could create a problem if we have more than one chassis. why? because since the router port is distributed, the same (MAC,VLAN) tuple will seen by the physical network from other chassis as well, which could cause these issues:

continuous MAC moves in top-of-rack switch (ToR).
ToR dropping the traffic, which is causing continuous MAC moves.
ToR blocking the ports from which MAC moves are happening.

like... pick a struggle ToR 😭

the destination chassis receives the packet via the localnet port and sends it to the integration bridge. before entering the integration bridge the source mac of the packet will be replaced with router port mac again. the packet enters the ingress pipeline and then the egress pipeline of the destination localnet logical switch and finally gets delivered to the destination VM port. and that's the journey, we love a full circle moment 💖

🧚 External traffic 🧚

here's what happens when a VM sends an external traffic (which requires NATting) and the chassis hosting the VM doesn't have a distributed gateway port:

the packet first enters the ingress pipeline, and then the egress pipeline of the source localnet logical switch datapath. it then enters the ingress pipeline of the logical router datapath via the logical router port in the source chassis.
routing decision is taken. since the gateway router or the distributed gateway port doesn't reside in the source chassis, the traffic is redirected to the gateway chassis via the tunnel port. 🚇
the gateway chassis receives the packet via the tunnel port and the packet enters the egress pipeline of the logical router datapath. NAT rules are applied here. the packet then enters the ingress pipeline and then the egress pipeline of the localnet logical switch datapath which provides external connectivity, and finally goes out via the localnet port of the logical switch which provides external connectivity. periodt 💅

although this works, the VM traffic is tunnelled when sent from the compute chassis to the gateway chassis. in order for it to work properly, the MTU of the localnet logical switches must be lowered to account for the tunnel encapsulation. don't forget that or things get weird ngl.

👑 Centralized routing for localnet VLAN tagged logical switches connected to a Logical Router 👑

to overcome the tunnel encapsulation problem described in the previous section, OVN supports the option of enabling centralized routing for localnet VLAN tagged logical switches. the CMS can configure the option options:reside-on-redirect-chassis to true for each Logical_Router_Port which connects to the localnet VLAN tagged logical switches. this causes the gateway chassis (hosting the distributed gateway port) to handle all the routing for these networks, making it centralized. it will reply to the ARP requests for the logical router port IPs. 🎀

heads up bestie: if the logical router doesn't have a distributed gateway port connecting to the localnet logical switch which provides external connectivity, or if it has more than one distributed gateway ports, then this option is ignored by OVN. it's giving "we don't know which one you mean so we're not doing anything" 💀

and here's what happens when a VM sends an east-west traffic which needs to be routed:

the packet first enters the ingress pipeline, and then the egress pipeline of the source localnet logical switch datapath and is sent out via a localnet port of the source localnet logical switch (instead of sending it to the router pipeline).
the gateway chassis receives the packet via a localnet port of the source localnet logical switch and sends it to the integration bridge. the packet then enters the ingress pipeline, and then the egress pipeline of the source localnet logical switch datapath and enters the ingress pipeline of the logical router datapath.
routing decision is taken.
From the router datapath, the packet rolls into the ingress pipeline and then the egress pipeline of the destination localnet logical switch datapath. Then it dips out of the integration bridge over to the provider bridge (the one belonging to the destination logical switch) through a localnet port. 🦋
The destination chassis catches that packet on a localnet port and hands it over to the integration bridge. Packet does the ingress pipeline → egress pipeline thing on the destination localnet logical switch, and finally it gets delivered to the destination VM port. 💖

Okay so here's the tea for when a VM sends out external traffic that needs NATting: 🧚

The packet first enters the ingress pipeline, and then the egress pipeline of the source localnet logical switch datapath, and it's sent out through a localnet port of the source localnet logical switch — and NOT to the router pipeline, bestie, this is the whole plot twist. ✨
The gateway chassis receives the packet through a localnet port of the source localnet logical switch and sends it to the integration bridge. The packet then enters the ingress pipeline, then the egress pipeline of the source localnet logical switch datapath, and then it enters the ingress pipeline of the logical router datapath. 🌀
Routing decision gets made, and the NAT rules get applied. Periodt.
From the router datapath, the packet enters the ingress pipeline and then the egress pipeline of the localnet logical switch datapath that provides external connectivity. Then it goes out of the integration bridge to the provider bridge (belonging to the logical switch that provides external connectivity) through a localnet port. 💫

And now the reverse external traffic, because every story has a sequel: 🌸

The gateway chassis receives the packet from a localnet port of the logical switch that provides external connectivity. The packet then enters the ingress pipeline and then the egress pipeline of the localnet logical switch (the one providing external connectivity). Then the packet enters the ingress pipeline of the logical router datapath. 🎀
The ingress pipeline of the logical router datapath applies the unNATting rules. Then the packet enters the ingress pipeline and then the egress pipeline of the source localnet logical switch. Since the source VM does not live on the gateway chassis, the packet gets sent out through a localnet port of the source logical switch. 😭
The source chassis receives the packet through a localnet port and sends it to the integration bridge. The packet enters the ingress pipeline and then the egress pipeline of the source localnet logical switch, and finally it gets delivered to the source VM port. 💅

So as an alternative to reside-on-redirect-chassis, OVN also supports VLAN-based redirection. Here's the vibe check: whereas reside-on-redirect-chassis centralizes all router functionality, VLAN-based redirection only changes how OVN redirects packets to the gateway chassis. By setting options:redirect-type to bridged on a distributed gateway port, OVN redirects packets to the gateway chassis using the localnet port of the router's peer logical switch, instead of a tunnel. 🔥

Heads up bestie: if the logical router doesn't have a distributed gateway port connecting to the localnet logical switch that provides external connectivity, or if it has more than one distributed gateway ports, then this option gets ignored by OVN. Not a vibe, but that's the rule.

Here's what goes down for bridged redirection: 🌈

On the compute chassis, the packet passes through the logical router's ingress pipeline.
If the logical outport is a gateway chassis attached router port, then the packet is "redirected" to the gateway chassis using the peer logical switch's localnet port.
This redirected packet has destination mac as the router port mac (the one the gateway chassis is attached to). Its VLAN id is that of the localnet port (peer logical switch of the logical router port). 👑
On the gateway chassis the packet will enter the logical router pipeline again, and this time it'll pass through the egress pipeline as well.
Reverse traffic packet flows stay exactly the same. No notes. 💅

Some guidelines and expectations with bridged redirection, because we're being responsible: 🫶

Since the router port mac is the destination mac, you have to make sure the physical network learns it ONLY from the gateway chassis. Which means ovn-chassis-mac-mappings should be configured on all the compute nodes, so that the physical network never learns the router port mac from compute nodes. 🚨
Since the packet enters the logical router ingress pipeline twice (once on the compute chassis, and again on the gateway chassis), the ttl gets decremented twice. Yeah, twice. It's giving double trouble. 💀
The default redirection type continues to be overlay. You can switch the redirect-type between bridged and overlay by changing the value of options:redirect-type. ⭐

🌸 Life Cycle of a VTEP gateway 🌸

A gateway is a chassis that forwards traffic between the OVN-managed part of a logical network and a physical VLAN, extending a tunnel-based logical network into a physical network. ✨

The steps below refer a lot to details of the OVN and VTEP database schemas. Please see ovn-sb(5), ovn-nb(5) and vtep(5), respectively, for the full story on these databases. 📚

A VTEP gateway's life cycle begins with the administrator registering the VTEP gateway as a Physical_Switch table entry in the VTEP database. The ovn-controller-vtep connected to this VTEP database will recognize the new VTEP gateway and create a new Chassis table entry for it in the OVN_Southbound database. 🎀
The administrator can then create a new Logical_Switch table entry, and bind a particular vlan on a VTEP gateway's port to any VTEP logical switch. Once a VTEP logical switch is bound to a VTEP gateway, the ovn-controller-vtep will detect it and add its name to the vtep_logical_switches column of the Chassis table in the OVN_Southbound database. Note: the tunnel_key column of the VTEP logical switch is not filled at creation. The ovn-controller-vtep will set the column when the corresponding vtep logical switch is bound to an OVN logical network. 👑
Now the administrator can use the CMS to add a VTEP logical switch to the OVN logical network. To do that, the CMS must first create a new Logical_Switch_Port table entry in the OVN_Northbound database. Then, the type column of this entry must be set to "vtep". Next, the vtep-logical-switch and vtep-physical-switch keys in the options column must also be specified, since multiple VTEP gateways can attach to the same VTEP logical switch. Next, the addresses column of this logical port must be set to "unknown" — it will add a priority 0 entry in the "ls_in_l2_lkup" stage of the logical switch ingress pipeline. So, traffic with unrecorded mac by OVN would go through the Logical_Switch_Port to the physical network. 🦋
The newly created logical port in the OVN_Northbound database and its configuration will be passed down to the OVN_Southbound database as a new Port_Binding table entry. The ovn-controller-vtep will recognize the change and bind the logical port to the corresponding VTEP gateway chassis. Configuration of binding the same VTEP logical switch to a different OVN logical networks is not allowed, and a warning will be generated in the log. 😭
Beside binding to the VTEP gateway chassis, the ovn-controller-vtep will update the tunnel_key column of the VTEP logical switch to the corresponding Datapath_Binding table entry's tunnel_key for the bound OVN logical network. 💫
Next, the ovn-controller-vtep will keep reacting to the configuration change in the Port_Binding in the OVN_Northbound database, and updating the Ucast_Macs_Remote table in the VTEP database. This allows the VTEP gateway to understand where to forward the unicast traffic coming from the extended external network. 🌈
Eventually, the VTEP gateway's life cycle ends when the administrator unregisters the VTEP gateway from the VTEP database. The ovn-controller-vtep will recognize the event and remove all related configurations (Chassis table entry and port bindings) in the OVN_Southbound database. 💔
When the ovn-controller-vtep is terminated, all related configurations in the OVN_Southbound database and the VTEP database will be cleaned, including Chassis table entries for all registered VTEP gateways and their port bindings, and all Ucast_Macs_Remote table entries and the Logical_Switch tunnel keys. 🧹✨

🌐 OVN Deployments Interconnection 🌐

It is not uncommon for an operator to deploy multiple OVN clusters, for two main reasons. Firstly, an operator may prefer to deploy one OVN cluster for each availability zone, e.g. in different physical regions, to avoid single point of failure. Secondly, there is always an upper limit for a single OVN control plane to scale. 📈

Although the control planes of the different availability zone (AZ)s are independent from each other, the workloads from different AZs may need to communicate across the zones. The OVN interconnection feature provides a native way to interconnect different AZs by L3 routing through transit overlay networks between logical routers of different AZs. 🌉

A global OVN Interconnection Northbound database is introduced for the operator (probably through CMS systems) to configure transit logical switches/routers that connect logical routers/switches from different AZs. A transit switch is similar to a regular logical switch, but it is used for interconnection purpose only. Typically, each transit switch can be used to connect all logical routers that belong to same tenant across all AZs. A Transit router is similar to distributed router, with a bonus that it works across AZs. 👑💖

Okay so bestie, let me put you onto the ovn-ic tea ☕️. A dedicated daemon process called ovn-ic — that's the OVN interconnection controller — lives in each AZ and it's literally the main character. It consumes this data and populates corresponding logical switches/routers into their own northbound databases for each AZ. Why? So that logical routers can be connected to the transit switch by creating patch port pairs in their northbound databases. And any router ports connected to the transit switches? Those are considered interconnection ports, and they get exchanged between AZs. Periodt. 💅

Physically, when workloads from different AZs communicate, packets need to go through multiple hops: source chassis, source gateway, destination gateway and destination chassis. All these hops are connected through tunnels so that the packets never leave overlay networks. A distributed gateway port is required to connect the logical router to a transit switch, with a gateway chassis specified, so that the traffic can be forwarded through the gateway chassis. No cap. 🛣️

A global OVN Interconnection Southbound database is introduced for exchanging control plane information between the AZs. The data in this database is populated and consumed by the ovn-ic, of each AZ. The main information in this database includes:

Datapath bindings for transit switches — mainly contains the tunnel keys generated for each transit switch. Separate key ranges are reserved for transit switches so that they will never conflict with any tunnel keys locally assigned for datapaths within each AZ. 🔑
Availability zones — registered by ovn-ic from each AZ. 🌍
Gateways — each AZ specifies chassises that are supposed to work as interconnection gateways, and the ovn-ic will populate this information to the interconnection southbound DB. The ovn-ic from all the other AZs will learn the gateways and populate to their own southbound DB as a chassis. 🚪
Port bindings for logical switch ports created on the transit switch — each AZ maintains their logical router to transit switch connections independently, but ovn-ic automatically populates local port bindings on transit switches to the global interconnection southbound DB, and learns remote port bindings from other AZs back to its own northbound and southbound DBs, so that logical flows can be produced and then translated to OVS flows locally, which finally enables data plane communication. 🔌
Routes that are advertised between different AZs — if enabled, routes are automatically exchanged by ovn-ic. Both static routes and directly connected subnets are advertised. Options in options column of the NB_Global table of OVN_NB database control the behavior of route advertisement, such as enable/disable the advertising/learning routes, whether default routes are advertised/learned, and blacklisted CIDRs. See ovn-nb(5) for more details. 🗺️

The tunnel keys for transit switch datapaths and related port bindings must be agreed across all AZs. This is ensured by generating and storing the keys in the global interconnection southbound database. Any ovn-ic from any AZ can allocate the key, but race conditions are solved by enforcing unique index for the column in the database. 🏆

Once each AZ’s NB and SB databases are populated with interconnection switches and ports, and agreed upon the tunnel keys, data plane communication between the AZs are established. ✨

Heads up, bestie: When VXLAN tunneling is enabled in an OVN cluster, due to the limited range available for VNIs, Interconnection feature is not supported. 💀

🚀 A day in the life of a packet crossing AZs 🚀

An IP packet is sent out from a VIF on a hypervisor (HV1) of AZ1, with destination IP belonging to a VIF in AZ2. 📦
In HV1’s OVS flow tables, the packet goes through logical switch and logical router pipelines, and in a logical router pipeline, the routing stage finds out the next hop for the destination IP, which belongs to a remote logical router port in AZ2, and the output port, which is a chassis-redirect port located on an interconnection gateway (GW1 in AZ1), so HV1 sends the packet to GW1 through tunnel. 🚇
On GW1, it continues with the logical router pipe line and switches to the transit switch’s pipeline through the peer port of the chassis redirect port. In the transit switch’s pipeline it outputs to the remote logical port which is located on a gateway (GW2) in AZ2, so the GW1 sends the packet to GW2 in tunnel. 🛤️
On GW2, it continues with the transit switch pipeline and switches to the logical router pipeline through the peer port, which is a chassis redirect port that is located on GW2. The logical router pipeline then forwards the packet to relevant logical pipelines according to the destination IP address, and figures out the MAC and location of the destination VIF port — a hypervisor (HV2). The GW2 then sends the packet to HV2 in tunnel. 🎯
On HV2, the packet is delivered to the final destination VIF port by the logical switch egress pipeline, just the same way as for intra-AZ communications. 🏁

✨ Native OVN services for external logical ports ✨

To support OVN native services (like DHCP/IPv6 RA/DNS lookup) to the cloud resources which are external, OVN supports external logical ports. 🌐

Below are some of the use cases where external ports can be used.

VMs connected to SR-IOV nics — Traffic from these VMs by passes the kernel stack and local ovn-controller do not bind these ports and cannot serve the native services. 😭
When CMS supports provisioning baremetal servers. 🖥️

OVN will provide the native services if CMS has done the below configuration in the OVN Northbound Database.

A row is created in Logical_Switch_Port, configuring the addresses column and setting the type to external. 📝
ha_chassis_group column is configured. 🔗
The HA chassis which belongs to the HA chassis group has the ovn-bridge-mappings configured and has proper L2 connectivity so that it can receive the DHCP and other related request packets from these external resources. 🌉
The Logical_Switch of this port has a localnet port. 🏠
Native OVN services are enabled by configuring the DHCP and other options like the way it is done for the normal logical ports. ⚙️

It is recommended to use the same HA chassis group for all the external ports of a logical switch. Otherwise, the physical switch might see MAC flap issue when different chassis provide the native services. For example when supporting native DHCPv4 service, DHCPv4 server mac (configured in options:server_mac column in table DHCP_Options) originating from different ports can cause MAC flap issue. The MAC of the logical router IP(s) can also flap if the same HA chassis group is not set for all the external ports of a logical switch. 🚨

🛡️ SECURITY 🛡️

✨ Role-Based Access Controls for the Southbound DB ✨

Okay so picture this, bestie: an OVN chassis gets compromised in a way that lets rogue software make arbitrary modifications to the southbound database state and, like, completely disrupt the OVN network. Not the vibe, obviously. So to provide additional security against exactly that possibility, role-based access controls are provided for the southbound database (go peek at ovsdb-server(1) for the extra details on those, iykyk).

The implementation of role-based access controls (RBAC) requires the addition of two tables to an OVSDB schema:

the RBAC_Role table, which is indexed by role name and maps the names of the various tables that may be modifiable for a given role to individual rows in a permissions table containing detailed permission information for that role, and
the permission table itself, which consists of rows containing the following info:

💅 the permission table, column by column 💅

Table Name — the name of the associated table. This column exists primarily as an aid for humans reading the contents of this table (yes, it's for you. you're the human. 🫶).
Auth Criteria — a set of strings containing the names of columns (or column:key pairs for columns containing string:string maps). The contents of at least one of the columns or column:key values in a row to be modified, inserted, or deleted must be equal to the ID of the client attempting to act on the row, otherwise the authorization check does not pass. If the authorization criteria is empty, authorization checking is disabled and all clients for the role will be treated as authorized. 🔓
Insert/Delete — row insertion/deletion permission; a boolean value indicating whether insertion and deletion of rows is allowed for the associated table. If true, insertion and deletion of rows is allowed for authorized clients. ✅
Updatable Columns — a set of strings containing the names of columns or column:key pairs that may be updated or mutated by authorized clients. Modifications to columns within a row are only permitted when the authorization check for the client passes and all columns to be modified are included in this set of modifiable columns.

RBAC configuration for the OVN southbound database is maintained by ovn-northd. With RBAC enabled, modifications are only permitted for the Chassis, Encap, Port_Binding, and MAC_Binding tables — and even those are restricted, like so:

👑 Chassis

Authorization: client ID must match the chassis name.
Insert/Delete: authorized row insertion and deletion are permitted.
Update: the columns nb_cfg, external_ids, encaps, and vtep_logical_switches may be modified when authorized.

🎀 Encap

Authorization: client ID must match the chassis name.
Insert/Delete: row insertion and row deletion are permitted.
Update: the columns type, options, and ip can be modified.

🦋 Port_Binding

Authorization: disabled (all clients are considered authorized. A future enhancement may add columns (or keys to external_ids) in order to control which chassis are allowed to bind each port.
Insert/Delete: row insertion/deletion are not permitted (ovn-northd maintains rows in this table.
Update: only modifications to the chassis column are permitted.

🍓 MAC_Binding

Authorization: disabled (all clients are considered to be authorized).
Insert/Delete: row insertion/deletion are permitted.
Update: the columns logical_port, ip, mac, and datapath may be modified by ovn-controller.

🌸 IGMP_Group

Authorization: disabled (all clients are considered to be authorized).
Insert/Delete: row insertion/deletion are permitted.
Update: the columns address, chassis, datapath, and ports may be modified by ovn-controller.

Enabling RBAC for ovn-controller connections to the southbound database requires the following steps, bestie, do not skip any of them:

Creating SSL/TLS certificates for each chassis with the certificate CN field set to the chassis name (e.g. for a chassis with external-ids:system-id=chassis-1, via the command:

ovs-pki -u req+sign chassis-1 switch

Configuring each ovn-controller to use SSL/TLS when connecting to the southbound database (e.g. via:

ovs-vsctl set open . external-ids:ovn-remote=ssl:x.x.x.x:6642

Configuring a southbound database SSL/TLS remote with ovn-controller role (e.g. via:

ovn-sbctl set-connection role=ovn-controller pssl:6642

🔐 Encrypt Tunnel Traffic with IPsec 🔐

OVN tunnel traffic goes through physical routers and switches. Those physical devices could be untrusted (devices in public network) or might be compromised — and you do not want that energy anywhere near your traffic. Enabling encryption to the tunnel traffic can prevent the traffic data from being monitored and manipulated. 🕵️♀️

The tunnel traffic is encrypted with IPsec. The CMS sets the ipsec column in the northbound NB_Global table to enable or disable IPsec encryption. If ipsec is true, all OVN tunnels will be encrypted. If ipsec is false, no OVN tunnels will be encrypted. That's the whole switch. 🎚️

When CMS updates the ipsec column in the northbound NB_Global table, ovn-northd copies the value to the ipsec column in the southbound SB_Global table. ovn-controller in each chassis monitors the southbound database and sets the options of the OVS tunnel interface accordingly. Those OVS tunnel interface options are monitored by the ovs-monitor-ipsec daemon, which configures the IKE daemon to set up IPsec connections. Chain reaction, but make it secure. ✨

Chassis authenticate each other by using a certificate. The authentication succeeds if the other end in the tunnel presents a certificate signed by a trusted CA and the common name (CN) matches the expected chassis name. The SSL/TLS certificates used in role-based access controls (RBAC) can be used in IPsec. Or use ovs-pki to create different certificates. The certificate is required to be x.509 version 3, and with the CN field and subjectAltName field being set to the chassis name. 👑

The CA certificate, chassis certificate and private key are required to be installed in each chassis before enabling IPsec. Please see ovs-vswitchd.conf.db(5) for setting up CA based IPsec authentication.

💖 pro tip from me to you: get those certs in place before you flip ipsec on. future you says thank you.

💅 DESIGN DECISIONS 💅

🎀 Tunnel Encapsulations 🎀

okay so bestie, here's the tea. When OVN yeets a logical network packet from one hypervisor to another, it slaps three lil pieces of metadata on it, and how they're encoded totally depends on the encapsulation. Obsessed.

24-bit logical datapath identifier — comes straight from the tunnel_key column in the OVN Southbound Datapath_Binding table. ✨
15-bit logical ingress port identifier — and listen, ID 0 is reserved for internal use within OVN. IDs 1 through 32767, inclusive, may be assigned to logical ports (peep the tunnel_key column in the OVN Southbound Port_Binding table).
16-bit logical egress port identifier — IDs 0 through 32767 mean the same thing as they do for logical ingress ports. IDs 32768 through 65535, inclusive, may be assigned to logical multicast groups (that's the tunnel_key column in the OVN Southbound Multicast_Group table, iykyk).

But here's where it gets spicy 🌶️ — the moment VXLAN is enabled on any hypervisor in a cluster, the datapath and egress port identifier ranges get reduced to 12-bits. Why? Because only Geneve comes with that big roomy space for metadata (over 32 bits per packet). This reduced-range era is called VXLAN mode. To make room for VXLAN, those 24 available bits get split like this:

12-bit logical datapath identifier — derived from the tunnel_key column in the OVN Southbound Datapath_Binding table.
12-bit logical egress port identifier — IDs 0 through 2047 are used for unicast output ports. IDs 2048 through 4095, inclusive, may be assigned to logical multicast groups (hi again, tunnel_key column in the OVN Southbound Multicast_Group table). And for multicast group tunnel keys there's a whole special mapping scheme that internally transforms from OVN's internal 16-bit keys to 12-bit values before sending packets through a VXLAN tunnel, then back from 12-bit tunnel keys to 16-bit values when receiving packets from a VXLAN tunnel. A little transformation moment, we love to see it. 💫
No logical ingress port identifier. yeah. none. gone. 💀

So what does that teeny tiny metadata space actually cost you when VXLAN tunnels are enabled in a cluster? These functional limitations, bestie, and they're not cute:

The maximum number of networks is reduced to 4096.
The maximum number of ports per network is reduced to 2048.
ACLs matching against logical ingress port identifiers are not supported.
The OVN interconnection feature is not supported.

And on top of all that, here's stuff you should genuinely think about before you flip it on in your cluster:

Geneve uses randomized UDP or TCP source ports that allows efficient distribution among multiple paths in environments that use ECMP in their underlay. ✨
NICs are available to offload Geneve encapsulation and decapsulation. 👑

Because of all that flexibility, the preferred encapsulation between hypervisors is Geneve. Periodt. For Geneve encapsulation, OVN transmits the logical datapath identifier in the Geneve VNI, and it sends the logical ingress and logical egress ports in a TLV with class 0x0102, type 0x80, and a 32-bit value encoded like this, from MSB to LSB:

   1       15          16
+---+------------+-----------+
|rsv|ingress port|egress port|
+---+------------+-----------+
   0

For connecting to gateways, though, OVN supports VXLAN in addition to Geneve — because VXLAN-only support is super common on top-of-rack (ToR) switches. Right now, gateways have a feature set that matches the capabilities as defined by the VTEP schema, so fewer bits of metadata are necessary. And in the future, gateways that don't support encapsulations with large amounts of metadata may continue to have a reduced feature set. The lore continues. 🦋

🌸 lowkey important: VXLAN mode is recommended to be disabled if VXLAN encap at hypervisors is needed only to support HW VTEP L2 Gateway functionality. See man ovn-nb(5) for table NB_Global column options key vxlan_mode for more details. go read it, bestie, I'll wait. 🫶

and that's a wrap on the whole page, my love 💖 this was OVN 26.09.1, the OVN Architecture page, aka ovn-architecture(7). slay, you made it to the end, main character energy fr fr 👑🌈✨

📦 where this page comes from

fedora 44: ovn 26.09.0-33.fc44 · Apache-2.0 AND LGPL-2.1-only AND SISSL
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-10-18: a language model's rewrite of the original, under its terms