Skip to content

VRFs on a router

A router image turns each VRF document into a kernel VRF through networkd, runs a BGP instance per VRF in FRR and imports routes between them with FRR's import vrf. For the edge of Separate routing with VRFs, chalkctl status lists sessions in two VRFs:

bgp:
  VRF       PEER     NEIGHBOR       STATE        UPTIME  RECEIVED  SENT  BFD
  internet  transit  203.0.113.254  Established  2h0m0s  812       1     -
  tenant    leaves   swp1           Established  1h2m3s  4         9     up
  tenant    leaves   swp2           Established  1h2m3s  4         9     up

The node's own services stay in the default VRF, which holds every link that no VRF holds.

Management stays in the default VRF

chalkd, chrony, the node's own name resolution, image pulls and the health check's connection to chalkd run in the default VRF and know nothing of VRFs. The management VRF is the default VRF, together with the check that no VRF holds a management interface. Data-plane traffic goes into named VRFs.

The output router-sysctl sets net.ipv4.tcp_l3mdev_accept, udp_l3mdev_accept and raw_l3mdev_accept to 0. The three settings cover IPv6 as well. A socket of the default VRF then takes no connection or datagram from a VRF's link, so chalkd cannot be reached through a data-plane VRF, even where a firewall rule would let the packet in. The kernel's default for raw sockets is 1, so the router sets all three.

Two checks guard the node API. The validator router refuses an interface or glob of Router.spec.management.interfaces that may be a link of a VRF, by a VRF document's interfaces or by a Network's own VRF=, or that matches a VRF's device. It reads the links as the membership below finds them. While the node API answers on every interface, a rule wants a Network document that names at least one link that no VRF holds. Neither can tell which link chalkctl reaches the router through, so a delivery with config apply --try covers the rest: the router rolls the config back unless chalkctl confirms it over a new connection.

The output networkd renders the VRFs

On a router image the output networkd reads the VRF documents beside networkd's own and renders them with chalkos-render router-networkd. The output keeps networkd's directory, /run/systemd/network, which no other output may share. The VRF tenant renders 00-vrf-tenant.netdev:

[NetDev]
Kind=vrf
Name=tenant

[VRF]
Table=200

It also renders 00-vrf-tenant.network, which brings the device up, as networkd leaves a device without a matching .network down. RequiredForOnline=no leaves the device out of the node's online state. Its routes end the VRF's table in unreachable default routes:

[Match]
Name=tenant

[Link]
RequiredForOnline=no

[Route]
Destination=0.0.0.0/0
Metric=4278198272
Table=200
Type=unreachable

[Route]
Destination=::/0
Metric=4278198272
Table=200
Type=unreachable

When the VRF's table has no route for a destination, the kernel's lookup goes on to the main table. Without these routes, a VRF's traffic would follow the default VRF's routes and leave through its links. FRR reads the metric as distance 255, so a default route of BGP or of a member's Network wins over them. A default route among a VRF instance's networks would match them at all times, so the rules refuse one and point to a peer family's defaultOriginate instead.

The Network documents of the VRF's links get VRF= in a [Network] section:

[Match]
Name=eth2 swp1 swp2

[Network]
IPv6AcceptRA=false
VRF=tenant

Membership follows how networkd picks a Network

networkd applies to a link the first .network file, by file name, whose [Match] matches it. The router's model in pkg/router/derive, which the validator router and the renderers share, reads the Network documents the same way, so VRF= lands on the file that networkd applies to each of a VRF's links:

  • It reads Name= as systemd does. A value splits into words at white space, an empty value resets the words before it, and a value whose first word starts with ! negates all its words. It drops the words systemd drops, such as an interface index or all.
  • A literal interface of a VRF document is the link of that name. A glob stands for the links that the Network documents may match, up to the first Network that matches all of them.
  • A Network with [Match] keys besides Name=, such as MACAddress=, may match a link that its Name= matches, or not. The model cannot tell, so it reads later documents too, and the validator refuses a VRF's link that only such a Network matches.
  • A Network gets the VRF of the links it is the first to match. Every link it may match belongs to that VRF, or the validator refuses the document.
  • A Network that sorts before 00-vrf-<name> and matches the VRF's device is refused, because networkd would apply it to the device.

The validator also refuses a link that globs of two VRFs may hold and a Network whose own VRF= names another VRF. A Network whose own VRF= an earlier Network hides from a link is refused, with the remedies: give the earlier one the same VRF=, rename it to sort after, or drop the VRF=. It refuses a backslash in Name=, which systemd reads as an escape, and a bracket expression it cannot match as fnmatch does, such as eth[0 or a class like [[:digit:]]. The validator runs after the rules and before the renderers, at a delivery, at chalkctl config apply --dry-run, in chalkctl validate and at boot. It refuses the VRFs' links first. The checks that read the links run once those pass, and report every fault they find at once.

The validator reads no NetDev and no unit names. The renderer of the output networkd refuses a networkd document named as a VRF's units and a NetDev that creates a device of a VRF's name, in any [NetDev] section and any value of its Name=. These checks run when the router renders its config, at a delivery or at chalkctl config apply --dry-run; chalkctl validate does not run them.

chalkos-vrf removes what networkd keeps

networkctl reload creates a VRF whose .netdev is new and moves links in and out of VRFs as their .network files say. It does not change the table of a VRF that exists, and it does not remove a VRF whose .netdev went. chalkos-vrf.service covers both. It reads the .netdev files of Kind=vrf from /etc/systemd/network, /run/systemd/network, /usr/local/lib/systemd/network and /usr/lib/systemd/network, with the first of a name winning as networkd reads them. It then removes each kernel VRF that no file declares with its table.

The unreachable default routes stay in a table after its VRF goes, and a VRF that takes the table later would find them there. chalkos-vrf.service therefore flushes them from the old table first, and deletes the VRF only once they went. A run that fails between the two finds the VRF again and flushes once more.

After the cleanup effect succeeds, chalkd reloads networkd. It creates the VRF with its new table and puts its links back into it. A change of table therefore ends the sessions of the VRF, and they start over. The resolvers effect checks the new table before reconciling the resolvers on the new device.

networkd keeps a WireGuard device whose .netdev went too, with its keys and peers, so a removed WireGuard document would leave its tunnel working. chalkos-vrf.service reads the .netdev files of Kind=wireguard the same way, and deletes each kernel WireGuard device that none declares.

The vrf and networkd effects fire together when the networkd output changes. The networkd effect waits for vrf to succeed, including when chalkd has to restart a failed chalkos-vrf.service. FRR reloads after its due network and firewall prerequisites. At boot, chalkos-vrf.service runs after chalkos-config.service and before networkd.

FRR runs an instance per VRF

The output frr reads the VRF and Network documents beside the BGP kinds. It renders a router bgp block per BGP document, the default VRF's first:

router bgp 64512
 bgp router-id 192.0.2.1
 no bgp default ipv4-unicast
 !
 address-family ipv4 unicast
  maximum-paths 1
  maximum-paths ibgp 1
 exit-address-family
 !
 address-family ipv6 unicast
  maximum-paths 1
  maximum-paths ibgp 1
  import vrf route-map rp-from-tenant
  import vrf tenant
 exit-address-family
exit
!
...
router bgp 64512 vrf tenant
 bgp router-id 192.0.2.1
 no bgp default ipv4-unicast
 neighbor peer-leaves peer-group
 neighbor peer-leaves remote-as external
 neighbor peer-leaves bfd
 neighbor peer-leaves bfd profile chalkos-tenant
 neighbor peer-leaves capability extended-nexthop
 neighbor swp1 interface peer-group peer-leaves
 neighbor swp2 interface peer-group peer-leaves

A peer group sits in the block of its peer's VRF. Prefix lists, community lists and route-maps are FRR's global objects, which every instance may use. Each instance with a BFD peer has a BFD profile of its timers: chalkos for the default VRF's, chalkos-<vrf> for another's. FRR finds the kernel's VRFs itself, so an instance whose VRF networkd has not created yet waits for it.

A peer's interfaces expand only over the links of the peer's VRF, as the membership rules above find them. For a peer of the default VRF, those are the links of no VRF. The validator router refuses a glob that matches links of other VRFs alone, a peer's interface that lies in another VRF and a peer that names a VRF's device, on which bgpd runs no session. It sees the VRF of a link as networkd puts it there, by the VRF documents and the Network documents' own VRF=, so the BGP kinds' rules leave these checks to it. chalkctl validate runs it too.

A route-map's set as-path prepend repeats an AS number, and one route-map serves every instance that names its policy. The renderer prepends the AS number of the instances that use the policy, and refuses a policy with prepend that instances of two AS numbers use.

FRR keeps its default instance while another VRF's runs, so a reload cannot remove it. A rule therefore refuses a delivery that drops the default VRF's BGP document and keeps another VRF's running instance.

Imported routes keep their next hop

import vrf copies the best paths of another VRF's instance into the importing instance. zebra installs them in the importing VRF's table with the next hop they had, on a link of the other VRF. The kernel routes a packet of the default VRF out of a VRF's link this way.

FRR marks an imported path as the instance's own and names the VRF it came from. chalkd reports that VRF, and chalkctl node routes prints it as the path's peer, such as vrf tenant. When FRR names no VRF of the next hop, chalkd reports the route distinguisher the path came from instead.

For a packet that a VRF's link delivers to the router, the input hook of nftables sees the VRF's device as iifname and the link as sdifname. The input chain therefore accepts the ports of a BGPPeer with vrf and interface or interfaces by both:

iifname "tenant" meta sdifname { "swp*" } tcp dport { 179 } accept comment "BGPPeer leaves"
iifname "tenant" meta sdifname { "swp*" } udp dport { 3784, 3785 } accept comment "BGPPeer leaves"

The rule matches the peer's globs against the links of its VRF alone, so a link of another VRF whose name the glob matches stays closed. A peer of the default VRF keeps its iifname rules, which match no VRF's link.

A peer of address with vrf is accepted from that address on the VRF's device, so the same address in another VRF stays closed:

iifname "internet" ip saddr { 203.0.113.254/32 } tcp dport { 179 } accept comment "BGPPeer transit"

The rule of a peer of address in the default VRF names no interface, as on a router without VRFs. No socket of the default VRF takes a connection from a VRF's link, so bgpd answers it in the default VRF alone.

Zones and conntrack zones take VRFs

A FirewallZone holds whole VRFs in vrfs or single links of a VRF in interfaces. A VRF with conntrackZone keeps its connections in a conntrack zone of its own, and the rules refuse an import into or out of such a VRF. The router's firewall shows how the ruleset matches them.

NAT64s and resolvers join their VRFs

A NAT64 with vrf runs in a network namespace of its own, as every NAT64 does. Its link n64-<name> is a member of the VRF that no VRF document names: chalkos-nat64.service puts it into the VRF's device itself, and the output networkd renders a .network that leaves it to the unit. The VRF's table routes the NAT64's prefix and explicit pool to the link, and the forward chain matches the link's packets by meta sdifname. The renderer adds the link to the VRF's conntrack zone, so a connection and its translation share the zone. No VRF takes a name that starts with n64-, and no FirewallZone names or globs a NAT64's link. Each instance runs in a namespace of its own shows the path of a packet.

A Resolver with vrf runs as unbound@<vrf>.service. Its unit sets BindNetworkInterface=<vrf>, so systemd binds each of Unbound's sockets to the VRF's device: the resolver answers on the VRF's links and sends its queries through the VRF's table, while the unit keeps Unbound's hardening. ip vrf exec would need privileges that the hardening takes away. systemd binds the sockets to the index the device has when the unit starts. When networkd creates the VRF anew, chalkos-resolvers.service restarts the resolver on the new device. The input chain opens the resolver's port 53 with the VRF's device as iifname.

chalkd reads the VRFs' state

chalkctl status lists the sessions of every VRF. chalkd asks vtysh for show bgp vrf all neighbors json and reports each session's VRF, the default VRF's sessions first.

The Routes call takes a VRF. chalkd reads its table from the node's running VRF document, below /run/chalkos/config/router.chalkos.dev/VRF, lists the kernel's routes of that table and asks for show bgp vrf <name> ipv4 unicast detail json and its IPv6 twin. A request names a VRF or a table, not both, and default is the main table and the default VRF's instance. A VRF of no running document is refused with the node has no VRF <name>.

The diagnostics bgp-summary, bgp-neighbor, bgp-route and bfd-peers take a vrf argument, default unless it is given, and pass it to vtysh as vrf <name>.

The status lists each VRF's resolver after the default VRF's, as resolver <vrf>: running. chalkd asks systemd whether unbound@<vrf>.service is active, and compares the device index that the resolver recorded with the index of the VRF's device. A resolver still bound to a device that networkd has since replaced does not count as running.

Limits

  • The kernel's rule for the local table comes before the VRFs' tables. A VRF's packet to an address of a default VRF's link is delivered to the router, and the kernel answers it, such as with an echo reply or a TCP reset. chalkd stays out of reach, as no socket of the default VRF takes it. The router's IPv6 echo reply to an address that a VRF and the default VRF both hold leaves through the default VRF.
  • No protected rule covers the router's own traffic to its addresses in a VRF. The VRF's policies to local decide it.
  • The rules read interface names and globs without knowing which links networkd matches. The validator router compares them with the names in the Network documents and cannot see the links the node has.
  • networkd also matches Name= against a link's alternative names, which the documents do not give. The validator and the renderers read interface names alone.
  • Routes leak between VRFs through BGP alone.