VRFs on a router¶
A router image turns each VRF document into a kernel VRF through networkd, runs a BGP instance
per VRF in FRR and imports routes between them with FRR's import vrf. For the edge of
Separate routing with VRFs, chalkctl status lists sessions in two VRFs:
bgp:
VRF PEER NEIGHBOR STATE UPTIME RECEIVED SENT BFD
internet transit 203.0.113.254 Established 2h0m0s 812 1 -
tenant leaves swp1 Established 1h2m3s 4 9 up
tenant leaves swp2 Established 1h2m3s 4 9 up
The node's own services stay in the default VRF, which holds every link that no VRF holds.
Management stays in the default VRF¶
chalkd, chrony, the node's own name resolution, image pulls and the health check's connection to chalkd run in the default VRF and know nothing of VRFs. The management VRF is the default VRF, together with the check that no VRF holds a management interface. Data-plane traffic goes into named VRFs.
The output router-sysctl sets net.ipv4.tcp_l3mdev_accept, udp_l3mdev_accept and
raw_l3mdev_accept to 0. The three settings cover IPv6 as well. A socket of the default VRF then
takes no connection or datagram from a VRF's link, so chalkd cannot be reached through a
data-plane VRF, even where a firewall rule would let the packet in. The kernel's default for raw
sockets is 1, so the router sets all three.
Two checks guard the node API. The
validator router refuses an interface or glob of
Router.spec.management.interfaces that may be a link of a VRF, by a VRF document's
interfaces or by a Network's own VRF=, or that matches a VRF's device. It reads the links
as the membership below finds them. While the
node API answers on every interface, a rule wants a Network document that names at least one
link that no VRF holds. Neither can tell which link chalkctl reaches the router through, so a
delivery with config apply --try covers the rest: the router rolls the config back unless
chalkctl confirms it over a new connection.
The output networkd renders the VRFs¶
On a router image the output networkd reads the VRF documents beside networkd's own and
renders them with chalkos-render router-networkd. The output keeps networkd's directory,
/run/systemd/network, which no other output may share. The VRF tenant renders
00-vrf-tenant.netdev:
It also renders 00-vrf-tenant.network, which brings the device up, as networkd leaves a device
without a matching .network down. RequiredForOnline=no leaves the device out of the node's
online state. Its routes end the VRF's table in unreachable default routes:
[Match]
Name=tenant
[Link]
RequiredForOnline=no
[Route]
Destination=0.0.0.0/0
Metric=4278198272
Table=200
Type=unreachable
[Route]
Destination=::/0
Metric=4278198272
Table=200
Type=unreachable
When the VRF's table has no route for a destination, the kernel's lookup goes on to the main
table. Without these routes, a VRF's traffic would follow the default VRF's routes and
leave through its links. FRR reads the metric as distance 255, so a default route of BGP or of a
member's Network wins over them. A default route among a VRF instance's networks would match
them at all times, so the rules refuse one and point to a peer family's defaultOriginate
instead.
The Network documents of the VRF's links get VRF= in a [Network] section:
Membership follows how networkd picks a Network¶
networkd applies to a link the first .network file, by file name, whose [Match] matches it.
The router's model in pkg/router/derive, which the validator router and the renderers share,
reads the Network documents the same way, so VRF= lands on the file that networkd applies to
each of a VRF's links:
- It reads
Name=as systemd does. A value splits into words at white space, an empty value resets the words before it, and a value whose first word starts with!negates all its words. It drops the words systemd drops, such as an interface index orall. - A literal interface of a
VRFdocument is the link of that name. A glob stands for the links that theNetworkdocuments may match, up to the firstNetworkthat matches all of them. - A
Networkwith[Match]keys besidesName=, such asMACAddress=, may match a link that itsName=matches, or not. The model cannot tell, so it reads later documents too, and the validator refuses a VRF's link that only such aNetworkmatches. - A
Networkgets the VRF of the links it is the first to match. Every link it may match belongs to that VRF, or the validator refuses the document. - A
Networkthat sorts before00-vrf-<name>and matches the VRF's device is refused, because networkd would apply it to the device.
The validator also refuses a link that globs of two VRFs may hold and a Network whose own VRF=
names another VRF. A Network whose own VRF= an earlier Network hides from a link is
refused, with the remedies: give the earlier one the same VRF=, rename it to sort after, or drop
the VRF=. It refuses a backslash in Name=, which systemd reads as an escape, and a bracket
expression it cannot match as fnmatch does, such as eth[0 or a class like [[:digit:]]. The
validator runs after the rules and before the renderers, at a delivery, at
chalkctl config apply --dry-run, in chalkctl validate and at boot. It refuses the VRFs' links
first. The checks that read the links run once those pass, and report every fault they find at
once.
The validator reads no NetDev and no unit names. The renderer of the output networkd refuses
a networkd document named as a VRF's units and a NetDev that creates a device of a VRF's name,
in any [NetDev] section and any value of its Name=. These checks run when the router renders
its config, at a delivery or at chalkctl config apply --dry-run; chalkctl validate does not
run them.
chalkos-vrf removes what networkd keeps¶
networkctl reload creates a VRF whose .netdev is new and moves links in and out of VRFs as
their .network files say. It does not change the table of a VRF that exists, and it does not
remove a VRF whose .netdev went. chalkos-vrf.service covers both. It reads the .netdev files
of Kind=vrf from /etc/systemd/network, /run/systemd/network, /usr/local/lib/systemd/network
and /usr/lib/systemd/network, with the first of a name winning as networkd reads them. It then
removes each kernel VRF that no file declares with its table.
The unreachable default routes stay in a table after its VRF goes, and a VRF that takes the table
later would find them there. chalkos-vrf.service therefore flushes them from the old table
first, and deletes the VRF only once they went. A run that fails between the two finds the VRF
again and flushes once more.
After the cleanup effect succeeds, chalkd reloads networkd. It creates the VRF with its
new table and puts its links back into it. A change of table therefore ends the sessions
of the VRF, and they start over. The resolvers effect checks the new table before
reconciling the resolvers on the new device.
networkd keeps a WireGuard device whose .netdev went too, with its keys and peers, so a removed
WireGuard document would leave its tunnel working. chalkos-vrf.service reads the .netdev files
of Kind=wireguard the same way, and deletes each kernel WireGuard device that none declares.
The vrf and networkd effects fire together when the networkd output changes. The
networkd effect waits for vrf to succeed, including when chalkd has to restart a failed
chalkos-vrf.service. FRR reloads after its due network and firewall prerequisites. At boot,
chalkos-vrf.service runs after chalkos-config.service and before networkd.
FRR runs an instance per VRF¶
The output frr reads the VRF and Network documents beside the BGP kinds. It renders a
router bgp block per BGP document, the default VRF's first:
router bgp 64512
bgp router-id 192.0.2.1
no bgp default ipv4-unicast
!
address-family ipv4 unicast
maximum-paths 1
maximum-paths ibgp 1
exit-address-family
!
address-family ipv6 unicast
maximum-paths 1
maximum-paths ibgp 1
import vrf route-map rp-from-tenant
import vrf tenant
exit-address-family
exit
!
...
router bgp 64512 vrf tenant
bgp router-id 192.0.2.1
no bgp default ipv4-unicast
neighbor peer-leaves peer-group
neighbor peer-leaves remote-as external
neighbor peer-leaves bfd
neighbor peer-leaves bfd profile chalkos-tenant
neighbor peer-leaves capability extended-nexthop
neighbor swp1 interface peer-group peer-leaves
neighbor swp2 interface peer-group peer-leaves
A peer group sits in the block of its peer's VRF. Prefix lists, community lists and route-maps are
FRR's global objects, which every instance may use. Each instance with a BFD peer has a BFD
profile of its timers: chalkos for the default VRF's, chalkos-<vrf> for another's. FRR finds
the kernel's VRFs itself, so an instance whose VRF networkd has not created yet waits for it.
A peer's interfaces expand only over the links of the peer's VRF, as the membership rules above
find them. For a peer of the default VRF, those are the links of no VRF. The validator router
refuses a glob that matches links of other VRFs alone, a peer's interface that lies in another
VRF and a peer that names a VRF's device, on which bgpd runs no session. It sees the VRF of a
link as networkd puts it there, by the VRF documents and the Network documents' own VRF=,
so the BGP kinds' rules leave these checks to it. chalkctl validate runs it too.
A route-map's set as-path prepend repeats an AS number, and one route-map serves every instance
that names its policy. The renderer prepends the AS number of the instances that use the policy,
and refuses a policy with prepend that instances of two AS numbers use.
FRR keeps its default instance while another VRF's runs, so a reload cannot remove it. A rule
therefore refuses a delivery that drops the default VRF's BGP document and keeps another VRF's
running instance.
Imported routes keep their next hop¶
import vrf copies the best paths of another VRF's instance into the importing instance. zebra
installs them in the importing VRF's table with the next hop they had, on a link of the other VRF.
The kernel routes a packet of the default VRF out of a VRF's link this way.
FRR marks an imported path as the instance's own and names the VRF it came from. chalkd reports
that VRF, and chalkctl node routes prints it as the path's peer, such as vrf tenant. When FRR
names no VRF of the next hop, chalkd reports the route distinguisher the path came from instead.
The firewall accepts a VRF peer's ports on its links¶
For a packet that a VRF's link delivers to the router, the input hook of nftables sees the VRF's
device as iifname and the link as sdifname. The input chain therefore accepts the ports of a
BGPPeer with vrf and interface or interfaces by both:
iifname "tenant" meta sdifname { "swp*" } tcp dport { 179 } accept comment "BGPPeer leaves"
iifname "tenant" meta sdifname { "swp*" } udp dport { 3784, 3785 } accept comment "BGPPeer leaves"
The rule matches the peer's globs against the links of its VRF alone, so a link of another VRF
whose name the glob matches stays closed. A peer of the default VRF keeps its iifname rules,
which match no VRF's link.
A peer of address with vrf is accepted from that address on the VRF's device, so the same
address in another VRF stays closed:
The rule of a peer of address in the default VRF names no interface, as on a router without
VRFs. No socket of the default VRF takes a connection from a VRF's link, so bgpd answers it in
the default VRF alone.
Zones and conntrack zones take VRFs¶
A FirewallZone holds whole VRFs in vrfs or single links of a VRF in interfaces. A VRF with
conntrackZone keeps its connections in a conntrack zone of its own, and the rules refuse an
import into or out of such a VRF. The router's firewall shows how
the ruleset matches them.
NAT64s and resolvers join their VRFs¶
A NAT64 with vrf runs in a network namespace of its own, as every NAT64 does. Its link
n64-<name> is a member of the VRF that no VRF document names: chalkos-nat64.service puts it
into the VRF's device itself, and the output networkd renders a .network that leaves it to the
unit. The VRF's table routes the NAT64's prefix and explicit pool to the link, and the forward
chain matches the link's packets by meta sdifname. The renderer adds the link to the VRF's
conntrack zone, so a connection and its translation share the zone. No VRF takes a name that
starts with n64-, and no FirewallZone names or globs a NAT64's link.
Each instance runs in a namespace of its own
shows the path of a packet.
A Resolver with vrf runs as unbound@<vrf>.service. Its unit sets
BindNetworkInterface=<vrf>, so systemd binds each of Unbound's sockets to the VRF's device: the
resolver answers on the VRF's links and sends its queries through the VRF's table, while the unit
keeps Unbound's hardening. ip vrf exec would need privileges that the hardening takes away.
systemd binds the sockets to the index the device has when the unit starts. When networkd creates
the VRF anew, chalkos-resolvers.service restarts the resolver on the new device. The input
chain opens the resolver's port 53 with the VRF's device as iifname.
chalkd reads the VRFs' state¶
chalkctl status lists the sessions of every VRF. chalkd asks vtysh for
show bgp vrf all neighbors json and reports each session's VRF, the default VRF's sessions
first.
The Routes call takes a VRF. chalkd reads its table from the node's running VRF document,
below /run/chalkos/config/router.chalkos.dev/VRF, lists the kernel's routes of that table and
asks for show bgp vrf <name> ipv4 unicast detail json and its IPv6 twin. A request names a VRF
or a table, not both, and default is the main table and the default VRF's instance. A VRF of no
running document is refused with the node has no VRF <name>.
The diagnostics bgp-summary, bgp-neighbor, bgp-route and bfd-peers take a vrf argument,
default unless it is given, and pass it to vtysh as vrf <name>.
The status lists each VRF's resolver after the default VRF's, as resolver <vrf>: running. chalkd
asks systemd whether unbound@<vrf>.service is active, and compares the device index that the
resolver recorded with the index of the VRF's device. A resolver still bound to a device that
networkd has since replaced does not count as running.
Limits¶
- The kernel's rule for the local table comes before the VRFs' tables. A VRF's packet to an address of a default VRF's link is delivered to the router, and the kernel answers it, such as with an echo reply or a TCP reset. chalkd stays out of reach, as no socket of the default VRF takes it. The router's IPv6 echo reply to an address that a VRF and the default VRF both hold leaves through the default VRF.
- No protected rule covers the router's own traffic to its addresses in a VRF. The VRF's
policies to
localdecide it. - The rules read interface names and globs without knowing which links networkd matches. The
validator
routercompares them with the names in theNetworkdocuments and cannot see the links the node has. - networkd also matches
Name=against a link's alternative names, which the documents do not give. The validator and the renderers read interface names alone. - Routes leak between VRFs through BGP alone.
Related pages¶
- Separate routing with VRFs configures VRFs.
- BGP on a router: FRR's config, the passwords and the reload.
- How a router works: the layer's outputs and the boot order.
- The router's firewall: the input chain, zones of VRFs and conntrack zones.