Skip to content

Separate routing with VRFs

A VRF gives a set of links a routing table of their own. The router routes a packet that arrives on a VRF's link by that table alone, so two VRFs may use the same addresses. This page puts links into VRFs, runs BGP in each, imports routes from one VRF into another and checks the result. VRFs on a router explains how chalkos renders them.

Before you begin

You need a router set up as Set up a router describes, and the secrets file. The examples configure the edge r1. Its management link eth0 stays out of every VRF. Its uplink eth1 goes into the VRF internet, and eth2 and the swp links to a tenant's leaves go into the VRF tenant. Each link has a Network document that names it in [Match] Name=:

apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-mgmt}
spec:
  match: {name: eth0}
  network: {dhcp: "yes"}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 20-internet}
spec:
  match: {name: eth1}
  network: {address: [203.0.113.1/24], ipv6AcceptRA: false}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 30-tenant}
spec:
  match: {name: "eth2 swp1 swp2"}
  network: {ipv6AcceptRA: false}

A VRF reaches a link through the Network document that networkd picks for it, so a link without one cannot join a VRF. Name links by their interface names, without other [Match] keys.

Keep the node API in the default VRF

The default VRF holds every link that no VRF holds. chalkd, chrony, the node's own name resolution, image pulls and every other service of the node run there and in no other VRF. The router sets net.ipv4.tcp_l3mdev_accept, udp_l3mdev_accept and raw_l3mdev_accept to 0, so a socket of the default VRF takes no connection or datagram from a VRF's link, whatever the firewall lets in.

Keep the link you reach the router through out of every VRF. The Router document of r1 names it as the node API's interface:

apiVersion: router.chalkos.dev/v1alpha1
kind: Router
metadata: {name: router}
spec:
  management: {interfaces: [eth0]}

The validator router refuses a management interface or glob that may be a link of a VRF. It reads the links that a VRF document holds and those that a Network's own VRF= puts in a VRF, so a VRF's eth* holds the management interface eth0:

chalkctl: r1: the config is refused: Router/router: spec.management.interfaces[0]: the management interface eth0 may be a link of VRF tenant, and chalkd answers in the default VRF alone; keep the node API's interfaces out of every VRF

It also refuses a management interface or glob that matches a VRF's device, which takes the VRF's name, such as ten* beside the VRF tenant:

chalkctl: r1: the config is refused: Router/router: spec.management.interfaces[0]: the management interface ten* matches tenant, the device of VRF tenant, and chalkd answers in the default VRF alone; name the default VRF's links

chalkctl validate and chalkctl config apply run the validator before chalkctl sends anything, and the router runs it again at the delivery.

A router without management interfaces answers the node API on every interface. The rules then want at least one link that a Network document names and no VRF holds. They cannot know which link chalkctl reaches the router through, so deliver every change of VRFs as a trial. The router rolls the trial back when chalkctl cannot reach it over a new connection:

chalkctl config apply r1 --try 2m --secrets secrets.age

Declare the VRFs

A VRF document names the VRF, which is also the name of its kernel device, and gives its routing table and its links:

apiVersion: router.chalkos.dev/v1alpha1
kind: VRF
metadata: {name: internet}
spec: {table: 100, interfaces: [eth1]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: VRF
metadata: {name: tenant}
spec: {table: 200, interfaces: [eth2, "swp*"]}

A name has 15 characters at most, and default names the default VRF. A table belongs to one VRF and is none of 253, 254 and 255, the kernel's tables of the default VRF. An interface is a name or a glob of a prefix and *, such as swp*. A glob stands for the links that the Network documents name, and an interface or glob belongs to one VRF.

The router adds VRF= to the Network that networkd picks for each of a VRF's links. All links that one Network may match belong to the same VRF, or to none. 30-tenant matches eth2, swp1 and swp2, which tenant holds, so it gets VRF=tenant. A Network of Name=swp* while tenant holds swp1 alone is refused, because it may match a link outside the VRF.

A Network may set network.vrf itself, for a VRF of a NetDev of kind: vrf that you write yourself. It may not name another VRF than the VRF document that holds its links.

Each VRF's table ends in unreachable default routes of both families. The router drops a packet that the VRF's table has no route for, so the packet never follows the default VRF's routes. A default route of BGP or of a member's Network takes precedence over them. chalkctl node routes --vrf lists them as routes of the type unreachable and the protocol static:

chalkctl node routes r1 --vrf tenant --protocol static --secrets secrets.age
DESTINATION  TYPE         PROTOCOL  TABLE  METRIC      NEXT HOPS
0.0.0.0/0    unreachable  static    200    4278198272  -
::/0         unreachable  static    200    4278198272  lo

Run BGP in a VRF

A router has one BGP document per VRF. spec.vrf names the VRF, and a document without it, or with vrf: default, serves the default VRF:

apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: bgp}
spec:
  asn: 64512
  routerID: 192.0.2.1
---
apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: internet}
spec:
  vrf: internet
  asn: 64512
  routerID: 192.0.2.1
  networks: [{prefix: 198.51.100.0/24}]
---
apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: tenant}
spec:
  vrf: tenant
  asn: 64512
  routerID: 192.0.2.1
  bfd: {receiveMs: 100, transmitMs: 100}

Each instance has its own AS number, router ID, multipath, BFD timers, graceful restart and networks. It announces a network while its VRF's table holds a route to it. The unreachable default routes would match a default route among a VRF instance's networks at all times, so the rules refuse one:

chalkctl: r1: the config is refused: BGP tenant announces the network ::/0 in VRF tenant, which would always be announced, as a VRF's table ends in an unreachable default route; use a peer family's defaultOriginate

Send a VRF's peers a default route with defaultOriginate in the peer's family instead.

A BGPPeer names its VRF the same way:

apiVersion: router.chalkos.dev/v1alpha1
kind: BGPPeer
metadata: {name: transit}
spec:
  vrf: internet
  address: 203.0.113.254
  remoteAS: 64496
  families: {ipv4: {}}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: BGPPeer
metadata: {name: leaves}
spec:
  vrf: tenant
  interfaces: ["swp*"]
  remoteAS: external
  bfd: true
  extendedNextHop: true
  families: {ipv4: {}, ipv6: {}}

A peer's VRF needs a BGP document. A peer's interfaces stand for links of its VRF, so swp* in tenant runs a session on swp1 and one on swp2. In the default VRF, a glob matches the links that no VRF holds. A peer's interface is a link of its own VRF, and a link that no Network names is in the default VRF. The validator router checks a peer's links against its VRF, and chalkctl validate runs it. The firewall accepts the BGP and BFD ports of a VRF's peer on the links of that VRF, and those of a peer of address from that address in its VRF alone.

Import routes from another VRF

spec.import of a BGP document takes the routes of other VRFs' BGP instances, per family:

apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: bgp}
spec:
  asn: 64512
  routerID: 192.0.2.1
  import:
    ipv6: {vrfs: [tenant], policy: from-tenant}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: tenant}
spec:
  vrf: tenant
  asn: 64512
  routerID: 192.0.2.1
  bfd: {receiveMs: 100, transmitMs: 100}
  import:
    ipv4: {vrfs: [internet]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: RoutingPolicy
metadata: {name: from-tenant}
spec:
  terms: [{match: {prefixList: tenant}, then: {action: accept}}]
---
apiVersion: router.chalkos.dev/v1alpha1
kind: PrefixList
metadata: {name: tenant}
spec:
  family: ipv6
  entries: [{prefix: "2001:db8:100::/40", le: 64}]

The default VRF takes the tenant's IPv6 routes within 2001:db8:100::/40, and tenant takes every IPv4 route of internet. The kernel installs an imported route in the importing VRF's table with the next hop it had, on a link of the VRF it came from. default names the default VRF's instance.

policy names a RoutingPolicy that decides which routes the instance takes from all of vrfs, and without one it takes every route. FRR has one such policy per family, so its prefix lists are of that family. An instance imports no routes of its own VRF, and each VRF it names needs a BGP document. Only BGP's routes are imported. The static routes of a VRF's Network documents stay in that VRF.

An import carries routes one way. A host of the importing VRF reaches the other VRF's networks through it, but the replies need a route back, so the other VRF imports the first one's routes as well. Filter and translate a VRF's traffic gives internet an import of tenant's IPv4 routes for this reason.

A RoutingPolicy with prepend repeats the AS number of the instances that use it, so instances of different AS numbers cannot share it.

Filter and translate a VRF's traffic

A FirewallZone holds whole VRFs or single links of a VRF, as Filter and translate traffic describes. The tenant reaches the internet through the VRF internet, which masquerades its IPv4. internet imports tenant's IPv4 routes, so the replies find their way back:

apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: internet}
spec:
  vrf: internet
  asn: 64512
  routerID: 192.0.2.1
  networks: [{prefix: 198.51.100.0/24}]
  import:
    ipv4: {vrfs: [tenant]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: FirewallZone
metadata: {name: internet}
spec: {vrfs: [internet]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: FirewallZone
metadata: {name: tenant}
spec: {vrfs: [tenant]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: FirewallPolicy
metadata: {name: tenant-internet}
spec: {from: tenant, to: internet, default: accept}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: FirewallPolicy
metadata: {name: internet-local}
spec: {from: internet, to: local}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: NATRule
metadata: {name: internet-out}
spec:
  type: masquerade
  out: {zone: internet}

internet-local drops every connection from the internet to the router, apart from what the firewall accepts before the policies, such as the transit's BGP session.

Keep a VRF's connections apart

conntrack tells connections apart by their addresses and ports. Two VRFs that use the same addresses therefore share connections. A host of one VRF could then pass the firewall as part of a connection that a host of the other VRF opened. A conntrack zone of its own keeps a VRF's connections, and their translations, apart:

apiVersion: router.chalkos.dev/v1alpha1
kind: VRF
metadata: {name: guest}
spec: {table: 300, interfaces: [eth3, eth4], conntrackZone: 1}

A zone is a number from 1 to 65535 that no other VRF uses. The default VRF and every VRF without a zone share zone 0. The link of a VRF's NAT64 joins the VRF's zone, so a connection and its translation share it. Giving a VRF a conntrackZone, or changing it, drops the VRF's established and translated connections, as their packets then belong to another zone. Deliver the change when the VRF's hosts can reconnect.

A VRF with a zone neither imports routes from another VRF nor gives its routes to one, because the replies of leaked connections would land in another zone. The rules refuse both:

VRF guest: a VRF with a conntrack zone leaks no routes through a BGP import, as its connections would answer in another zone; drop the zone or the import

Check that it worked

Deliver the documents as a trial, as above. chalkctl status lists the sessions with their VRF, the default VRF's first:

chalkctl status r1 --secrets secrets.age
bgp:
  VRF       PEER     NEIGHBOR       STATE        UPTIME  RECEIVED  SENT  BFD
  internet  transit  203.0.113.254  Established  2h0m0s  812       1     -
  tenant    leaves   swp1           Established  1h2m3s  4         9     up
  tenant    leaves   swp2           Established  1h2m3s  4         9     up

chalkctl node routes --vrf lists a VRF's table and the paths of its instance:

chalkctl node routes r1 --vrf tenant --secrets secrets.age

Without --vrf, the command lists the main table and the default VRF's paths. A path that an instance imported names its VRF as its peer:

chalkctl node routes r1 2001:db8:100::/40 --secrets secrets.age
DESTINATION        TYPE     PROTOCOL  TABLE  METRIC  NEXT HOPS
2001:db8:100::/48  unicast  bgp       254    20      fe80::b817:4cff:fe8b:9cf on swp1

BGP PREFIX         BEST  AS PATH  PEER        NEXT HOPS                 LOCAL PREF  MED  COMMUNITIES
2001:db8:100::/48  *     64601    vrf tenant  fe80::b817:4cff:fe8b:9cf  -           0    -

The route sits in the main table, and its next hop is on swp1, a link of tenant. --vrf and --table exclude each other, and --vrf default is the main table. A node whose image does not know VRFs cannot list a VRF's routes, and --vrf fails on it. The diagnostics bgp-summary, bgp-neighbor, bgp-route and bfd-peers take vrf, which is default when it is left out:

chalkctl node diag r1 --secrets secrets.age bgp-summary vrf=tenant
chalkctl node diag r1 --secrets secrets.age bgp-route vrf=internet family=ipv4 prefix=198.51.100.0/24

Change a VRF's table

A change of table creates the VRF anew. Its links leave it and join it again, and the sessions in it end and start over. Deliver the change as a trial, as any change of VRFs.

Remove a VRF

Remove a VRF's BGPPeer, BGP, NAT64, Resolver and VRF documents in one delivery. The rules refuse a VRF document's removal while another document still names the VRF. Once the VRF is gone, its links are in the default VRF.

FRR keeps the default VRF's instance while another VRF's runs. The rules therefore refuse a delivery that removes the default VRF's BGP document and keeps another VRF's instance running:

chalkctl: r1: the config is refused: the config drops the default VRF's BGP document and keeps a running BGP instance of another VRF, which FRR does not allow; drop the other VRFs' BGP documents in the same delivery, or keep the default VRF's

Fix a refused delivery

The rules read the documents' names and globs without knowing which links networkd matches. The validator router checks the rest against the Network documents. chalkctl validate and chalkctl config apply run it before chalkctl sends anything, and the router runs it again at the delivery, after the rules and before the renderers. It checks the VRFs' links first, and what reads the links only once they pass. Within each step it reports every fault it finds at once, and each fault once. chalkctl prints the refusals after the config is refused:, as it prints a rule's, with the document and the field each is about, and joins several with ;:

chalkctl: r1: the config is refused: Network/20-both: spec.match: Network 20-both matches links of no VRF and VRF a; the links of one Network are in one VRF

The validator refuses what networkd, FRR or nftables would apply otherwise than the documents say:

Message Fix
VRF/tenant: spec.interfaces[0]: VRF tenant names the interface eth9, which no Network document's [Match] Name= matches without other [Match] keys; VRF= reaches a link through its Network document, so give it one Add a Network that names the link in Name= without other [Match] keys.
Network/20-both: spec.match: Network 20-both matches links of no VRF and VRF a; the links of one Network are in one VRF Split the Network, so that each matches the links of one VRF.
Network/10-swp: spec.match.name: Network 10-swp's Name=swp* may match links of VRF tenant and links outside it; name its links Name the links in Name=, or put the whole glob into the VRF.
Network/00-all: metadata.name: Network 00-all matches the device of VRF tenant before 00-vrf-tenant; rename it to sort after 00-vrf-tenant Rename the Network, or narrow its Name=.
Network/10-a: spec.match: Network 10-a is the first to match eth5, so networkd never applies Network 20-b's VRF=mgmt to it; give 10-a VRF=mgmt, rename it to sort after 20-b, or drop 20-b's VRF= Take one of the remedies the message lists.
Network/20-x: spec.match.name: Name= holds "eth[0", which is no glob the router can match as networkd does Write Name= without backslashes and bracket expressions.
BGPPeer/leaves: spec.interfaces[0]: swp* matches no link of VRF default, only links of other VRFs Give the peer the VRF of its links.
BGPPeer/x: spec.interface: interface tenant is the device of VRF tenant Name the VRF's links, not its device.
BGPPeer/x: spec.interface: interface eth2 is in the VRF tenant, not in the peer's VRF default Give the peer the VRF of its link, or name a link of the peer's VRF.
FirewallZone/x: spec.interfaces[0]: ten* matches tenant, the device of VRF tenant; name the VRF in vrfs Put the VRF into the zone's vrfs, or name its links.
FirewallZone/x: spec.interfaces[0]: FirewallZone x names eth2, which may be a link of VRF tenant, which FirewallZone tenant holds; give each link one zone Take the VRF out of the other zone's vrfs and name its links in each zone, or drop the link here.
Network/50-lab: spec.match.name: VRF tenant holds the links of Network 50-lab's Name=lab[12], which nftables cannot match; name them by a name or a prefix and * Write the Network's Name= as names or a prefix and *.
NATRule/n: spec.out.interface: inter* matches no link, only internet, the device of VRF internet; name the VRF's links or a zone of the VRF Name the VRF's links, or a zone that holds the VRF.
Router/router: spec.management.interfaces[0]: the management interface tenant matches tenant, the device of VRF tenant, and chalkd answers in the default VRF alone; name the default VRF's links Name links of the default VRF in management.interfaces.
Router/router: spec.management.interfaces[0]: the management interface eth0 may be a link of VRF tenant, and chalkd answers in the default VRF alone; keep the node API's interfaces out of every VRF Take the link out of the VRF's interfaces and out of its Network's VRF=, or name links of the default VRF in management.interfaces.

The renderer of the output networkd checks the unit names and the NetDev documents, which the validator does not read. It checks when the router renders its config, at a delivery or at chalkctl config apply --dry-run, and chalkctl validate does not run it. A renderer's refusal fails the delivery, and chalkctl prints it in the same form:

Message Fix
Network/00-all: metadata.name: Network 00-all matches the link n64-guest of NAT64 guest before 00-n64-guest; rename it to sort after 00-n64-guest Rename the Network, or narrow its Name=.
NetDev/20-dummy: spec.netDev.name: NetDev 20-dummy creates the device tenant, which VRF tenant creates; rename the VRF or the NetDev Rename the VRF, or the device in the NetDev's Name=.

Limits

  • The kernel looks up the router's own addresses before a VRF's table. A VRF's hosts therefore get the kernel's answers from the addresses of the default VRF's links, such as echo replies and TCP resets. No socket of the default VRF takes their connections, so chalkd stays out of their reach. The router's IPv6 echo reply to an address that a VRF and the default VRF both hold leaves through the default VRF.
  • No protected rule covers the router's own connections to its addresses in a VRF. They reach the input chain as connections from the VRF's links, and the VRF's policies to local decide them.
  • A VRF instance cannot announce a default route from its networks. Use a peer family's defaultOriginate.
  • networkd also matches Name= against a link's alternative names. The documents do not give them, so the validator router cannot tell which link a Network matches by an alternative name. Name links by their interface names.
  • Routes leak between VRFs through BGP alone.

What next