Separate routing with VRFs¶
A VRF gives a set of links a routing table of their own. The router routes a packet that arrives on a VRF's link by that table alone, so two VRFs may use the same addresses. This page puts links into VRFs, runs BGP in each, imports routes from one VRF into another and checks the result. VRFs on a router explains how chalkos renders them.
Before you begin¶
You need a router set up as Set up a router describes, and the secrets file. The
examples configure the edge r1. Its management link eth0 stays out of every VRF. Its uplink
eth1 goes into the VRF internet, and eth2 and the swp links to a tenant's leaves go into
the VRF tenant. Each link has a Network document that names it in [Match] Name=:
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-mgmt}
spec:
match: {name: eth0}
network: {dhcp: "yes"}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 20-internet}
spec:
match: {name: eth1}
network: {address: [203.0.113.1/24], ipv6AcceptRA: false}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 30-tenant}
spec:
match: {name: "eth2 swp1 swp2"}
network: {ipv6AcceptRA: false}
A VRF reaches a link through the Network document that networkd picks for it, so a link without
one cannot join a VRF. Name links by their interface names, without other [Match] keys.
Keep the node API in the default VRF¶
The default VRF holds every link that no VRF holds. chalkd, chrony, the node's own name
resolution, image pulls and every other service of the node run there and in no other VRF. The
router sets net.ipv4.tcp_l3mdev_accept, udp_l3mdev_accept and raw_l3mdev_accept to 0, so a
socket of the default VRF takes no connection or datagram from a VRF's link, whatever the firewall
lets in.
Keep the link you reach the router through out of every VRF. The Router document of r1 names
it as the node API's interface:
apiVersion: router.chalkos.dev/v1alpha1
kind: Router
metadata: {name: router}
spec:
management: {interfaces: [eth0]}
The validator router refuses a management
interface or glob that may be a link of a VRF. It reads the links that a VRF document holds and
those that a Network's own VRF= puts in a VRF, so a VRF's eth* holds the management
interface eth0:
chalkctl: r1: the config is refused: Router/router: spec.management.interfaces[0]: the management interface eth0 may be a link of VRF tenant, and chalkd answers in the default VRF alone; keep the node API's interfaces out of every VRF
It also refuses a management interface or glob that matches a VRF's device, which takes the VRF's
name, such as ten* beside the VRF tenant:
chalkctl: r1: the config is refused: Router/router: spec.management.interfaces[0]: the management interface ten* matches tenant, the device of VRF tenant, and chalkd answers in the default VRF alone; name the default VRF's links
chalkctl validate and chalkctl config apply run the validator before chalkctl sends anything,
and the router runs it again at the delivery.
A router without management interfaces answers the node API on every interface. The rules then
want at least one link that a Network document names and no VRF holds. They cannot know which
link chalkctl reaches the router through, so deliver every change of VRFs as a trial. The router
rolls the trial back when chalkctl cannot reach it over a new connection:
Declare the VRFs¶
A VRF document names the VRF, which is also the name of its kernel device, and gives its routing
table and its links:
apiVersion: router.chalkos.dev/v1alpha1
kind: VRF
metadata: {name: internet}
spec: {table: 100, interfaces: [eth1]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: VRF
metadata: {name: tenant}
spec: {table: 200, interfaces: [eth2, "swp*"]}
A name has 15 characters at most, and default names the default VRF. A table belongs to one VRF
and is none of 253, 254 and 255, the kernel's tables of the default VRF. An interface is a name or
a glob of a prefix and *, such as swp*. A glob stands for the links that the Network
documents name, and an interface or glob belongs to one VRF.
The router adds VRF= to the Network that networkd picks for each of a VRF's links. All links
that one Network may match belong to the same VRF, or to none. 30-tenant matches eth2,
swp1 and swp2, which tenant holds, so it gets VRF=tenant. A Network of Name=swp* while
tenant holds swp1 alone is refused, because it may match a link outside the VRF.
A Network may set network.vrf itself, for a VRF of a NetDev of kind: vrf that you write
yourself. It may not name another VRF than the VRF document that holds its links.
Each VRF's table ends in unreachable default routes of both families. The router drops a packet
that the VRF's table has no route for, so the packet never follows the default VRF's routes. A
default route of BGP or of a member's Network takes precedence over them.
chalkctl node routes --vrf lists them as routes of the type unreachable and the protocol
static:
DESTINATION TYPE PROTOCOL TABLE METRIC NEXT HOPS
0.0.0.0/0 unreachable static 200 4278198272 -
::/0 unreachable static 200 4278198272 lo
Run BGP in a VRF¶
A router has one BGP document per VRF. spec.vrf names the VRF, and a document without it,
or with vrf: default, serves the default VRF:
apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: bgp}
spec:
asn: 64512
routerID: 192.0.2.1
---
apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: internet}
spec:
vrf: internet
asn: 64512
routerID: 192.0.2.1
networks: [{prefix: 198.51.100.0/24}]
---
apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: tenant}
spec:
vrf: tenant
asn: 64512
routerID: 192.0.2.1
bfd: {receiveMs: 100, transmitMs: 100}
Each instance has its own AS number, router ID, multipath, BFD timers, graceful restart and
networks. It announces a network while its VRF's table holds a route to it. The unreachable
default routes would match a default route among a VRF instance's networks at all times, so the
rules refuse one:
chalkctl: r1: the config is refused: BGP tenant announces the network ::/0 in VRF tenant, which would always be announced, as a VRF's table ends in an unreachable default route; use a peer family's defaultOriginate
Send a VRF's peers a default route with defaultOriginate in the peer's family instead.
A BGPPeer names its VRF the same way:
apiVersion: router.chalkos.dev/v1alpha1
kind: BGPPeer
metadata: {name: transit}
spec:
vrf: internet
address: 203.0.113.254
remoteAS: 64496
families: {ipv4: {}}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: BGPPeer
metadata: {name: leaves}
spec:
vrf: tenant
interfaces: ["swp*"]
remoteAS: external
bfd: true
extendedNextHop: true
families: {ipv4: {}, ipv6: {}}
A peer's VRF needs a BGP document. A peer's interfaces stand for links of its VRF, so swp*
in tenant runs a session on swp1 and one on swp2. In the default VRF, a glob matches the
links that no VRF holds. A peer's interface is a link of its own VRF, and a link that no
Network names is in the default VRF. The validator router checks a peer's links against its
VRF, and chalkctl validate runs it. The firewall accepts the BGP and BFD ports of a VRF's peer
on the links of that VRF, and those of a peer of address from that address in its VRF alone.
Import routes from another VRF¶
spec.import of a BGP document takes the routes of other VRFs' BGP instances, per family:
apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: bgp}
spec:
asn: 64512
routerID: 192.0.2.1
import:
ipv6: {vrfs: [tenant], policy: from-tenant}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: tenant}
spec:
vrf: tenant
asn: 64512
routerID: 192.0.2.1
bfd: {receiveMs: 100, transmitMs: 100}
import:
ipv4: {vrfs: [internet]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: RoutingPolicy
metadata: {name: from-tenant}
spec:
terms: [{match: {prefixList: tenant}, then: {action: accept}}]
---
apiVersion: router.chalkos.dev/v1alpha1
kind: PrefixList
metadata: {name: tenant}
spec:
family: ipv6
entries: [{prefix: "2001:db8:100::/40", le: 64}]
The default VRF takes the tenant's IPv6 routes within 2001:db8:100::/40, and tenant takes
every IPv4 route of internet. The kernel installs an imported route in the importing VRF's table
with the next hop it had, on a link of the VRF it came from. default names the default VRF's
instance.
policy names a RoutingPolicy that decides which routes the instance takes from all of vrfs,
and without one it takes every route. FRR has one such policy per family, so its prefix lists are
of that family. An instance imports no routes of its own VRF, and each VRF it names needs a BGP
document. Only BGP's routes are imported. The static routes of a VRF's Network documents stay in
that VRF.
An import carries routes one way. A host of the importing VRF reaches the other VRF's networks
through it, but the replies need a route back, so the other VRF imports the first one's routes as
well. Filter and translate a VRF's traffic gives
internet an import of tenant's IPv4 routes for this reason.
A RoutingPolicy with prepend repeats the AS number of the instances that use it, so instances
of different AS numbers cannot share it.
Filter and translate a VRF's traffic¶
A FirewallZone holds whole VRFs or single links of a VRF, as
Filter and translate traffic describes. The tenant
reaches the internet through the VRF internet, which masquerades its IPv4. internet imports
tenant's IPv4 routes, so the replies find their way back:
apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: internet}
spec:
vrf: internet
asn: 64512
routerID: 192.0.2.1
networks: [{prefix: 198.51.100.0/24}]
import:
ipv4: {vrfs: [tenant]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: FirewallZone
metadata: {name: internet}
spec: {vrfs: [internet]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: FirewallZone
metadata: {name: tenant}
spec: {vrfs: [tenant]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: FirewallPolicy
metadata: {name: tenant-internet}
spec: {from: tenant, to: internet, default: accept}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: FirewallPolicy
metadata: {name: internet-local}
spec: {from: internet, to: local}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: NATRule
metadata: {name: internet-out}
spec:
type: masquerade
out: {zone: internet}
internet-local drops every connection from the internet to the router, apart from what the
firewall accepts before the policies, such as the transit's BGP session.
Keep a VRF's connections apart¶
conntrack tells connections apart by their addresses and ports. Two VRFs that use the same addresses therefore share connections. A host of one VRF could then pass the firewall as part of a connection that a host of the other VRF opened. A conntrack zone of its own keeps a VRF's connections, and their translations, apart:
apiVersion: router.chalkos.dev/v1alpha1
kind: VRF
metadata: {name: guest}
spec: {table: 300, interfaces: [eth3, eth4], conntrackZone: 1}
A zone is a number from 1 to 65535 that no other VRF uses. The default VRF and every VRF without
a zone share zone 0. The link of a VRF's NAT64 joins the VRF's zone, so a connection and its
translation share it. Giving a VRF a conntrackZone, or changing it, drops the VRF's established
and translated connections, as their packets then belong to another zone. Deliver the change when
the VRF's hosts can reconnect.
A VRF with a zone neither imports routes from another VRF nor gives its routes to one, because the replies of leaked connections would land in another zone. The rules refuse both:
VRF guest: a VRF with a conntrack zone leaks no routes through a BGP import, as its connections would answer in another zone; drop the zone or the import
Check that it worked¶
Deliver the documents as a trial, as above. chalkctl status lists the sessions with their VRF,
the default VRF's first:
bgp:
VRF PEER NEIGHBOR STATE UPTIME RECEIVED SENT BFD
internet transit 203.0.113.254 Established 2h0m0s 812 1 -
tenant leaves swp1 Established 1h2m3s 4 9 up
tenant leaves swp2 Established 1h2m3s 4 9 up
chalkctl node routes --vrf lists a VRF's table and the paths of its instance:
Without --vrf, the command lists the main table and the default VRF's paths. A path that an
instance imported names its VRF as its peer:
DESTINATION TYPE PROTOCOL TABLE METRIC NEXT HOPS
2001:db8:100::/48 unicast bgp 254 20 fe80::b817:4cff:fe8b:9cf on swp1
BGP PREFIX BEST AS PATH PEER NEXT HOPS LOCAL PREF MED COMMUNITIES
2001:db8:100::/48 * 64601 vrf tenant fe80::b817:4cff:fe8b:9cf - 0 -
The route sits in the main table, and its next hop is on swp1, a link of tenant. --vrf and
--table exclude each other, and --vrf default is the main table. A node whose image does
not know VRFs cannot list a VRF's routes, and --vrf fails on it. The diagnostics bgp-summary,
bgp-neighbor, bgp-route and bfd-peers take vrf, which is default when it is left out:
chalkctl node diag r1 --secrets secrets.age bgp-summary vrf=tenant
chalkctl node diag r1 --secrets secrets.age bgp-route vrf=internet family=ipv4 prefix=198.51.100.0/24
Change a VRF's table¶
A change of table creates the VRF anew. Its links leave it and join it again, and the sessions
in it end and start over. Deliver the change as a trial, as any change of VRFs.
Remove a VRF¶
Remove a VRF's BGPPeer, BGP, NAT64, Resolver and VRF documents in one delivery. The
rules refuse a VRF document's removal while another document still names the VRF. Once the VRF
is gone, its links are in the default VRF.
FRR keeps the default VRF's instance while another VRF's runs. The rules therefore refuse a
delivery that removes the default VRF's BGP document and keeps another VRF's instance running:
chalkctl: r1: the config is refused: the config drops the default VRF's BGP document and keeps a running BGP instance of another VRF, which FRR does not allow; drop the other VRFs' BGP documents in the same delivery, or keep the default VRF's
Fix a refused delivery¶
The rules read the documents' names and globs without knowing which links networkd matches. The
validator router checks the rest against the
Network documents. chalkctl validate and chalkctl config apply run it before chalkctl sends
anything, and the router runs it again at the delivery, after the rules and before the renderers.
It checks the VRFs' links first, and what reads the links only once they pass. Within each
step it reports every fault it finds at once, and each fault once. chalkctl prints the refusals
after the config is refused:, as it prints a rule's, with the document and the field each is
about, and joins several with ;:
chalkctl: r1: the config is refused: Network/20-both: spec.match: Network 20-both matches links of no VRF and VRF a; the links of one Network are in one VRF
The validator refuses what networkd, FRR or nftables would apply otherwise than the documents say:
| Message | Fix |
|---|---|
VRF/tenant: spec.interfaces[0]: VRF tenant names the interface eth9, which no Network document's [Match] Name= matches without other [Match] keys; VRF= reaches a link through its Network document, so give it one |
Add a Network that names the link in Name= without other [Match] keys. |
Network/20-both: spec.match: Network 20-both matches links of no VRF and VRF a; the links of one Network are in one VRF |
Split the Network, so that each matches the links of one VRF. |
Network/10-swp: spec.match.name: Network 10-swp's Name=swp* may match links of VRF tenant and links outside it; name its links |
Name the links in Name=, or put the whole glob into the VRF. |
Network/00-all: metadata.name: Network 00-all matches the device of VRF tenant before 00-vrf-tenant; rename it to sort after 00-vrf-tenant |
Rename the Network, or narrow its Name=. |
Network/10-a: spec.match: Network 10-a is the first to match eth5, so networkd never applies Network 20-b's VRF=mgmt to it; give 10-a VRF=mgmt, rename it to sort after 20-b, or drop 20-b's VRF= |
Take one of the remedies the message lists. |
Network/20-x: spec.match.name: Name= holds "eth[0", which is no glob the router can match as networkd does |
Write Name= without backslashes and bracket expressions. |
BGPPeer/leaves: spec.interfaces[0]: swp* matches no link of VRF default, only links of other VRFs |
Give the peer the VRF of its links. |
BGPPeer/x: spec.interface: interface tenant is the device of VRF tenant |
Name the VRF's links, not its device. |
BGPPeer/x: spec.interface: interface eth2 is in the VRF tenant, not in the peer's VRF default |
Give the peer the VRF of its link, or name a link of the peer's VRF. |
FirewallZone/x: spec.interfaces[0]: ten* matches tenant, the device of VRF tenant; name the VRF in vrfs |
Put the VRF into the zone's vrfs, or name its links. |
FirewallZone/x: spec.interfaces[0]: FirewallZone x names eth2, which may be a link of VRF tenant, which FirewallZone tenant holds; give each link one zone |
Take the VRF out of the other zone's vrfs and name its links in each zone, or drop the link here. |
Network/50-lab: spec.match.name: VRF tenant holds the links of Network 50-lab's Name=lab[12], which nftables cannot match; name them by a name or a prefix and * |
Write the Network's Name= as names or a prefix and *. |
NATRule/n: spec.out.interface: inter* matches no link, only internet, the device of VRF internet; name the VRF's links or a zone of the VRF |
Name the VRF's links, or a zone that holds the VRF. |
Router/router: spec.management.interfaces[0]: the management interface tenant matches tenant, the device of VRF tenant, and chalkd answers in the default VRF alone; name the default VRF's links |
Name links of the default VRF in management.interfaces. |
Router/router: spec.management.interfaces[0]: the management interface eth0 may be a link of VRF tenant, and chalkd answers in the default VRF alone; keep the node API's interfaces out of every VRF |
Take the link out of the VRF's interfaces and out of its Network's VRF=, or name links of the default VRF in management.interfaces. |
The renderer of the output networkd checks the unit names and the NetDev documents,
which the validator does not read. It checks when the router renders its config, at a delivery
or at chalkctl config apply --dry-run, and chalkctl validate does not run it. A renderer's
refusal fails the delivery, and chalkctl prints it in the same form:
| Message | Fix |
|---|---|
Network/00-all: metadata.name: Network 00-all matches the link n64-guest of NAT64 guest before 00-n64-guest; rename it to sort after 00-n64-guest |
Rename the Network, or narrow its Name=. |
NetDev/20-dummy: spec.netDev.name: NetDev 20-dummy creates the device tenant, which VRF tenant creates; rename the VRF or the NetDev |
Rename the VRF, or the device in the NetDev's Name=. |
Limits¶
- The kernel looks up the router's own addresses before a VRF's table. A VRF's hosts therefore get the kernel's answers from the addresses of the default VRF's links, such as echo replies and TCP resets. No socket of the default VRF takes their connections, so chalkd stays out of their reach. The router's IPv6 echo reply to an address that a VRF and the default VRF both hold leaves through the default VRF.
- No protected rule covers the router's own connections to its addresses in a VRF. They reach the
input chain as connections from the VRF's links, and the VRF's policies to
localdecide them. - A VRF instance cannot announce a default route from its
networks. Use a peer family'sdefaultOriginate. - networkd also matches
Name=against a link's alternative names. The documents do not give them, so the validatorroutercannot tell which link aNetworkmatches by an alternative name. Name links by their interface names. - Routes leak between VRFs through BGP alone.
What next¶
- Translate and resolve in a VRF gives a VRF a NAT64 and a resolver of its own.
- VRFs on a router explains the rendered units, the removal of VRFs, FRR's instances and the state chalkd reads.
- Route with BGP configures an instance, its peers and its policies.
- Route traffic by policy sends chosen traffic into a VRF or out of it.
- Lease addresses with DHCP runs a DHCP server of a VRF.
- Config kinds describes each field.