Route with BGP¶
This page shows how to run BGP on a router: its instance, its peers, the policies on their routes
and how to check them. A router without a BGP document runs no BGP.
BGP on a router explains how chalkos turns the documents into
FRR's config.
Before you begin¶
You need a router set up as Set up a router describes, and the secrets file. Its
image must carry FRR. Router images do unless their modules set the image option
chalkos.router.bgp.enable to false.
An image without FRR does not accept the BGP kinds: chalkctl validate reports that it
does not accept kind "BGP". Ask the image author for an image with BGP.
The examples configure the fabric edge r1. It peers with the leaves of a fabric on eth1 and
eth2 without addresses of its own there, and with a transit provider at 203.0.113.254 on its
WAN. Each of its interfaces has a Network document:
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 20-fabric}
spec:
match: {name: [eth1, eth2]}
network: {ipv6AcceptRA: false}
ipv6AcceptRA: false keeps networkd from installing a default route from the router
advertisements on the link. FRR sends and reads those advertisements to find its neighbours. FRR's
zebra prefers a route that the kernel already holds from elsewhere, such as a default route from
router advertisements, over the path BGP learns, and then installs no route of its own for that
prefix. Turn router advertisements off on each interface whose default route should come from BGP.
Give the router a BGP instance¶
A node has one BGP document per VRF. This page uses the default VRF, whose document has no
vrf; Separate routing with VRFs runs instances in others. The document gives the
instance its AS number and router ID:
apiVersion: router.chalkos.dev/v1alpha1
kind: BGP
metadata: {name: bgp}
spec:
asn: 64512
routerID: 192.0.2.1
multipath: {ebgp: 64, asPathRelax: true}
networks:
- {prefix: "2001:db8:ffff::1/128"}
- {prefix: "64:ff9b::/96", policy: nat64}
- {prefix: 192.0.2.0/24}
multipath installs up to 64 equal paths from external peers for one prefix, and asPathRelax
counts paths through different leaves' ASes as equal. The router announces each of networks
while its own table holds a route to it, such as the address of a loopback or a static route. A
network's policy sets the attributes of its announcement, as a term of
a routing policy does. Two networks of one range are refused.
routerID is required. BGP names a router by an IPv4 address, which need not be one of its own,
so give an IPv6-only router one too. The rules refuse 0.0.0.0, a multicast address and
255.255.255.255.
Peer on the fabric's links¶
A BGPPeer of interfaces runs a session on each link it names, with the neighbour it finds
there by its IPv6 link-local address:
apiVersion: router.chalkos.dev/v1alpha1
kind: BGPPeer
metadata: {name: fabric}
spec:
interfaces: [eth1, eth2]
remoteAS: external
bfd: true
extendedNextHop: true
password: {secretRef: {name: bgp, key: fabric}}
description: the fabric's leaves
families:
ipv6: {defaultOriginate: true, import: from-fabric, export: to-fabric}
remoteAS: external takes a neighbour of any AS but the router's own, and internal one of the
router's own. With bfd, BFD checks the link at the timers of the BGP document and ends the
session when it finds the link down. With the default timers of 300 ms and a multiplier of 3 at
both ends, that takes about 900 ms. defaultOriginate sends the leaves a default route whatever
the router's own table holds, unless defaultOriginateWhen says when, as
Send a default only while a route exists shows.
extendedNextHop carries IPv4 routes over IPv6 sessions, for a peer whose families gives
ipv4 too.
interfaces and interface find their links in different ways:
interfacestakes names and globs, such aseth*. Each matches the interface names in the[Match]sections'Name=of the node'sNetworkdocuments. A link that noNetworknames that way gets no session, also when the interface exists.interface: eth1names one link and runs its session whether or not aNetworknames it.
Both stand for links of the peer's VRF. Here that is the default VRF, which holds the links that
no VRF document holds.
description holds up to 80 printable ASCII characters. An @ and two spaces in a row are
refused.
The password is a key of a Secret:
apiVersion: chalkos.dev/v1alpha1
kind: Secret
metadata: {name: bgp}
spec:
data: {fabric: <password>}
<password> stands for the session's TCP MD5 password, which the neighbour sets too. FRR takes
one of 1 to 80 printable characters without white space. Another password fails frr.service
when the config is delivered or the node boots. The name and key of the Secret hold letters,
digits, ., _ and - alone.
Peer at an address¶
A peer of address runs one session with that neighbour:
apiVersion: router.chalkos.dev/v1alpha1
kind: BGPPeer
metadata: {name: transit}
spec:
address: 203.0.113.254
remoteAS: 65001
families:
ipv4: {import: from-transit, export: to-transit, maxPrefix: 1000}
maxPrefix ends the session once the neighbour sends more than 1000 routes. A peer names exactly
one of address, interface, interfaces and listen, and one family at least. A family
without import accepts every route, and one without export sends every route the router has.
The address is a unicast address beyond the router. The rules refuse a loopback, multicast or
broadcast address and an IPv4-mapped IPv6 address such as ::ffff:203.0.113.254; give an IPv4
neighbour its IPv4 address. A link-local address is refused too, as a neighbour on a link takes
interface or interfaces. A peer at an IPv6 address that exchanges IPv4 routes needs
extendedNextHop: true. The validator router refuses a session that two peers name, as
Fix a refused delivery describes.
Accept every neighbour of a prefix¶
A router that serves many hosts on one LAN, such as servers that announce their own addresses,
can take a session from each of them with one peer. A BGPPeer of listen accepts a session from
every neighbour whose address lies in range:
apiVersion: router.chalkos.dev/v1alpha1
kind: BGPPeer
metadata: {name: hosts}
spec:
listen: {range: "2001:db8:11::/64"}
remoteAS: 65001
bfd: true
families:
ipv6: {defaultOriginate: true}
Every neighbour in the range shares the peer's settings: its AS, password, BFD, families and
policies. The router opens no session itself. It waits for each host to connect, so each host
peers with the router's address on the LAN. The firewall accepts BGP from the range, and BFD too
with bfd.
The LAN needs a prefix other than link-local, such as a unique local or a global one, that the
router and the hosts have addresses of, as FRR takes no link-local range. The rules refuse a range
that starts in link-local, loopback, multicast or IPv4-mapped space, and a range of length 0, such
as ::/0, which would accept a session from every address. A peer with an IPv6 range that
exchanges IPv4 routes needs extendedNextHop: true, as one at an IPv6 address does. The validator
router refuses two ranges of one VRF that overlap, as a neighbour in both would have two peers.
A BGP instance accepts at most 100 neighbours from its listen ranges together, FRR's default
limit; a neighbour past it gets no session.
chalkctl status lists each host's session under the peer's name, with the host's address as the
neighbour:
bgp:
VRF PEER NEIGHBOR STATE UPTIME RECEIVED SENT BFD
default hosts 2001:db8:11::10 Established 5m12s 1 1 up
default hosts 2001:db8:11::11 Established 5m9s 1 1 up
A host that has not connected has no line.
Filter routes with policies¶
A RoutingPolicy decides which routes a peer's family accepts or sends. Its terms match in
order, and the first that matches decides. default decides the rest, and is reject unless the
policy says otherwise:
apiVersion: router.chalkos.dev/v1alpha1
kind: RoutingPolicy
metadata: {name: from-fabric}
spec:
terms:
- {match: {prefixList: fabric}, then: {action: accept}}
default: reject
---
apiVersion: router.chalkos.dev/v1alpha1
kind: RoutingPolicy
metadata: {name: to-fabric}
spec:
terms:
- {match: {prefixList: edge-originated}, then: {action: accept}}
default: reject
---
apiVersion: router.chalkos.dev/v1alpha1
kind: PrefixList
metadata: {name: fabric}
spec:
family: ipv6
entries: [{prefix: "2001:db8::/32", le: 128}]
---
apiVersion: router.chalkos.dev/v1alpha1
kind: PrefixList
metadata: {name: edge-originated}
spec:
family: ipv6
entries:
- {prefix: "2001:db8:ffff::/48", le: 128}
- {prefix: "64:ff9b::/96"}
- {prefix: "::/0", action: deny}
A PrefixList entry without ge or le matches its prefix alone. le: 128 also matches the
routes within 2001:db8::/32 up to /128. Both ge and le lie from the prefix's length to the
family's longest prefix, and ge is no higher than le. Two entries with the same prefix, ge
and le are refused, because the later one never decides. The PrefixLists of a policy are of one
family, and a peer's family and a network take policies of their own family alone. A policy holds
6000 terms at most.
A term may set attributes of the routes it accepts. The transit's policies keep its blackholed
routes out, prefer the routes of its own AS and tag the rest. They announce the router's IPv4
range, and the ranges within it from /25, with a MED of 50 and a longer AS path:
apiVersion: router.chalkos.dev/v1alpha1
kind: RoutingPolicy
metadata: {name: from-transit}
spec:
terms:
- {match: {communityList: blackhole}, then: {action: reject}}
- match: {asPath: "^65001_"}
then: {action: accept, localPref: 200, removeCommunities: ["65001:100"]}
- then: {action: accept, addCommunities: ["64512:100", "64512:100:1"]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: RoutingPolicy
metadata: {name: to-transit}
spec:
terms:
- {match: {prefixList: transit-out}, then: {action: accept, med: 50, prepend: 2}}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: PrefixList
metadata: {name: transit-out}
spec:
family: ipv4
entries: [{prefix: 192.0.2.0/24, ge: 25, le: 32}, {prefix: 192.0.2.0/24}]
---
apiVersion: router.chalkos.dev/v1alpha1
kind: CommunityList
metadata: {name: blackhole}
spec: {communities: ["65001:666"]}
---
apiVersion: router.chalkos.dev/v1alpha1
kind: RoutingPolicy
metadata: {name: nat64}
spec:
terms:
- then: {action: accept, addCommunities: ["64512:64"]}
asPath is a regular expression over the route's AS path, in which _ stands for the start, the
end or a space between two ASes. It holds digits, single spaces and _^$.*+?()|{},-, without
brackets. FRR's bgpd compiles it as a POSIX extended regular expression of glibc. The output
frr refuses one that bgpd would not compile, such as ^*, when the router renders its config;
chalkctl validate does not check it.
prepend: 2 puts the router's AS in front of the path twice. A CommunityList holds standard
communities, <AS>:<value>, or large ones, <AS>:<value>:<value>, not both, and each community
once. A community's numbers have no leading zeros. A term that rejects sets nothing.
The order of terms and entries gives their sequence numbers in FRR's config. A term or entry put before others renumbers those after it, and the reload rewrites them.
Send a default only while a route exists¶
A leaf should get the edge's default route only while the edge itself has a way out. Here the
way out is the IPv6 default route that the WAN's router advertisements give the edge. A
PrefixList matches that route, and the fabric peer's family names it in
defaultOriginateWhen:
apiVersion: router.chalkos.dev/v1alpha1
kind: PrefixList
metadata: {name: wan-default}
spec:
family: ipv6
entries: [{prefix: "::/0"}]
---
apiVersion: router.chalkos.dev/v1alpha1
kind: BGPPeer
metadata: {name: fabric}
spec:
interfaces: [eth1, eth2]
remoteAS: external
bfd: true
extendedNextHop: true
password: {secretRef: {name: bgp, key: fabric}}
description: the fabric's leaves
families:
ipv6: {defaultOriginate: true, defaultOriginateWhen: wan-default, import: from-fabric, export: to-fabric}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-wan}
spec:
match: {name: eth0}
network: {address: [203.0.113.1/24], ipv6AcceptRA: true}
The router sends the default route while the peer's VRF holds a route that the list matches. The
route can be one BGP learned, a kernel route or the prefix of a link that is up. Kernel routes
include those from DHCP and router advertisements, a Network's [Route] and a route someone
added by hand. Blackhole, unreachable and prohibit routes do not count. That includes the
unreachable default that ends every VRF's table, so a VRF's peer can track ::/0 too.
The router does not announce the routes it tracks. The list decides when the default route goes out, and the peer's policies still decide which other routes do. The default route follows the tracked route within about 5 seconds, when it appears and when it goes.
The list is of the family of the block that names it. The rules refuse a list of the other
family, a list the config lacks and defaultOriginateWhen without defaultOriginate: true:
chalkctl: r1: the config is refused: BGPPeer/fabric: spec.families: the ipv6 family names defaultOriginateWhen without defaultOriginate: true; defaultOriginateWhen says when the router sends the default route that defaultOriginate originates
chalkctl node routes on the edge lists the tracked route as a path of its own, with the
communities noAdvertise and 4200000000:0:1. Among the other IPv6 routes, it shows:
DESTINATION TYPE PROTOCOL TABLE METRIC NEXT HOPS
::/0 unicast ra 254 1024 fe80::1 on eth0
BGP PREFIX BEST AS PATH PEER NEXT HOPS LOCAL PREF MED COMMUNITIES
::/0 * - local fe80::1 - 1024 noAdvertise 4200000000:0:1
chalkctl node routes on a leaf shows ::/0 from the edge while the WAN gives the edge its
default route, and none from that edge after the route goes.
Keep the list narrow. A broad list also matches the paths BGP holds, such as the routes peers send, so the default route goes out while any of them exists. Each kernel or connected route the list matches becomes one more BGP path, and the check every 5 seconds walks them all. The limits of BGP on a router list what else the condition counts.
The router marks the routes it tracks with the large community 4200000000:0:1. An import of
the VRF's routes into another VRF drops every route that carries it. Leave that community out of
your policies' addCommunities. When a peer sends it, remove it in the import policy with
removeCommunities.
Deliver and check the sessions¶
Validate the files and deliver them:
A delivery reloads FRR, which applies the changes and keeps the sessions they leave alone.
chalkctl status lists the router's sessions under bgp::
bgp:
VRF PEER NEIGHBOR STATE UPTIME RECEIVED SENT BFD
default fabric eth1 Established 1h2m3s 1 2 up
default fabric eth2 Idle - 0 0 down
default transit 203.0.113.254 Active - 0 0 -
Each line is a session of a VRF: on a link, named by its interface, or with an address.
RECEIVED counts the routes the router accepted from the neighbour and SENT those it sends,
over every family. STATE is BGP's state of the session, and a session in any state but
Established, such as Active or Idle, exchanges no routes. bgp: no sessions means the
router has no BGP instance or no session. A router whose image carries no FRR prints no bgp
line, and chalkctl node routes shows its kernel's routes alone. bgp: unknown: gives vtysh's
error when chalkd could not read FRR's state, such as failed to connect to any daemons while FRR
does not run. A node in maintenance mode runs no FRR.
chalkctl node routes shows the kernel's routes and the paths BGP selected, to a prefix and
within it when one is given, so ::/0 lists every IPv6 route. A leaf r2 has sessions to two
edges of AS 64512. Each edge originates a default route and announces 2001:db8:ffff::1/128, and
r2 announces its own 2001:db8:2::/48:
DESTINATION TYPE PROTOCOL TABLE METRIC NEXT HOPS
::/0 unicast bgp 254 20 fe80::e06b:dcff:fe54:f465 on eth2 (weight 1), fe80::b817:4cff:fe8b:9cf on eth1 (weight 1)
2001:db8:2::/48 blackhole boot 254 1024 -
2001:db8:ffff::1/128 unicast bgp 254 20 fe80::e06b:dcff:fe54:f465 on eth2 (weight 1), fe80::b817:4cff:fe8b:9cf on eth1 (weight 1)
BGP PREFIX BEST AS PATH PEER NEXT HOPS LOCAL PREF MED COMMUNITIES
::/0 * 64512 eth2 fe80::e06b:dcff:fe54:f465 200 0 64512:1 64512:1:2
::/0 64512 eth1 fe80::b817:4cff:fe8b:9cf 200 0 -
2001:db8:2::/48 * - local :: - 0 -
2001:db8:ffff::1/128 * 64512 eth2 fe80::e06b:dcff:fe54:f465 200 0 -
2001:db8:ffff::1/128 64512 eth1 fe80::b817:4cff:fe8b:9cf 200 0 -
The kernel's routes from BGP have two next hops each, one for each path BGP installed. * marks
the best path of a prefix, and local a path of the router's own networks. --protocol and
--table narrow the kernel's routes, and --vrf lists another VRF's table and paths, as
chalkctl node routes describes.
The diagnostics bgp-summary, bgp-neighbor, bgp-route and bfd-peers print FRR's own views
of the sessions, one session, the paths to a prefix and the BFD sessions:
chalkctl node diag r1 --secrets secrets.age bgp-neighbor peer=eth1
chalkctl node diag r1 --secrets secrets.age bgp-route prefix=::/0
chalkctl node diag r1 --secrets secrets.age bgp-route family=ipv4 prefix=192.0.2.0/24
bgp-route asks for IPv6 unless family says ipv4, and each asks the default VRF unless
vrf names another. The status, the routes and these diagnostics need a reader client.
Fix a refused delivery¶
chalkctl validate runs the schemas, the rules and the
validator router. The validator checks the peers'
sessions against the Network and VRF documents and against each other. chalkctl validate
and chalkctl config apply run it before chalkctl sends anything, and the router runs it again
at the delivery. It checks the VRFs' links first, and the peers only once the links pass.
Within each step it reports every fault it finds at once. chalkctl prints the refusals
after the config is refused:, as it prints a rule's. Each refusal names the document and the
field:
chalkctl: r1: the config is refused: BGPPeer/spare: spec.interface: BGPPeer fabric and BGPPeer spare both name eth1 in the VRF default; give each session one peer
| Message | Fix |
|---|---|
BGPPeer/spare: spec.interface: BGPPeer fabric and BGPPeer spare both name eth1 in the VRF default; give each session one peer |
Drop the interface or address from one of the peers. |
BGPPeer/fabric: spec.interfaces[0]: interface "192.0.2.10" is an address, which FRR cannot tell from a neighbour's |
Narrow the peer's glob, or rename the link that a Network names. |
BGPPeer/servers: spec.listen.range: the listen range 2001:db8:11::/64 of BGPPeer servers overlaps 2001:db8::/32 of BGPPeer hosts in the VRF default; give each neighbour one peer |
Narrow one of the ranges so that they do not overlap, or merge the peers. |
The renderer of the output frr checks the rest when the router renders its config, at a
delivery or at chalkctl config apply --dry-run, before the delivery changes anything: the names
of the peer groups, the AS numbers of a policy's users, and how bgpd compiles an AS path.
chalkctl validate does not run it. Its refusal fails the delivery, and chalkctl prints it in
the same form:
| Message | Fix |
|---|---|
BGPPeer/fabric: metadata.name: BGPPeer x names peer-fabric, the peer group of BGPPeer fabric; rename the document or the interface |
Rename the BGPPeer or the interface. |
RoutingPolicy/from-transit: spec.terms[1].match.asPath: asPath "^*1" is no regular expression bgpd compiles: the * at 2 repeats nothing |
Write the expression so that each repetition follows what it repeats. |
RoutingPolicy/p: spec.terms[0].then.prepend: RoutingPolicy p prepends the AS number, and BGP instances of AS 64512 and 64513 use it; give each AS a policy of its own |
Give each AS a policy of its own. |
Separate routing with VRFs lists the validator's messages about the links of a peer's VRF.
Change the AS number¶
The rules refuse a delivery that changes spec.asn of the BGP document:
chalkctl: r1: the config is refused: BGP/bgp: spec.asn: the AS number stays 64512; to change it, deliver the config without the BGP document, which ends the router's sessions, and then with the new AS number
Deliver the config without the BGP and BGPPeer documents first, which ends every session, as
a BGPPeer needs a BGP document. Then deliver it with the new number. The rule compares the
BGP document of one name. A BGP document of another name replaces the old one in one
delivery, AS number included, and the change of AS ends the sessions too.
The rule guards the AS number of each VRF's instance the same way, by the name of its BGP
document. While an instance of another VRF runs, the delivery without the default VRF's BGP
document leaves out the other VRFs' BGP documents too, as
Separate routing with VRFs explains.
What next¶
- BGP on a router explains the rendered config, the passwords and the state chalkd reads.
- Config kinds describes each field.
- Filter and translate traffic filters what the router forwards. The firewall accepts each peer's BGP and BFD ports by itself.