BGP on a router¶
A router image runs BGP with FRR: its daemons zebra, bgpd, bfdd and staticd, and vtysh, its shell.
The output frr renders FRR's whole config, an instance per VRF, from the node's BGP, BGPPeer,
PrefixList, CommunityList, RoutingPolicy and VRF documents. No document holds FRR commands
of its own.
The output frr renders FRR's whole config¶
chalkos-render frr writes frr.conf from the documents. The edge of
Route with BGP renders, in part:
frr defaults traditional
hostname r1
log syslog informational
service integrated-vtysh-config
!
ipv6 prefix-list pl-fabric seq 5 permit 2001:db8::/32 le 128
!
route-map rp-from-fabric permit 10
match ipv6 address prefix-list pl-fabric
exit
route-map rp-from-fabric deny 65535
exit
!
router bgp 64512
bgp router-id 192.0.2.1
no bgp default ipv4-unicast
bgp bestpath as-path multipath-relax
neighbor peer-fabric peer-group
neighbor peer-fabric remote-as external
neighbor peer-fabric description the fabric's leaves
neighbor peer-fabric password @secret_bgp.fabric@
neighbor peer-fabric bfd
neighbor peer-fabric bfd profile chalkos
neighbor peer-fabric capability extended-nexthop
neighbor eth1 interface peer-group peer-fabric
neighbor eth2 interface peer-group peer-fabric
!
address-family ipv6 unicast
network 2001:db8:ffff::1/128
network 64:ff9b::/96 route-map rp-nat64
maximum-paths 64
maximum-paths ibgp 1
neighbor peer-fabric activate
neighbor peer-fabric default-originate
neighbor peer-fabric route-map rp-from-fabric in
neighbor peer-fabric route-map rp-to-fabric out
exit-address-family
exit
!
bfd
profile chalkos
receive-interval 300
transmit-interval 300
detect-multiplier 3
exit
exit
!
end
The config follows these rules:
| Documents | Config |
|---|---|
BGP |
router bgp <asn>, or router bgp <asn> vrf <name> for a document with vrf, with frr defaults traditional and every count and timer of the document written out, so FRR's defaults decide nothing the document sets. Both address families list the document's networks of the family and its maximum-paths. |
BGPPeer |
A peer group peer-<name> in the block of its VRF's instance, and a neighbour in it for the peer's address, its interface or each link its interfaces match, or a bgp listen range line that puts each neighbour of its listen range in the group. |
| A peer's family | activate, and a route-map in each direction: the family's policy, or accept-all without one. FRR's ebgp-requires-policy then always holds. |
A peer's family with defaultOriginateWhen |
default-originate route-map do-<list>, and in its instance's address family redistribute kernel and redistribute connected through track-<vrf>-<family>. |
PrefixList |
ip prefix-list or ipv6 prefix-list pl-<name>, the entries at sequence numbers 5, 10, 15 and on. |
CommunityList |
bgp community-list standard cl-<name>, or bgp large-community-list for large communities. |
RoutingPolicy |
route-map rp-<name>, the terms at sequence numbers 10, 20, 30 and on, and the default at 65535. A term's AS path and the communities it removes get lists of their own, asp-<policy>-<seq> and rm-<policy>-<seq>. |
bfd of a peer |
The profile of its instance's BFD timers: chalkos for the default VRF, chalkos-<vrf> for another. |
The prefixes keep the names apart. FRR names peers and interfaces alike, and without them a peer
group could take the name of an interface or of another list. The renderer refuses an interface
named like another peer's group, such as peer-fabric. A node without a BGP document renders
the first lines and end alone, and FRR runs no BGP.
The sequence numbers follow the positions of the entries and terms. A term put before others renumbers those after it, and the reload rewrites the renumbered entries.
A peer's interfaces expand against the Network documents¶
A peer's interfaces expand when the config renders. Each name or glob matches the names in the
[Match] sections' Name= of the node's Network documents, with the globs and negations among
those names left out. The leaf example's interfaces: ["swp*"] gives a session on each of swp1
and swp2, which its documents 20-swp1 and 20-swp2 name. Its document 30-swp matches the
glob swp* and names no link of its own. A link that only a Link or NetDev names gets no
session. A glob expands over the links of the peer's VRF alone, as
VRFs on a router explains.
A peer's single interface renders as it is written, whether or not a Network names it, and
must be a link of the peer's VRF. A name that reads as an address, such as fe80::1, is refused
in both, because FRR could not tell the interface from a neighbour's address: the rules refuse
one that the peer writes out, and the validator router a link that a glob expands to. The
validator also refuses two peers that name one session.
A default route while a route exists¶
A peer family's defaultOriginateWhen names a PrefixList, and the router sends the peer its
default route while the peer's VRF holds a route that the list matches. The edge of
Route with BGP tracks the WAN's
IPv6 default route and renders, in part:
ipv6 prefix-list pl-wan-default seq 5 permit ::/0
!
route-map do-wan-default permit 10
match ipv6 address prefix-list pl-wan-default
exit
route-map track-default-ipv6 deny 5
match ipv6 next-hop type blackhole
exit
route-map track-default-ipv6 permit 10
match ipv6 address prefix-list pl-wan-default
set community no-advertise additive
set distance 255
set large-community 4200000000:0:1 additive
set weight 0
exit
!
router bgp 64512
address-family ipv6 unicast
redistribute kernel route-map track-default-ipv6
redistribute connected route-map track-default-ipv6
neighbor peer-fabric default-originate route-map do-wan-default
exit-address-family
exit
bgpd's default-originate route-map sends the default route while a path in the instance's BGP
table passes the route-map, and do-<list> matches the list. zebra's kernel and connected
routes reach that table only through redistribution. The instance therefore redistributes them
through track-<vrf>-<family>, where <vrf> is default for the default VRF. Its first entry
denies blackhole routes. Its other entries permit the routes of each list its peers track and
set four values on them:
no-advertisekeeps them from every peer.distance 255keeps a path a peer sends for the same prefix the best path. bgpd compares the administrative distance before weight and local preference. Without it, the redistributed path, of distance 0, would win, and the peer's path would no longer be installed, announced or leaked.weight 0keeps a network of the instance the best path for its prefix. bgpd compares no distance between a network and a redistributed path, and both would have the weight 32768. A network whose policy prepends the AS path or lowers the local preference would then lose to the redistributed path, and no peer would get it.- The large community
4200000000:0:1, of the private-use AS 4200000000, marks them alike on every router.
The VRF tables end in unreachable default routes of the kernel metric 4278198272. zebra reads that metric as distance 255 and metric 8192, so a match on the metric would miss them. The route-map matches the next hop's type instead: zebra treats unreachable, prohibit and blackhole routes alike as blackhole next hops.
A tracked route that another VRF's instance imports would land in that VRF's kernel table, with
its next hop in this one. An import from a VRF that tracks routes of the family goes through
leak-<vrf>-<family>. It denies chalkos-tracked, the large-community list of
4200000000:0:1, and then calls the import's policy:
bgp large-community-list standard chalkos-tracked seq 5 permit 4200000000:0:1
!
route-map leak-default-ipv6 deny 5
match large-community chalkos-tracked
exit
route-map leak-default-ipv6 permit 10
call rp-from-tenant
exit
Here the default VRF's instance imports the VRF tenant, which tracks IPv6 routes, with the
policy from-tenant. An import without a policy gets permit 10 alone. Imports from VRFs that
track nothing of the family use no leak route-map.
The large community 4200000000:0:1 is reserved for tracked routes. The leak route-maps deny
every path that carries it, also one a peer sent with it or one an import policy marked with it.
Such a route stays out of a VRF that imports it from a VRF that tracks routes of its family. A
policy's removeCommunities strips the community from a peer's routes.
bgpd checks the condition every 5 seconds, FRR's bgp default-originate timer. The default route
therefore follows a tracked route within about 5 seconds. A delivery that adds, changes or drops
a peer family's defaultOriginateWhen withdraws the default route at once, and the next check
sends it again if the route exists.
The check parses the config, and the validator and the renderer check the rest¶
A delivery renders frr.conf and checks it with vtysh --dryrun before it changes anything.
vtysh parses each line with the daemons' commands and needs no daemon. It catches a command that
no daemon takes, and a value outside a command's range, such as le 129. It does not check what
the daemons would refuse when they apply it, such as a route-map that names a missing list.
At boot, FRR's mgmtd applies the prefix lists, route-maps and static routes of the config in one
transaction, so one value it refuses would leave all of them out. The schemas, the rules and the
config's uniqueness check therefore refuse the values FRR's daemons refuse, and
chalkctl validate runs them all. The uniqueness check refuses two entries of a PrefixList with
one prefix, ge and le, one community twice in a CommunityList and two networks of one range,
before any rule runs. The rules refuse a ge or le outside the prefix's length and the
family's longest prefix, a policy whose PrefixLists are of both families, and a peer's or
network's policy of another family. The schemas refuse a community with a part above 65535 in a
standard community or above 4294967295 in a large one, and a description with an @ or two
spaces in a row.
The validator router checks the peers' sessions
against the Network and VRF documents and against each other. It runs after the rules and
before the renderers, at a delivery, at chalkctl config apply --dry-run and in
chalkctl validate, and refuses:
- a session that two peers name;
- a link that a peer's glob expands to and that is named like an address;
- a peer's interface in another VRF than the peer's, or of a VRF's device, and a glob that matches links of other VRFs alone.
The renderer trusts what the schemas, the rules and the validator check. It checks what depends
on the names FRR gives its objects or on bgpd itself, when the router renders its config, at a
delivery or at chalkctl config apply --dry-run, and chalkctl validate does not run it:
- an
asPaththat bgpd cannot compile; - an interface named like a peer's group;
- a policy with
prependthat instances of two AS numbers use.
A value that would add a command to the config, such as a line break in a description, cannot
pass the schemas. Should a document bypass them, the renderer fails with
internal error: ...; chalkd checks the config against the schemas, the rules and the validators before it renders
rather than write it.
bgpd compiles an asPath with glibc's regcomp as a POSIX extended regular expression, each _
standing for the start, the end or a separator of the path. bgpd drops an as-path list it cannot
compile, and its term then matches no route. The renderer checks each expression as glibc reads
it. It refuses a repetition of nothing, such as ^*, parentheses that do not pair up and an
interval that ends below its start. The schema leaves out brackets, whose meaning differs between
POSIX and other regular expressions.
Passwords reach FRR as credentials¶
The renderer sees no Secret. A peer's password renders as the placeholder
@secret_<secret>.<key>@ on its peer group's password line. frr.service gets the passwords
that the BGPPeer documents name as its credentials. Before it starts and on each reload, a
helper of the unit writes /run/frr/frr.conf with each password line's placeholder replaced by
its credential. Only root reads the file, mode 0600, and /etc/frr/frr.conf links to it.
The helper fills in the password lines alone. A placeholder elsewhere stays as written, and a
description cannot hold one, as it holds no @. A password that is empty, longer than 80
characters, or holds white space, a line break or anything but printable ASCII fails the unit and
leaves the file as it was. So does a placeholder whose credential the unit lacks.
A rotated password changes no rendered file. The effect of frr fires all the same, because the
value of a Secret that its documents reference changed. The reload loads the credentials again
before the helper runs.
A delivery reloads FRR¶
The effect of frr reloads frr.service once the delivery is stored. The reload fills in the
passwords again and runs chalkos-frr-reload, which compares the new config with the running
config that vtysh shows and applies the difference with vtysh -f, so a session that the change
leaves alone stays up. A command whose value changes, such as a password, a peer's route-map or
the route-map of a network, is given its new value without being deleted first. Commands, peers,
peer groups and instances that the new config lacks are deleted before the new ones are added.
FRR keeps a community list's or AS path list's entries by value, so an entry that moves to
another sequence number is deleted before the entries are added. Route-map entries, BFD profiles,
prefix lists and lists the new config drops go last, once nothing names them. Afterwards
chalkos-frr-reload reads the running config again. A deleted peer group takes its members and
their config with it, so a second pass may follow.
FRR shows the system's hostname, so the comparison leaves the hostname out. A reload that changes something sets the new config's hostname as well, and a new hostname alone takes effect at the next such reload or at a restart.
chalkos-frr-reload knows the commands the renderer writes and the lines FRR shows for them. It
fails the reload on a line it does not know, in either config, and when the running config still
differs after two passes or after 90 seconds. Its log in the journal names the commands it ran
and those vtysh refused, with the passwords hidden. A reload that fails fails the delivery,
with the config stored. FRR keeps the commands the reload applied, and a restart of
frr.service loads the new config whole. A failed frr.service is restarted instead of
reloaded.
chalkd reloads the units of one delivery in one systemctl reload. frr.service orders after
chalkos-firewall.service, so the firewall accepts a new peer's ports before FRR dials it.
FRR starts after the config and the firewall¶
At boot, frr.service starts after chalkos-config.service has rendered the outputs and after
chalkos-firewall.service. It orders before chalkos-node-addresses.target and is wanted by it,
as a unit that announces the node's addresses. Without a rendered frr.conf, the helper writes
the image's fallback config, which runs no BGP.
frr.service starts only when chalkd has written its credentials, which it does once the node
has config. In maintenance mode, which loads no config, the node runs no FRR.
networkd leaves zebra's routes and next hops alone. By default it removes the routes it did not
configure itself each time it configures a link, so router images turn ManageForeignRoutes= and
ManageForeignNextHops= off.
The firewall accepts each peer's ports¶
The input chain of inet chalkos accepts BGP, TCP port 179, from each BGPPeer, and BFD from
each with bfd. A peer of interface or interfaces is accepted on those interfaces, globs as
written, a peer of address from that address alone, and a peer of listen from its range.
BFD takes UDP ports 3784 and 3785 of single-hop sessions, and 4784 of multi-hop ones from a peer
of address or listen, whose neighbours may lie beyond the link. A peer on a link is one hop
away. The edge renders:
iifname { "eth1", "eth2" } tcp dport { 179 } accept comment "BGPPeer fabric"
iifname { "eth1", "eth2" } udp dport { 3784, 3785 } accept comment "BGPPeer fabric"
ip saddr { 203.0.113.254/32 } tcp dport { 179 } accept comment "BGPPeer transit"
These rules come after chalkd's port and before the policies to local, which cannot close them.
The router's firewall describes the chain. A peer of a VRF is
accepted on that VRF's links by meta sdifname, as
VRFs on a router explains.
chalkd reads FRR's state¶
chalkctl status shows a router's BGP sessions. chalkd asks vtysh for
show bgp vrf all neighbors json, within 5 seconds, and reports each session's VRF, peer,
neighbour, state, uptime, routes received and sent, and BFD state. It finds a session's BGPPeer
by the name of its peer group. A neighbour of a listen range is in its peer's group too, and the
status names it by its address. When vtysh fails, the status gives its error, and the status's
other parts stand.
An image whose modules set
chalkos.router.bgp.enable to false
carries no FRR and refuses the kinds BGP, BGPPeer, PrefixList, CommunityList and RoutingPolicy.
chalkd finds no /etc/frr/frr.conf there and asks vtysh nothing. The status sets bgp_absent,
and chalkctl status prints no bgp line. A router with FRR and no sessions prints
bgp: no sessions instead. On such an image the Routes call lists the kernel's routes alone.
The Routes call lists the kernel's routes, which chalkd reads through netlink, and the paths BGP
selected, from show bgp ipv4 unicast detail json and its IPv6 twin. For another VRF it lists that
VRF's table and asks for show bgp vrf <name> ipv4 unicast detail json. A route of several next
hops lists each with its weight, and an IPv4 route over an IPv6 next hop names it.
chalkctl node routes prints both. A node of another image answers that the router layer is not
in its image.
The diagnostics bgp-summary, bgp-neighbor, bgp-route and bfd-peers run vtysh as a member
of frrvty, the group of FRR's sockets, and need a reader.
Limits¶
- FRR announces a network of
BGP.spec.networksonly while the router's table holds a route to it, whatever the policy says. A VRF's table always holds a default route, the unreachable one that ends it, so the rules refuse a default route among a VRF instance's networks. A peer family'sdefaultOriginatesends one instead. - zebra prefers a route the kernel holds from elsewhere over BGP's path to the same prefix, and
then installs none of BGP's. A default route that networkd takes from router advertisements on
an uplink keeps BGP's default route out of the kernel.
ipv6AcceptRA: falseon thatNetworklets it in. - FRR negotiates the extended next hop on sessions of
interfaceandinterfaceswhateverextendedNextHopsays. - A peer's
interfacesmatch the names ofNetworkdocuments, not the links that exist, and aNetworkthat matches links by anything butName=names none. - The AS number rule compares a
BGPdocument with the running one of the same name. ABGPdocument of another name replaces the old one, AS number included. - The condition of
defaultOriginateWhencounts BGP's paths too. A list that matches a prefix of the instance'snetworks, a peer's route or an imported route holds while BGP holds that path. - The condition counts every path that passes the route-map. That includes an invalid path, such as one whose next hop does not resolve, and a stale path that graceful restart keeps after its session went down. Each check, up to every 5 seconds, walks the family's whole BGP table. That costs CPU when the instance holds a full Internet table in that family.
- Blackhole, unreachable and prohibit routes never satisfy
defaultOriginateWhen.
Related pages¶
- Route with BGP configures BGP.
- How a router works: the layer's outputs and the boot order.
- VRFs on a router: an instance per VRF and imported routes.
- Config kinds for every field of the BGP kinds.
- Built-ins for the output, its effect, the rules and the diagnostics.