Skip to content

BGP on a router

A router image runs BGP with FRR: its daemons zebra, bgpd, bfdd and staticd, and vtysh, its shell. The output frr renders FRR's whole config, an instance per VRF, from the node's BGP, BGPPeer, PrefixList, CommunityList, RoutingPolicy and VRF documents. No document holds FRR commands of its own.

The output frr renders FRR's whole config

chalkos-render frr writes frr.conf from the documents. The edge of Route with BGP renders, in part:

frr defaults traditional
hostname r1
log syslog informational
service integrated-vtysh-config
!
ipv6 prefix-list pl-fabric seq 5 permit 2001:db8::/32 le 128
!
route-map rp-from-fabric permit 10
 match ipv6 address prefix-list pl-fabric
exit
route-map rp-from-fabric deny 65535
exit
!
router bgp 64512
 bgp router-id 192.0.2.1
 no bgp default ipv4-unicast
 bgp bestpath as-path multipath-relax
 neighbor peer-fabric peer-group
 neighbor peer-fabric remote-as external
 neighbor peer-fabric description the fabric's leaves
 neighbor peer-fabric password @secret_bgp.fabric@
 neighbor peer-fabric bfd
 neighbor peer-fabric bfd profile chalkos
 neighbor peer-fabric capability extended-nexthop
 neighbor eth1 interface peer-group peer-fabric
 neighbor eth2 interface peer-group peer-fabric
 !
 address-family ipv6 unicast
  network 2001:db8:ffff::1/128
  network 64:ff9b::/96 route-map rp-nat64
  maximum-paths 64
  maximum-paths ibgp 1
  neighbor peer-fabric activate
  neighbor peer-fabric default-originate
  neighbor peer-fabric route-map rp-from-fabric in
  neighbor peer-fabric route-map rp-to-fabric out
 exit-address-family
exit
!
bfd
 profile chalkos
  receive-interval 300
  transmit-interval 300
  detect-multiplier 3
 exit
exit
!
end

The config follows these rules:

Documents Config
BGP router bgp <asn>, or router bgp <asn> vrf <name> for a document with vrf, with frr defaults traditional and every count and timer of the document written out, so FRR's defaults decide nothing the document sets. Both address families list the document's networks of the family and its maximum-paths.
BGPPeer A peer group peer-<name> in the block of its VRF's instance, and a neighbour in it for the peer's address, its interface or each link its interfaces match, or a bgp listen range line that puts each neighbour of its listen range in the group.
A peer's family activate, and a route-map in each direction: the family's policy, or accept-all without one. FRR's ebgp-requires-policy then always holds.
A peer's family with defaultOriginateWhen default-originate route-map do-<list>, and in its instance's address family redistribute kernel and redistribute connected through track-<vrf>-<family>.
PrefixList ip prefix-list or ipv6 prefix-list pl-<name>, the entries at sequence numbers 5, 10, 15 and on.
CommunityList bgp community-list standard cl-<name>, or bgp large-community-list for large communities.
RoutingPolicy route-map rp-<name>, the terms at sequence numbers 10, 20, 30 and on, and the default at 65535. A term's AS path and the communities it removes get lists of their own, asp-<policy>-<seq> and rm-<policy>-<seq>.
bfd of a peer The profile of its instance's BFD timers: chalkos for the default VRF, chalkos-<vrf> for another.

The prefixes keep the names apart. FRR names peers and interfaces alike, and without them a peer group could take the name of an interface or of another list. The renderer refuses an interface named like another peer's group, such as peer-fabric. A node without a BGP document renders the first lines and end alone, and FRR runs no BGP.

The sequence numbers follow the positions of the entries and terms. A term put before others renumbers those after it, and the reload rewrites the renumbered entries.

A peer's interfaces expand against the Network documents

A peer's interfaces expand when the config renders. Each name or glob matches the names in the [Match] sections' Name= of the node's Network documents, with the globs and negations among those names left out. The leaf example's interfaces: ["swp*"] gives a session on each of swp1 and swp2, which its documents 20-swp1 and 20-swp2 name. Its document 30-swp matches the glob swp* and names no link of its own. A link that only a Link or NetDev names gets no session. A glob expands over the links of the peer's VRF alone, as VRFs on a router explains.

A peer's single interface renders as it is written, whether or not a Network names it, and must be a link of the peer's VRF. A name that reads as an address, such as fe80::1, is refused in both, because FRR could not tell the interface from a neighbour's address: the rules refuse one that the peer writes out, and the validator router a link that a glob expands to. The validator also refuses two peers that name one session.

A default route while a route exists

A peer family's defaultOriginateWhen names a PrefixList, and the router sends the peer its default route while the peer's VRF holds a route that the list matches. The edge of Route with BGP tracks the WAN's IPv6 default route and renders, in part:

ipv6 prefix-list pl-wan-default seq 5 permit ::/0
!
route-map do-wan-default permit 10
 match ipv6 address prefix-list pl-wan-default
exit
route-map track-default-ipv6 deny 5
 match ipv6 next-hop type blackhole
exit
route-map track-default-ipv6 permit 10
 match ipv6 address prefix-list pl-wan-default
 set community no-advertise additive
 set distance 255
 set large-community 4200000000:0:1 additive
 set weight 0
exit
!
router bgp 64512
 address-family ipv6 unicast
  redistribute kernel route-map track-default-ipv6
  redistribute connected route-map track-default-ipv6
  neighbor peer-fabric default-originate route-map do-wan-default
 exit-address-family
exit

bgpd's default-originate route-map sends the default route while a path in the instance's BGP table passes the route-map, and do-<list> matches the list. zebra's kernel and connected routes reach that table only through redistribution. The instance therefore redistributes them through track-<vrf>-<family>, where <vrf> is default for the default VRF. Its first entry denies blackhole routes. Its other entries permit the routes of each list its peers track and set four values on them:

  • no-advertise keeps them from every peer.
  • distance 255 keeps a path a peer sends for the same prefix the best path. bgpd compares the administrative distance before weight and local preference. Without it, the redistributed path, of distance 0, would win, and the peer's path would no longer be installed, announced or leaked.
  • weight 0 keeps a network of the instance the best path for its prefix. bgpd compares no distance between a network and a redistributed path, and both would have the weight 32768. A network whose policy prepends the AS path or lowers the local preference would then lose to the redistributed path, and no peer would get it.
  • The large community 4200000000:0:1, of the private-use AS 4200000000, marks them alike on every router.

The VRF tables end in unreachable default routes of the kernel metric 4278198272. zebra reads that metric as distance 255 and metric 8192, so a match on the metric would miss them. The route-map matches the next hop's type instead: zebra treats unreachable, prohibit and blackhole routes alike as blackhole next hops.

A tracked route that another VRF's instance imports would land in that VRF's kernel table, with its next hop in this one. An import from a VRF that tracks routes of the family goes through leak-<vrf>-<family>. It denies chalkos-tracked, the large-community list of 4200000000:0:1, and then calls the import's policy:

bgp large-community-list standard chalkos-tracked seq 5 permit 4200000000:0:1
!
route-map leak-default-ipv6 deny 5
 match large-community chalkos-tracked
exit
route-map leak-default-ipv6 permit 10
 call rp-from-tenant
exit

Here the default VRF's instance imports the VRF tenant, which tracks IPv6 routes, with the policy from-tenant. An import without a policy gets permit 10 alone. Imports from VRFs that track nothing of the family use no leak route-map.

The large community 4200000000:0:1 is reserved for tracked routes. The leak route-maps deny every path that carries it, also one a peer sent with it or one an import policy marked with it. Such a route stays out of a VRF that imports it from a VRF that tracks routes of its family. A policy's removeCommunities strips the community from a peer's routes.

bgpd checks the condition every 5 seconds, FRR's bgp default-originate timer. The default route therefore follows a tracked route within about 5 seconds. A delivery that adds, changes or drops a peer family's defaultOriginateWhen withdraws the default route at once, and the next check sends it again if the route exists.

The check parses the config, and the validator and the renderer check the rest

A delivery renders frr.conf and checks it with vtysh --dryrun before it changes anything. vtysh parses each line with the daemons' commands and needs no daemon. It catches a command that no daemon takes, and a value outside a command's range, such as le 129. It does not check what the daemons would refuse when they apply it, such as a route-map that names a missing list.

At boot, FRR's mgmtd applies the prefix lists, route-maps and static routes of the config in one transaction, so one value it refuses would leave all of them out. The schemas, the rules and the config's uniqueness check therefore refuse the values FRR's daemons refuse, and chalkctl validate runs them all. The uniqueness check refuses two entries of a PrefixList with one prefix, ge and le, one community twice in a CommunityList and two networks of one range, before any rule runs. The rules refuse a ge or le outside the prefix's length and the family's longest prefix, a policy whose PrefixLists are of both families, and a peer's or network's policy of another family. The schemas refuse a community with a part above 65535 in a standard community or above 4294967295 in a large one, and a description with an @ or two spaces in a row.

The validator router checks the peers' sessions against the Network and VRF documents and against each other. It runs after the rules and before the renderers, at a delivery, at chalkctl config apply --dry-run and in chalkctl validate, and refuses:

  • a session that two peers name;
  • a link that a peer's glob expands to and that is named like an address;
  • a peer's interface in another VRF than the peer's, or of a VRF's device, and a glob that matches links of other VRFs alone.

The renderer trusts what the schemas, the rules and the validator check. It checks what depends on the names FRR gives its objects or on bgpd itself, when the router renders its config, at a delivery or at chalkctl config apply --dry-run, and chalkctl validate does not run it:

  • an asPath that bgpd cannot compile;
  • an interface named like a peer's group;
  • a policy with prepend that instances of two AS numbers use.

A value that would add a command to the config, such as a line break in a description, cannot pass the schemas. Should a document bypass them, the renderer fails with internal error: ...; chalkd checks the config against the schemas, the rules and the validators before it renders rather than write it.

bgpd compiles an asPath with glibc's regcomp as a POSIX extended regular expression, each _ standing for the start, the end or a separator of the path. bgpd drops an as-path list it cannot compile, and its term then matches no route. The renderer checks each expression as glibc reads it. It refuses a repetition of nothing, such as ^*, parentheses that do not pair up and an interval that ends below its start. The schema leaves out brackets, whose meaning differs between POSIX and other regular expressions.

Passwords reach FRR as credentials

The renderer sees no Secret. A peer's password renders as the placeholder @secret_<secret>.<key>@ on its peer group's password line. frr.service gets the passwords that the BGPPeer documents name as its credentials. Before it starts and on each reload, a helper of the unit writes /run/frr/frr.conf with each password line's placeholder replaced by its credential. Only root reads the file, mode 0600, and /etc/frr/frr.conf links to it.

The helper fills in the password lines alone. A placeholder elsewhere stays as written, and a description cannot hold one, as it holds no @. A password that is empty, longer than 80 characters, or holds white space, a line break or anything but printable ASCII fails the unit and leaves the file as it was. So does a placeholder whose credential the unit lacks.

A rotated password changes no rendered file. The effect of frr fires all the same, because the value of a Secret that its documents reference changed. The reload loads the credentials again before the helper runs.

A delivery reloads FRR

The effect of frr reloads frr.service once the delivery is stored. The reload fills in the passwords again and runs chalkos-frr-reload, which compares the new config with the running config that vtysh shows and applies the difference with vtysh -f, so a session that the change leaves alone stays up. A command whose value changes, such as a password, a peer's route-map or the route-map of a network, is given its new value without being deleted first. Commands, peers, peer groups and instances that the new config lacks are deleted before the new ones are added. FRR keeps a community list's or AS path list's entries by value, so an entry that moves to another sequence number is deleted before the entries are added. Route-map entries, BFD profiles, prefix lists and lists the new config drops go last, once nothing names them. Afterwards chalkos-frr-reload reads the running config again. A deleted peer group takes its members and their config with it, so a second pass may follow.

FRR shows the system's hostname, so the comparison leaves the hostname out. A reload that changes something sets the new config's hostname as well, and a new hostname alone takes effect at the next such reload or at a restart.

chalkos-frr-reload knows the commands the renderer writes and the lines FRR shows for them. It fails the reload on a line it does not know, in either config, and when the running config still differs after two passes or after 90 seconds. Its log in the journal names the commands it ran and those vtysh refused, with the passwords hidden. A reload that fails fails the delivery, with the config stored. FRR keeps the commands the reload applied, and a restart of frr.service loads the new config whole. A failed frr.service is restarted instead of reloaded.

chalkd reloads the units of one delivery in one systemctl reload. frr.service orders after chalkos-firewall.service, so the firewall accepts a new peer's ports before FRR dials it.

FRR starts after the config and the firewall

At boot, frr.service starts after chalkos-config.service has rendered the outputs and after chalkos-firewall.service. It orders before chalkos-node-addresses.target and is wanted by it, as a unit that announces the node's addresses. Without a rendered frr.conf, the helper writes the image's fallback config, which runs no BGP.

frr.service starts only when chalkd has written its credentials, which it does once the node has config. In maintenance mode, which loads no config, the node runs no FRR.

networkd leaves zebra's routes and next hops alone. By default it removes the routes it did not configure itself each time it configures a link, so router images turn ManageForeignRoutes= and ManageForeignNextHops= off.

The firewall accepts each peer's ports

The input chain of inet chalkos accepts BGP, TCP port 179, from each BGPPeer, and BFD from each with bfd. A peer of interface or interfaces is accepted on those interfaces, globs as written, a peer of address from that address alone, and a peer of listen from its range. BFD takes UDP ports 3784 and 3785 of single-hop sessions, and 4784 of multi-hop ones from a peer of address or listen, whose neighbours may lie beyond the link. A peer on a link is one hop away. The edge renders:

iifname { "eth1", "eth2" } tcp dport { 179 } accept comment "BGPPeer fabric"
iifname { "eth1", "eth2" } udp dport { 3784, 3785 } accept comment "BGPPeer fabric"
ip saddr { 203.0.113.254/32 } tcp dport { 179 } accept comment "BGPPeer transit"

These rules come after chalkd's port and before the policies to local, which cannot close them. The router's firewall describes the chain. A peer of a VRF is accepted on that VRF's links by meta sdifname, as VRFs on a router explains.

chalkd reads FRR's state

chalkctl status shows a router's BGP sessions. chalkd asks vtysh for show bgp vrf all neighbors json, within 5 seconds, and reports each session's VRF, peer, neighbour, state, uptime, routes received and sent, and BFD state. It finds a session's BGPPeer by the name of its peer group. A neighbour of a listen range is in its peer's group too, and the status names it by its address. When vtysh fails, the status gives its error, and the status's other parts stand.

An image whose modules set chalkos.router.bgp.enable to false carries no FRR and refuses the kinds BGP, BGPPeer, PrefixList, CommunityList and RoutingPolicy. chalkd finds no /etc/frr/frr.conf there and asks vtysh nothing. The status sets bgp_absent, and chalkctl status prints no bgp line. A router with FRR and no sessions prints bgp: no sessions instead. On such an image the Routes call lists the kernel's routes alone.

The Routes call lists the kernel's routes, which chalkd reads through netlink, and the paths BGP selected, from show bgp ipv4 unicast detail json and its IPv6 twin. For another VRF it lists that VRF's table and asks for show bgp vrf <name> ipv4 unicast detail json. A route of several next hops lists each with its weight, and an IPv4 route over an IPv6 next hop names it. chalkctl node routes prints both. A node of another image answers that the router layer is not in its image.

The diagnostics bgp-summary, bgp-neighbor, bgp-route and bfd-peers run vtysh as a member of frrvty, the group of FRR's sockets, and need a reader.

Limits

  • FRR announces a network of BGP.spec.networks only while the router's table holds a route to it, whatever the policy says. A VRF's table always holds a default route, the unreachable one that ends it, so the rules refuse a default route among a VRF instance's networks. A peer family's defaultOriginate sends one instead.
  • zebra prefers a route the kernel holds from elsewhere over BGP's path to the same prefix, and then installs none of BGP's. A default route that networkd takes from router advertisements on an uplink keeps BGP's default route out of the kernel. ipv6AcceptRA: false on that Network lets it in.
  • FRR negotiates the extended next hop on sessions of interface and interfaces whatever extendedNextHop says.
  • A peer's interfaces match the names of Network documents, not the links that exist, and a Network that matches links by anything but Name= names none.
  • The AS number rule compares a BGP document with the running one of the same name. A BGP document of another name replaces the old one, AS number included.
  • The condition of defaultOriginateWhen counts BGP's paths too. A list that matches a prefix of the instance's networks, a peer's route or an imported route holds while BGP holds that path.
  • The condition counts every path that passes the route-map. That includes an invalid path, such as one whose next hop does not resolve, and a stale path that graceful restart keeps after its session went down. Each check, up to every 5 seconds, walks the family's whole BGP table. That costs CPU when the instance holds a full Internet table in that family.
  • Blackhole, unreachable and prohibit routes never satisfy defaultOriginateWhen.