Skip to content

Write a BGP extension

This tutorial builds the BGP extension in examples/bgp-extension from an empty module. Its image, node-bird-metal, uses BIRD in place of the Kubernetes layer's built-in BGP speaker. The extension runs bird on each node, with two config kinds of its own: Speaker for the node's side of its sessions and Peer for its neighbours. At the end node-bird-metal declares both kinds, renders bird.conf from them, runs bird with the sessions' password, reloads it on a change, refuses config bird cannot run, and shows bird's sessions to an operator. Each step adds one part, shows how to look at what it produced, and tests it before the next. The last part debugs the extension on a running node. Plan about an hour, most of it for builds.

For a smaller first step, Write your first extension adds one kind, a rendered file, a test and a unit to the quick start's lab in about 15 minutes.

The Build images pages explain each part in depth. This page puts them in the order of the work and links to them where the details matter.

Before you begin

You need:

  • a ChalkOS checkout and the homelab example in a git repository of its own, with the flake's shell, as Prepare the files and Get the tools of the home lab set it up. Run every command of this page in that shell, in the repository;
  • some Nix: NixOS modules, options and let bindings;
  • for the last part, a node of the home lab, which Try it on a node moves to node-bird-metal. Build a home lab installs three. The extension needs a BGP neighbour on an IPv6 network, which the quick start's lab of virtual machines lacks, so that lab can run the steps up to the node but not the sessions.

The steps up to Try it on a node need no node: they run in the Nix sandbox.

Start from an empty module

Keep a copy of the finished extension nearby while you build your own. These commands assume you made the homelab beside the ChalkOS checkout as shown in the home lab guide.

cp -r ../chalkos/examples/bgp-extension/extensions extensions.done
mkdir extensions

Add this entry to images in the homelab's flake.nix. It builds node-bird-metal with your extensions/ directory and turns off the Kubernetes layer's built-in BGP in that image.

{
  name = "node-bird-metal";
  platform = "metal";
  modules = [
    chalkos.nixosModules.kubernetes
    { chalkos.kubernetes.bgp.enable = false; }
    ./extensions
  ];
}

Add node-bird-metal to the list inherited from images in packages.${system}. Create extensions/default.nix to import the extension.

extensions/default.nix
# The guide's BGP extension, bird with its Speaker and Peer kinds and the config rendered from
# them, which the image node-bird-metal carries in place of the built-in speaker.
{
  imports = [ ./bgp.nix ];
}

extensions/bgp.nix starts with the module's arguments and two types the kinds use:

extensions/bgp.nix
{
  chalkosLib,
  config,
  lib,
  pkgs,
  ...
}:
let
  # An interface's name as bird.conf quotes it: the characters of a Linux interface name, 15 at
  # most, without globs.
  interfaceName = lib.types.strMatching "[A-Za-z0-9_.-]{1,15}";
  # An octet of an IPv4 address in decimal, without leading zeros.
  octet = "25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9]";
in
{
}

chalkosLib is chalkos's library for image modules: the kinds' option types, the renderers and the effects' helpers. Every later step adds attributes to the module's body. Add the new files to git, because a flake sees only the files git tracks:

git add flake.nix extensions

A file that git does not track is missing from the build, and the error names a path that does not exist. Run git add after each new file of the steps below.

Declare the kinds

A kind is the schema of a config document. The image converts its options into a JSON Schema that chalkctl and chalkd check every document against. Add Peer to the module's body:

chalkos.kinds.Peer = {
  group = "bgp.example.org";
  version = "v1alpha1";
  description = "A BGP neighbour that the node's bird runs a session with on each of the given interfaces.";
  multiple = true;
  spec = {
    # The address lands in bird.conf. chalkctl and chalkd accept only an IPv4 or IPv6 address,
    # which holds nothing that could end bird's statement, and the rule keeps it to IPv6, the
    # family of the sessions bird.conf sets up.
    address = chalkosLib.mkOption {
      type = chalkosLib.types.address;
      example = "fdc8:a1b2:c3d4:1::1";
      description = "IPv6 address of the neighbour, on the subnet of the interfaces the sessions run on.";
      rules = [
        {
          description = "A neighbour has an IPv6 address.";
          rule = ''ip(self).family() == "ipv6"'';
          messageExpression = ''self + " is no IPv6 address; bird runs IPv6 sessions alone"'';
        }
      ];
    };
    # A neighbour's AS number is part of who it is: a new one is a new Peer.
    asn = chalkosLib.mkOption {
      type = lib.types.ints.between 1 4294967295;
      example = 65000;
      description = "AS number of the neighbour.";
      rules = [
        {
          description = "A Peer keeps its AS number.";
          rule = "self == oldSelf";
          messageExpression = ''"the AS number stays " + string(oldSelf) + "; add a Peer of another name for " + string(self)'';
        }
      ];
    };
    # Each name lands in bird.conf in quotes. A name holds no glob, so a Peer lists each port by its
    # exact name.
    interfaces = chalkosLib.mkOption {
      type = lib.types.nonEmptyListOf interfaceName;
      example = [
        "enp1s0"
        "enp2s0"
      ];
      unique = {
        across = "list";
        hint = "list each interface once";
      };
      description = "Interfaces on which bird runs a session with the neighbour, one each, bound to the interface.";
    };
    password = lib.mkOption {
      type = lib.types.nullOr chalkosLib.types.secretRef;
      default = null;
      example = {
        secretRef = {
          name = "bgp";
          key = "password";
        };
      };
      description = "Key of a Secret that holds the sessions' TCP MD5 password; null for none.";
    };
    bfd = lib.mkOption {
      type = lib.types.bool;
      default = true;
      description = "Whether BFD watches each session, so that bird ends it within about a second of its link failing.";
    };
  };
};

Each field is a NixOS option. The type decides what the schema accepts, so pick the narrowest one: every value here lands in bird.conf, and chalkosLib.types.address and interfaceName keep a value from adding a statement of its own. Without them, a value such as fdc8:a1b2:c3d4:1::1; import all would add a statement to bird.conf. The rule of address keeps it to IPv6, and the pattern of interfaceName keeps each name of interfaces to the characters of an interface name. A field without a default is required. password and bfd have defaults, so a Peer may leave both out. rules and unique need chalkosLib.mkOption in place of lib.mkOption.

Add Speaker, of which a node holds one, since it has no multiple:

# bird takes its router ID from an IPv4 address of the node unless its config gives one, and an
# IPv6-only node has none, so the Speaker gives it.
chalkos.kinds.Speaker = {
  group = "bgp.example.org";
  version = "v1alpha1";
  description = "The node's side of its BGP sessions: its AS number, its router ID and the interfaces whose addresses bird announces.";
  spec = {
    asn = lib.mkOption {
      type = lib.types.ints.between 1 4294967295;
      example = 65001;
      description = "AS number of the node.";
    };
    routerID = chalkosLib.mkOption {
      type = lib.types.strMatching "(${octet})([.](${octet})){3}";
      example = "0.0.0.11";
      description = "Router ID of the node in the form of an IPv4 address, unique within the node's AS.";
      rules = [
        {
          description = "A router ID is not 0.0.0.0.";
          rule = ''self != "0.0.0.0"'';
          message = "bird takes no router ID 0.0.0.0; give the node another one, unique within its AS";
        }
      ];
    };
    export.interfaces = lib.mkOption {
      type = lib.types.nonEmptyListOf interfaceName;
      example = [ "node0" ];
      description = "Interfaces whose addresses bird announces to every Peer, such as a dummy interface that holds the node's address.";
    };
  };
};

chalkctl refuses a second Speaker on a node. export.interfaces is a nested field: an attribute set of options in spec becomes an object of the schema. The rule of routerID refuses 0.0.0.0, which the pattern of an IPv4 address accepts. Each node needs a router ID of its own, so its Speaker sits in a file of that node alone. node1's holds the fields of its built-in BGPSpeaker:

apiVersion: bgp.example.org/v1alpha1
kind: Speaker
metadata: {name: bgp}
spec:
  asn: 65001
  routerID: 0.0.0.11
  export: {interfaces: [node0]}

Look at the schemas

The schemas are a build output of their own, which takes seconds and needs no image. Build them and print a field:

$ nix build .#nixosConfigurations.node-bird-metal.config.system.build.chalkosSchemas
$ jq .properties.spec.properties.asn result/bgp.example.org_v1alpha1_Peer.json
{
  "description": "AS number of the neighbour.",
  "examples": [
    65000
  ],
  "maximum": 4294967295,
  "minimum": 1,
  "type": "integer",
  "x-chalkos-rules": [
    {
      "description": "A Peer keeps its AS number.",
      "messageExpression": "\"the AS number stays \" + string(oldSelf) + \"; add a Peer of another name for \" + string(self)",
      "rule": "self == oldSelf",
      "takes": "refuse"
    }
  ]
}

result/ holds a file per kind the image accepts. A type that has no JSON Schema, a rule that does not compile or a misspelled option fails this build, with the option's path in the message. It is the fastest check of a kind: run it after each change to one.

Validate a node's files

The homelab's files hold the built-in BGPSpeaker and BGPPeer documents, which node-bird-metal does not accept. Give node1 a file of its own with the extension's documents in their place. Start from its current file:

cp nodes/node1.yaml nodes/node1-bird.yaml

In nodes/node1-bird.yaml, replace the BGPSpeaker document with the Speaker above, and add the ports' Network documents of bgp/fabric.yaml and a Peer for the router. bird runs a session on each interface of the Peer, both to the router's first address. The file's header comment and the comment above the Speaker speak of FRR, BGPPeer documents and bgp/fabric.yaml: adapt them to bird and to the documents the file now holds.

apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 20-enp1s0}
spec:
  match: {name: enp1s0}
  network: {ipv6AcceptRA: true}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 20-enp2s0}
spec:
  match: {name: enp2s0}
  network: {ipv6AcceptRA: true}
---
apiVersion: bgp.example.org/v1alpha1
kind: Peer
metadata: {name: router}
spec:
  address: "fdc8:a1b2:c3d4:1::1"
  asn: 65000
  interfaces: [enp1s0, enp2s0]
  password: {secretRef: {name: bgp, key: password}}

In chalkos.yaml, add the image and give node1 that image and its files, outside the group nodes, whose bgp/fabric.yaml holds BGPPeer documents:

chalkos.yaml
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: homelab}
spec:
  images:
    router-metal: {source: {directory: images/router-metal}}
    node-metal: {source: {directory: images/node-metal}}
    node-bird-metal: {source: {directory: images/node-bird-metal}}
  groups:
    router: {imageName: router-metal, files: [router/*.yaml]}
    nodes: {imageName: node-metal, files: [cluster/*.yaml, bgp/fabric.yaml, bgp/secrets.yaml]}
  nodes:
    router: {address: 192.168.1.2, groups: [router]}
    node1:
      address: "fdc8:a1b2:c3d4:ff::11"
      imageName: node-bird-metal
      files: [cluster/*.yaml, bgp/secrets.yaml, nodes/node1-bird.yaml]
    node2: {address: "fdc8:a1b2:c3d4:ff::12", groups: [nodes], files: [nodes/node2.yaml]}
    node3: {address: "fdc8:a1b2:c3d4:ff::13", groups: [nodes], files: [nodes/node3.yaml]}

chalkctl validate checks each node's documents against the schemas of the image the inventory names. It reads the built image, so build it first:

nix build .#node-bird-metal -o images/node-bird-metal
chalkctl validate

The image build takes minutes, so keep it for the files of the nodes. With asn misspelled as asm in the Peer, chalkctl validate refuses node1:

$ chalkctl validate
...
chalkctl: the config of 1 node is invalid:
node node1: Peer "router": - at '/spec': additional properties 'asm' not allowed
- at '/spec': missing property 'asn'

Render bird.conf

bird reads a config file of its own format, so an output renders it from the documents. Copy the template from the finished extension:

cp extensions.done/bird.conf.tmpl extensions/
git add extensions
extensions/bird.conf.tmpl
# bird's config for {{ .node.name }}, rendered by chalkos from its Speaker and Peer documents.
{{- if and .documents.Peer (not .documents.Speaker) }}
{{- refuse "Peer" (index .documents.Peer 0).metadata.name "metadata.name" "a node with Peer documents needs a Speaker document, which gives bird its AS number and router ID" }}
{{- end }}
{{- range .documents.Speaker }}
router id {{ .spec.routerID }};
{{- else }}
# Without a Speaker bird runs no session. The router ID lets it start and reaches no peer.
router id 0.0.0.1;
{{- end }}

protocol device {
}
{{- range .documents.Speaker }}
{{- $asn := .spec.asn }}

# The addresses of the Speaker's interfaces, which every session announces.
protocol direct node_addresses {
  ipv6;
{{- range .spec.export.interfaces }}
  interface {{ quote . }};
{{- end }}
}

# The default route the peers send, installed into the kernel with a next hop per session.
protocol kernel {
  ipv6 {
    import none;
    export where source = RTS_BGP;
  };
  merge paths on;
}

protocol bfd {
}
{{- range $.documents.Peer }}
{{- $peer := . }}
{{- range .spec.interfaces }}

protocol bgp '{{ $peer.metadata.name }}_{{ . }}' {
  local as {{ $asn }};
  neighbor {{ $peer.spec.address }} as {{ $peer.spec.asn }};
  interface {{ quote . }};
  direct;
  connect retry time 5;
{{- if $peer.spec.bfd }}
  bfd on;
{{- end }}
{{- with $peer.spec.password }}
  authentication md5;
  password 'secret_{{ .secretRef.name }}.{{ .secretRef.key }}';
{{- end }}
  ipv6 {
    # The router sends a link-local next hop beside its global one. bird would resolve the global
    # one through the first port on the fabric's prefix for every session; the link-local one
    # keeps each session's routes on its own port.
    next hop prefer local;
    import where net = ::/0;
    export where proto = "node_addresses";
  };
}
{{- end }}
{{- end }}
{{- end }}

The template is Go's text/template. It reads the node's name as .node.name and each input kind's documents as a list, .documents.Speaker and .documents.Peer, sorted by name and with the schemas' defaults filled in. quote and refuse are chalkos's own functions. The template's first lines refuse Peer documents on a node without a Speaker, whose AS number every session needs: refuse prints a refusal that names the first Peer, and the delivery is refused.

The output's check runs bird's parser on the rendered file, so the module needs bird's package. Add it to the let bindings:

bird = config.services.bird.package;

Then declare the output in the body:

# bird.conf, with a session per Peer and interface. bird -p parses a config without running it;
# the constants the password lines name exist on the node alone, so the check defines each first.
chalkos.outputs.bird-conf = {
  inputs = [
    "Speaker"
    "Peer"
  ];
  render = chalkosLib.render.template ./bird.conf.tmpl;
  check = ''
    conf=$(mktemp)
    grep -o "'secret_[^']*'" "$OUT/bird.conf" | sort -u | sed 's/.*/define & = "checked";/' >"$conf"
    echo "include \"$OUT/bird.conf\";" >>"$conf"
    ${bird}/bin/bird -p -c "$conf"
  '';
};

On a node, chalkd renders and checks every output before a delivery changes anything, and refuses the delivery when one fails. The same renderer and check run in the image's tests, without a node.

See what the output renders

A test gives the image a config and records what the image makes of it. Copy the example config of a homelab node that runs the extension, which holds a Speaker, the Peer, the Secret with the password and the Network documents of the ports, and declare a test of it in the body:

mkdir extensions/bgp-examples
cp extensions.done/bgp-examples/fabric.yaml extensions/bgp-examples/
git add extensions
chalkos.tests.bgp-fabric.config = ./bgp-examples/fabric.yaml;

Build the results of the image's tests. The build takes seconds, needs no image, and succeeds whatever the tests got:

$ nix build .#nixosConfigurations.node-bird-metal.config.system.build.chalkosTestResults
$ cat result/summary.txt
PASS bgp-fabric
PASS chrony-empty
PASS chrony-servers
PASS networkd-none
PASS networkd-static
PASS networkd-vlan

The chrony-* and networkd-* tests are chalkos's own, which every image carries. bgp-fabric passes because it expects nothing yet beyond an accepted config. Read what the output rendered:

$ cat result/bgp-fabric/outputs/bird-conf/bird.conf
# bird's config for example, rendered by chalkos from its Speaker and Peer documents.
router id 0.0.0.11;

protocol device {
}

# The addresses of the Speaker's interfaces, which every session announces.
protocol direct node_addresses {
  ipv6;
  interface "node0";
}

# The default route the peers send, installed into the kernel with a next hop per session.
protocol kernel {
  ipv6 {
    import none;
    export where source = RTS_BGP;
  };
  merge paths on;
}

protocol bfd {
}

protocol bgp 'router_enp1s0' {
  local as 65001;
  neighbor fdc8:a1b2:c3d4:1::1 as 65000;
  interface "enp1s0";
  direct;
  connect retry time 5;
  bfd on;
  authentication md5;
  password 'secret_bgp.password';
  ipv6 {
    # The router sends a link-local next hop beside its global one. bird would resolve the global
    # one through the first port on the fabric's prefix for every session; the link-local one
    # keeps each session's routes on its own port.
    next hop prefer local;
    import where net = ::/0;
    export where proto = "node_addresses";
  };
}

protocol bgp 'router_enp2s0' {
  local as 65001;
  neighbor fdc8:a1b2:c3d4:1::1 as 65000;
  interface "enp2s0";
  direct;
  connect retry time 5;
  bfd on;
  authentication md5;
  password 'secret_bgp.password';
  ipv6 {
    # The router sends a link-local next hop beside its global one. bird would resolve the global
    # one through the first port on the fabric's prefix for every session; the link-local one
    # keeps each session's routes on its own port.
    next hop prefer local;
    import where net = ::/0;
    export where proto = "node_addresses";
  };
}

It holds a protocol bgp per interface of the Peer, named after the Peer and the interface and bound to that interface. The Speaker gives the router ID, the AS number of every session and the interfaces of node_addresses, whose addresses each session announces. If a neighbour sends a default route, the extension accepts it and merge paths can install it with a next hop per session. The homelab router sends its gateway in router advertisements. The test's node is named example. After each change to the template, build the results again and read the new file.

Find out why a render fails

Say the template loses the ; after router id. bird's parser refuses the file, and the test gets the refusal a delivery would get:

$ cat result/summary.txt
FAIL bgp-fabric: refusals differ: expected [], got [{"kind":"","name":"","path":"","message":"output bird-conf: its check refused the rendered files (exit 1)"}]
...

The refusal names the output and the exit status alone, as on a node, because a check's output can echo the documents. What bird printed is in the build's log. Build with -L to print it:

$ nix build -L .#nixosConfigurations.node-bird-metal.config.system.build.chalkosTestResults
...
chalkos-test-results> 2026/11/16 07:52:35 output bird-conf: its check /nix/store/39kg0jfg9wiagx56jrf3hpnwjvh6xv44-chalkos-output-bird-conf-check failed (exit 1): "bird: 2026-11-16 07:52:35.151 [0000] <FATAL> /build/chalkos-tests-2581098952/.staging/bird-conf.1551592570/bird.conf:4:1 syntax error, unexpected PROTOCOL, expecting ';'"
...

Line 4 of the rendered file starts with protocol, where bird expected the ; of the line before. A render that fails leaves no files in result/<test>/outputs, so the log is the place to look.

Keep the rendered file as golden files

Once bird.conf is the file bird should get, keep it in the repository, and have the test compare every later render with it:

mkdir -p extensions/tests/fabric
cp -r --no-preserve=mode result/bgp-fabric/outputs/bird-conf extensions/tests/fabric/bird-conf
git add extensions
chalkos.tests.bgp-fabric = {
  config = ./bgp-examples/fabric.yaml;
  outputs.bird-conf = ./tests/fabric/bird-conf;
};

From now on a template change that alters the file fails bgp-fabric with a diff, and prints the command that takes the new file.

Run bird

The image needs bird itself, reading the rendered file. Add the service to the body:

services.bird = {
  enable = true;
  # The node's constants, then the rendered config. Neither exists when the image is built, so
  # bird-conf's check parses the config in place of the module's.
  config = ''
    include "/run/bird/credentials.conf";
    include "${config.chalkos.outputs.bird-conf.dir}/bird.conf";
  '';
  checkConfig = false;
};
systemd.services.bird.serviceConfig = {
  ExecStartPre = lib.getExe credentials;
  # A reload loads the credentials again and writes the constants anew, so a rotated password
  # reaches bird without a restart.
  RefreshOnReload = "credentials";
  ExecReload = lib.mkForce [
    (lib.getExe credentials)
    "${bird}/bin/birdc configure"
  ];
};

bird.conf exists on the node alone, so the NixOS module's check of the config is off, and the output's check takes its place. The include of the output's dir names the output by its option, so a misspelled name fails the evaluation. The service's settings run bird-credentials, which turns the sessions' password into bird constants. Add it to the let bindings:

# bird reads no password from a file, and its strings take no escapes: this writes each of the
# unit's credentials as a constant of bird's config, named as the credential, which bird.conf's
# password lines name. A name outside letters, digits, '.', '_' and '-', or a value with a quote
# or a line break, fails the unit. The constants go to a new file that replaces the old one, so
# bird never reads a file half written or emptied by a failed run.
credentials = pkgs.writeShellApplication {
  name = "bird-credentials";
  text = ''
    shopt -s nullglob
    umask 077
    tmp=$(mktemp /run/bird/credentials.conf.XXXXXX)
    trap 'rm -f "$tmp"' EXIT
    for file in "''${CREDENTIALS_DIRECTORY-}"/secret_*; do
      name=''${file##*/}
      if [[ ! ''${name#secret_} =~ ^[A-Za-z0-9._-]+$ ]]; then
        echo "$name: bird takes a credential name of letters, digits, '.', '_' and '-' alone" >&2
        exit 1
      fi
      value=$(<"$file")
      if [[ $value == *[$'"\n']* ]]; then
        echo "$name: bird takes no password with a quote or a line break" >&2
        exit 1
      fi
      printf "define '%s' = \"%s\";\n" "$name" "$value"
    done >"$tmp"
    mv "$tmp" /run/bird/credentials.conf
  '';
};

bird-credentials runs before bird starts and on each reload. It reads each secret_* file in $CREDENTIALS_DIRECTORY and writes a line define 'secret_bgp.password' = "..."; to /run/bird/credentials.conf, which only bird's user may read. bird's config includes that file before the rendered one. chalkd accepts more characters in a Secret's name and key than bird-credentials does, so keep both to letters, digits, ., _ and -.

A renderer never sees a secret's value, so bird.conf names the password by its credential, 'secret_bgp.password', and the constant resolves it. A grant of the Peer kind has chalkd hand bird the values that the Peer documents' password fields name, as systemd credentials:

# chalkd loads the values the Peers' password fields name as bird's credentials.
systemd.services.bird.chalkos.kinds = [ "Peer" ];

The homelab keeps the password in bgp/secrets.yaml, a file of its own, so that the file with the Peer can go into a public repository. The router's side of the sessions needs the same password, which router/secrets.yaml holds in a Secret of the router's own.

Last, two settings outside bird: networkd leaves the routes bird installs alone, and the firewall lets BFD's packets in:

# bird installs the default route its peers send. networkd removes the routes it did not
# configure itself each time it configures a link, so it leaves them alone here, as on router
# images. BFD's control packets arrive on UDP port 3784, from a source port of the neighbour's
# choosing, so no connection the firewall tracks covers them.
systemd.network.config.networkConfig = {
  ManageForeignRoutes = false;
  ManageForeignNextHops = false;
};
networking.firewall.allowedUDPPorts = [ 3784 ];

Build the image. The build checks what the evaluation cannot, such as a granted unit that runs nothing:

nix build .#node-bird-metal -o images/node-bird-metal

Reload bird on a change

A delivery that changes bird.conf must reach bird. An effect says what a delivery does to a service once it is applied. Add it to the body:

# bird reads its config again on birdc configure, which keeps the sessions a restart would drop.
chalkos.effects.bird = chalkosLib.reloadOn { outputs = [ "bird-conf" ]; } [ "bird.service" ];

bird reloads when the files of bird-conf change, or when a value of a Secret that the output's documents reference changes. Check the effect with a test of a change: before holds the config the node runs, and config the one delivered. Copy the example's config with a new password, and replace chalkos.tests.bgp-fabric with chalkos.tests = { ... }; holding these two tests:

cp extensions.done/tests/fabric-new-password.yaml extensions/tests/
git add extensions
# The image accepts the example: bgp-peers refuses nothing, bird parses the bird.conf it
# renders, and that is the golden file's.
bgp-fabric = {
  config = ./bgp-examples/fabric.yaml;
  outputs.bird-conf = ./tests/fabric/bird-conf;
};
# A new password leaves bird.conf as it was, which names the credential alone, and bird
# reloads to load the credential anew.
bgp-new-password = {
  before = ./bgp-examples/fabric.yaml;
  config = ./tests/fabric-new-password.yaml;
  outputs.bird-conf = ./tests/fabric/bird-conf;
  effects = [ "bird" ];
};

bird.conf names the credential, not its value, so the new password leaves the file as it was, and the test compares it with the same golden files. The effect fires all the same, and the results show it:

$ cat result/bgp-new-password/effects.json
["bird"]

Refuse what bird cannot run

A schema checks each document on its own. Two more checks refuse a config before it reaches bird: a change rule, which compares a document with the one the node runs, and a validator, a program that reads documents of several kinds.

Keep a Peer's AS number

The asn field of the Peer you declared carries a rule, self == oldSelf: a neighbour with a new AS number is a new neighbour, so the node refuses the change. Copy the example's config with a new AS number, and test the refusal:

cp extensions.done/tests/new-asn.yaml extensions/tests/
git add extensions
chalkos.tests.bgp-new-asn = {
  before = ./bgp-examples/fabric.yaml;
  config = ./tests/new-asn.yaml;
  refusals = [
    {
      kind = "Peer";
      name = "router";
      path = "spec.asn";
      message = "the AS number stays 65000; add a Peer of another name for 65002";
    }
  ];
};

The test passes when the refusals are exactly these. result/bgp-new-asn/refusals.json holds what the test got:

$ cat result/bgp-new-asn/refusals.json
[{"kind":"Peer","name":"router","path":"spec.asn","message":"the AS number stays 65000; add a Peer of another name for 65002"}]

Refuse an interface the node does not configure

bird binds each session to an interface. A Peer on an interface that no Network document configures gets no session, and no schema can tell, because the names sit in documents of another kind. A validator reads the documents of several kinds and prints the refusals it finds. Copy the example's jq program and declare it:

cp extensions.done/bgp-peers.jq extensions/
git add extensions
# bird binds each session to an interface and announces the addresses of the Speaker's
# interfaces, which the schemas cannot tell the node configures: the validator looks for the
# names in the node's Network documents, and refuses Peers without a Speaker.
chalkos.validators.bgp-peers = {
  inputs = [
    "Peer"
    "Speaker"
    "Network"
  ];
  program = chalkosLib.validate.jq ./bgp-peers.jq;
};

The program reads the same input as a renderer, and prints a report of refusals, one per interface that no Network names:

extensions/bgp-peers.jq
# The validator bgp-peers: it reads the node's Peer, Speaker and Network documents, by kind.
# bird binds each session of a Peer to one of the Peer's interfaces, and announces the addresses
# of the Speaker's interfaces. Neither works on an interface that networkd leaves down, so the
# validator refuses an interface that no Network document names. It compares the names as
# written: a glob such as fabric* in a Network may or may not match an interface, so it names
# none. It also refuses Peer documents without a Speaker, which gives bird its AS number and
# router ID.

# A value as a list: a section or key of a networkd document is one value or a list of them.
def each: if type == "array" then .[] else . end;

# The names the [Match] sections of the Network documents give, where one string may hold
# several, separated by spaces.
[ .documents.Network[] | .spec.match? // [] | each | .name? // [] | each
  | strings | splits(" +") | select(. != "") ] as $names

# A refusal of each item of a list of interfaces that no Network names, as the field path names
# the list.
| def unconfigured($document; $path; $why):
    to_entries[] | select(.value as $name | $names | index([$name]) | not)
    | { kind: $document.kind, name: $document.metadata.name, path: "\($path)[\(.key)]",
        message: "\(.value) has no Network document; \($why)" };

{ refusals: [
    ( .documents.Peer[] | . as $peer | .spec.interfaces
      | unconfigured($peer; "spec.interfaces"; "bird binds the session to an interface the node configures") ),
    ( .documents.Speaker[] | . as $speaker | .spec.export.interfaces
      | unconfigured($speaker; "spec.export.interfaces"; "bird announces the addresses of an interface the node configures") ),
    ( select(.documents.Peer != [] and .documents.Speaker == [])
      | { kind: "Peer", name: .documents.Peer[0].metadata.name, path: "metadata.name",
          message: "a node with Peer documents needs a Speaker document, which gives bird its AS number and router ID" } )
  ] }

A section of a networkd document may be one object or a list of them, and a key one value or a list, so the program reads name from every [Match] section, and splits a value that holds several names. It compares the names as written. A glob such as enp* may or may not match an interface, so the program takes it for no name, and a Peer on such an interface is refused.

The program also refuses Peer documents without a Speaker, as the template does. The validators run before the renderers, so the operator sees the validator's refusal. Test this one with a config whose Peer names a third interface, enp3s0, without a Network document:

cp extensions.done/tests/unconfigured-peer-interface.yaml extensions/tests/
git add extensions
chalkos.tests.bgp-unconfigured-peer-interface = {
  config = ./tests/unconfigured-peer-interface.yaml;
  refusals = [
    {
      kind = "Peer";
      name = "router";
      path = "spec.interfaces[2]";
      message = "enp3s0 has no Network document; bird binds the session to an interface the node configures";
    }
  ];
};

result/bgp-unconfigured-peer-interface/refusals.json holds what the test got:

$ cat result/bgp-unconfigured-peer-interface/refusals.json
[{"kind":"Peer","name":"router","path":"spec.interfaces[2]","message":"enp3s0 has no Network document; bird binds the session to an interface the node configures"}]

chalkctl validate accepts this config, because chalkctl runs no validator of the image. The node refuses it on delivery.

The finished extension tests the validator on more configs: interfaces matched by lists, globs and names separated by spaces, a Peer without a Speaker, a Speaker that exports an interface without a Network document and a node without any Peer. It also tests the example config, bgp-examples/fabric.yaml, which the validator accepts. It declares the tests of tests/ through a function of its own, validatorTest, that takes a file's name and the refusals. Cover each branch of a program this way: a config it accepts, and one per refusal.

Add a diagnostic

A node has no logins, so an operator reads bird's state through a diagnostic, a read-only command of the image that chalkd runs on request. Add it to the body:

# chalkctl node diag <node> bird-protocols shows bird's sessions. birdc -r asks bird read-only,
# on bird's socket, which bird makes writable for its group. bird names each session after its
# Peer and interface, such as router_enp1s0, in quotes, as a name with a hyphen needs them.
chalkos.diagnostics.bird-protocols = {
  description = "The protocols bird runs and the state of their BGP sessions.";
  command = [
    "${bird}/bin/birdc"
    "-r"
    "show"
    "protocols"
    "'{protocol}'"
  ];
  parameters.protocol = {
    type = "pattern";
    regex = "^[A-Za-z0-9][A-Za-z0-9._-]*$";
    optional = true;
    description = "Name of a protocol, such as router_enp1s0 for the session of the Peer router on enp1s0, which alone is shown; every protocol when left out.";
  };
  role = "reader";
  timeout = "10s";
  groups = [ "bird" ];
};

birdc -r asks bird in its read-only mode, through bird's socket, which groups makes reachable. protocol takes only names of the form bird.conf gives the sessions, so no value closes bird's quotes. The image's build checks the declaration: a placeholder without a parameter, or a group the image lacks, fails it.

Run every test

The flake exports the images' tests as its checks. Run them, and compare the module with the finished one:

nix flake check
diff -r extensions.done extensions

nix flake check ends with all checks passed!. The diff shows what you left out, such as the comments and the other tests of the validator. Once you are done comparing, remove the copy:

rm -r extensions.done

Try it on a node

Build the image and check node1's files against it, as in Validate a node's files, then move node1 to the image with an upgrade, which carries node1's files:

nix build .#node-bird-metal -o images/node-bird-metal
chalkctl validate
chalkctl upgrade --nodes node1 --secrets=secrets.age

The node checks its config with the new image's validators, renderers and checks before it installs anything, and boots the image with bird in place of the built-in speaker. Change the documents alone, such as a new Peer, with chalkctl config apply, which needs no new image.

Debug on the node

The tests catch what the image makes of a config. What remains are the node's own state and the network: an interface, a password, a neighbour that does not answer. Each part of the extension shows its state in another place:

Part Where it shows
Kinds and rules chalkctl validate, and the refusal of chalkctl config apply --dry-run
Validator the refusal of chalkctl config apply --dry-run
Output chalkctl status, and the journal of the render's unit
bird and bird-credentials chalkctl logs of bird.service
Effect the output of chalkctl config apply, and bird's journal
Sessions and routes chalkctl node diag

A delivery is refused

Ask the node to check a delivery without making it:

chalkctl config apply node1 --secrets=secrets.age --dry-run

It prints the refusal a delivery would get, with the document and field, such as the validator's:

chalkctl: node1: the config is refused: Peer/router: spec.interfaces[2]: enp3s0 has no Network document; bird binds the session to an interface the node configures

Fix the file and check again. To find out why a config is refused, copy the node's documents into a test's config file: the renderers and validators read nothing but the documents, so the test gets the refusal the node gets, and nix build -L of the test results shows what the programs printed.

An output fails

chalkctl status node1 --secrets=secrets.age lists the outputs, bird-conf among them, each with the hash of the files the node serves and its state. failed: names the last failure. The refusal names the output and the exit status alone, as in the tests. The render ran as a transient unit, chalkos-render-<n>.service, which chalkd's journal names beside the failure, and that unit's journal holds what the renderer or the check printed:

chalkctl logs node1 --secrets=secrets.age --unit=chalkd.service
chalkctl logs node1 --secrets=secrets.age --unit=chalkos-render-<n>.service

A node renders every output again at boot, from the config it boots with. A renderer reads its input alone, so the files at boot are those the delivery checked, and a test of the node's documents renders the same bird.conf the node serves.

bird does not run

bird's journal shows its start, each reload and the errors of bird-credentials:

chalkctl logs node1 --secrets=secrets.age --unit=bird.service

bird takes no password with a quote or a line break needs another value in the Secret. bird takes a credential name of letters, digits, '.', '_' and '-' alone needs a Secret name and key of those characters. chalkctl status names a unit that failed as failed unit bird.service. --follow keeps printing new entries while you deliver a change, so a reload shows as it happens.

A session does not come up

The diagnostic shows bird's sessions, each named after its Peer and interface:

$ chalkctl node diag node1 --secrets=secrets.age bird-protocols
...
Name           Proto      Table      State  Since         Info
...
router_enp1s0  BGP        ---        up     10:02:16.731  Established
router_enp2s0  BGP        ---        up     10:02:16.902  Established

A session in Connect or Active reaches no neighbour. protocol=router_enp1s0 shows one session alone. The base diagnostics show the rest of the path: addresses that the ports and node0 hold their addresses, neighbours that the router answers on the port, and routes that the node has a default route with protocol ra:

chalkctl node diag node1 --secrets=secrets.age addresses
chalkctl node diag node1 --secrets=secrets.age neighbours
chalkctl node diag node1 --secrets=secrets.age routes family=inet6

The kernel drops TCP segments whose MD5 signature does not match, so a wrong password logs nothing on either side: the session stays in Connect or Active. Compare the passwords of the two sides. A BGP session does not come up of the home lab lists the other causes. not_found: the image declares no diagnostic "bird-protocols" means the node still runs an image without it.

Check that it worked

nix flake check passes, chalkctl validate prints a line per node, and on the upgraded node chalkctl status lists bird-conf as rendered and bird-protocols shows both sessions Established.

What next

These pages explain the steps of this tutorial in more depth: