Skip to content

Build a home lab

This tutorial builds the home lab of the homelab example on real machines: a router on your home network and three Kubernetes control planes behind it, on an IPv6 fabric of their own. You discover each machine's ports and disks, adapt the example's files to them, install the four machines and bootstrap the cluster. At the end you test two failures: a node keeps its address when one of its cables is pulled, and the Kubernetes API stays up when one control plane goes down. Plan an afternoon; most of it goes into the first image builds and the installs.

An advanced IPv6 and BGP setup

This lab is a maintainer's own home lab. Its BGP fabric, IPv6-only nodes, NAT64 and WireGuard tunnel are there for fun and to try those features out, not because a home network needs them. The guide shares the setup in case it helps you test chalkos on a similar network. For a first cluster on real machines, follow Build a cluster on real machines instead: one network, IPv4 and no router of chalkos.

What you build

The router connects the home network to the fabric, a switch that holds the router's second port and both ports of each node. The workstation reaches the router at its address on the home network and the fabric through a WireGuard tunnel to the router.

flowchart TB
    home["home network, IPv4 by DHCP"]
    ws["workstation"]
    router["router: NAT64, DNS64, router advertisements, BGP, WireGuard"]
    switch["unmanaged switch, the fabric"]
    node1["node1"]
    node2["node2"]
    node3["node3"]
    ws --- home
    home ---|wan| router
    ws -.->|WireGuard| router
    router ---|fabric| switch
    switch ---|enp1s0, enp2s0| node1
    switch ---|enp1s0, enp2s0| node2
    switch ---|enp1s0, enp2s0| node3

Every address comes from one unique local /48, fdc8:a1b2:c3d4::/48 in the example. You replace it with a random /48 of your own in Prepare the files.

Use Prefix or address
The fabric, which the router advertises fdc8:a1b2:c3d4:1::/64
The router on the fabric: resolver and the BGP neighbour of each node's enp1s0 fdc8:a1b2:c3d4:1::1
The router's second address on the fabric: the BGP neighbour of each node's enp2s0 fdc8:a1b2:c3d4:1::2
node1, node2 and node3, each on its dummy interface node0 fdc8:a1b2:c3d4:ff::11, ::12 and ::13
The Kubernetes API's VIP fdc8:a1b2:c3d4:ff::10
The WireGuard tunnel: the router and the workstation fdc8:a1b2:c3d4:f0::1 and ::2, in fdc8:a1b2:c3d4:f0::/64
NAT64 prefix 64:ff9b::/96
AS numbers 65000 for the router, 65001 for every node
The router on the home network an IPv4 address by DHCP, 192.168.1.2 in the example

The fabric carries IPv6 alone. The router's advertisements give every port on the fabric an address by SLAAC, a default gateway, the resolver and the NAT64 prefix. The resolver answers a name that has only IPv4 addresses with addresses in 64:ff9b::/96, and the NAT64 carries connections to them to the IPv4 internet, out of the router's port on the home network. A node's own address is a /128 on its dummy interface node0. FRR, the BGP speaker of the nodes' image, announces it to the router over a session on each port, each to one of the router's two addresses on the fabric, and the control plane that holds the API's VIP announces the VIP the same way. The router then has two equal paths to each address. When a cable or port fails, BFD ends that port's session within about a second, and the router forwards over the other port. In the other direction, networkd uses the default gateway from the surviving port's advertisements when a port loses carrier. The installer and installed nodes can reach the workstation and IPv4 internet before BGP comes up. Traffic between nodes' routed addresses passes through the router, so the cluster still depends on the router.

Before you begin

You need:

  • four amd64 machines with UEFI firmware, a TPM 2.0 and two Ethernet ports each, and a disk of at least 16 GiB for the router and a larger one for each node, which holds the containers' images and etcd; the requirements list the rest;
  • an unmanaged switch with at least seven ports, and Ethernet cables;
  • a USB stick of at least 1 GiB;
  • your home network, whose router hands out IPv4 addresses by DHCP and lets you reserve an address for a MAC address;
  • a workstation on the home network, an x86-64 Linux machine with Nix and flakes enabled, git, and WireGuard's tools wg and wg-quick from your distribution;
  • an age key or an SSH key of type ed25519 or RSA to encrypt the cluster's secrets to.

Secure Boot is optional

The tutorial installs the machines without Secure Boot, with the unsigned images and installer. Turn Secure Boot off in each machine's firmware setup. The steps marked optional add Secure Boot with keys of your own: Sign the images and Enrol the Secure Boot keys. A machine whose firmware is in Setup Mode gets the keys from chalkctl. For any other machine, enrol the db certificate in its firmware as Enrol the certificate without Setup Mode describes before you install it.

Without Secure Boot, the TPM still unlocks the disks at every boot. It seals their keys to PCR 7, which then measures Secure Boot as off, so a machine that boots another system from a USB stick unlocks them too. The encryption protects a disk taken out of its machine, and no longer the machine itself. Security model describes what else Secure Boot protects.

Decide before you install a machine. Turning Secure Boot on later changes PCR 7: the machine then asks for its recovery key at every boot until you install it again.

Prepare the files

The example is part of the chalkos repository. Copy it into a repository of your own, and put it under git, because Nix reads a flake from the files git tracks:

git clone https://github.com/trevex/chalkos
cp -r chalkos/examples/homelab homelab
cd homelab
git init && git add .

Run every later command in homelab. The example's files explain themselves in comments:

File Holds
flake.nix Builds router-metal, node-metal and the installer.
chalkos.yaml The inventory: the four nodes, the address chalkctl reaches each at, their images and files.
router/router.yaml The router's ports, firewall, resolver, BGP and WireGuard.
router/nat64.yaml The router's NAT64, the fabric's way to the IPv4 internet.
router/secrets.yaml The router's WireGuard key and BGP password.
cluster/cluster.yaml The Kubernetes cluster that all three nodes share.
bgp/fabric.yaml, bgp/secrets.yaml The nodes' ports, their BGP sessions with the router, and the password.
nodes/node1.yaml to nodes/node3.yaml Each node's disk, address and BGP speaker.
chalkos-air-gapped.yaml, air-gapped/ The lab without internet access, an optional variant near the end of this tutorial.

Get the tools

The flake's shell holds chalkctl and chalkbuild of the chalkos revision that flake.lock pins, with kubectl, jq, openssl, wg, arp-scan, nc, crane and yq. Lock the flake's chalkos input and enter the shell:

nix flake lock
nix develop

Run every later command of this tutorial in that shell, so that chalkctl and the images come from the same chalkos release.

The diagnostic addresses, which shows a machine's ports, prints JSON, as ip -json address show does. Keep a jq filter in the shell that turns it into a line per port, with the port's name, MAC address, state and addresses beyond the link:

ports='.[] | [.ifname, .address, .operstate,
  ([.addr_info[] | select(.scope == "global") | .local] | join(" "))] | @tsv'

Generate the cluster's secrets

The secrets file holds the cluster's CAs and keys. Generate it encrypted to your key, and to a backup key that you keep offline:

$ chalkctl gen secrets --recipient "$(cat ~/.ssh/id_ed25519.pub)" \
    --recipient age1nmrqj76cqyf4ft8xtczufd3mtdk0xf2j7tevgwnzjp3zq9tffu3slx5eux
encrypted secrets.age to ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKMcR1DCFPGSdWXphQ3We8UmkMp2KdufUzVxxC6Hvo9k alice@workstation, age1nmrqj76cqyf4ft8xtczufd3mtdk0xf2j7tevgwnzjp3zq9tffu3slx5eux
wrote secrets.age and secrets.pub.json

The age1 recipient here stands for your offline backup key's public key. Then give the images and the installer the cluster's OS CA, so that a machine waiting for its install accepts only a chalkctl that holds this cluster's secrets. Add the line to the arguments of chalkos.lib.mkImages in flake.nix:

osCA = ./secrets.pub.json;

Without it, any machine on your home network could install the router while it waits in the installer.

Choose your own prefix

Generate the first 48 bits of a unique local prefix, and replace the example's in every file:

set -- $(od -An -N5 -tu1 /dev/urandom)
ula=$(printf 'fd%02x:%x:%x' "$1" $(($2 * 256 + $3)) $(($4 * 256 + $5)))
echo "$ula"
grep -rlF fdc8:a1b2:c3d4 --exclude-dir=.git . | xargs sed -i "s/fdc8:a1b2:c3d4/$ula/g"

The command prints a prefix such as fd3b:99df:9be2. The rest of this tutorial shows the example's fdc8:a1b2:c3d4; read yours in its place.

Make the WireGuard keys and the BGP password

Make a key pair for the router and one for the workstation. The example's .gitignore keeps the private keys, *.key, out of git, as it keeps the images, the installer and the kubeconfig:

(umask 077 && wg genkey | tee router.key | wg pubkey > router.pub)
(umask 077 && wg genkey | tee workstation.key | wg pubkey > workstation.pub)

Replace the example's keys: the router's private key goes into router/secrets.yaml, and the workstation's public key into the WireGuardPeer of router/router.yaml:

sed -i "s|uBjWXU2Bu/LvLVRo+dtAte3VHt75d9W7DPxsEhZrm24=|$(cat router.key)|" router/secrets.yaml
sed -i "s|cK0R7bvqvEfk07xvf9i71Jj0ATJEGaKZ3gs1zBXr1m0=|$(cat workstation.pub)|" router/router.yaml

The router and the nodes share one BGP password, in router/secrets.yaml and bgp/secrets.yaml. Replace the example's with a random one in both:

sed -i "s/example-password/$(openssl rand -hex 16)/" router/secrets.yaml bgp/secrets.yaml

The two files now hold real secrets in plain text. Keep the repository private, or keep the two files out of it.

Build the images and the installer

Build the two images the inventory names where it expects them, and the installer:

git add .
nix build .#router-metal -o images/router-metal
nix build .#node-metal -o images/node-metal
nix build .#installer -o installer

The first build takes a while. The router's image carries the router layer and no Kubernetes; the nodes' image carries Kubernetes and its BGP speaker, FRR.

Sign the images (optional)

Skip this step without Secure Boot. With it, create your Secure Boot keys in keys/, and sign copies of the three builds with the db key:

chalkctl secureboot new keys --name homelab
chalkbuild sign --key keys/db.key --cert keys/db.crt images/router-metal images/router-metal-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt images/node-metal images/node-metal-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt installer installer-signed

The example's .gitignore keeps the .key files out of git. Keep a backup of keys/ offline: whoever holds its keys decides what the machines boot.

Point the inventory's images at the signed copies. chalkos.yaml then reads:

chalkos.yaml
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: homelab}
spec:
  images:
    # chalkbuild sign writes the signed copies of the builds here.
    router-metal: {source: {directory: images/router-metal-signed}}
    node-metal: {source: {directory: images/node-metal-signed}}
  groups:
    router: {imageName: router-metal, files: [router/*.yaml]}
    nodes: {imageName: node-metal, files: [cluster/*.yaml, bgp/fabric.yaml, bgp/secrets.yaml]}
  nodes:
    router: {address: 192.168.1.2, groups: [router]}
    node1: {address: "fdc8:a1b2:c3d4:ff::11", groups: [nodes], files: [nodes/node1.yaml]}
    node2: {address: "fdc8:a1b2:c3d4:ff::12", groups: [nodes], files: [nodes/node2.yaml]}
    node3: {address: "fdc8:a1b2:c3d4:ff::13", groups: [nodes], files: [nodes/node3.yaml]}

Write the installer to a USB stick

Write the installer's ISO to the USB stick:

sudo dd if=installer/chalkos-installer.iso of=/dev/<usb-disk> \
  bs=4M conv=fsync status=progress

With Secure Boot, write the signed copy, installer-signed/chalkos-installer.iso, instead. <usb-disk> is the stick's device, such as sdb; lsblk lists the candidates. The command overwrites everything on the stick. The installer holds no config of a node: it takes addresses on every Ethernet port, by DHCP and from router advertisements, and keeps the kernel's names for the ports.

Install the router

The router goes first, because the nodes' installers take their addresses from its advertisements on the fabric.

Boot the installer

Cable one of the router's ports to the home network and the other to the switch. Which port is which does not matter yet: the next step finds out. Boot the router from the USB stick through the firmware's boot menu. Once chalkd runs, the screen shows the console. Its header keeps chalkd's fingerprint on screen, and the address that the home network's DHCP server gave the port it reaches:

nixos  installer  up 1m
image 81bd81bd81bd
installer: chalkctl install installs nodes from here
fingerprint <fingerprint>
addresses 192.168.1.57

Note the address and the fingerprint, 64 hex characters. chalkctl reaches a node that a command names at the node's address in chalkos.yaml, so put the installer's address there as the router's for now, and the commands below need no --address:

chalkos.yaml
    router: {address: 192.168.1.57, groups: [router]}

The commands check the machine's certificate with --fingerprint.

Find the installer without a screen

On a machine without a screen, look for the installer on the home network: chalkd listens on TCP port 50000. The flake's shell holds arp-scan and nc; this lists the home network's machines that answer on that port:

sudo "$(command -v arp-scan)" --localnet --plain --quiet | awk '{print $1}' \
  | xargs -I {} nc -zv -w 1 {} 50000 2>&1 | grep -E "succeeded|open"

--insecure in place of --fingerprint

When reading the console is impractical, pass --insecure in place of --fingerprint. chalkctl then accepts any certificate and prints the fingerprint it saw, as chalkctl: the node's certificate fingerprint is <fingerprint>. Compare it with the console when you can. chalkctl install sends the secrets only over a connection pinned to that fingerprint.

Enrol the Secure Boot keys (optional)

Skip this step without Secure Boot, or when the router's firmware holds your db certificate already. When its firmware is in Setup Mode, as after you clear its keys in the firmware setup, it boots the installer without checking it. Enrol your keys; secureboot enroll names no node, so it takes the address with --address:

$ chalkctl secureboot enroll --keys keys --address=192.168.1.57 --fingerprint=<fingerprint> \
    --secrets=secrets.age
...
the node reboots with Secure Boot enforced
chalkctl: the node's certificate fingerprint is now <new fingerprint>; compare it with the node's console and install with --fingerprint <new fingerprint>
Secure Boot is on; install the node with chalkctl install

The router reboots into the installer, now with Secure Boot on; pick the USB stick in the boot menu if it boots something else. chalkd makes a new certificate at that boot: use the new fingerprint from here on.

When the router's network cards carry option ROMs, the node refuses the keys and names them: add --option-roms to enrol their digests.

Find the disk and the ports

List the disks:

$ chalkctl disks router --fingerprint=<fingerprint> --secrets=secrets.age
DEVICE            SIZE    TYPE  MODEL                    SERIAL          WWN                   USE
/dev/nvme0n1      476.9G  nvme  Samsung SSD 980 512GB    S64DNF0R812345  eui.002538b811b2c3d4
...

The list goes on with the installer's own stick, whose use is boot. Pick the disk for the system, and find its path below /dev/disk/by-id, which stays the same across boots. For an NVMe disk it is nvme-<model>_<serial>, and for a SATA disk ata-<model>_<serial>, with each space of the model as an underscore: /dev/disk/by-id/nvme-Samsung_SSD_980_512GB_S64DNF0R812345 here. When the path is in doubt, select the disk by its serial number instead, with disk: {serial: S64DNF0R812345} in place of path. The installer refuses a disk that holds data.

The diagnostic addresses lists the ports, and the jq filter of "Get the tools" puts each on a line:

$ chalkctl node diag router --fingerprint=<fingerprint> --secrets=secrets.age addresses \
  | jq -r "$ports"
lo  00:00:00:00:00:00   UNKNOWN
enp1s0  02:00:00:00:01:01   UP  192.168.1.57
enp2s0  02:00:00:00:01:02   UP

The port with the home network's address is the router's wan, and the other one, on the switch, is its fabric. A port that shows DOWN has no carrier: check its cable.

Adapt the router's files

router/router.yaml names the ports by their permanent MAC addresses in two Link documents, which rename them to wan and fabric at the installed router's first boot. Every other document names the ports by those names. The addresses diagnostic shows each port's current MAC address, which on a plain port, without a bond or a VLAN, is its permanent one. Put the two into the Link documents, and your disk into the SystemDisk:

router/router.yaml
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Link
metadata: {name: 10-wan}
spec:
  match: {permanentMACAddress: "02:00:00:00:01:01"}
  link: {name: wan}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Link
metadata: {name: 10-fabric}
spec:
  match: {permanentMACAddress: "02:00:00:00:01:02"}
  link: {name: fabric}
---
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
  disk: {path: /dev/disk/by-id/nvme-Samsung_SSD_980_512GB_S64DNF0R812345}

Reserve an IPv4 address for the wan port's MAC address on your home network's DHCP server. The installed router takes it on wan, and chalkctl reaches it there once you put it into chalkos.yaml after the install. The example reserves 192.168.1.2.

Check the files against the images:

chalkctl validate

It prints a line per node, and names the document and field of each mistake.

Install

Install the router at the installer's address, which chalkos.yaml names for now:

$ chalkctl install router --secrets=secrets.age --fingerprint=<fingerprint>
installing router onto /dev/disk/by-id/nvme-Samsung_SSD_980_512GB_S64DNF0R812345 from /home/alice/homelab/images/router-metal: chalkos <image>, sending <n> bytes of store, hash tree, UKI and boot loader
router is installed and reboots
chalkctl recovery-key router prints the key that unlocks it when its TPM fails

<image> is the start of the image's root hash, and <n> the size of what chalkctl sends. The router reboots into its image, renames its ports and takes the reserved address on wan. Remove the USB stick once the console's header shows normal, and put the reserved address back into chalkos.yaml:

chalkos.yaml
    router: {address: 192.168.1.2, groups: [router]}

Wipe a disk that holds data

The installer refuses a disk that holds data, such as a disk from another machine or an earlier install of another image, and names what it found:

install: the target disk /dev/nvme0n1 (model "Samsung SSD 980 512GB", size 476.9G, serial "S64DNF0R812345") carries data (gpt ext4); pass --wipe-disk to replace it

Check in chalkctl disks that the disk is the one you mean, and install again with --wipe-disk. The installer then clears the disk's partition tables and lays it out anew. Nothing on the disk can be recovered afterwards:

chalkctl install router --secrets=secrets.age --fingerprint=<fingerprint> --wipe-disk

Check the router

chalkctl status reaches the router at its address in the inventory:

$ chalkctl status router --secrets=secrets.age
...
bgp: no sessions
nat64:
  INSTANCE  SESSIONS  BINDINGS  POOL TCP  POOL UDP  POOL ICMP
  nat64     0         0         4535      4535      4535
resolver: running
...

The router runs its config, its NAT64 and its resolver. It has no BGP session yet, because no node runs. The addresses diagnostic shows the renamed ports, wan with the reserved address and fabric with its two addresses, fdc8:a1b2:c3d4:1::1 and ::2:

$ chalkctl node diag router --secrets=secrets.age addresses | jq -r "$ports"
lo  00:00:00:00:00:00   UNKNOWN
wan 02:00:00:00:01:01   UP  192.168.1.2
fabric  02:00:00:00:01:02   UP  fdc8:a1b2:c3d4:1::2 fdc8:a1b2:c3d4:1::1
wg0     UNKNOWN fdc8:a1b2:c3d4:f0::1
...

Connect the workstation

The workstation reaches the fabric through the router's WireGuard interface wg0. Its config for wg-quick holds the workstation's private key, so create the file readable by root alone first:

sudo install -m 600 /dev/null /etc/wireguard/homelab.conf

Then write the config into it, with the workstation's private key and the router's public key:

/etc/wireguard/homelab.conf
[Interface]
PrivateKey = <contents of workstation.key>
Address = fdc8:a1b2:c3d4:f0::2/64

[Peer]
PublicKey = <contents of router.pub>
Endpoint = 192.168.1.2:51820
AllowedIPs = fdc8:a1b2:c3d4::/48
PersistentKeepalive = 25

AllowedIPs sends the whole /48 through the tunnel, and nothing else: the workstation reaches the internet as before. Keep PersistentKeepalive: every config delivery to the router sets its WireGuard peers anew, and the router reaches the workstation again only once the workstation sends a packet. Bring the tunnel up and ping the router's fabric address:

$ sudo wg-quick up homelab
$ ping -c 3 fdc8:a1b2:c3d4:1::1
PING fdc8:a1b2:c3d4:1::1 (fdc8:a1b2:c3d4:1::1) 56 data bytes
64 bytes from fdc8:a1b2:c3d4:1::1: icmp_seq=1 ttl=64 time=0.612 ms
...

The diagnostic wireguard shows the router's side of the tunnel, with the workstation's latest handshake. The public keys are the example's; yours are those of router.pub and workstation.pub:

$ chalkctl node diag router --secrets=secrets.age wireguard interface=wg0
interface: wg0
  public key: N1efwZV4rLKHLJQIeOXqOmCOlJVUhEnnXMOqRPH2L0k=
  private key: (hidden)
  listening port: 51820

peer: cK0R7bvqvEfk07xvf9i71Jj0ATJEGaKZ3gs1zBXr1m0=
  endpoint: 192.168.1.20:41641
  allowed ips: fdc8:a1b2:c3d4:f0::2/128
  latest handshake: 9 seconds ago
  transfer: 1.02 KiB received, 892 B sent

Install the nodes

Install node1, node2 and node3 one after another, with the same steps.

Boot the installer

Cable both of the node's ports to the switch and boot it from the USB stick. The router's advertisements give each port an address of the fabric's prefix, and the console's header shows both:

nixos  installer  up 1m
image 81bd81bd81bd
installer: chalkctl install installs nodes from here
fingerprint <fingerprint>
addresses fdc8:a1b2:c3d4:1:0:ff:fe00:1101 fdc8:a1b2:c3d4:1:0:ff:fe00:1102

The header shows no link-local address. A port without an address of the fabric has no cable, or the router sends no advertisements: the section "Troubleshooting" covers both. Note one of the addresses and the fingerprint. Put the address into chalkos.yaml as node1's for now, quoted, as YAML needs for an IPv6 address; the workstation reaches it over the tunnel:

chalkos.yaml
    node1: {address: "fdc8:a1b2:c3d4:1:0:ff:fe00:1101", groups: [nodes], files: [nodes/node1.yaml]}

Find the installer through the router

When the node has no screen, find its ports in the router's neighbour table. Each of the installer's ports takes an address of the fabric's prefix with the same interface ID as its link-local address, so this lists every machine the router sees on fabric with its MAC address and its address of the fabric:

chalkctl node diag router --secrets=secrets.age neighbours \
  | jq -r '.[] | select(.dev == "fabric" and (.dst | startswith("fe80::")))
      | [.lladdr, (.dst | sub("^fe80::"; "fdc8:a1b2:c3d4:1:"))] | @tsv'

The output has a line for each port, here the two of node1:

02:00:00:00:11:01   fdc8:a1b2:c3d4:1:0:ff:fe00:1101
02:00:00:00:11:02   fdc8:a1b2:c3d4:1:0:ff:fe00:1102

The table also holds the ports of nodes installed before, whose MAC addresses the addresses diagnostic shows on each of them. A port the router has not talked to yet is missing from the table: boot one node at a time, and look again after a minute. Check that chalkd answers at an address before you use it:

chalkctl node diag router --secrets=secrets.age tcp-connect \
  address=fdc8:a1b2:c3d4:1:0:ff:fe00:1101 port=50000

Without the console you have no fingerprint to compare: reach the installer with --insecure in place of --fingerprint, as the note under the router's "Boot the installer" describes.

With Secure Boot, and when the node's firmware is in Setup Mode, enrol your keys at that address as for the router, in "Enrol the Secure Boot keys", and note the new fingerprint.

Find the disk and the ports

List the disks and the ports:

$ chalkctl disks node1 --fingerprint=<fingerprint> --secrets=secrets.age
DEVICE            SIZE    TYPE  MODEL                    SERIAL          WWN                   USE
/dev/nvme0n1      931.5G  nvme  WD Blue SN580 1TB        24131D801234    eui.e8238fa6bf530001
...
$ chalkctl node diag node1 --fingerprint=<fingerprint> --secrets=secrets.age addresses \
  | jq -r "$ports"
lo  00:00:00:00:00:00   UNKNOWN
enp1s0  02:00:00:00:11:01   UP  fdc8:a1b2:c3d4:1:0:ff:fe00:1101
enp2s0  02:00:00:00:11:02   UP  fdc8:a1b2:c3d4:1:0:ff:fe00:1102

The disk's path is /dev/disk/by-id/nvme-WD_Blue_SN580_1TB_24131D801234 here. The two ports are enp1s0 and enp2s0, the names the kernel gives them. The names follow from the ports' PCI slots, so every machine of the same model shows the same two.

Adapt the node's file

bgp/fabric.yaml, which every node gets, configures the ports: a Network document for each port, which takes the default gateway and other settings from the router's advertisements, and a BGPPeer for each port, which advertises node and VIP addresses to the router. It names the ports enp1s0 and enp2s0. When the addresses diagnostic shows other names, replace enp1s0 and enp2s0 throughout bgp/fabric.yaml with them. You do this once, for node1: node2 and node3 have the same names when they are the same model. A node whose ports have other names, such as a machine of another model, takes Network and BGPPeer documents of its own in place of bgp/fabric.yaml, as the file's comment describes.

nodes/node1.yaml holds everything that differs between the nodes:

nodes/node1.yaml
# node1: a control plane whose two ports are both on the fabric's switch. Its address sits on the
# dummy interface node0, and FRR announces it over a session on each port, so the address stays
# reachable while one port or cable works.
#
# The ports' Network documents and the BGPPeers lie in bgp/fabric.yaml, under the names the kernel
# gives the ports, which identical machines share. A node whose ports have other names leaves
# bgp/fabric.yaml out and brings its own Network and BGPPeer documents.
#
# Replace the disk with the node's own, by its path below /dev/disk/by-id.
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
  disk: {path: /dev/disk/by-id/nvme-EXAMPLE_SSD_node1}
---
# node0 holds the node's address, and the API's VIP while the node holds it. chalkd adds the VIP
# with a preferred lifetime of 0, which networkd counts as a dynamic address, so keepConfiguration
# yes has networkd keep every address it did not configure itself when it configures the link again.
apiVersion: networkd.chalkos.dev/v1alpha1
kind: NetDev
metadata: {name: 30-node0}
spec:
  netDev: {name: node0, kind: dummy}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 30-node0}
spec:
  match: {name: node0}
  network: {address: ["fdc8:a1b2:c3d4:ff::11/128"], keepConfiguration: "yes"}
---
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
  type: controlplane
  nodeIP: {addresses: ["fdc8:a1b2:c3d4:ff::11"]}
  vip: {interface: node0}
---
# The node's side of its BGP sessions. The nodes share an AS number, and each has a router ID of
# its own. FRR announces the addresses of node0: the node's own and, while the node holds it, the
# API's VIP.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: BGPSpeaker
metadata: {name: bgp}
spec:
  asn: 65001
  routerID: 0.0.0.11
  export: {interfaces: [node0]}

Put your disk's path into the SystemDisk.

The rest of the file stays as it is. node0 is a dummy interface that holds the node's address, fdc8:a1b2:c3d4:ff::11, which no cable can take away. The Node document makes the node a control plane, gives Kubernetes that address as the node's address, and names node0 as the interface the node adds the API's VIP to while it holds it. The BGPSpeaker gives FRR the node's AS number and router ID and announces the addresses of node0. node2 and node3 differ in their address, ::12 and ::13, and their router ID, 0.0.0.12 and 0.0.0.13. Each node needs an address and a router ID of its own, and the address in its Network and Node documents is the one chalkos.yaml names for it.

Install

Validate the files and install the node at the installer's address, which chalkos.yaml names for now. Add --wipe-disk when the installer refuses the disk, as for the router:

chalkctl validate
chalkctl install node1 --secrets=secrets.age --fingerprint=<fingerprint>

chalkctl prints the lines that the router's install printed. The node reboots into its image, brings up node0 and starts FRR, whose sessions announce the node's address to the router. Put the node's own address back into chalkos.yaml, where chalkctl reaches it from now on:

chalkos.yaml
    node1: {address: "fdc8:a1b2:c3d4:ff::11", groups: [nodes], files: [nodes/node1.yaml]}

Until the bootstrap, its status shows that it waits:

$ chalkctl status node1 --secrets=secrets.age
...
kubernetes controlplane: waiting for bootstrap or for the cluster at https://[fdc8:a1b2:c3d4:ff::10]:6443
...

Repeat the steps for node2 and node3.

Bootstrap Kubernetes

Once the three nodes wait, bootstrap one of them and write a kubeconfig:

$ chalkctl bootstrap node1 --secrets=secrets.age
bootstrapping node1; the control plane pulls its images and starts
node1 is bootstrapped; applied 19 objects
$ chalkctl kubeconfig --secrets=secrets.age
wrote kubeconfig; its certificate is valid for 8760h0m0s

The bootstrap starts etcd and the control plane on node1. node1 then wins the election for the VIP, adds fdc8:a1b2:c3d4:ff::10 to node0, and FRR announces it. node2 and node3 find the cluster at the endpoint and join etcd one after the other. Pulling the images through the NAT64 takes most of the bootstrap's time.

List the nodes through the VIP:

$ export KUBECONFIG=$PWD/kubeconfig
$ kubectl get nodes
NAME    STATUS   ROLES    AGE   VERSION
node1   Ready    <none>   4m    v1.37.2
node2   Ready    <none>   2m    v1.37.2
node3   Ready    <none>   1m    v1.37.2

A node may show NotReady for the first minute, until flannel runs on it. kubectl get nodes -o wide shows each node's address, its /128 on node0, as INTERNAL-IP. The three nodes are control planes and run workloads too.

The nodes reach the IPv4 internet through the router. The diagnostic dns-query asks the node's resolver, the router's, for the name ipv4only.arpa, which has IPv4 addresses alone; DNS64 answers with addresses in the NAT64's prefix:

$ chalkctl node diag node1 --secrets=secrets.age dns-query name=ipv4only.arpa type=AAAA
...
;; ANSWER SECTION:
ipv4only.arpa.  3600    IN  AAAA    64:ff9b::c000:ab
ipv4only.arpa.  3600    IN  AAAA    64:ff9b::c000:aa
...
$ chalkctl node diag node1 --secrets=secrets.age tcp-connect address=64:ff9b::101:101 port=443
connected to 64:ff9b::101:101 port 443

64:ff9b::101:101 is the IPv4 address 1.1.1.1 in the NAT64's prefix. Pods resolve names through CoreDNS, which asks the same resolver, so a pod reaches an IPv4-only host by its name:

kubectl run nat64-check --rm -it --restart=Never --image=busybox:1.37 -- nslookup github.com

The answer holds an address in 64:ff9b::/96 beside the host's IPv4 address.

Check BGP and failover

The sessions

Each node runs a session on each port, to the router's address that the port's BGPPeer names. The diagnostic bgp-summary of the nodes' image shows them as FRR's show bgp summary prints them, a line for each of the router's addresses, fdc8:a1b2:c3d4:1::1 and fdc8:a1b2:c3d4:1::2. A session that is up shows the number of prefixes received where a session that is down shows its state. The IPv4 section shows both sessions as NoNeg, because the router's peer nodes carries IPv6 alone. bfd-peers shows the BFD session that watches each:

chalkctl node diag node1 --secrets=secrets.age bgp-summary
chalkctl node diag node1 --secrets=secrets.age bfd-peers

The router accepts a session from every address of the fabric, and chalkctl status lists the six under its peer nodes, each with the address of a node's port:

$ chalkctl status router --secrets=secrets.age
...
bgp:
  VRF      PEER   NEIGHBOR                         STATE        UPTIME  RECEIVED  SENT  BFD
  default  nodes  fdc8:a1b2:c3d4:1:0:ff:fe00:1101  Established  12m4s   2         0     up
  default  nodes  fdc8:a1b2:c3d4:1:0:ff:fe00:1102  Established  12m3s   2         0     up
  default  nodes  fdc8:a1b2:c3d4:1:0:ff:fe00:1201  Established  9m40s   1         0     up
  default  nodes  fdc8:a1b2:c3d4:1:0:ff:fe00:1202  Established  9m40s   1         0     up
  default  nodes  fdc8:a1b2:c3d4:1:0:ff:fe00:1301  Established  7m12s   1         0     up
  default  nodes  fdc8:a1b2:c3d4:1:0:ff:fe00:1302  Established  7m11s   1         0     up
...

Each node sends its address on each session, and the holder of the VIP the VIP as well, so node1 shows 2 routes received. The router sends zero BGP routes. The routes to the nodes' addresses have two next hops each, one per port:

$ chalkctl node routes router fdc8:a1b2:c3d4:ff::/64 --secrets=secrets.age
DESTINATION                TYPE     PROTOCOL  TABLE  METRIC  NEXT HOPS
fdc8:a1b2:c3d4:ff::10/128  unicast  bgp       254    20      fe80::ff:fe00:1101 on fabric (weight 1), fe80::ff:fe00:1102 on fabric (weight 1)
fdc8:a1b2:c3d4:ff::11/128  unicast  bgp       254    20      fe80::ff:fe00:1101 on fabric (weight 1), fe80::ff:fe00:1102 on fabric (weight 1)
...

Pull a cable

Start a ping of node1's address from the workstation, and pull one of node1's cables:

ping fdc8:a1b2:c3d4:ff::11

Within about a second, BFD ends the session on that port: bgp-summary on node1 shows a state in place of that session's count of received prefixes, the router's bgp: table loses the port's session, and the route to node1 keeps one next hop. The ping goes on, losing a packet or two at most. Plug the cable back in, and the session comes back within seconds.

Take the VIP's holder down

chalkctl status shows which control plane holds the VIP:

$ chalkctl status node1 --secrets=secrets.age
...
kubernetes controlplane: bootstrapped, node ready: True, vip holder, control plane current
...

The other two show vip standby. Reboot the holder:

chalkctl node reboot node1 --secrets=secrets.age

chalkctl drains node1 and keeps etcd's quorum while it reboots. Another control plane wins the election, adds the VIP to its node0, and its FRR announces it. kubectl get nodes answers again once the router has the new route, within about 30 seconds. For a harder test, cut the holder's power instead: its sessions end within BFD's second, and the VIP moves when its lease runs out. Power it on again afterwards; the VIP stays with its new holder.

Change things later

To change the lab, change its files, validate them, and deliver them to the nodes whose config changes:

chalkctl validate
chalkctl apply node1 node2 node3 --secrets=secrets.age

Change a node's config covers previews, trials and rollbacks. Deliver a change of the router's management interfaces or firewall with --try, so that the router rolls it back when chalkctl loses it. A delivery renames a router port at once when its Link document gives it a new name. Change every document that names the port in the same delivery, its Network, firewall zone and management interfaces, and deliver it with chalkctl config apply router --try, so that the router rolls it back when chalkctl loses it.

A new chalkos release or a change to the images' modules in flake.nix takes new images. For a new release, change the tag in the chalkos input's URL and update the flake's lock with nix flake update chalkos. Enter the shell of Get the tools again, so that chalkctl matches the images. Build the images again, as in Build the images and the installer. With Secure Boot, remove the signed copies of the old builds, which chalkbuild sign does not overwrite, and sign the new ones:

rm -rf images/router-metal-signed images/node-metal-signed installer-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt images/router-metal images/router-metal-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt images/node-metal images/node-metal-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt installer installer-signed

Then roll the images out to the nodes with chalkctl upgrade, as Upgrade nodes describes. chalkctl upgrades the control planes one at a time, so the Kubernetes API stays up. While the router reboots, traffic between the nodes' routed addresses, to the Kubernetes API, and to the internet is interrupted. Upgrade the router in a step of its own, at a time when that pause suits you:

chalkctl upgrade --nodes node1,node2,node3 --secrets=secrets.age
chalkctl upgrade --nodes router --secrets=secrets.age

Run the lab without internet access (optional)

This optional variant cuts the fabric off the internet. The router drops its NAT64, the control planes serve the container images from the cluster registry, and the router serves the nodes' time. The workstation keeps its tunnel and reaches every node as before. Follow it in place of Bootstrap Kubernetes once the three nodes wait for their bootstrap, and run its chalkctl bootstrap and chalkctl kubeconfig commands; its NAT64 checks fail in the variant. On a running lab, the bootstrap step drops out. Build an air-gapped cluster explains the approach for any cluster.

What connects the fabric to the internet

The NAT64 is the fabric's only way out for connections; DNS queries still leave through the router's resolver, so the variant is no strict air gap. The fabric holds unique local addresses alone, and the router takes no IPv6 route from the home network, so the router has no IPv6 route beyond the lab's /48. Its firewall policies forward between the fabric and the tunnel alone. The NAT64's connections pass ahead of the policies, out of wan from the router's IPv4 address. The router sends the nodes no BGP routes: their default route from its advertisements leads to the router and no further.

Without the NAT64, the router has no route to 64:ff9b::/96, so the nodes' connections there fail. The resolver stays: it answers names from the root servers over wan, as the router's own connections still leave there, and DNS64 still answers with addresses in 64:ff9b::/96, which lead nowhere now. CoreDNS keeps a server to forward to. The nodes reach the cluster registry, the router's time and each other by their addresses, and need no names. In a real air gap the router has no uplink either, and its resolver answers no name outside the lab. Its chrony then has no server to follow and answers the nodes as unsynchronised, at stratum 16, which they refuse. Give the router a reference clock, or add localStratum: 10 to its TimeServer, which serves the router's own clock.

Two things the nodes take from the internet stop with the NAT64:

What Through the NAT64 In the variant
Container images registry.k8s.io, ghcr.io and docker.io the cluster registry on the three control planes
Time ptbtime1.ptb.de to ptbtime3.ptb.de the router's chrony at fdc8:a1b2:c3d4:1::1, over plain NTP

The images and the installer need no change: chalkctl reads them from images/ on the workstation and sends them to the nodes.

Look at the variant's files

The example keeps the variant beside the lab's files, so the default inventory stays as it is. chalkos-air-gapped.yaml is the variant's inventory. Its router leaves out router/nat64.yaml and takes air-gapped/router.yaml, and its nodes take air-gapped/nodes.yaml as well:

chalkos-air-gapped.yaml
# The home lab without internet access, the optional variant of the guide: the inventory of
# chalkos.yaml, whose router leaves out router/nat64.yaml and whose nodes take the cluster
# registry and the router's time from air-gapped/. chalkctl reads it with
# --inventory chalkos-air-gapped.yaml or CHALKOSINVENTORY=chalkos-air-gapped.yaml, and
# chalkos.yaml's config connects the lab again. Keep the nodes' addresses and the image sources the
# same as there, such as the signed copies, and list every file of router/ but nat64.yaml here too.
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: homelab}
spec:
  images:
    router-metal: {source: {directory: images/router-metal}}
    node-metal: {source: {directory: images/node-metal}}
  groups:
    router:
      imageName: router-metal
      files: [router/router.yaml, router/secrets.yaml, air-gapped/router.yaml]
    nodes:
      imageName: node-metal
      files: [cluster/*.yaml, bgp/fabric.yaml, bgp/secrets.yaml, air-gapped/nodes.yaml]
  nodes:
    router: {address: 192.168.1.2, groups: [router]}
    node1: {address: "fdc8:a1b2:c3d4:ff::11", groups: [nodes], files: [nodes/node1.yaml]}
    node2: {address: "fdc8:a1b2:c3d4:ff::12", groups: [nodes], files: [nodes/node2.yaml]}
    node3: {address: "fdc8:a1b2:c3d4:ff::13", groups: [nodes], files: [nodes/node3.yaml]}

Keep its nodes' addresses and its image sources the same as those of chalkos.yaml, such as the signed copies of Sign the images. air-gapped/nodes.yaml turns the cluster registry on, with the three control planes' addresses as its endpoints and the hosts of the lab's images as its mirrors, and names the router as the nodes' time server:

air-gapped/nodes.yaml
# What the nodes of the air-gapped lab, chalkos-air-gapped.yaml, get beside the files of
# chalkos.yaml: the cluster registry, from which they pull every container image, and the router
# as their time server.
#
# The control planes serve the cluster registry at their node addresses, and containerd on every
# node looks there first for the images of the mirrored hosts: registry.k8s.io for the control
# plane, CoreDNS and the pod sandbox, ghcr.io for flannel, and docker.io for the workloads, such as
# the guide's busybox. chalkctl registry push puts the images there through the tunnel.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: ClusterRegistry
metadata: {name: cluster}
spec:
  mirrors: [docker.io, ghcr.io, registry.k8s.io]
  endpoints: ["fdc8:a1b2:c3d4:ff::11", "fdc8:a1b2:c3d4:ff::12", "fdc8:a1b2:c3d4:ff::13"]
---
# The nodes' default time servers lie on the internet. The router keeps its own clock over wan and
# answers NTP from the fabric, as the TimeServer of air-gapped/router.yaml sets it up. Plain NTP,
# as the router's chrony serves no NTS. A router without an uplink answers as unsynchronised,
# which the nodes refuse, unless it has a reference clock or the TimeServer sets localStratum.
apiVersion: time.chalkos.dev/v1alpha1
kind: TimeSync
metadata: {name: default}
spec:
  servers:
    - {host: "fdc8:a1b2:c3d4:1::1", nts: false}

air-gapped/router.yaml makes the router the fabric's time server:

air-gapped/router.yaml
# What the router of the air-gapped lab, chalkos-air-gapped.yaml, gets beside router/router.yaml
# and router/secrets.yaml. It leaves out router/nat64.yaml, so the fabric reaches no internet,
# and answers the nodes' NTP: the router's chrony serves the time it keeps over wan, and the
# firewall accepts NTP from the lab's addresses on the fabric's first address alone. A router
# without an uplink would need `localStratum: 10` here to serve its own clock, which drifts from
# the real time; an air gap that needs accurate time wants a reference clock instead.
apiVersion: router.chalkos.dev/v1alpha1
kind: TimeServer
metadata: {name: ntp}
spec:
  listen: ["fdc8:a1b2:c3d4:1::1"]
  allow: ["fdc8:a1b2:c3d4::/48"]

The TimeServer lets the router's chrony answer the lab's addresses, and the firewall accepts NTP from them on the fabric's first address alone. The router keeps following the internet's time servers over wan, so it answers as synchronised. The image needs no change: router images carry the time server, as Serve time with NTP describes.

Copy the images to the workstation

The workstation keeps its own internet access, so it fetches the images. Copy every image of the Cluster document into an OCI image layout of its own with crane, and busybox, which the checks below run:

mkdir -p images/oci
for ref in $(yq 'select(.kind == "Cluster") | .spec.images[]' cluster/cluster.yaml); do
  repo=${ref%%@*}; name=${repo##*/}; name=${name%%:*}
  rm -rf "images/oci/$name"
  crane pull --format=oci "$ref" "images/oci/$name"
done
rm -rf images/oci/busybox
crane pull --format=oci docker.io/library/busybox:1.37.0 images/oci/busybox

The copies keep every platform of each image's index and take about 1.2 GiB. Bring the images into the air gap explains why the loop uses crane and not skopeo.

Cut the fabric off

Run chalkctl with the variant's inventory for the rest of this section:

export CHALKOSINVENTORY=chalkos-air-gapped.yaml
chalkctl validate

Have the router check its new config first. The dry run prints the documents that change, the NAT64 removed and the TimeServer added, and the outputs and effects that follow:

chalkctl config apply router --secrets=secrets.age --dry-run

Deliver it on trial, because it changes the router's firewall, and then the nodes' config:

chalkctl config apply router --secrets=secrets.age --try 5m
chalkctl config apply node1 node2 node3 --secrets=secrets.age

The router stops its NAT64 and answers NTP from the fabric. Each control plane starts the cluster registry at its node address, and every node's containerd looks there first for the images of registry.k8s.io, ghcr.io and docker.io.

Push the images

Push every image to the three endpoints through the tunnel:

for ref in $(yq 'select(.kind == "Cluster") | .spec.images[]' cluster/cluster.yaml); do
  repo=${ref%%@*}; name=${repo##*/}; name=${name%%:*}
  chalkctl registry push "images/oci/$name" "$ref" --secrets=secrets.age
done
chalkctl registry push images/oci/busybox docker.io/library/busybox:1.37.0 --secrets=secrets.age

Each push prints the image's digest and a line for each endpoint, such as those of the API server's image:

pushing registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4 as registry.k8s.io/kube-apiserver@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
fdc8:a1b2:c3d4:ff::11: pushed [fdc8:a1b2:c3d4:ff::11]:5000/registry.k8s.io/kube-apiserver:v1.37.2 as sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
fdc8:a1b2:c3d4:ff::12: pushed [fdc8:a1b2:c3d4:ff::12]:5000/registry.k8s.io/kube-apiserver:v1.37.2 as sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
fdc8:a1b2:c3d4:ff::13: pushed [fdc8:a1b2:c3d4:ff::13]:5000/registry.k8s.io/kube-apiserver:v1.37.2 as sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4

Check that every endpoint holds every image:

$ chalkctl registry list --secrets=secrets.age
REPOSITORY                               TAG      DIGEST               ENDPOINTS
docker.io/library/busybox                1.37.0   sha256:a7dbc319b143  all
ghcr.io/flannel-io/flannel               v0.28.9  sha256:708a2c9c1cfb  all
registry.k8s.io/coredns/coredns          v1.14.6  sha256:900f9c109f7a  all
registry.k8s.io/etcd                     3.7.0-0  sha256:6ecefbe2510c  all
registry.k8s.io/kube-apiserver           v1.37.2  sha256:3e563afb5cb5  all
registry.k8s.io/kube-controller-manager  v1.37.2  sha256:28e62daf706e  all
registry.k8s.io/kube-proxy               v1.37.2  sha256:b2e075697d7a  all
registry.k8s.io/kube-scheduler           v1.37.2  sha256:028c847fc321  all
registry.k8s.io/pause                    3.10.2   sha256:f548e0e8e3dc  all

Bootstrap and check the cut

On a lab that waits for its bootstrap, bootstrap node1 and write a kubeconfig, as in Bootstrap Kubernetes. The control planes pull their images from the cluster registry. Then check that the nodes run and that a pod pulls busybox from the registry:

kubectl get nodes
kubectl run airgap-check --rm -it --restart=Never --image=busybox:1.37.0 -- true

Check that the fabric no longer reaches the internet. The tcp-connect diagnostic to 64:ff9b::101:101, which connects on the connected lab, fails now, and chalkctl status router shows no nat64: table:

chalkctl node diag node1 --secrets=secrets.age tcp-connect address=64:ff9b::101:101 port=443
chalkctl status router --secrets=secrets.age

chalkctl status node1 --secrets=secrets.age prints a line time: that names fdc8:a1b2:c3d4:1::1, the router, as the server the node's clock follows.

A pod whose image is not in the registry stays in ImagePullBackOff. Copy and push its image as above; If something goes wrong covers other failures.

Connect the lab again

Deliver the config of chalkos.yaml to the router and the nodes:

unset CHALKOSINVENTORY
chalkctl config apply router --secrets=secrets.age --try 5m
chalkctl config apply node1 node2 node3 --secrets=secrets.age

The router runs its NAT64 again, the nodes take their time from the internet, and containerd pulls from the registries themselves. The pushed images stay on the control planes, in /var/lib/chalkos-registry, and the registry serves them again when the variant's config returns.

Switch an existing lab to advertised gateways

If the nodes still take their default route over BGP, apply the new bgp/fabric.yaml to the nodes one at a time while the router still sends a BGP default. Try the node config with chalkctl config apply node1 --secrets=secrets.age --try 5m --no-confirm. Use chalkctl node diag node1 --secrets=secrets.age routes family=inet6 to check for an IPv6 default route with protocol ra on each connected fabric port. Check management and internet access before keeping the trial with chalkctl config confirm node1 --secrets=secrets.age. After every node uses an advertised gateway, apply the router config to stop announcing the BGP default.

Troubleshooting

A node's port gets no address of the fabric

The router's advertisements give the ports their addresses. Check that the router's fabric port holds fdc8:a1b2:c3d4:1::1 with the addresses diagnostic, as in Check the router. When fabric is missing and a port of the kernel's name shows instead, the MAC address of the router's Link document is not the port's: correct it, and deliver it with chalkctl config apply router --try, which renames the port at once. A node port that shows DOWN has no carrier on the switch. An installed node configures only the ports that bgp/fabric.yaml names: a port of another name takes no address, so put the names the addresses diagnostic shows into bgp/fabric.yaml, as in Adapt the node's file.

A BGP session does not come up

The router's bgp: table shows no line for a port that never connected. Without a session the router has no route to the node's address, so chalkctl does not reach the node there. The node still answers at its ports' addresses of the fabric, through its advertised default gateway. The router's neighbours diagnostic lists them, as the tip "Find the installer through the router" in Install the nodes shows; pass one with --address:

chalkctl node diag node1 --secrets=secrets.age \
  --address=fdc8:a1b2:c3d4:1:0:ff:fe00:1101 bgp-summary

Check, on the node:

  • the ports' names: the addresses diagnostic shows the names that the BGPPeer documents of bgp/fabric.yaml list, enp1s0 and enp2s0 in the example. FRR runs no session on a port of another name;
  • the password: the kernel drops TCP segments whose MD5 signature does not match, so neither side logs a refused session. The router has no bgp: line for the node, and on the node chalkctl node diag node1 bgp-summary shows the sessions in Connect or Active instead of a count of received prefixes. Compare the passwords of router/secrets.yaml and bgp/secrets.yaml, which must be the same.

The router's firewall accepts BGP and BFD from the fabric by itself, and the nodes' image opens BFD's port.

A name does not resolve, or a connection to the internet fails

On the router, chalkctl status router shows resolver: running and the NAT64's line. The diagnostics dns-query and nat64-sessions show the resolver's answers and the NAT64's connections, as Translate to IPv4 and resolve names describes. A NAT64 connection needs the router's wan address and default route from the home network's DHCP server.

The router takes another address on the home network

The router identifies itself to the DHCP server with a DUID made from the wan port's MAC address, the same at every boot, as the installer does for each port. Check that the reservation names the MAC address that the addresses diagnostic shows for wan. A DHCP server that matches a reservation against a client identifier of the bare MAC address finds none in a DUID. Have the router send its MAC address as the identifier, in the wan port's Network of router/router.yaml:

router/router.yaml
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-wan}
spec:
  match: {name: wan}
  network: {dhcp: ipv4, ipv6AcceptRA: false}
  dhcpv4: {clientIdentifier: mac}

Until the router has the reserved address, put the address it took, which its console shows, into chalkos.yaml as the router's.

The tunnel has no handshake

The wireguard diagnostic shows no latest handshake for the workstation's peer. Check that the WireGuardPeer holds the contents of workstation.pub, that the workstation's config holds the router's public key from router.pub, and that its Endpoint is the router's address on the home network. After a delivery to the router, the tunnel works again once the workstation sends; PersistentKeepalive makes it send within 25 seconds.

What next

These pages explain the steps of this tutorial in more depth: