Build a home lab¶
This tutorial builds the home lab of the homelab example on real machines: a router on your home network and three Kubernetes control planes behind it, on an IPv6 fabric of their own. You discover each machine's ports and disks, adapt the example's files to them, install the four machines and bootstrap the cluster. At the end you test two failures: a node keeps its address when one of its cables is pulled, and the Kubernetes API stays up when one control plane goes down. Plan an afternoon; most of it goes into the first image builds and the installs.
An advanced IPv6 and BGP setup
This lab is a maintainer's own home lab. Its BGP fabric, IPv6-only nodes, NAT64 and WireGuard tunnel are there for fun and to try those features out, not because a home network needs them. The guide shares the setup in case it helps you test chalkos on a similar network. For a first cluster on real machines, follow Build a cluster on real machines instead: one network, IPv4 and no router of chalkos.
What you build¶
The router connects the home network to the fabric, a switch that holds the router's second port and both ports of each node. The workstation reaches the router at its address on the home network and the fabric through a WireGuard tunnel to the router.
flowchart TB
home["home network, IPv4 by DHCP"]
ws["workstation"]
router["router: NAT64, DNS64, router advertisements, BGP, WireGuard"]
switch["unmanaged switch, the fabric"]
node1["node1"]
node2["node2"]
node3["node3"]
ws --- home
home ---|wan| router
ws -.->|WireGuard| router
router ---|fabric| switch
switch ---|enp1s0, enp2s0| node1
switch ---|enp1s0, enp2s0| node2
switch ---|enp1s0, enp2s0| node3
Every address comes from one unique local /48, fdc8:a1b2:c3d4::/48 in the example. You replace it
with a random /48 of your own in Prepare the files.
| Use | Prefix or address |
|---|---|
| The fabric, which the router advertises | fdc8:a1b2:c3d4:1::/64 |
The router on the fabric: resolver and the BGP neighbour of each node's enp1s0 |
fdc8:a1b2:c3d4:1::1 |
The router's second address on the fabric: the BGP neighbour of each node's enp2s0 |
fdc8:a1b2:c3d4:1::2 |
node1, node2 and node3, each on its dummy interface node0 |
fdc8:a1b2:c3d4:ff::11, ::12 and ::13 |
| The Kubernetes API's VIP | fdc8:a1b2:c3d4:ff::10 |
| The WireGuard tunnel: the router and the workstation | fdc8:a1b2:c3d4:f0::1 and ::2, in fdc8:a1b2:c3d4:f0::/64 |
| NAT64 prefix | 64:ff9b::/96 |
| AS numbers | 65000 for the router, 65001 for every node |
| The router on the home network | an IPv4 address by DHCP, 192.168.1.2 in the example |
The fabric carries IPv6 alone. The router's advertisements give every port on the fabric an
address by SLAAC, a default gateway, the resolver and the NAT64 prefix. The resolver answers a name that has only IPv4
addresses with addresses in 64:ff9b::/96, and the NAT64 carries connections to them to the IPv4
internet, out of the router's port on the home network. A node's own address is a /128 on its dummy
interface node0. FRR, the BGP speaker of the nodes' image, announces it to the router over a
session on each port, each to one of the router's two addresses on the fabric, and the control
plane that holds the API's VIP announces the VIP the same way. The router then has two equal paths
to each address. When a cable or port fails, BFD ends that port's session within about a second,
and the router forwards over the other port. In the other direction, networkd uses the default
gateway from the surviving port's advertisements when a port loses carrier. The installer and
installed nodes can reach the workstation and IPv4 internet before BGP comes up. Traffic between
nodes' routed addresses passes through the router, so the cluster still depends on the router.
Before you begin¶
You need:
- four amd64 machines with UEFI firmware, a TPM 2.0 and two Ethernet ports each, and a disk of at least 16 GiB for the router and a larger one for each node, which holds the containers' images and etcd; the requirements list the rest;
- an unmanaged switch with at least seven ports, and Ethernet cables;
- a USB stick of at least 1 GiB;
- your home network, whose router hands out IPv4 addresses by DHCP and lets you reserve an address for a MAC address;
- a workstation on the home network, an x86-64 Linux machine with
Nix and flakes enabled, git, and WireGuard's tools
wgandwg-quickfrom your distribution; - an age key or an SSH key of type ed25519 or RSA to encrypt the cluster's secrets to.
Secure Boot is optional¶
The tutorial installs the machines without Secure Boot, with the unsigned images and installer. Turn Secure Boot off in each machine's firmware setup. The steps marked optional add Secure Boot with keys of your own: Sign the images and Enrol the Secure Boot keys. A machine whose firmware is in Setup Mode gets the keys from chalkctl. For any other machine, enrol the db certificate in its firmware as Enrol the certificate without Setup Mode describes before you install it.
Without Secure Boot, the TPM still unlocks the disks at every boot. It seals their keys to PCR 7, which then measures Secure Boot as off, so a machine that boots another system from a USB stick unlocks them too. The encryption protects a disk taken out of its machine, and no longer the machine itself. Security model describes what else Secure Boot protects.
Decide before you install a machine. Turning Secure Boot on later changes PCR 7: the machine then asks for its recovery key at every boot until you install it again.
Prepare the files¶
The example is part of the chalkos repository. Copy it into a repository of your own, and put it under git, because Nix reads a flake from the files git tracks:
git clone https://github.com/trevex/chalkos
cp -r chalkos/examples/homelab homelab
cd homelab
git init && git add .
Run every later command in homelab. The example's files explain themselves in comments:
| File | Holds |
|---|---|
flake.nix |
Builds router-metal, node-metal and the installer. |
chalkos.yaml |
The inventory: the four nodes, the address chalkctl reaches each at, their images and files. |
router/router.yaml |
The router's ports, firewall, resolver, BGP and WireGuard. |
router/nat64.yaml |
The router's NAT64, the fabric's way to the IPv4 internet. |
router/secrets.yaml |
The router's WireGuard key and BGP password. |
cluster/cluster.yaml |
The Kubernetes cluster that all three nodes share. |
bgp/fabric.yaml, bgp/secrets.yaml |
The nodes' ports, their BGP sessions with the router, and the password. |
nodes/node1.yaml to nodes/node3.yaml |
Each node's disk, address and BGP speaker. |
chalkos-air-gapped.yaml, air-gapped/ |
The lab without internet access, an optional variant near the end of this tutorial. |
Get the tools¶
The flake's shell holds chalkctl and chalkbuild of the chalkos revision that flake.lock pins,
with kubectl, jq, openssl, wg, arp-scan, nc, crane and yq. Lock the flake's chalkos input and
enter the shell:
Run every later command of this tutorial in that shell, so that chalkctl and the images come from the same chalkos release.
The diagnostic addresses, which shows a machine's ports, prints JSON, as ip -json address show
does. Keep a jq filter in the shell that turns it into a line per port, with the port's name, MAC
address, state and addresses beyond the link:
ports='.[] | [.ifname, .address, .operstate,
([.addr_info[] | select(.scope == "global") | .local] | join(" "))] | @tsv'
Generate the cluster's secrets¶
The secrets file holds the cluster's CAs and keys. Generate it encrypted to your key, and to a backup key that you keep offline:
$ chalkctl gen secrets --recipient "$(cat ~/.ssh/id_ed25519.pub)" \
--recipient age1nmrqj76cqyf4ft8xtczufd3mtdk0xf2j7tevgwnzjp3zq9tffu3slx5eux
encrypted secrets.age to ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKMcR1DCFPGSdWXphQ3We8UmkMp2KdufUzVxxC6Hvo9k alice@workstation, age1nmrqj76cqyf4ft8xtczufd3mtdk0xf2j7tevgwnzjp3zq9tffu3slx5eux
wrote secrets.age and secrets.pub.json
The age1 recipient here stands for your offline backup
key's public key. Then give the images and the installer the cluster's OS CA, so
that a machine waiting for its install accepts only a chalkctl that holds this cluster's secrets.
Add the line to the arguments of chalkos.lib.mkImages in flake.nix:
Without it, any machine on your home network could install the router while it waits in the installer.
Choose your own prefix¶
Generate the first 48 bits of a unique local prefix, and replace the example's in every file:
set -- $(od -An -N5 -tu1 /dev/urandom)
ula=$(printf 'fd%02x:%x:%x' "$1" $(($2 * 256 + $3)) $(($4 * 256 + $5)))
echo "$ula"
grep -rlF fdc8:a1b2:c3d4 --exclude-dir=.git . | xargs sed -i "s/fdc8:a1b2:c3d4/$ula/g"
The command prints a prefix such as fd3b:99df:9be2. The rest of this tutorial shows the example's
fdc8:a1b2:c3d4; read yours in its place.
Make the WireGuard keys and the BGP password¶
Make a key pair for the router and one for the workstation. The example's .gitignore keeps the
private keys, *.key, out of git, as it keeps the images, the installer and the kubeconfig:
(umask 077 && wg genkey | tee router.key | wg pubkey > router.pub)
(umask 077 && wg genkey | tee workstation.key | wg pubkey > workstation.pub)
Replace the example's keys: the router's private key goes into router/secrets.yaml, and the
workstation's public key into the WireGuardPeer of router/router.yaml:
sed -i "s|uBjWXU2Bu/LvLVRo+dtAte3VHt75d9W7DPxsEhZrm24=|$(cat router.key)|" router/secrets.yaml
sed -i "s|cK0R7bvqvEfk07xvf9i71Jj0ATJEGaKZ3gs1zBXr1m0=|$(cat workstation.pub)|" router/router.yaml
The router and the nodes share one BGP password, in router/secrets.yaml and bgp/secrets.yaml.
Replace the example's with a random one in both:
The two files now hold real secrets in plain text. Keep the repository private, or keep the two files out of it.
Build the images and the installer¶
Build the two images the inventory names where it expects them, and the installer:
git add .
nix build .#router-metal -o images/router-metal
nix build .#node-metal -o images/node-metal
nix build .#installer -o installer
The first build takes a while. The router's image carries the router layer and no Kubernetes; the nodes' image carries Kubernetes and its BGP speaker, FRR.
Sign the images (optional)¶
Skip this step without Secure Boot. With it, create your Secure Boot keys in keys/, and sign
copies of the three builds with the db key:
chalkctl secureboot new keys --name homelab
chalkbuild sign --key keys/db.key --cert keys/db.crt images/router-metal images/router-metal-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt images/node-metal images/node-metal-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt installer installer-signed
The example's .gitignore keeps the .key files out of git. Keep a backup of keys/ offline:
whoever holds its keys decides what the machines boot.
Point the inventory's images at the signed copies. chalkos.yaml then reads:
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: homelab}
spec:
images:
# chalkbuild sign writes the signed copies of the builds here.
router-metal: {source: {directory: images/router-metal-signed}}
node-metal: {source: {directory: images/node-metal-signed}}
groups:
router: {imageName: router-metal, files: [router/*.yaml]}
nodes: {imageName: node-metal, files: [cluster/*.yaml, bgp/fabric.yaml, bgp/secrets.yaml]}
nodes:
router: {address: 192.168.1.2, groups: [router]}
node1: {address: "fdc8:a1b2:c3d4:ff::11", groups: [nodes], files: [nodes/node1.yaml]}
node2: {address: "fdc8:a1b2:c3d4:ff::12", groups: [nodes], files: [nodes/node2.yaml]}
node3: {address: "fdc8:a1b2:c3d4:ff::13", groups: [nodes], files: [nodes/node3.yaml]}
Write the installer to a USB stick¶
Write the installer's ISO to the USB stick:
With Secure Boot, write the signed copy, installer-signed/chalkos-installer.iso, instead.
<usb-disk> is the stick's device, such as sdb; lsblk lists the candidates. The command
overwrites everything on the stick. The installer holds no config of a node: it takes addresses on
every Ethernet port, by DHCP and from router advertisements, and keeps the kernel's names for the
ports.
Install the router¶
The router goes first, because the nodes' installers take their addresses from its advertisements on the fabric.
Boot the installer¶
Cable one of the router's ports to the home network and the other to the switch. Which port is which does not matter yet: the next step finds out. Boot the router from the USB stick through the firmware's boot menu. Once chalkd runs, the screen shows the console. Its header keeps chalkd's fingerprint on screen, and the address that the home network's DHCP server gave the port it reaches:
nixos installer up 1m
image 81bd81bd81bd
installer: chalkctl install installs nodes from here
fingerprint <fingerprint>
addresses 192.168.1.57
Note the address and the fingerprint, 64 hex characters. chalkctl reaches a node that a command
names at the node's address in chalkos.yaml, so put the installer's address there as the
router's for now, and the commands below need no --address:
The commands check the machine's certificate with --fingerprint.
Find the installer without a screen
On a machine without a screen, look for the installer on the home network: chalkd listens on
TCP port 50000. The flake's shell holds arp-scan and nc; this lists the home network's
machines that answer on that port:
--insecure in place of --fingerprint
When reading the console is impractical, pass --insecure in place of --fingerprint. chalkctl
then accepts any certificate and prints the fingerprint it saw, as
chalkctl: the node's certificate fingerprint is <fingerprint>. Compare it with the console
when you can. chalkctl install sends the secrets only over a connection pinned to that
fingerprint.
Enrol the Secure Boot keys (optional)¶
Skip this step without Secure Boot, or when the router's firmware holds your db certificate
already. When its firmware is
in Setup Mode, as after you clear its keys in the firmware setup, it boots the installer without
checking it. Enrol your keys; secureboot enroll names no node, so it takes the address with
--address:
$ chalkctl secureboot enroll --keys keys --address=192.168.1.57 --fingerprint=<fingerprint> \
--secrets=secrets.age
...
the node reboots with Secure Boot enforced
chalkctl: the node's certificate fingerprint is now <new fingerprint>; compare it with the node's console and install with --fingerprint <new fingerprint>
Secure Boot is on; install the node with chalkctl install
The router reboots into the installer, now with Secure Boot on; pick the USB stick in the boot menu if it boots something else. chalkd makes a new certificate at that boot: use the new fingerprint from here on.
When the router's network cards carry option ROMs, the node refuses the keys and names them: add
--option-roms to enrol their digests.
Find the disk and the ports¶
List the disks:
$ chalkctl disks router --fingerprint=<fingerprint> --secrets=secrets.age
DEVICE SIZE TYPE MODEL SERIAL WWN USE
/dev/nvme0n1 476.9G nvme Samsung SSD 980 512GB S64DNF0R812345 eui.002538b811b2c3d4
...
The list goes on with the installer's own stick, whose use is boot. Pick the disk for the
system, and find its path below /dev/disk/by-id, which stays the same across boots. For an NVMe
disk it is nvme-<model>_<serial>, and for a SATA disk ata-<model>_<serial>, with each space of
the model as an underscore: /dev/disk/by-id/nvme-Samsung_SSD_980_512GB_S64DNF0R812345 here. When
the path is in doubt, select the disk by its serial number instead, with
disk: {serial: S64DNF0R812345} in place of path. The installer refuses a disk
that holds data.
The diagnostic addresses lists the ports, and the jq filter of
"Get the tools" puts each on a line:
$ chalkctl node diag router --fingerprint=<fingerprint> --secrets=secrets.age addresses \
| jq -r "$ports"
lo 00:00:00:00:00:00 UNKNOWN
enp1s0 02:00:00:00:01:01 UP 192.168.1.57
enp2s0 02:00:00:00:01:02 UP
The port with the home network's address is the router's wan, and the other one, on the switch,
is its fabric. A port that shows DOWN has no carrier: check its cable.
Adapt the router's files¶
router/router.yaml names the ports by their permanent MAC addresses in two Link documents,
which rename them to wan and fabric at the installed router's first boot. Every other document
names the ports by those names. The addresses diagnostic shows each port's current MAC address,
which on a plain port, without a bond or a VLAN, is its permanent one. Put the two into the Link
documents, and your disk into the SystemDisk:
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Link
metadata: {name: 10-wan}
spec:
match: {permanentMACAddress: "02:00:00:00:01:01"}
link: {name: wan}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Link
metadata: {name: 10-fabric}
spec:
match: {permanentMACAddress: "02:00:00:00:01:02"}
link: {name: fabric}
---
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
disk: {path: /dev/disk/by-id/nvme-Samsung_SSD_980_512GB_S64DNF0R812345}
Reserve an IPv4 address for the wan port's MAC address on your home network's DHCP server. The
installed router takes it on wan, and chalkctl reaches it there once you put it into
chalkos.yaml after the install. The example reserves 192.168.1.2.
Check the files against the images:
It prints a line per node, and names the document and field of each mistake.
Install¶
Install the router at the installer's address, which chalkos.yaml names for now:
$ chalkctl install router --secrets=secrets.age --fingerprint=<fingerprint>
installing router onto /dev/disk/by-id/nvme-Samsung_SSD_980_512GB_S64DNF0R812345 from /home/alice/homelab/images/router-metal: chalkos <image>, sending <n> bytes of store, hash tree, UKI and boot loader
router is installed and reboots
chalkctl recovery-key router prints the key that unlocks it when its TPM fails
<image> is the start of the image's root hash, and <n> the size of what chalkctl sends. The
router reboots into its image, renames its ports and takes the reserved address on wan. Remove
the USB stick once the console's header shows normal, and put the reserved address back into
chalkos.yaml:
Wipe a disk that holds data
The installer refuses a disk that holds data, such as a disk from another machine or an earlier install of another image, and names what it found:
install: the target disk /dev/nvme0n1 (model "Samsung SSD 980 512GB", size 476.9G, serial "S64DNF0R812345") carries data (gpt ext4); pass --wipe-disk to replace it
Check in chalkctl disks that the disk is the one you mean, and install again with
--wipe-disk. The installer then clears the disk's partition tables and lays it out anew.
Nothing on the disk can be recovered afterwards:
Check the router¶
chalkctl status reaches the router at its address in the inventory:
$ chalkctl status router --secrets=secrets.age
...
bgp: no sessions
nat64:
INSTANCE SESSIONS BINDINGS POOL TCP POOL UDP POOL ICMP
nat64 0 0 4535 4535 4535
resolver: running
...
The router runs its config, its NAT64 and its resolver. It has no BGP session yet, because no
node runs. The addresses diagnostic shows the renamed ports, wan with the reserved address and
fabric with its two addresses, fdc8:a1b2:c3d4:1::1 and ::2:
$ chalkctl node diag router --secrets=secrets.age addresses | jq -r "$ports"
lo 00:00:00:00:00:00 UNKNOWN
wan 02:00:00:00:01:01 UP 192.168.1.2
fabric 02:00:00:00:01:02 UP fdc8:a1b2:c3d4:1::2 fdc8:a1b2:c3d4:1::1
wg0 UNKNOWN fdc8:a1b2:c3d4:f0::1
...
Connect the workstation¶
The workstation reaches the fabric through the router's WireGuard interface wg0. Its config for
wg-quick holds the workstation's private key, so create the file readable by root alone first:
Then write the config into it, with the workstation's private key and the router's public key:
[Interface]
PrivateKey = <contents of workstation.key>
Address = fdc8:a1b2:c3d4:f0::2/64
[Peer]
PublicKey = <contents of router.pub>
Endpoint = 192.168.1.2:51820
AllowedIPs = fdc8:a1b2:c3d4::/48
PersistentKeepalive = 25
AllowedIPs sends the whole /48 through the tunnel, and nothing else: the workstation reaches the
internet as before. Keep PersistentKeepalive: every config delivery to the router sets its
WireGuard peers anew, and the router reaches the workstation again only once the workstation sends
a packet. Bring the tunnel up and ping the router's fabric address:
$ sudo wg-quick up homelab
$ ping -c 3 fdc8:a1b2:c3d4:1::1
PING fdc8:a1b2:c3d4:1::1 (fdc8:a1b2:c3d4:1::1) 56 data bytes
64 bytes from fdc8:a1b2:c3d4:1::1: icmp_seq=1 ttl=64 time=0.612 ms
...
The diagnostic wireguard shows the router's side of the tunnel, with the workstation's latest
handshake. The public keys are the example's; yours are those of router.pub and
workstation.pub:
$ chalkctl node diag router --secrets=secrets.age wireguard interface=wg0
interface: wg0
public key: N1efwZV4rLKHLJQIeOXqOmCOlJVUhEnnXMOqRPH2L0k=
private key: (hidden)
listening port: 51820
peer: cK0R7bvqvEfk07xvf9i71Jj0ATJEGaKZ3gs1zBXr1m0=
endpoint: 192.168.1.20:41641
allowed ips: fdc8:a1b2:c3d4:f0::2/128
latest handshake: 9 seconds ago
transfer: 1.02 KiB received, 892 B sent
Install the nodes¶
Install node1, node2 and node3 one after another, with the same steps.
Boot the installer¶
Cable both of the node's ports to the switch and boot it from the USB stick. The router's advertisements give each port an address of the fabric's prefix, and the console's header shows both:
nixos installer up 1m
image 81bd81bd81bd
installer: chalkctl install installs nodes from here
fingerprint <fingerprint>
addresses fdc8:a1b2:c3d4:1:0:ff:fe00:1101 fdc8:a1b2:c3d4:1:0:ff:fe00:1102
The header shows no link-local address. A port without an address of the fabric has no cable, or
the router sends no advertisements: the section "Troubleshooting" covers both. Note one of the
addresses and the fingerprint. Put the address into chalkos.yaml as node1's for now, quoted, as
YAML needs for an IPv6 address; the workstation reaches it over the tunnel:
node1: {address: "fdc8:a1b2:c3d4:1:0:ff:fe00:1101", groups: [nodes], files: [nodes/node1.yaml]}
Find the installer through the router
When the node has no screen, find its ports in the router's neighbour table. Each of the
installer's ports takes an address of the fabric's prefix with the same interface ID as its
link-local address, so this lists every machine the router sees on fabric with its MAC
address and its address of the fabric:
chalkctl node diag router --secrets=secrets.age neighbours \
| jq -r '.[] | select(.dev == "fabric" and (.dst | startswith("fe80::")))
| [.lladdr, (.dst | sub("^fe80::"; "fdc8:a1b2:c3d4:1:"))] | @tsv'
The output has a line for each port, here the two of node1:
The table also holds the ports of nodes installed before, whose MAC addresses the addresses
diagnostic shows on each of them. A port the router has not talked to yet is missing from the
table: boot one node at a time, and look again after a minute. Check that chalkd answers at an
address before you use it:
chalkctl node diag router --secrets=secrets.age tcp-connect \
address=fdc8:a1b2:c3d4:1:0:ff:fe00:1101 port=50000
Without the console you have no fingerprint to compare: reach the installer with --insecure
in place of --fingerprint, as the note under the router's "Boot the installer" describes.
With Secure Boot, and when the node's firmware is in Setup Mode, enrol your keys at that address as for the router, in "Enrol the Secure Boot keys", and note the new fingerprint.
Find the disk and the ports¶
List the disks and the ports:
$ chalkctl disks node1 --fingerprint=<fingerprint> --secrets=secrets.age
DEVICE SIZE TYPE MODEL SERIAL WWN USE
/dev/nvme0n1 931.5G nvme WD Blue SN580 1TB 24131D801234 eui.e8238fa6bf530001
...
$ chalkctl node diag node1 --fingerprint=<fingerprint> --secrets=secrets.age addresses \
| jq -r "$ports"
lo 00:00:00:00:00:00 UNKNOWN
enp1s0 02:00:00:00:11:01 UP fdc8:a1b2:c3d4:1:0:ff:fe00:1101
enp2s0 02:00:00:00:11:02 UP fdc8:a1b2:c3d4:1:0:ff:fe00:1102
The disk's path is /dev/disk/by-id/nvme-WD_Blue_SN580_1TB_24131D801234 here. The two ports are
enp1s0 and enp2s0, the names the kernel gives them. The names follow from the ports' PCI slots,
so every machine of the same model shows the same two.
Adapt the node's file¶
bgp/fabric.yaml, which every node gets, configures the ports: a Network document for each port,
which takes the default gateway and other settings from the router's advertisements, and a BGPPeer
for each port, which advertises node and VIP addresses to the router. It names the ports enp1s0 and enp2s0. When the
addresses diagnostic shows other names, replace enp1s0 and enp2s0 throughout
bgp/fabric.yaml with them. You do this once, for node1: node2 and node3 have the same names when
they are the same model. A node whose ports have other names, such as a machine of another model,
takes Network and BGPPeer documents of its own in place of bgp/fabric.yaml, as the file's
comment describes.
nodes/node1.yaml holds everything that differs between the nodes:
# node1: a control plane whose two ports are both on the fabric's switch. Its address sits on the
# dummy interface node0, and FRR announces it over a session on each port, so the address stays
# reachable while one port or cable works.
#
# The ports' Network documents and the BGPPeers lie in bgp/fabric.yaml, under the names the kernel
# gives the ports, which identical machines share. A node whose ports have other names leaves
# bgp/fabric.yaml out and brings its own Network and BGPPeer documents.
#
# Replace the disk with the node's own, by its path below /dev/disk/by-id.
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
disk: {path: /dev/disk/by-id/nvme-EXAMPLE_SSD_node1}
---
# node0 holds the node's address, and the API's VIP while the node holds it. chalkd adds the VIP
# with a preferred lifetime of 0, which networkd counts as a dynamic address, so keepConfiguration
# yes has networkd keep every address it did not configure itself when it configures the link again.
apiVersion: networkd.chalkos.dev/v1alpha1
kind: NetDev
metadata: {name: 30-node0}
spec:
netDev: {name: node0, kind: dummy}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 30-node0}
spec:
match: {name: node0}
network: {address: ["fdc8:a1b2:c3d4:ff::11/128"], keepConfiguration: "yes"}
---
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
type: controlplane
nodeIP: {addresses: ["fdc8:a1b2:c3d4:ff::11"]}
vip: {interface: node0}
---
# The node's side of its BGP sessions. The nodes share an AS number, and each has a router ID of
# its own. FRR announces the addresses of node0: the node's own and, while the node holds it, the
# API's VIP.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: BGPSpeaker
metadata: {name: bgp}
spec:
asn: 65001
routerID: 0.0.0.11
export: {interfaces: [node0]}
Put your disk's path into the SystemDisk.
The rest of the file stays as it is. node0 is a dummy interface that holds the node's address,
fdc8:a1b2:c3d4:ff::11, which no cable can take away. The Node document makes the node a
control plane, gives Kubernetes that address as the node's address, and names node0 as the
interface the node adds the API's VIP to while it holds it. The BGPSpeaker gives FRR the node's
AS number and router ID and announces the addresses of node0. node2 and node3 differ in their
address, ::12 and ::13, and their router ID, 0.0.0.12 and 0.0.0.13. Each node needs an
address and a router ID of its own, and the address in its Network and Node documents is the
one chalkos.yaml names for it.
Install¶
Validate the files and install the node at the installer's address, which chalkos.yaml names for
now. Add --wipe-disk when the installer refuses the disk, as for the router:
chalkctl prints the lines that the router's install printed. The node reboots into its image,
brings up node0 and starts FRR, whose sessions announce the node's address to the router. Put
the node's own address back into chalkos.yaml, where chalkctl reaches it from now on:
Until the bootstrap, its status shows that it waits:
$ chalkctl status node1 --secrets=secrets.age
...
kubernetes controlplane: waiting for bootstrap or for the cluster at https://[fdc8:a1b2:c3d4:ff::10]:6443
...
Repeat the steps for node2 and node3.
Bootstrap Kubernetes¶
Once the three nodes wait, bootstrap one of them and write a kubeconfig:
$ chalkctl bootstrap node1 --secrets=secrets.age
bootstrapping node1; the control plane pulls its images and starts
node1 is bootstrapped; applied 19 objects
$ chalkctl kubeconfig --secrets=secrets.age
wrote kubeconfig; its certificate is valid for 8760h0m0s
The bootstrap starts etcd and the control plane on node1. node1 then wins the election for the VIP,
adds fdc8:a1b2:c3d4:ff::10 to node0, and FRR announces it. node2 and node3 find the cluster at
the endpoint and join etcd one after the other. Pulling the images through the NAT64 takes most of
the bootstrap's time.
List the nodes through the VIP:
$ export KUBECONFIG=$PWD/kubeconfig
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready <none> 4m v1.37.2
node2 Ready <none> 2m v1.37.2
node3 Ready <none> 1m v1.37.2
A node may show NotReady for the first minute, until flannel runs on it. kubectl get nodes -o
wide shows each node's address, its /128 on node0, as INTERNAL-IP. The three nodes are
control planes and run workloads too.
The nodes reach the IPv4 internet through the router. The diagnostic dns-query asks the node's
resolver, the router's, for the name ipv4only.arpa, which has IPv4 addresses alone; DNS64 answers
with addresses in the NAT64's prefix:
$ chalkctl node diag node1 --secrets=secrets.age dns-query name=ipv4only.arpa type=AAAA
...
;; ANSWER SECTION:
ipv4only.arpa. 3600 IN AAAA 64:ff9b::c000:ab
ipv4only.arpa. 3600 IN AAAA 64:ff9b::c000:aa
...
$ chalkctl node diag node1 --secrets=secrets.age tcp-connect address=64:ff9b::101:101 port=443
connected to 64:ff9b::101:101 port 443
64:ff9b::101:101 is the IPv4 address 1.1.1.1 in the NAT64's prefix. Pods resolve names through
CoreDNS, which asks the same resolver, so a pod reaches an IPv4-only host by its name:
The answer holds an address in 64:ff9b::/96 beside the host's IPv4 address.
Check BGP and failover¶
The sessions¶
Each node runs a session on each port, to the router's address that the port's BGPPeer names.
The diagnostic bgp-summary of the nodes' image shows them as FRR's show bgp summary prints
them, a line for each of the router's addresses, fdc8:a1b2:c3d4:1::1 and fdc8:a1b2:c3d4:1::2.
A session that is up shows the number of prefixes received where a session that is down shows its
state. The IPv4 section shows both sessions as NoNeg, because the router's peer nodes carries
IPv6 alone. bfd-peers shows the BFD session that watches each:
chalkctl node diag node1 --secrets=secrets.age bgp-summary
chalkctl node diag node1 --secrets=secrets.age bfd-peers
The router accepts a session from every address of the fabric, and chalkctl status lists the six
under its peer nodes, each with the address of a node's port:
$ chalkctl status router --secrets=secrets.age
...
bgp:
VRF PEER NEIGHBOR STATE UPTIME RECEIVED SENT BFD
default nodes fdc8:a1b2:c3d4:1:0:ff:fe00:1101 Established 12m4s 2 0 up
default nodes fdc8:a1b2:c3d4:1:0:ff:fe00:1102 Established 12m3s 2 0 up
default nodes fdc8:a1b2:c3d4:1:0:ff:fe00:1201 Established 9m40s 1 0 up
default nodes fdc8:a1b2:c3d4:1:0:ff:fe00:1202 Established 9m40s 1 0 up
default nodes fdc8:a1b2:c3d4:1:0:ff:fe00:1301 Established 7m12s 1 0 up
default nodes fdc8:a1b2:c3d4:1:0:ff:fe00:1302 Established 7m11s 1 0 up
...
Each node sends its address on each session, and the holder of the VIP the VIP as well, so node1 shows 2 routes received. The router sends zero BGP routes. The routes to the nodes' addresses have two next hops each, one per port:
$ chalkctl node routes router fdc8:a1b2:c3d4:ff::/64 --secrets=secrets.age
DESTINATION TYPE PROTOCOL TABLE METRIC NEXT HOPS
fdc8:a1b2:c3d4:ff::10/128 unicast bgp 254 20 fe80::ff:fe00:1101 on fabric (weight 1), fe80::ff:fe00:1102 on fabric (weight 1)
fdc8:a1b2:c3d4:ff::11/128 unicast bgp 254 20 fe80::ff:fe00:1101 on fabric (weight 1), fe80::ff:fe00:1102 on fabric (weight 1)
...
Pull a cable¶
Start a ping of node1's address from the workstation, and pull one of node1's cables:
Within about a second, BFD ends the session on that port: bgp-summary on node1 shows a state in
place of that session's count of received prefixes, the router's bgp: table loses the port's
session, and the route to node1 keeps one next hop. The ping goes on, losing a packet or two at
most. Plug the cable back in, and the session comes back within seconds.
Take the VIP's holder down¶
chalkctl status shows which control plane holds the VIP:
$ chalkctl status node1 --secrets=secrets.age
...
kubernetes controlplane: bootstrapped, node ready: True, vip holder, control plane current
...
The other two show vip standby. Reboot the holder:
chalkctl drains node1 and keeps etcd's quorum while it reboots. Another control plane wins the
election, adds the VIP to its node0, and its FRR announces it. kubectl get nodes answers again
once the router has the new route, within about 30 seconds. For a harder test, cut the holder's
power instead: its sessions end within BFD's second, and the VIP moves when its lease runs out.
Power it on again afterwards; the VIP stays with its new holder.
Change things later¶
To change the lab, change its files, validate them, and deliver them to the nodes whose config changes:
Change a node's config covers previews, trials and rollbacks.
Deliver a change of the router's management interfaces or firewall with --try, so that the
router rolls it back when chalkctl loses it. A delivery renames a router port at once when its
Link document gives it a new name. Change every document that names the port in the same
delivery, its Network, firewall zone and management interfaces, and deliver it with
chalkctl config apply router --try, so that the router rolls it back when chalkctl loses it.
A new chalkos release or a change to the images' modules in flake.nix takes new images. For a
new release, change the tag in the chalkos input's URL and update the flake's lock with
nix flake update chalkos. Enter the shell of Get the tools again, so that
chalkctl matches the images. Build the images again, as in
Build the images and the installer. With Secure Boot, remove
the signed copies of the old builds, which chalkbuild sign does not overwrite, and sign the new
ones:
rm -rf images/router-metal-signed images/node-metal-signed installer-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt images/router-metal images/router-metal-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt images/node-metal images/node-metal-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt installer installer-signed
Then roll the images out to the nodes with chalkctl upgrade, as
Upgrade nodes describes. chalkctl upgrades the control planes one at a
time, so the Kubernetes API stays up. While the router reboots, traffic between the nodes'
routed addresses, to the Kubernetes API, and to the internet is interrupted. Upgrade the router
in a step of its own, at a time when that pause suits you:
chalkctl upgrade --nodes node1,node2,node3 --secrets=secrets.age
chalkctl upgrade --nodes router --secrets=secrets.age
Run the lab without internet access (optional)¶
This optional variant cuts the fabric off the internet. The router drops its NAT64, the control
planes serve the container images from the cluster registry, and the router serves the nodes'
time. The workstation keeps its tunnel and reaches every node as before. Follow it in place of
Bootstrap Kubernetes once the three nodes wait for their bootstrap, and
run its chalkctl bootstrap and chalkctl kubeconfig commands; its NAT64 checks fail in the
variant. On a running lab, the bootstrap step drops out.
Build an air-gapped cluster explains the approach for any cluster.
What connects the fabric to the internet¶
The NAT64 is the fabric's only way out for connections; DNS queries still leave through the
router's resolver, so the variant is no strict air gap. The fabric holds unique local addresses
alone, and the router takes no IPv6 route from the home network, so the router has no IPv6 route beyond the lab's
/48. Its firewall policies forward between the fabric and the tunnel alone. The NAT64's
connections pass ahead of the policies, out of wan from the router's IPv4 address. The router
sends the nodes no BGP routes: their default route from its advertisements leads to the router
and no further.
Without the NAT64, the router has no route to 64:ff9b::/96, so the nodes' connections there fail.
The resolver stays: it answers names from the root servers over wan, as the router's own
connections still leave there, and DNS64 still answers with addresses in 64:ff9b::/96, which
lead nowhere now. CoreDNS keeps a server to forward to. The nodes reach the cluster registry, the
router's time and each other by their addresses, and need no names. In a real air gap the router
has no uplink either, and its resolver answers no name outside the lab. Its chrony then has no
server to follow and answers the nodes as unsynchronised, at stratum 16, which they refuse. Give
the router a reference clock, or add localStratum: 10 to its TimeServer, which serves the
router's own clock.
Two things the nodes take from the internet stop with the NAT64:
| What | Through the NAT64 | In the variant |
|---|---|---|
| Container images | registry.k8s.io, ghcr.io and docker.io |
the cluster registry on the three control planes |
| Time | ptbtime1.ptb.de to ptbtime3.ptb.de |
the router's chrony at fdc8:a1b2:c3d4:1::1, over plain NTP |
The images and the installer need no change: chalkctl reads them from images/ on the workstation
and sends them to the nodes.
Look at the variant's files¶
The example keeps the variant beside the lab's files, so the default inventory stays as it is.
chalkos-air-gapped.yaml is the variant's inventory. Its router leaves out router/nat64.yaml
and takes air-gapped/router.yaml, and its nodes take air-gapped/nodes.yaml as well:
# The home lab without internet access, the optional variant of the guide: the inventory of
# chalkos.yaml, whose router leaves out router/nat64.yaml and whose nodes take the cluster
# registry and the router's time from air-gapped/. chalkctl reads it with
# --inventory chalkos-air-gapped.yaml or CHALKOSINVENTORY=chalkos-air-gapped.yaml, and
# chalkos.yaml's config connects the lab again. Keep the nodes' addresses and the image sources the
# same as there, such as the signed copies, and list every file of router/ but nat64.yaml here too.
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: homelab}
spec:
images:
router-metal: {source: {directory: images/router-metal}}
node-metal: {source: {directory: images/node-metal}}
groups:
router:
imageName: router-metal
files: [router/router.yaml, router/secrets.yaml, air-gapped/router.yaml]
nodes:
imageName: node-metal
files: [cluster/*.yaml, bgp/fabric.yaml, bgp/secrets.yaml, air-gapped/nodes.yaml]
nodes:
router: {address: 192.168.1.2, groups: [router]}
node1: {address: "fdc8:a1b2:c3d4:ff::11", groups: [nodes], files: [nodes/node1.yaml]}
node2: {address: "fdc8:a1b2:c3d4:ff::12", groups: [nodes], files: [nodes/node2.yaml]}
node3: {address: "fdc8:a1b2:c3d4:ff::13", groups: [nodes], files: [nodes/node3.yaml]}
Keep its nodes' addresses and its image sources the same as those of chalkos.yaml, such as the
signed copies of Sign the images. air-gapped/nodes.yaml turns the
cluster registry on, with the three control planes' addresses as its endpoints and the hosts of
the lab's images as its mirrors, and names the router as the nodes' time server:
# What the nodes of the air-gapped lab, chalkos-air-gapped.yaml, get beside the files of
# chalkos.yaml: the cluster registry, from which they pull every container image, and the router
# as their time server.
#
# The control planes serve the cluster registry at their node addresses, and containerd on every
# node looks there first for the images of the mirrored hosts: registry.k8s.io for the control
# plane, CoreDNS and the pod sandbox, ghcr.io for flannel, and docker.io for the workloads, such as
# the guide's busybox. chalkctl registry push puts the images there through the tunnel.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: ClusterRegistry
metadata: {name: cluster}
spec:
mirrors: [docker.io, ghcr.io, registry.k8s.io]
endpoints: ["fdc8:a1b2:c3d4:ff::11", "fdc8:a1b2:c3d4:ff::12", "fdc8:a1b2:c3d4:ff::13"]
---
# The nodes' default time servers lie on the internet. The router keeps its own clock over wan and
# answers NTP from the fabric, as the TimeServer of air-gapped/router.yaml sets it up. Plain NTP,
# as the router's chrony serves no NTS. A router without an uplink answers as unsynchronised,
# which the nodes refuse, unless it has a reference clock or the TimeServer sets localStratum.
apiVersion: time.chalkos.dev/v1alpha1
kind: TimeSync
metadata: {name: default}
spec:
servers:
- {host: "fdc8:a1b2:c3d4:1::1", nts: false}
air-gapped/router.yaml makes the router the fabric's time server:
# What the router of the air-gapped lab, chalkos-air-gapped.yaml, gets beside router/router.yaml
# and router/secrets.yaml. It leaves out router/nat64.yaml, so the fabric reaches no internet,
# and answers the nodes' NTP: the router's chrony serves the time it keeps over wan, and the
# firewall accepts NTP from the lab's addresses on the fabric's first address alone. A router
# without an uplink would need `localStratum: 10` here to serve its own clock, which drifts from
# the real time; an air gap that needs accurate time wants a reference clock instead.
apiVersion: router.chalkos.dev/v1alpha1
kind: TimeServer
metadata: {name: ntp}
spec:
listen: ["fdc8:a1b2:c3d4:1::1"]
allow: ["fdc8:a1b2:c3d4::/48"]
The TimeServer lets the router's chrony answer the lab's addresses, and the firewall accepts NTP
from them on the fabric's first address alone. The router keeps following the internet's time
servers over wan, so it answers as synchronised. The image needs no change: router images carry
the time server, as Serve time with NTP describes.
Copy the images to the workstation¶
The workstation keeps its own internet access, so it fetches the images. Copy every image of the
Cluster document into an OCI image layout of its own with crane, and busybox, which the checks
below run:
mkdir -p images/oci
for ref in $(yq 'select(.kind == "Cluster") | .spec.images[]' cluster/cluster.yaml); do
repo=${ref%%@*}; name=${repo##*/}; name=${name%%:*}
rm -rf "images/oci/$name"
crane pull --format=oci "$ref" "images/oci/$name"
done
rm -rf images/oci/busybox
crane pull --format=oci docker.io/library/busybox:1.37.0 images/oci/busybox
The copies keep every platform of each image's index and take about 1.2 GiB. Bring the images into the air gap explains why the loop uses crane and not skopeo.
Cut the fabric off¶
Run chalkctl with the variant's inventory for the rest of this section:
Have the router check its new config first. The dry run prints the documents that change, the
NAT64 removed and the TimeServer added, and the outputs and effects that follow:
Deliver it on trial, because it changes the router's firewall, and then the nodes' config:
chalkctl config apply router --secrets=secrets.age --try 5m
chalkctl config apply node1 node2 node3 --secrets=secrets.age
The router stops its NAT64 and answers NTP from the fabric. Each control plane starts the cluster
registry at its node address, and every node's containerd looks there first for the images of
registry.k8s.io, ghcr.io and docker.io.
Push the images¶
Push every image to the three endpoints through the tunnel:
for ref in $(yq 'select(.kind == "Cluster") | .spec.images[]' cluster/cluster.yaml); do
repo=${ref%%@*}; name=${repo##*/}; name=${name%%:*}
chalkctl registry push "images/oci/$name" "$ref" --secrets=secrets.age
done
chalkctl registry push images/oci/busybox docker.io/library/busybox:1.37.0 --secrets=secrets.age
Each push prints the image's digest and a line for each endpoint, such as those of the API server's image:
pushing registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4 as registry.k8s.io/kube-apiserver@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
fdc8:a1b2:c3d4:ff::11: pushed [fdc8:a1b2:c3d4:ff::11]:5000/registry.k8s.io/kube-apiserver:v1.37.2 as sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
fdc8:a1b2:c3d4:ff::12: pushed [fdc8:a1b2:c3d4:ff::12]:5000/registry.k8s.io/kube-apiserver:v1.37.2 as sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
fdc8:a1b2:c3d4:ff::13: pushed [fdc8:a1b2:c3d4:ff::13]:5000/registry.k8s.io/kube-apiserver:v1.37.2 as sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
Check that every endpoint holds every image:
$ chalkctl registry list --secrets=secrets.age
REPOSITORY TAG DIGEST ENDPOINTS
docker.io/library/busybox 1.37.0 sha256:a7dbc319b143 all
ghcr.io/flannel-io/flannel v0.28.9 sha256:708a2c9c1cfb all
registry.k8s.io/coredns/coredns v1.14.6 sha256:900f9c109f7a all
registry.k8s.io/etcd 3.7.0-0 sha256:6ecefbe2510c all
registry.k8s.io/kube-apiserver v1.37.2 sha256:3e563afb5cb5 all
registry.k8s.io/kube-controller-manager v1.37.2 sha256:28e62daf706e all
registry.k8s.io/kube-proxy v1.37.2 sha256:b2e075697d7a all
registry.k8s.io/kube-scheduler v1.37.2 sha256:028c847fc321 all
registry.k8s.io/pause 3.10.2 sha256:f548e0e8e3dc all
Bootstrap and check the cut¶
On a lab that waits for its bootstrap, bootstrap node1 and write a kubeconfig, as in Bootstrap Kubernetes. The control planes pull their images from the cluster registry. Then check that the nodes run and that a pod pulls busybox from the registry:
Check that the fabric no longer reaches the internet. The tcp-connect diagnostic to
64:ff9b::101:101, which connects on the connected lab, fails now,
and chalkctl status router shows no nat64: table:
chalkctl node diag node1 --secrets=secrets.age tcp-connect address=64:ff9b::101:101 port=443
chalkctl status router --secrets=secrets.age
chalkctl status node1 --secrets=secrets.age prints a line time: that names
fdc8:a1b2:c3d4:1::1, the router, as the server the node's clock follows.
A pod whose image is not in the registry stays in ImagePullBackOff. Copy and push its image as
above;
If something goes wrong covers other
failures.
Connect the lab again¶
Deliver the config of chalkos.yaml to the router and the nodes:
unset CHALKOSINVENTORY
chalkctl config apply router --secrets=secrets.age --try 5m
chalkctl config apply node1 node2 node3 --secrets=secrets.age
The router runs its NAT64 again, the nodes take their time from the internet, and containerd pulls
from the registries themselves. The pushed images stay on the control planes, in
/var/lib/chalkos-registry, and the registry serves them again when the variant's config returns.
Switch an existing lab to advertised gateways¶
If the nodes still take their default route over BGP, apply the new bgp/fabric.yaml to the nodes
one at a time while the router still sends a BGP default. Try the node config with
chalkctl config apply node1 --secrets=secrets.age --try 5m --no-confirm. Use
chalkctl node diag node1 --secrets=secrets.age routes family=inet6 to check for an IPv6
default route with protocol ra on each connected fabric port. Check management and internet
access before keeping the trial with chalkctl config confirm node1 --secrets=secrets.age.
After every node uses an advertised gateway, apply the router config to stop announcing the BGP
default.
Troubleshooting¶
A node's port gets no address of the fabric¶
The router's advertisements give the ports their addresses. Check that the router's fabric port
holds fdc8:a1b2:c3d4:1::1 with the addresses diagnostic, as in
Check the router. When fabric is missing and a port of the kernel's name
shows instead, the MAC address of the router's Link document is not the port's: correct it, and
deliver it with chalkctl config apply router --try, which renames the port at once. A node port
that shows DOWN has no carrier on the switch. An installed node configures only the ports that
bgp/fabric.yaml names: a port of another name takes no address, so put the names the addresses
diagnostic shows into bgp/fabric.yaml, as in Adapt the node's file.
A BGP session does not come up¶
The router's bgp: table shows no line for a port that never connected. Without a session the
router has no route to the node's address, so chalkctl does not reach the node there. The node
still answers at its ports' addresses of the fabric, through its advertised default gateway.
The router's neighbours diagnostic lists them, as the tip "Find the installer through
the router" in Install the nodes shows; pass one with --address:
chalkctl node diag node1 --secrets=secrets.age \
--address=fdc8:a1b2:c3d4:1:0:ff:fe00:1101 bgp-summary
Check, on the node:
- the ports' names: the
addressesdiagnostic shows the names that theBGPPeerdocuments ofbgp/fabric.yamllist,enp1s0andenp2s0in the example. FRR runs no session on a port of another name; - the password: the kernel drops TCP segments whose MD5 signature does not match, so neither side
logs a refused session. The router has no
bgp:line for the node, and on the nodechalkctl node diag node1 bgp-summaryshows the sessions inConnectorActiveinstead of a count of received prefixes. Compare the passwords ofrouter/secrets.yamlandbgp/secrets.yaml, which must be the same.
The router's firewall accepts BGP and BFD from the fabric by itself, and the nodes' image opens BFD's port.
A name does not resolve, or a connection to the internet fails¶
On the router, chalkctl status router shows resolver: running and the NAT64's line. The
diagnostics dns-query and nat64-sessions show the resolver's answers and the NAT64's
connections, as Translate to IPv4 and resolve names
describes. A NAT64 connection needs the router's wan address and default route from the home
network's DHCP server.
The router takes another address on the home network¶
The router identifies itself to the DHCP server with a DUID made from the wan port's MAC
address, the same at every boot, as the installer does for each port. Check that the reservation
names the MAC address that the addresses diagnostic shows for wan. A DHCP server that matches
a reservation against a client identifier of the bare MAC address finds none in a DUID. Have the
router send its MAC address as the identifier, in the wan port's Network of
router/router.yaml:
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-wan}
spec:
match: {name: wan}
network: {dhcp: ipv4, ipv6AcceptRA: false}
dhcpv4: {clientIdentifier: mac}
Until the router has the reserved address, put the address it took, which its console shows, into
chalkos.yaml as the router's.
The tunnel has no handshake¶
The wireguard diagnostic shows no latest handshake for the workstation's peer. Check that the
WireGuardPeer holds the contents of workstation.pub, that the workstation's config holds the
router's public key from router.pub, and that its Endpoint is the router's address on the home
network. After a delivery to the router, the tunnel works again once the workstation sends;
PersistentKeepalive makes it send within 25 seconds.
What next¶
- Status, logs and the console reads the rest of
chalkctl status. - Run a router covers the router's firewall, BGP, NAT64 and WireGuard.
- Run a highly available control plane explains the VIP's election, and how to replace a control plane.
- Announce addresses with BGP covers the nodes' BGP speaker: its peers, the default route, its diagnostics and the deliveries it refuses.
- Declare a config kind follows the BIRD extension example, which
declares the
SpeakerandPeerkinds.
These pages explain the steps of this tutorial in more depth:
- Generate the cluster's secrets with age and Keep the services' secrets in Secret documents explain the secrets file, its backup and the Secret documents.
- Sign images for Secure Boot and Enrol Secure Boot keys cover the keys, a failed enrolment and the cards' option ROMs.
- The console on the machine's screen describes what the installer shows.
- Install a node covers a disk that holds data,
and Write a node's config the output of
chalkctl validate. - Run WireGuard covers more peers, preshared keys and key rotation.
- Bootstrap and get a kubeconfig explains each step of the bootstrap.
- Route with BGP explains the columns of the
router's
bgp:table and the BGP paths that follow the kernel's routes.