Install a node¶
This page is for an operator who turns a physical machine into a node of the inventory. At the end the node runs its image from
its system disk with its config, and chalkctl reaches it at the address the inventory names. The
page installs cp1 of the cluster-metal template, as Write a node's config
describes it. Build a cluster on real machines installs all
three of the template's machines.
Install a node in a VM runs a node under QEMU, libvirt or Proxmox instead.
In short:
nix build .#installer -o installer
chalkbuild sign --key keys/db.key --cert keys/db.crt installer installer-signed
chalkctl disks --address=<address> --fingerprint=<fingerprint> --secrets=secrets.age
chalkctl install cp1 --secrets=secrets.age --address=<address> --fingerprint=<fingerprint>
chalkctl status cp1 --secrets=secrets.age
Before you begin¶
You need a workstation set up as Set up your workstation describes: chalkctl,
the repository with the inventory, the nodes' config files, the image directory and the secrets
file secrets.age. chalkctl validate accepts every node's config.
The image and the installer carry the cluster's OS CA, as Build the images with the cluster's OS CA describes. A machine in maintenance mode then accepts only a chalkctl that holds the secrets file or a client file of this cluster.
Decisions before you install lists the choices to make before the first install, such as whether the machines enforce Secure Boot.
Each machine needs UEFI firmware, Secure Boot and a TPM 2.0. The requirements list the rest. For a machine that enforces Secure Boot, you also need:
- Secure Boot keys, with the db certificate enrolled in the machine's firmware.
Sign images for Secure Boot creates them in
keys/and explains the enrolment. A machine whose firmware is in Setup Mode gets them fromchalkctl secureboot enrollbefore the install; - chalkbuild on the machine that holds
keys/db.key, fromnix shell github:trevex/chalkos#chalkbuildor the cluster flake's development shell.
A physical machine boots the installer from a USB stick larger than the installer ISO, about 450 MiB, or from a BMC that mounts an ISO as virtual media.
Installing without Secure Boot
The steps work on machines with Secure Boot off: leave out the signing, and keep the inventory on the unsigned image directory. Anyone who holds an operator certificate can then run an image of their own on the node, as Secure Boot limits a node to images the key holder signed explains.
A machine passes through maintenance mode once: it boots the installer, which has no config,
chalkctl install gives it its config and secrets, and it reboots as an installed node in normal
mode. A node passes through stages
shows the whole life of a node.
Sign the image for machines that enforce Secure Boot¶
Skip this step on a machine that does not enforce Secure Boot: it takes the unsigned
images/node-metal as it is. The firmware of a machine that enforces Secure Boot boots only a
boot loader and a UKI signed by a certificate in its db. Sign a copy of the build with
chalkbuild sign, which also checks the copy against
db.crt:
$ rm -rf images/node-metal-signed
$ chalkbuild sign --key keys/db.key --cert keys/db.crt images/node-metal images/node-metal-signed
signed images/node-metal-signed/chalkos.raw
The output directory must not exist, so rm -rf removes the copy of an earlier build. Point the
inventory's image at the signed copy, as
Point the inventory at the signed copy
shows, and sign every new build again before you install or upgrade with it.
Boot the machine into maintenance mode¶
A physical machine boots the installer. Build the installer from the repository's directory:
$ nix build .#installer -o installer
$ ls installer/
chalkos-installer.iso chalkos-installer.iso.json chalkos-installer.raw metadata.json repart-output.json
The .iso boots from a CD, from virtual media and written to a USB stick. The .raw is the same
system as a plain disk image, for machines that boot better from one. The installer takes its
addresses by DHCP and SLAAC; Customise the installer gives it
static addresses, VLANs, bonds or extra drivers. Use the cluster's installer: the generic one of
the chalkos flake, github:trevex/chalkos#installer, has no OS CA and accepts any client.
Sign a copy of the installer's directory, its raw image and its ISO, with the same key as the images:
rm -rf installer-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt installer installer-signed
An image signed with another key than the installer boots, but its first boot asks for the recovery key, as Sign the installer explains.
Write the signed ISO to the whole stick, not to a partition of it:
sudo dd if=installer-signed/chalkos-installer.iso of=/dev/<usb-disk> \
bs=4M conv=fsync status=progress
<usb-disk> is the stick's device, for example sdb; lsblk lists the candidates. The
command overwrites everything on the stick. With a BMC, mount
installer-signed/chalkos-installer.iso as virtual media instead.
Boot the machine from the stick through the firmware's boot menu. The screen shows the boot's status and then the console. The first serial port, which a BMC's serial-over-LAN shows, gets the kernel's and chalkd's lines.
Note the address and the fingerprint¶
Once chalkd starts in maintenance mode, the machine's screen shows the console. Its header keeps the SHA-256 fingerprint of chalkd's self-signed certificate and the machine's addresses on screen, and follows the addresses when they change:
nixos maintenance up 2m
image 81bd81bd81bd
maintenance mode: chalkctl install installs this node
fingerprint <fingerprint>
addresses 192.168.1.50
<fingerprint> is 64 hex characters. Note it and the address. A reboot into maintenance mode
shows a new fingerprint. chalkd prints both on the serial console too, as
A node in maintenance mode is trusted by its fingerprint
shows. When its first line there says accepting any client, the installer or image has no OS CA:
it is the generic installer, or the flake's mkImages has no osCA.
In maintenance mode the machine takes its addresses by DHCP, and by SLAAC where a router sends
advertisements. Such an address is not yet the one the inventory names, so the commands below
reach it with --address.
Enrol the Secure Boot keys¶
Skip this step when the machine's firmware holds your db certificate already, or when the machine
runs without Secure Boot. A machine whose firmware is in Setup Mode boots the signed installer
without checking it and gets its keys from
chalkctl secureboot enroll, at the address
and fingerprint from the console:
chalkctl secureboot enroll --keys keys --address=<address> --fingerprint=<fingerprint> \
--secrets=secrets.age
The node writes the keys and reboots with Secure Boot on. It boots with a new certificate, so
chalkctl prints a new fingerprint: compare it with the console and use it from here on.
Enrol the keys in Setup Mode
explains the checks, --microsoft and --option-roms, and what to do when the enrolment fails.
Choose the system disk¶
The installer writes only to the node's system disk: spec.disk of the SystemDisk document in
the node's file. List the machine's disks with
chalkctl disks, at the address and fingerprint from the
console:
--secrets gives chalkctl the certificate that the machine in maintenance mode asks for. The
output lists each disk with the properties a selector matches, as
Find the disks shows. The installer's own medium shows the use
boot.
Name the disk by a /dev/disk/by-id/ path or a selector, because device names such as
/dev/sda can change between boots. Here cp1 names its NVMe disk by its by-id path:
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
disk: {path: /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_2TB_S7KHNJ0W100001}
chalkctl reads the file as it is when a command runs. After a change, check it again with
chalkctl validate cp1.
Install the node¶
Run chalkctl install with the node's name, the address
and the fingerprint:
When reading the console is impractical, --insecure accepts any certificate and prints
chalkctl: the node's certificate fingerprint is <fingerprint>. Compare it with the console
afterwards. chalkctl sends the node its config, certificates and secrets, with the Kubernetes layer
its Kubernetes share
too. It refuses a node that is installed already, with
cp1 at <address> is installed already.
With the Kubernetes layer, a revoked name is refused
chalkctl asks the other control planes of the inventory for the revoked nodes, and refuses a
name that chalkctl node revoke revoked, with
cp3 is revoked since <time>, and install installs cp3 only once chalkctl node unrevoke cp3
lets it back in, or with --force.
It also refuses while control planes were bootstrapped and none says which nodes are revoked.
A control plane that does not answer as an installed node, such as one in maintenance mode or
switched off, counts as not installed yet, so the cluster's first nodes install while the
others wait. --force installs without asking.
Reuse a removed node's name frees the name.
The installer gets the image as well, from the directory of the node's image in the inventory or
the one --image names. chalkctl refuses a directory whose raw image is not the build its
metadata.json names, and sends the store, hash tree, UKI and boot loader. The output below is
illustrative: its lines are chalkctl's, and the disk, the image hash and the size are an
example's.
installing cp1 onto /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_2TB_S7KHNJ0W100001 from /home/alice/cluster/images/node-metal-signed: chalkos 5f1e60980b55, sending 311717904 bytes of store, hash tree, UKI and boot loader
cp1 is installed and reboots
chalkctl recovery-key cp1 prints the key that unlocks it when its TPM fails
The first line names the disk, the image directory, the image's ID chalkos and the image's name,
whose form Status, logs and the console
explains.
When Secure Boot is enforced, the installer checks the UKI and the boot loader against the
machine's db and dbx, and refuses an unsigned or wrongly signed image. It writes the image to the
system disk, and the image checks the config with its own checker, as
An image that does not run checks its own config
describes. Then the installer sets up STATE, VAR and the volumes, and adds a UEFI boot entry named
chalkos, as
The install in code
follows step by step. A refused config leaves the machine in maintenance mode, as
If it fails describes.
The boot entry goes first in the boot order, so the stick may stay in. Remove it before the next
install elsewhere. When the inventory names the node's image in a registry, the installer pulls it,
and the first line ends in which the installer pulls from <reference>.
Install from the registry covers the
machine's route to the registry and its credentials.
The machine reboots into its image. The node then runs in normal mode with the network of its
config, and chalkctl reaches it at the address the inventory names. Keep that address the same as
the static address of the node's Network, because chalkctl never reads it from the node's
config.
Continue an install that stopped¶
An install that stops halfway, because chalkctl was interrupted, the network dropped or the machine
lost power, continues where it stopped. Run the same chalkctl install command again, with the
fingerprint the console prints now. The installer logs continuing the install on <disk> on the
console and skips a slot that holds the image already. Until the boot loader is written, the disk
boots nothing.
An install with another image that lays the disk out the same way continues on it too, while
STATE holds neither an installed node nor the earlier image's config, as after a refused config.
The installer writes the new image over slot A and removes the earlier image's UKI, whatever its
image ID, as Files on the ESP describes. Once an
earlier attempt wrote the node's config to STATE, another image needs --wipe-disk.
Replace what a disk holds¶
The installer takes a disk on which blkid finds no signature or only an empty partition table,
or one that an earlier attempt of this install left. It refuses any other disk and names what it
found:
the target disk /dev/nvme0n1 (model "...", size ..., serial "...") carries data (dos); pass --wipe-disk to replace it
the target disk /dev/nvme0n1 (model "...", size ..., serial "...") is not one an earlier install of this image left: ...; pass --wipe-disk to replace it
the target disk /dev/nvme0n1 (model "...", size ..., serial "...") holds an installed node; pass --wipe-disk to replace it
slot A of the target disk /dev/nvme0n1 (model "...", size ..., serial "...") holds the store 3df89200aef8615c of another image, whose config STATE holds; pass --wipe-disk to replace it
--wipe-disk destroys the disk's data
With --wipe-disk, the installer replaces whatever the disk holds, an installed node
included, and nothing on it can be recovered. Check the disk in chalkctl disks first. A
disk on which blkid finds no signature counts as empty without the flag, even when it
holds data.
Add and remove nodes covers reinstalling a node that is part of the cluster.
Keep the recovery keys at hand¶
Each node's encrypted volumes have a second keyslot, by default its recovery key, which unlocks
them when the TPM does not. chalkctl recovery-key
derives the key from the secrets file:
No file stores the keys, and installing never changes the secrets file, so a backup of the secrets file backs up every recovery key. Keep the nodes' keys where you can read them at a node's console, such as in a password manager. The console asks for a passphrase at boot covers the prompt.
Install the other nodes¶
Install the remaining nodes the same way. Boot each into maintenance mode, note the address and
the fingerprint, choose the disk and run chalkctl install.
With the Kubernetes layer, only the first control plane needs a bootstrap
Once cp1 is back up in normal mode, Bootstrap and get a kubeconfig
starts the cluster on it. The other nodes need no bootstrap. Workers join at the cluster
endpoint with the kubelet certificate chalkctl gave them. Further control planes join etcd
and the control plane on their own, as
Run a highly available control plane describes.
Check that it worked¶
chalkctl status shows the node's config, image, platform,
volumes, certificates and failed units, as
Status, logs and the console explains:
The node shows its volumes as mounted and no failed units.
If it fails¶
A config that the image's checker refuses ends chalkctl install with the image's message, such as
that of the homelab example's validator bgp-peers from
Validate config with a program, on its node node1:
chalkctl: install node1: failed_precondition: install: the config is refused: Peer/router: spec.interfaces[2]: enp3s0 has no Network document; bird binds the session to an interface the node configures; the target disk boots nothing, and an install of the image continues on it
The machine stays in maintenance mode. Fix the config and run the same chalkctl install again: the
install continues on the disk without --wipe-disk, also with an image rebuilt with a fixed
validator. Build the installer from the same chalkos release as the image, so that both lay out
the storage the same way. Otherwise the installer refuses the config once the image's store is on
the disk, names the parts of the storage that differ, and leaves a disk that boots nothing. Then
install with the installer built with the image, with --wipe-disk if it refuses the disk's
layout.
chalkctl reports the installer's other refusals as its error. The common errors and symptoms:
| Message or symptom | Cause and fix |
|---|---|
the maintenance image trusts another OS CA than the secrets file, so it refuses chalkctl's certificate |
The installer or the image was built before the OS CA was rotated, or from another cluster's secrets.pub.json. Build and sign both again from the current one. |
Secure Boot would refuse the UKI: ... or Secure Boot would refuse the boot loader: ... |
The image is unsigned, or signed with a key whose certificate is not in this machine's db. Sign the build again with chalkbuild sign and the enrolled key, or enrol the key's certificate. |
under the new db, Secure Boot would refuse the drivers the firmware loaded ... from chalkctl secureboot enroll |
The machine's cards carry option ROMs that db would not allow under the new keys. Enrol their digests with --option-roms or Microsoft's CAs with --microsoft, as Allow the cards' option ROMs explains. |
STATE on the target disk ... does not open on this machine, so it belongs to another node |
The disk was installed in another machine. Use --wipe-disk if its data may go. |
| The first boot asks for a recovery key | The installer and the image were signed with different keys, or Secure Boot changed between install and boot. Type the key that chalkctl recovery-key cp1 prints. |
The console asks for a passphrase at boot explains the recovery-key prompt and how to make it go away. Troubleshooting covers a node that does not come up after the reboot.
What next¶
- Add and remove nodes grows or shrinks the cluster later.
- The install in code follows an install step by step through chalkctl, chalkd and the disk.
- Sign images for Secure Boot covers the key's life and why it must not change for an installed node.
With the Kubernetes layer, continue at Bootstrap and get a kubeconfig.