Write a node's config¶
This page is for an operator who describes the cluster's nodes before installing them. At the
end each node of the inventory has its config files, with its system disk, network, time servers
and volumes, and chalkctl validate accepts them. The page follows the
cluster-metal template:
the three Kubernetes control planes cp1, cp2 and cp3, on one image. Where it needs groups
or a second image, it shows the
homelab example, a router beside
three Kubernetes nodes.
Before you begin¶
You need a workstation set up as Set up your workstation describes: chalkctl,
the repository and the image directory the inventory names. chalkctl validate checks the config
against that image, so the image author builds it first.
You need to know each machine's disks and network interfaces. On a machine that runs the
installer, chalkctl disks lists the disks, as
Choose the system disk shows.
Give each node its files in the inventory¶
A node's config is the set of YAML documents that makes it this node. The inventory names the
files each node's documents come from. The template's gives every node the files of cluster/, and
each node its own file in nodes/:
# The cluster's three nodes, the address chalkctl reaches each at, the image they run and their
# config files: every node gets cluster/, each its own file in nodes/. Replace the addresses with
# free ones of your network, here and in the nodes' files.
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: cluster}
spec:
images:
# nix build .#node-metal -o images/node-metal puts the image here.
node-metal: {source: {directory: images/node-metal}}
all: {imageName: node-metal, files: [cluster/*.yaml]}
nodes:
cp1: {address: 192.168.1.11, files: [nodes/cp1.yaml]}
cp2: {address: 192.168.1.12, files: [nodes/cp2.yaml]}
cp3: {address: 192.168.1.13, files: [nodes/cp3.yaml]}
The inventory's metadata.name, cluster here, is a label for the cluster. It reaches no node:
chalkctl uses it in its messages and as the default name of a kubeconfig's context.
The inventory's spec has four parts:
spec.imagesnames each image and the directory of its build, relative to the inventory's directory.spec.allis what every node gets: the imageimageNamenames, and the config filesfilesnames. The template gives every node the imagenode-metaland the files ofcluster/.spec.groupsholds named groups of nodes, each with its files and its image. The template has none, and the homelab example two,routerandnodes.spec.nodesholds each node by its name, with theaddresschalkctl reaches its chalkd at, as a host orhost:port, and its own files.
The key of spec.nodes, such as cp1, is the node's name. Install writes it into the node's
certificate. The hostname defaults to it, and the recovery key and the seeds of the partition
UUIDs derive from it. Renaming a node therefore takes a reinstall.
address is where chalkctl reaches the node, also in maintenance mode before the node has any
config. chalkctl never reads it from the node's Network documents. A node that installs with a
DHCP address and then gets a static one names the static one here, as the template's nodes do.
--address overrides it for one command.
Here is cp1's own file, nodes/cp1.yaml:
# cp1: a control plane with one Ethernet port on the network. Replace the disk with the node's own,
# by its path below /dev/disk/by-id, and the MAC address with its port's, which chalkctl disks and
# the diagnostic addresses show while the machine waits in the installer.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
type: controlplane
---
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
disk: {path: /dev/disk/by-id/nvme-EXAMPLE_SSD_cp1}
---
# The port's static address, the one chalkos.yaml names for cp1, its gateway and its DNS server.
# The node holds the VIP on this port while it is the VIP's holder.
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-lan}
spec:
match: {macAddress: "02:00:00:00:00:11"}
network: {address: [192.168.1.11/24], gateway: 192.168.1.1, dns: [192.168.1.1]}
Each document has an apiVersion with its group and version, a kind, a metadata.name and a
spec. A kind of which a node holds one takes any metadata.name; the examples use
default or a short name such as system.
Keep each document in one file¶
Documents never merge, so a node's set must hold each document once. Two documents of the same kind and name in one node's set are an error that names both files:
chalkctl: the config of 3 nodes is invalid:
node cp1: cluster/net.yaml and nodes/cp1.yaml both hold Network "10-lan"
node cp2: cluster/net.yaml and nodes/cp2.yaml both hold Network "10-lan"
node cp3: cluster/net.yaml and nodes/cp3.yaml both hold Network "10-lan"
Put a value that every node shares into a shared file once. When one node needs a different value, that node's own file holds its own document of that kind, and the shared file leaves the kind out. The inventory gives each node a set of files explains the order in which a node's set takes its files.
Share files and an image through a group¶
A group holds what some of the nodes share. The template needs none, because its nodes share one
image and one set of files. The homelab example has two. Its group router gives its one node
the image router-metal and the files of router/. The group nodes gives the three Kubernetes
nodes the image node-metal and the files of cluster/ and bgp/, and each of them names its
own file beside the group.
A node runs the image named in the first of three places:
- the node's own
imageName; - the
imageNameof its groups; spec.all.imageName.
The diagram shows how the homelab's inventory gives each node its files and its image:
flowchart LR
routerg["group router: router/*.yaml"] --> router["router's set"]
nodesg["group nodes: cluster/*.yaml, bgp/"] --> node1["node1's set"]
nodesg --> node2["node2's set"]
nodesg --> node3["node3's set"]
node1f["nodes/node1.yaml"] --> node1
node2f["nodes/node2.yaml"] --> node2
node3f["nodes/node3.yaml"] --> node3
routermetal["router-metal"] -->|the group's imageName| router
nodemetal["node-metal"] -->|the group's imageName| node1
nodemetal -->|the group's imageName| node2
nodemetal -->|the group's imageName| node3
The router gets the files of its group and runs router-metal. Each Kubernetes node gets the files
of nodes, then its own, and runs node-metal.
chalkctl refuses the inventory when a node is in two groups that name different images, also when
the node names its own. It also refuses a node that gets no image, and an imageName that names
no image of spec.images:
The image stays out of the node's config. A new build of an image, or a node's move to another image, changes no node's config.
Choose the system disk and the size of VAR¶
A SystemDisk names the disk that holds the system region and VAR, the partition for the node's
data. Every node holds exactly one. The template's take the whole disk for VAR. This one gives VAR
200 GiB and leaves the rest of the disk to volumes:
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
disk: {path: /dev/disk/by-id/nvme-EXAMPLE_SSD_cp1}
var: {size: 200G}
Name the disk by a path below /dev/disk/by-id, which stays the same across boots, or by
selectors such as model, serial or size. Without var.size, VAR fills the rest of the
disk. By default STATE, VAR and every volume are encrypted with a key sealed to the TPM, with the
node's recovery key as the second keyslot. Add storage volumes covers the
disk selectors, the encryption and the sizes.
Configure the interfaces¶
A node's network is a set of Network, NetDev and Link documents. Each is one
systemd-networkd unit file: a .network, a .netdev or a .link file. The document's
metadata.name is the file's name without its suffix, so networkd orders the files by it. Its
spec holds networkd's sections and keys in camelCase: [Match] with MACAddress= is
match: {macAddress: …}.
The template's nodes give their port a static address, a gateway and a DNS server, and match the port by its MAC address:
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-lan}
spec:
match: {macAddress: "02:00:00:00:00:11"}
network: {address: [192.168.1.11/24], gateway: 192.168.1.1, dns: [192.168.1.1]}
The image brings one unit of its own, 99-ethernet-default-dhcp, which runs DHCP on every
physical Ethernet interface that no Network document matches. A node whose config holds no
Network document therefore runs DHCP on all of them. The documents' units take precedence over
the image's units.
A bond, a VLAN and a dummy interface are NetDev documents, each
with the Network documents that attach it or configure it. On this machine, two ports form an
LACP bond, and the machine's address is on VLAN 20 of the bond:
apiVersion: networkd.chalkos.dev/v1alpha1
kind: NetDev
metadata: {name: 10-bond0}
spec:
netDev: {name: bond0, kind: bond}
bond: {mode: 802.3ad, transmitHashPolicy: layer3+4, miiMonitorSec: 100ms, lacpTransmitRate: fast}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: NetDev
metadata: {name: 20-vlan20}
spec:
netDev: {name: vlan20, kind: vlan}
vlan: {id: 20}
---
# The two ports join the bond and get no address of their own.
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-bond0-ports}
spec:
match: {name: enp1s0 enp2s0}
network: {bond: bond0}
---
# The bond carries the VLAN and no address of its own.
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 20-bond0}
spec:
match: {name: bond0}
network: {vlan: vlan20, linkLocalAddressing: "no"}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 30-vlan20}
spec:
match: {name: vlan20}
network: {address: [192.0.2.21/24], gateway: 192.0.2.1, dns: [192.0.2.1]}
The NetDev documents create the devices bond0 and vlan20. The Network documents attach
the ports to the bond and the VLAN to the bond, and give the VLAN device the node's address. The
switch ports must form an LACP group too.
A Link document is a .link file, which sets the properties of a physical device, such as its
name or its MTU. The homelab's router names its ports by permanent MAC address, so that its other
documents name the ports wan and fabric, whatever the kernel calls them:
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Link
metadata: {name: 10-fabric}
spec:
match: {permanentMACAddress: "02:00:00:00:01:02"}
link: {name: fabric}
The installer holds no config, so a machine's ports keep the kernel's names until the installed node's first boot, which renames them. The template's nodes keep the kernel's names and match their port by its MAC address instead.
The schemas of the three kinds come from the networkd of the image's own systemd, so they accept exactly the sections and keys that networkd accepts. Config kinds lists them, and Networking describes how a node applies them.
A node applies its Link documents at every boot, before networkd brings the links up. A
delivery applies a changed Link document to the running ports at once, as far as their drivers
allow, and that includes a new name. Every document that names the port must change in the same
delivery: the match of its Network, and the interfaces of a VRF, a BGP peer or a firewall zone.
Deliver a rename as a trial, with
chalkctl config apply --try, so a
mistake that cuts the node off rolls back. The rollback renames the port back when the previous
config names it in a Link document; otherwise the port keeps the new name until the next boot.
An added alternative name or setting also takes effect at once, and a removed one at the next
boot.
A VLAN takes the MAC address of its port, and the ports of a bond take the bond's, so a Link
document that matches by macAddress can match the wrong device. Match a port by
permanentMACAddress, which only the physical device has.
Set the hostname¶
The hostname is the node's name unless a Host document gives another:
The Host document changes the hostname alone. The node's name, its certificate and, with the
Kubernetes layer, its Kubernetes node name stay as the inventory gives them.
Name the time servers¶
Nodes check certificates against their clocks. Without a TimeSync document, a node synchronises
its clock with three servers of the German national metrology institute, PTB, authenticated with
NTS. NTS needs the nodes to reach the servers on TCP port 4460 as well as UDP port 123. A network
without outbound NTP names its own servers, in a file every node gets:
apiVersion: time.chalkos.dev/v1alpha1
kind: TimeSync
metadata: {name: default}
spec:
servers:
- host: ntp1.example.internal
- host: ntp2.example.internal
nts: false
A server authenticates with NTS unless nts: false. dhcp: true also takes the servers that
DHCP announces, which nothing authenticates.
Add storage beside VAR¶
A Volume is a further partition, on the system disk or on a disk of its own. This one takes a
SATA SSD, which a selector finds by its model and type, for a storage system's replicas:
apiVersion: storage.chalkos.dev/v1alpha1
kind: Volume
metadata: {name: replicas}
spec:
disk: {model: "Samsung SSD 870*", type: ssd}
format: xfs
mountPoint: /var/lib/replicas
A node may hold several volumes, each with a name of its own. Some later changes to a volume take a reset or a reinstall. Add storage volumes covers the formats, the encryption and which changes a node accepts.
Keep secret values in Secret documents¶
A Secret document holds secret values by key in spec.data. A field of a kind, built in or an
extension's, references one of its keys instead of holding the value. The homelab's BGPPeer
documents in bgp/fabric.yaml do so with password: {secretRef: {name: bgp, key: password}}, and
bgp/secrets.yaml holds the Secret named bgp. chalkctl refuses a set in which a reference names a Secret or a key that
the set lacks.
Secret documents are config, separate from the cluster's secrets file.
Keep the services' secrets in Secret documents
says where to keep the files that hold them.
With the Kubernetes layer, every node also holds a Cluster and a Node document
An image with the Kubernetes layer runs Kubernetes. Each node's set then holds exactly one
Cluster document, the same on every node, and one Node document with the node's type,
controlplane or worker. The template puts the Cluster in cluster/cluster.yaml and each
Node in the node's own file.
Configure Kubernetes covers both.
Check that it worked¶
chalkctl validate builds each node's set and checks it against the schemas of the node's image,
as chalkd will. It reaches no node and needs no credentials, so it also runs in CI:
$ chalkctl validate
cp1: valid for the image node-metal (node-metal on metal, root hash 98bb85096ed9), config 91a07f018b112dd92925b568babd16a428cee713fbb9e9c393ac11a2f4f5976e
cp2: valid for the image node-metal (node-metal on metal, root hash 98bb85096ed9), config 138189d54c5a0f2d15e197b349f23afbe71ded1b6179bde77bfdaf63a7ed93d7
cp3: valid for the image node-metal (node-metal on metal, root hash 98bb85096ed9), config 52a62aa28b05f78a15e09f984db2709f79a89fe79a87a87c3d8a9066f0d8e7d5
Each line names the node's image, its own name and platform, the first 12 hex characters of its
root hash, and the hash of the node's config. chalkctl status later compares that hash with the
config the node runs. chalkctl validate cp1 checks cp1 alone.
validate checks that each set:
- holds exactly one
SystemDisk, and every other kind at most once, unless the kind allows several; - holds only kinds that the node's image declares;
- holds no two documents of a kind with the same name;
- names only
Secretdocuments and keys that it holds; - passes each kind's schema, and the checks that chalkd runs when it translates the set, such as a volume that does not fit its disk.
It also runs the change rules of the image that read no facts about the node. It warns of a node
whose SystemDisk leaves STATE unencrypted.
validate checks no signature, so CI needs no db key. When the inventory names a signed copy, a CI
job puts the unsigned build of the same image in that directory instead.
If validate refuses the config¶
validate names each node whose config is invalid and exits with status 1. A misspelled field
names the document and the field:
cp1: valid for the image node-metal (node-metal on metal, root hash 98bb85096ed9), config 91a07f018b112dd92925b568babd16a428cee713fbb9e9c393ac11a2f4f5976e
cp3: valid for the image node-metal (node-metal on metal, root hash 98bb85096ed9), config 52a62aa28b05f78a15e09f984db2709f79a89fe79a87a87c3d8a9066f0d8e7d5
chalkctl: the config of 1 node is invalid:
node cp2: Network "10-lan": - at '/spec/network': additional properties 'keepconfiguration' not allowed
Here keepconfiguration should be keepConfiguration. The common refusals and their fixes:
- An unknown kind means that the node's image does not declare it. Check the
apiVersion, or ask the image author for the extension that brings the kind. names no filemeans that a path or glob of the inventory matches no file. Check the path, which is relative to the inventory's directory.both holdnames two files with a document of the same kind and name. Remove one, or move the shared document to the shared file alone.- An inventory error names the node and the image. Check
spec.imagesand theimageNameof the node, its groups andspec.all.
What next¶
- Install a node installs a node with its config.
- Add storage volumes plans the disks and volumes in depth.
- Config and the inventory explains how a node checks, keeps and applies its config.