Skip to content

Write a node's config

This page is for an operator who describes the cluster's nodes before installing them. At the end each node of the inventory has its config files, with its system disk, network, time servers and volumes, and chalkctl validate accepts them. The page follows the cluster-metal template: the three Kubernetes control planes cp1, cp2 and cp3, on one image. Where it needs groups or a second image, it shows the homelab example, a router beside three Kubernetes nodes.

Before you begin

You need a workstation set up as Set up your workstation describes: chalkctl, the repository and the image directory the inventory names. chalkctl validate checks the config against that image, so the image author builds it first.

You need to know each machine's disks and network interfaces. On a machine that runs the installer, chalkctl disks lists the disks, as Choose the system disk shows.

Give each node its files in the inventory

A node's config is the set of YAML documents that makes it this node. The inventory names the files each node's documents come from. The template's gives every node the files of cluster/, and each node its own file in nodes/:

chalkos.yaml
# The cluster's three nodes, the address chalkctl reaches each at, the image they run and their
# config files: every node gets cluster/, each its own file in nodes/. Replace the addresses with
# free ones of your network, here and in the nodes' files.
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: cluster}
spec:
  images:
    # nix build .#node-metal -o images/node-metal puts the image here.
    node-metal: {source: {directory: images/node-metal}}
  all: {imageName: node-metal, files: [cluster/*.yaml]}
  nodes:
    cp1: {address: 192.168.1.11, files: [nodes/cp1.yaml]}
    cp2: {address: 192.168.1.12, files: [nodes/cp2.yaml]}
    cp3: {address: 192.168.1.13, files: [nodes/cp3.yaml]}

The inventory's metadata.name, cluster here, is a label for the cluster. It reaches no node: chalkctl uses it in its messages and as the default name of a kubeconfig's context.

The inventory's spec has four parts:

  • spec.images names each image and the directory of its build, relative to the inventory's directory.
  • spec.all is what every node gets: the image imageName names, and the config files files names. The template gives every node the image node-metal and the files of cluster/.
  • spec.groups holds named groups of nodes, each with its files and its image. The template has none, and the homelab example two, router and nodes.
  • spec.nodes holds each node by its name, with the address chalkctl reaches its chalkd at, as a host or host:port, and its own files.

The key of spec.nodes, such as cp1, is the node's name. Install writes it into the node's certificate. The hostname defaults to it, and the recovery key and the seeds of the partition UUIDs derive from it. Renaming a node therefore takes a reinstall.

address is where chalkctl reaches the node, also in maintenance mode before the node has any config. chalkctl never reads it from the node's Network documents. A node that installs with a DHCP address and then gets a static one names the static one here, as the template's nodes do. --address overrides it for one command.

Here is cp1's own file, nodes/cp1.yaml:

nodes/cp1.yaml
# cp1: a control plane with one Ethernet port on the network. Replace the disk with the node's own,
# by its path below /dev/disk/by-id, and the MAC address with its port's, which chalkctl disks and
# the diagnostic addresses show while the machine waits in the installer.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
  type: controlplane
---
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
  disk: {path: /dev/disk/by-id/nvme-EXAMPLE_SSD_cp1}
---
# The port's static address, the one chalkos.yaml names for cp1, its gateway and its DNS server.
# The node holds the VIP on this port while it is the VIP's holder.
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-lan}
spec:
  match: {macAddress: "02:00:00:00:00:11"}
  network: {address: [192.168.1.11/24], gateway: 192.168.1.1, dns: [192.168.1.1]}

Each document has an apiVersion with its group and version, a kind, a metadata.name and a spec. A kind of which a node holds one takes any metadata.name; the examples use default or a short name such as system.

Keep each document in one file

Documents never merge, so a node's set must hold each document once. Two documents of the same kind and name in one node's set are an error that names both files:

chalkctl: the config of 3 nodes is invalid:
node cp1: cluster/net.yaml and nodes/cp1.yaml both hold Network "10-lan"
node cp2: cluster/net.yaml and nodes/cp2.yaml both hold Network "10-lan"
node cp3: cluster/net.yaml and nodes/cp3.yaml both hold Network "10-lan"

Put a value that every node shares into a shared file once. When one node needs a different value, that node's own file holds its own document of that kind, and the shared file leaves the kind out. The inventory gives each node a set of files explains the order in which a node's set takes its files.

Share files and an image through a group

A group holds what some of the nodes share. The template needs none, because its nodes share one image and one set of files. The homelab example has two. Its group router gives its one node the image router-metal and the files of router/. The group nodes gives the three Kubernetes nodes the image node-metal and the files of cluster/ and bgp/, and each of them names its own file beside the group.

A node runs the image named in the first of three places:

  1. the node's own imageName;
  2. the imageName of its groups;
  3. spec.all.imageName.

The diagram shows how the homelab's inventory gives each node its files and its image:

flowchart LR
    routerg["group router: router/*.yaml"] --> router["router's set"]
    nodesg["group nodes: cluster/*.yaml, bgp/"] --> node1["node1's set"]
    nodesg --> node2["node2's set"]
    nodesg --> node3["node3's set"]
    node1f["nodes/node1.yaml"] --> node1
    node2f["nodes/node2.yaml"] --> node2
    node3f["nodes/node3.yaml"] --> node3
    routermetal["router-metal"] -->|the group's imageName| router
    nodemetal["node-metal"] -->|the group's imageName| node1
    nodemetal -->|the group's imageName| node2
    nodemetal -->|the group's imageName| node3

The router gets the files of its group and runs router-metal. Each Kubernetes node gets the files of nodes, then its own, and runs node-metal.

chalkctl refuses the inventory when a node is in two groups that name different images, also when the node names its own. It also refuses a node that gets no image, and an imageName that names no image of spec.images:

chalkctl: chalkos.yaml: node cp1 names the image node-kvm, which the inventory does not define

The image stays out of the node's config. A new build of an image, or a node's move to another image, changes no node's config.

Choose the system disk and the size of VAR

A SystemDisk names the disk that holds the system region and VAR, the partition for the node's data. Every node holds exactly one. The template's take the whole disk for VAR. This one gives VAR 200 GiB and leaves the rest of the disk to volumes:

apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
  disk: {path: /dev/disk/by-id/nvme-EXAMPLE_SSD_cp1}
  var: {size: 200G}

Name the disk by a path below /dev/disk/by-id, which stays the same across boots, or by selectors such as model, serial or size. Without var.size, VAR fills the rest of the disk. By default STATE, VAR and every volume are encrypted with a key sealed to the TPM, with the node's recovery key as the second keyslot. Add storage volumes covers the disk selectors, the encryption and the sizes.

Configure the interfaces

A node's network is a set of Network, NetDev and Link documents. Each is one systemd-networkd unit file: a .network, a .netdev or a .link file. The document's metadata.name is the file's name without its suffix, so networkd orders the files by it. Its spec holds networkd's sections and keys in camelCase: [Match] with MACAddress= is match: {macAddress: …}.

The template's nodes give their port a static address, a gateway and a DNS server, and match the port by its MAC address:

apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-lan}
spec:
  match: {macAddress: "02:00:00:00:00:11"}
  network: {address: [192.168.1.11/24], gateway: 192.168.1.1, dns: [192.168.1.1]}

The image brings one unit of its own, 99-ethernet-default-dhcp, which runs DHCP on every physical Ethernet interface that no Network document matches. A node whose config holds no Network document therefore runs DHCP on all of them. The documents' units take precedence over the image's units.

A bond, a VLAN and a dummy interface are NetDev documents, each with the Network documents that attach it or configure it. On this machine, two ports form an LACP bond, and the machine's address is on VLAN 20 of the bond:

apiVersion: networkd.chalkos.dev/v1alpha1
kind: NetDev
metadata: {name: 10-bond0}
spec:
  netDev: {name: bond0, kind: bond}
  bond: {mode: 802.3ad, transmitHashPolicy: layer3+4, miiMonitorSec: 100ms, lacpTransmitRate: fast}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: NetDev
metadata: {name: 20-vlan20}
spec:
  netDev: {name: vlan20, kind: vlan}
  vlan: {id: 20}
---
# The two ports join the bond and get no address of their own.
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-bond0-ports}
spec:
  match: {name: enp1s0 enp2s0}
  network: {bond: bond0}
---
# The bond carries the VLAN and no address of its own.
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 20-bond0}
spec:
  match: {name: bond0}
  network: {vlan: vlan20, linkLocalAddressing: "no"}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 30-vlan20}
spec:
  match: {name: vlan20}
  network: {address: [192.0.2.21/24], gateway: 192.0.2.1, dns: [192.0.2.1]}

The NetDev documents create the devices bond0 and vlan20. The Network documents attach the ports to the bond and the VLAN to the bond, and give the VLAN device the node's address. The switch ports must form an LACP group too.

A Link document is a .link file, which sets the properties of a physical device, such as its name or its MTU. The homelab's router names its ports by permanent MAC address, so that its other documents name the ports wan and fabric, whatever the kernel calls them:

apiVersion: networkd.chalkos.dev/v1alpha1
kind: Link
metadata: {name: 10-fabric}
spec:
  match: {permanentMACAddress: "02:00:00:00:01:02"}
  link: {name: fabric}

The installer holds no config, so a machine's ports keep the kernel's names until the installed node's first boot, which renames them. The template's nodes keep the kernel's names and match their port by its MAC address instead.

The schemas of the three kinds come from the networkd of the image's own systemd, so they accept exactly the sections and keys that networkd accepts. Config kinds lists them, and Networking describes how a node applies them.

A node applies its Link documents at every boot, before networkd brings the links up. A delivery applies a changed Link document to the running ports at once, as far as their drivers allow, and that includes a new name. Every document that names the port must change in the same delivery: the match of its Network, and the interfaces of a VRF, a BGP peer or a firewall zone. Deliver a rename as a trial, with chalkctl config apply --try, so a mistake that cuts the node off rolls back. The rollback renames the port back when the previous config names it in a Link document; otherwise the port keeps the new name until the next boot. An added alternative name or setting also takes effect at once, and a removed one at the next boot.

A VLAN takes the MAC address of its port, and the ports of a bond take the bond's, so a Link document that matches by macAddress can match the wrong device. Match a port by permanentMACAddress, which only the physical device has.

Set the hostname

The hostname is the node's name unless a Host document gives another:

apiVersion: chalkos.dev/v1alpha1
kind: Host
metadata: {name: default}
spec: {hostname: cp1-shelf-a}

The Host document changes the hostname alone. The node's name, its certificate and, with the Kubernetes layer, its Kubernetes node name stay as the inventory gives them.

Name the time servers

Nodes check certificates against their clocks. Without a TimeSync document, a node synchronises its clock with three servers of the German national metrology institute, PTB, authenticated with NTS. NTS needs the nodes to reach the servers on TCP port 4460 as well as UDP port 123. A network without outbound NTP names its own servers, in a file every node gets:

cluster/time.yaml
apiVersion: time.chalkos.dev/v1alpha1
kind: TimeSync
metadata: {name: default}
spec:
  servers:
    - host: ntp1.example.internal
    - host: ntp2.example.internal
      nts: false

A server authenticates with NTS unless nts: false. dhcp: true also takes the servers that DHCP announces, which nothing authenticates.

Add storage beside VAR

A Volume is a further partition, on the system disk or on a disk of its own. This one takes a SATA SSD, which a selector finds by its model and type, for a storage system's replicas:

apiVersion: storage.chalkos.dev/v1alpha1
kind: Volume
metadata: {name: replicas}
spec:
  disk: {model: "Samsung SSD 870*", type: ssd}
  format: xfs
  mountPoint: /var/lib/replicas

A node may hold several volumes, each with a name of its own. Some later changes to a volume take a reset or a reinstall. Add storage volumes covers the formats, the encryption and which changes a node accepts.

Keep secret values in Secret documents

A Secret document holds secret values by key in spec.data. A field of a kind, built in or an extension's, references one of its keys instead of holding the value. The homelab's BGPPeer documents in bgp/fabric.yaml do so with password: {secretRef: {name: bgp, key: password}}, and bgp/secrets.yaml holds the Secret named bgp. chalkctl refuses a set in which a reference names a Secret or a key that the set lacks.

Secret documents are config, separate from the cluster's secrets file. Keep the services' secrets in Secret documents says where to keep the files that hold them.

With the Kubernetes layer, every node also holds a Cluster and a Node document

An image with the Kubernetes layer runs Kubernetes. Each node's set then holds exactly one Cluster document, the same on every node, and one Node document with the node's type, controlplane or worker. The template puts the Cluster in cluster/cluster.yaml and each Node in the node's own file. Configure Kubernetes covers both.

Check that it worked

chalkctl validate builds each node's set and checks it against the schemas of the node's image, as chalkd will. It reaches no node and needs no credentials, so it also runs in CI:

$ chalkctl validate
cp1: valid for the image node-metal (node-metal on metal, root hash 98bb85096ed9), config 91a07f018b112dd92925b568babd16a428cee713fbb9e9c393ac11a2f4f5976e
cp2: valid for the image node-metal (node-metal on metal, root hash 98bb85096ed9), config 138189d54c5a0f2d15e197b349f23afbe71ded1b6179bde77bfdaf63a7ed93d7
cp3: valid for the image node-metal (node-metal on metal, root hash 98bb85096ed9), config 52a62aa28b05f78a15e09f984db2709f79a89fe79a87a87c3d8a9066f0d8e7d5

Each line names the node's image, its own name and platform, the first 12 hex characters of its root hash, and the hash of the node's config. chalkctl status later compares that hash with the config the node runs. chalkctl validate cp1 checks cp1 alone.

validate checks that each set:

  • holds exactly one SystemDisk, and every other kind at most once, unless the kind allows several;
  • holds only kinds that the node's image declares;
  • holds no two documents of a kind with the same name;
  • names only Secret documents and keys that it holds;
  • passes each kind's schema, and the checks that chalkd runs when it translates the set, such as a volume that does not fit its disk.

It also runs the change rules of the image that read no facts about the node. It warns of a node whose SystemDisk leaves STATE unencrypted.

validate checks no signature, so CI needs no db key. When the inventory names a signed copy, a CI job puts the unsigned build of the same image in that directory instead.

If validate refuses the config

validate names each node whose config is invalid and exits with status 1. A misspelled field names the document and the field:

cp1: valid for the image node-metal (node-metal on metal, root hash 98bb85096ed9), config 91a07f018b112dd92925b568babd16a428cee713fbb9e9c393ac11a2f4f5976e
cp3: valid for the image node-metal (node-metal on metal, root hash 98bb85096ed9), config 52a62aa28b05f78a15e09f984db2709f79a89fe79a87a87c3d8a9066f0d8e7d5
chalkctl: the config of 1 node is invalid:
node cp2: Network "10-lan": - at '/spec/network': additional properties 'keepconfiguration' not allowed

Here keepconfiguration should be keepConfiguration. The common refusals and their fixes:

  • An unknown kind means that the node's image does not declare it. Check the apiVersion, or ask the image author for the extension that brings the kind.
  • names no file means that a path or glob of the inventory matches no file. Check the path, which is relative to the inventory's directory.
  • both hold names two files with a document of the same kind and name. Remove one, or move the shared document to the shared file alone.
  • An inventory error names the node and the image. Check spec.images and the imageName of the node, its groups and spec.all.

What next