Skip to content

Images and platforms

This page is for an image author whose nodes do not all run the same software, or do not all run on the same kind of machine. At the end your flake builds one image for each group of nodes, and the inventory gives each node the image it needs. It starts from the flake of Your first image flake.

An image has a name, a platform and NixOS modules. The modules are the software a group of nodes runs. A platform is the kind of machine an image runs on, such as metal or kvm. mkImages builds the images its images list names.

List the images

This call builds three images: Kubernetes nodes for metal and kvm, and storage nodes without Kubernetes for metal:

images = chalkos.lib.mkImages {
  osCA = ./secrets.pub.json;
  # Every image carries these.
  modules = [ ./site.nix ];
  images = [
    {
      name = "node-metal";
      platform = "metal";
      modules = [ chalkos.nixosModules.kubernetes ];
    }
    {
      name = "node-kvm";
      platform = "kvm";
      modules = [ chalkos.nixosModules.kubernetes ];
    }
    {
      name = "storage-metal";
      platform = "metal";
      modules = [ ./storage.nix ];
    }
  ];
};

It returns images.node-metal, images.node-kvm, images.storage-metal and images.installer. Expose each image the cluster runs as a package, as the first flake exposes node-metal, and build it the same way:

nix build .#storage-metal -o images/storage-metal

An entry of images sets three things:

  • name, which names the image in the flake, in its os-release and in chalkctl upgrade.
  • platform, the kind of machine it runs on.
  • modules, the NixOS modules of the image: a layer, a monitoring agent, a storage daemon, more kernel module groups or an extension's kinds and services.

A name is lower-case letters, digits and dashes, and starts with a letter or digit. It becomes CHALKOS_IMAGE in the image's os-release and name in its metadata.json. mkImages refuses two entries of one name, and an entry named installer, which names the installer. The names above end in their platform only by convention; platform alone sets the platform.

chalkctl upgrade --image without --nodes installs the image on the nodes that run an image of the same name. A renamed image matches no running node, so its first upgrade names the nodes with --nodes.

Add a layer

A layer is a predefined set of NixOS modules that chalkos ships. An image lists it in its modules like any other module, and it enables chalkos's own modules for the layer:

  • chalkos.nixosModules.kubernetes enables the Kubernetes layer. Its images run containerd and the kubelet, with etcd on control planes, and accept the kinds of kubernetes.chalkos.dev: Node, Cluster, Flannel and Addon. They also run FRR, the layer's BGP speaker, and accept its kinds BGPSpeaker and BGPPeer, unless the image's modules set chalkos.kubernetes.bgp.enable to false.
  • chalkos.nixosModules.router enables the router layer. Its images forward IPv4 and IPv6 packets between their interfaces, accept the kinds of router.chalkos.dev, filter with the router's own firewall in place of NixOS's and run FRR for BGP, Jool for NAT64, Unbound and Kea, unless the image leaves router features out. How a router works describes the layer.

An image lists one layer at most, and the build refuses an image with both. An image without a layer, such as storage-metal above, runs chalkos's base alone: chalkd, config, storage, networking and upgrades, with no kubelet, etcd or containerd.

Leave router features out

A router image carries every router feature unless its modules leave one out. Each feature has an image option, true by default. Set it to false in the image's modules to leave the feature's software, kinds, outputs, units and diagnostics out of the image:

{
  name = "edge-metal";
  platform = "metal";
  modules = [
    chalkos.nixosModules.router
    {
      chalkos.router = {
        nat64.enable = false;
        dhcp.enable = false;
        dhcpRelay.enable = false;
      };
    }
  ];
}
Option Leaves out Kinds the nodes refuse
chalkos.router.bgp.enable FRR: zebra, bgpd, bfdd, staticd, vtysh and chalkos-frr-reload BGP, BGPPeer, PrefixList, CommunityList, RoutingPolicy
chalkos.router.nat64.enable Jool's kernel module and jool NAT64
chalkos.router.resolver.enable Unbound Resolver
chalkos.router.dhcp.enable Kea DHCPSubnet
chalkos.router.dhcpRelay.enable dnsmasq DHCPRelay

The firewall, VRFs, policy routing and the kernel's forwarding settings run no daemon of their own, so every router image carries them. A metal router image with every feature takes 320 MiB of store; one without all five takes 205 MiB. FRR accounts for 97 MiB of the difference.

The binary cache does not hold Kea built without its database backends, so the first build of a router image with DHCP after a nixpkgs update compiles it, about 5 minutes on 16 cores.

A node whose image leaves a feature out refuses documents of its kinds. chalkctl validate and chalkctl config apply report that the image does not accept kind "BGP", and chalkd refuses an upgrade to such an image while the node's config holds them. To drop a feature from running nodes, remove its documents from their config with chalkctl config apply first, then upgrade them to the new image. chalkctl status prints no line of a feature the image leaves out.

Add an image only for other software

An image fixes the software a node runs. The node's config sets every value that names the cluster, the node or its surroundings:

Fixed when the image is built Set at runtime by the node's config
Kernel, kernel modules and initrd Hostname
NixOS services and the image's modules systemd-networkd units: Network, NetDev and Link documents
The OS CA for maintenance mode, from osCA Storage: system disk, VAR, volumes and encryption
Partition sizes of the system region, from chalkos.disk Time servers and whether DHCP may add more
Boot tries and health timeout, from chalkos.upgrade Secret documents and documents of the kinds extensions declare
The image's revision and tag, which only metadata.json records

A change in the left column is a new image and an upgrade. A change in the right column reaches the node with chalkctl config apply and needs no reboot; a storage change that would destroy data is refused there and needs chalkctl storage reset.

So a cluster needs a second image on a platform only for nodes that run other software, such as storage nodes with a daemon the others should not carry, or GPU nodes with drivers. A cluster whose nodes run the same software has one image per platform, and its nodes differ only in config. Because no cluster value is in the image, a cluster of fifty nodes on one platform builds one image.

Kubernetes: what the image fixes and what config sets

An image with Kubernetes also fixes the kubelet's release, which the next section chooses, and the pod network from chalkos.cni.provider: flannel or none, and the CNI plugins. The node's config sets whether the node is a control plane or a worker, its labels, taints and addresses, and the cluster's endpoint and Kubernetes settings: the images of every other component, ranges, VIP, extra flags, flannel's MTU, registry mirrors and addons.

So one image with Kubernetes serves control planes and workers. Each node's Node document makes it one or the other.

Choose the kubelet's release

An image with Kubernetes runs the kubelet of one Kubernetes release. chalkos pins each of the three newest minor releases, the ones Kubernetes supports, to one patch release, and the image's nixpkgs does not change it. By default the image runs the newest. chalkos.kubernetes.version picks another of the three. Set it in the image's modules:

{
  name = "node-metal";
  platform = "metal";
  modules = [
    chalkos.nixosModules.kubernetes
    { chalkos.kubernetes.version = "1.36"; }
  ];
}

The flake's lib.kubernetesReleases lists the minors with the patch release of each. Unless a binary cache holds the kubelet, the image's build compiles it from source, about two minutes on 16 cores. The image directory's facts/kubernetes.json names the kubelet's version, the patch release of 1.36 here.

The Cluster document names the images of the control plane and kube-proxy apart from the kubelet. They may stay at a newer minor than the kubelet, within Kubernetes's version skew policy, or follow the kubelet's minor. The image's checker holds them to that policy against its own kubelet, so chalkctl validate refuses an API server older than the kubelet or more than three minors newer. The lab template's cluster/cluster.yaml names the images of the newest release, which a kubelet two minors older accepts.

A release the option does not offer, such as another patch release, needs chalkos.kubernetes.package: a Kubernetes package whose bin/kubelet the image runs. Set one of the two options. An image that sets both to different minors fails to build and names both.

Give each node its image

A node runs the image the inventory names for it. Add the new image to spec.images, beside node-metal, and name it on the node that runs it, here storage1:

chalkos.yaml
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: storage}
spec:
  images:
    node-metal: {source: {directory: images/node-metal}}
    storage-metal: {source: {directory: images/storage-metal}}
  all: {imageName: node-metal, files: [cluster/*.yaml]}
  nodes:
    node1: {address: 192.0.2.11, files: [nodes/node1.yaml]}
    node2: {address: 192.0.2.12, files: [nodes/node2.yaml]}
    storage1: {address: 192.0.2.31, imageName: storage-metal, files: [nodes/storage1.yaml]}

A node's own imageName wins, then the imageName of its groups, then spec.all.imageName. chalkctl refuses the inventory when a node is in two groups that name different images, or when a node ends up with no image. Several storage nodes can share a group with imageName: storage-metal instead.

No check compares the software of two images. A node may move to another image with an upgrade. The node checks the config the upgrade carries against the new image's schemas before the image becomes its next boot, so an image that does not accept the node's config never boots.

Build an image per kind of machine

A platform's NixOS modules add what one kind of machine needs and other machines do not: drivers, firmware, agents and the console. chalkos has modules for two platforms, which an image of the platform carries before its own modules:

  • metal sets the kernel's consoles to the first serial port and the screen, the screen last.
  • kvm sets the kernel's console to the first serial port alone, adds the virtio disk drivers to the initrd and runs a QEMU guest agent, which starts only under KVM. The host may ask the agent about the node and shut it down. It never runs commands or reads and writes files on the node.

Any other name is a platform of the image's own modules. This image adds modules for one server model as a platform r650:

platforms/r650.nix
{
  chalkos.consoles = [
    "tty0"
    "ttyS1,115200"
  ];
  chalkos.kernel.extraModules = [ "mlx5_ib" ];
}
{
  name = "node-r650";
  platform = "r650";
  modules = [
    ./platforms/r650.nix
    chalkos.nixosModules.kubernetes
  ];
}

Only node-r650 carries the module and the consoles. Here the BMC's serial-over-LAN on the second serial port is the console people watch, so that port comes last. A platform of your own starts from chalkos's node modules alone, without metal's modules; its consoles default to the first serial port and the screen, as on metal. Its name follows the rules of an image's name and becomes CHALKOS_PLATFORM in the image's os-release.

A node refuses an image of a platform other than the one it runs, so a node changes platform only by being installed again. Its config files stay as they are on any platform. One image per platform keeps each image small, because a universal image would carry every platform's agents and drivers.

Choose the console people watch

chalkos.consoles lists the kernel's consoles in order, each as the kernel's console= parameter takes it, such as ttyS0,115200 or tty0 for the screen. The kernel prints its messages on every console, and systemd asks for each passphrase and recovery key on every console, so any of them can answer. The last entry becomes /dev/console: systemd's boot status and the emergency shell appear there alone.

metal, a platform of your own and the installer put the screen last, so a machine without a serial port shows its boot and its emergency shell on the screen. While the last console is a virtual terminal such as the screen, the unit chalkos-serial-log@ copies chalkd's lines, with its fingerprint and addresses, from the journal to every serial port of the list, such as ttyS0, ttyAMA0 or hvc0, so a BMC's serial-over-LAN shows them. An entry of another form, such as uart8250,mmio32,0xfe215040, gets only the kernel's messages. On the screen, chalkd's lines keep off the console, which shows the fingerprint and addresses itself; an image without the console shows chalkd's lines there. While the last console is a serial port, chalkd's lines reach it as /dev/console.

A machine whose BMC's serial-over-LAN is the console people watch can put that port last, so the boot's status and the emergency shell appear there too. Set it in the image's modules, or in modules for every image:

{
  chalkos.consoles = [
    "tty0"
    "ttyS0,115200"
  ];
}

The build refuses a console= entry in boot.kernelParams and names chalkos.consoles.

Override a platform's value

An image is one NixOS system of four lists of modules: chalkos's, its platform's, modules and its own. Module order lists them, and the installer's. NixOS merges a value by priority, not by module order. So an image that sets a value its platform also sets plainly gets a conflict, and overrides that value with lib.mkForce. A value the platform forces itself, such as the kvm platform's ExecStart of the guest agent, needs lib.mkOverride with a priority below 50, for example lib.mkOverride 40.

A flake can also build each image with chalkos.lib.mkImage and the installer with chalkos.lib.mkInstaller itself, as mkImage and mkInstaller shows. Customise the installer adds modules to the installer.

Tie the images to one cluster

osCA names secrets.pub.json, the public part of the cluster's secrets that chalkctl gen secrets writes. It holds certificates and no keys, so it belongs in the repository beside flake.nix. Every image and the installer carry its OS CA, so chalkd in maintenance mode accepts only clients with a certificate from it. chalkos reads version 3 of the file only. The evaluation fails on another version and says to generate new secrets.

Left null, images accept any client until they are installed. That suits a lab on a private network and nothing else. The OS CA is the only cluster value an image carries, so images built with one serve that cluster alone, and images built without one serve any.

No module reads the secrets file itself. chalkctl reads it where --secrets names it. The lab template's .gitignore keeps the plaintext secrets.json out of git, and so out of /nix/store, which copies a flake's tracked files.

Limits

  • One image serves every node it is given. A per-node kernel parameter, package or driver needs an image of its own.
  • mkImages builds images for one Nix system per call. A cluster of x86-64 and arm64 machines calls it once for each.
  • mkImages builds the list it is given. A flake that builds the same software for several platforms writes an entry for each, or maps over a list of platforms itself.
  • A flake rebuilds an image whenever its modules or flake.lock change. A build whose store changed has a new root hash and is a new image to chalkctl upgrade. A build whose store did not change is the image the nodes run already.

If it fails

Failure Cause and fix
mkImages refuses an entry Two entries share a name, or one is named installer. Rename it, as List the images describes.
The build refuses an image with both layers An image lists one layer at most. Put each layer in an image of its own.
does not accept kind "BGP" The image leaves the feature out. Remove its documents first, as Leave router features out describes.
chalkctl refuses the inventory A node is in two groups that name different images, or has no image. See Give each node its image.
The build names chalkos.kubernetes.version and chalkos.kubernetes.package Both are set, to different minors. Set one of them.
The build names chalkos.consoles boot.kernelParams holds a console= entry. Set the consoles in chalkos.consoles.
The evaluation says to generate new secrets osCA names a secrets file of another version than 3. Generate new secrets.
chalkctl upgrade --image of a renamed image A renamed image matches no running node. Name the nodes of its first upgrade with --nodes.

What next