Skip to content

Architecture

chalkos builds signed, read-only images with Nix and configures the machines that boot them with YAML documents, through one agent on each node. This page names the parts, says where each one runs and explains the two modes of the agent. Three machines take part shows the machines and the trust between them. The diagram here shows the parts inside them and what each one drives, chalklab and the node's own parts included.

flowchart LR
    flake["Flake calling mkImages"] -->|nix build| images["Image directories and installer"]
    inventory["Inventory and YAML config"] --> chalkctl
    images --> chalkbuild
    chalkbuild -->|signed image directories| chalkctl
    chalklab -->|starts VMs, installs with| chalkctl
    chalkctl -->|node API, mutual TLS, port 50000| chalkd
    subgraph node["Node"]
        chalkd --> systemd
        chalkd --> disk["Disk: slots, STATE, VAR"]
        chalkd --> tpm["TPM"]
        console["chalkos-console on tty1"] -->|node API, local socket| chalkd
    end

Nix and chalkbuild run on the build host, chalkctl and chalklab on the operator's workstation, and chalkd on every node. The sections below take them in that order.

Nix builds the software and the config carries the rest

Nix builds the software a node runs: the operating system, Kubernetes and chalkd. chalkos.lib.mkImages evaluates one NixOS system per image the flake lists, each with a name and a platform. Each system comes from chalkos's node modules, the platform's modules, the modules every image shares and the image's own. Nix builds each into an unsigned raw disk image in an image directory. It also builds one installer, which installs images of every platform. mkImages takes no node, address or cluster setting. Its only input of a cluster is the optional public part of its secrets, for the OS CA. So one image serves every node the inventory runs on it, control planes and workers alike. Images and platforms explains images, platforms and how their modules combine.

Everything that differs between nodes is config, delivered at runtime. The split keeps the images few, so a cluster of three control planes and fifty workers that run the same software builds and signs one image per platform. A change to what the image fixes, such as the kernel or a service, is a new image and an upgrade. A change to the config, such as the hostname or the time servers, is a change to YAML files, delivered with chalkctl config apply without a reboot. A storage change that would destroy data is refused there and takes a reset or a reinstall, as Additive changes apply while the node runs describes. Boot and the image lists what falls on each side, and Images and platforms draws the line in detail. Config and the inventory explains the config side.

chalkctl and chalklab run on the operator's machine

chalkctl reads the cluster from an inventory: the file --inventory names, else the file $CHALKOSINVENTORY names, else chalkos.yaml in the current directory. The inventory names each node, the address chalkctl reaches it at, the YAML files of its config and the image it runs, by the image directory of its build. chalkctl reads the image directory when a command needs the image. Without an inventory, a command that only reaches a node works with a client file and --address. Config and the inventory describes the files.

chalkctl authenticates with the secrets file, which holds every CA key of the cluster, or with a client file, which holds one certificate of the reader, operator or admin role. Installs, config changes and rotations need the secrets file. Upgrades, reboots, status and logs work with a client file, so the people who operate a cluster need not hold its CA keys. Clients authenticate with the secrets file or a client file explains both.

chalklab runs the nodes a lab file names as QEMU virtual machines on one machine, each with Secure Boot and a TPM. It never builds images. It copies the kvm image the inventory names for each node, signs the copy with lab keys it enrols in the VMs' firmware, boots each node and installs it with chalkctl. A lab is how the quick start runs.

A node runs its image from a read-only store

A node runs its image from a read-only store checked by dm-verity, with its root file system in memory. What it writes goes to two encrypted partitions. STATE, 128 MiB by default, holds the node's config, certificates and Kubernetes secrets. VAR, mounted at /var, holds etcd, containerd, the kubelet and logs. The image describes the disk, and Storage and encryption describes STATE and VAR.

At every boot, two units in the initrd unlock STATE and mount VAR. A unit before networkd applies the config on STATE. Then chalkd starts and serves the node API, and on a counted boot a health check decides whether a new image is healthy. Boot and the image lists the units.

chalkd runs as root, because it partitions disks, enrols TPM keyslots, writes EFI variables, mounts volumes and sets the hostname. It restarts whenever it stops, with no limit, because it is the only way to reach the node. chalkd's sandbox lists what its unit allows.

The machine's screen shows chalkos-console, the console, in place of a login prompt. It reaches chalkd through a local socket and calls the node API there, for the reads and the few actions the console may call. It changes nothing on the node itself.

With the Kubernetes layer, chalkd also runs the control plane

On a control plane, chalkd renders the static pods of etcd, the API server, the controller-manager and the scheduler. It holds the VIP when elected, renews the node certificates of the cluster and drains nodes for upgrades. The control plane covers that side.

An image carries the base and at most one layer

Every image carries the base: chalkd, the config pipeline, storage, upgrades and the node API. A layer is a predefined set of NixOS modules that chalkos ships, which an image lists in its modules like any other module. It enables chalkos's modules of one role on top of the base. The Kubernetes layer, chalkos.nixosModules.kubernetes, runs containerd and the kubelet and makes a node a control plane or a worker. The router layer, chalkos.nixosModules.router, forwards packets between the node's interfaces. An image carries one layer at most, and one that lists two fails to build.

Each layer writes an empty marker, /etc/chalkos/layers/<layer>. chalkd reads the directory when it starts and reports the layers in Info and Contract.

chalkd and chalkctl each carry a plugin of every layer, compiled in. chalkd starts a layer's plugin only on an image that carries the layer. In chalkd, the plugin serves the layer's API beside NodeService: KubernetesService for the Kubernetes layer, RouterService for the router layer. It also runs the layer's loops, adds its certificates and health check to the node's, and takes its part of a delivery, such as the Kubernetes share. On an image without the layer, a method of its API fails with UNIMPLEMENTED and the message the kubernetes layer is not in this image, before chalkd checks the mode or the role. In chalkctl, the plugin adds the layer's commands, such as chalkctl etcd, its section of chalkctl status, and its steps of an upgrade or a rotation.

A layer's units run its chalkd subcommands as chalkd <layer> <command>, such as chalkd kubernetes prepare. An effect may name an action of a layer as <layer>:<action>, such as kubernetes:restart, which acts on what only the running chalkd holds; React to changes covers both. An image whose effects name a layer it does not carry fails to evaluate.

The installed marker decides chalkd's mode

chalkd runs in one of two modes, and the file installed on STATE decides which. The flowchart shows how chalkd picks the mode when it starts.

flowchart TD
    start["chalkd starts"] --> read{"Can it tell whether STATE holds installed?"}
    read -->|no| fail["Fails, systemd restarts it"]
    read -->|yes, installed| cert{"Node certificate on STATE loads?"}
    cert -->|no| fail
    cert -->|yes| normal["Normal mode: clients of the OS CAs on STATE"]
    read -->|yes, not installed| ca{"Image carries an OS CA?"}
    ca -->|yes| maintca["Maintenance mode: clients of that OS CA"]
    ca -->|no| maintany["Maintenance mode: any client"]

Maintenance mode serves a node that is not installed

A machine reaches maintenance mode in one of two ways:

  • It boots the installer, from a USB disk, an ISO or a network boot. The installer never touches the medium it boots from. It writes the node's image to another disk.
  • It boots its image directly, written to its disk with dd, used as a VM's disk or created by chalklab. On the first boot, systemd-repart adds slot B and an empty STATE behind the partitions the image ships with. STATE has no installed file, so chalkd starts in maintenance mode.

In maintenance mode, chalkd serves a self-signed certificate and accepts only reads (node information, the image's contract, disks, logs and the drivers the firmware loaded), diagnostics, a reboot, a shutdown, the enrolment of Secure Boot keys and an install. How chalkctl trusts that certificate by the fingerprint on the console, and whom chalkd accepts, is explained in A node in maintenance mode is trusted by its fingerprint. Install a node covers the install as a task.

Normal mode serves an installed node

In normal mode, chalkd serves its node certificate and accepts only clients whose certificate chains to the OS CAs on STATE. Each method is open to the client roles allowed to call it, and the install method is closed.

When chalkd cannot read STATE well enough to tell whether the node is installed, it fails and restarts. It never falls back to maintenance mode, which would offer an installed node for a new install.

chalkctl and chalkd talk over the node API

They use the node API: Connect RPC over HTTPS with TLS 1.3 on TCP port 50000, with a client certificate on every call. chalkd takes the client's role from the certificate and checks the certificate again for every request, as chalkd checks the role of every request explains. Roles and credentials lists what each role may call.

Limits

  • An image without an OS CA accepts any client in maintenance mode, so anyone on the machine's network can install it. mkImages' osCA closes that.
  • How a node works lists what chalkos leaves to the operator, such as enrolling Secure Boot keys.