Architecture¶
chalkos builds signed, read-only images with Nix and configures the machines that boot them with YAML documents, through one agent on each node. This page names the parts, says where each one runs and explains the two modes of the agent. Three machines take part shows the machines and the trust between them. The diagram here shows the parts inside them and what each one drives, chalklab and the node's own parts included.
flowchart LR
flake["Flake calling mkImages"] -->|nix build| images["Image directories and installer"]
inventory["Inventory and YAML config"] --> chalkctl
images --> chalkbuild
chalkbuild -->|signed image directories| chalkctl
chalklab -->|starts VMs, installs with| chalkctl
chalkctl -->|node API, mutual TLS, port 50000| chalkd
subgraph node["Node"]
chalkd --> systemd
chalkd --> disk["Disk: slots, STATE, VAR"]
chalkd --> tpm["TPM"]
console["chalkos-console on tty1"] -->|node API, local socket| chalkd
end
Nix and chalkbuild run on the build host, chalkctl and chalklab on the operator's workstation, and chalkd on every node. The sections below take them in that order.
Nix builds the software and the config carries the rest¶
Nix builds the software a node runs: the operating system, Kubernetes and chalkd.
chalkos.lib.mkImages evaluates one NixOS system
per image the flake lists, each with a name and a platform. Each system comes from chalkos's node
modules, the platform's modules, the modules every image shares and the image's own. Nix builds each
into an unsigned raw disk image in an image directory. It also builds one installer, which installs
images of every platform. mkImages takes no node, address or cluster setting. Its only input of a
cluster is the optional public part of its secrets, for the OS CA. So one image serves every node
the inventory runs on it, control planes and workers alike.
Images and platforms explains images, platforms and how their
modules combine.
Everything that differs between nodes is config, delivered at runtime. The split keeps the images
few, so a cluster of three control planes and fifty workers that run the same software builds and
signs one image per platform. A change to what the image fixes, such as the kernel or a
service, is a new image and an upgrade. A change to the config, such as the
hostname or the time servers, is a change to YAML files, delivered with
chalkctl config apply without a reboot. A storage
change that would destroy data is refused there and takes a reset or a reinstall, as
Additive changes apply while the node runs
describes.
Boot and the image lists
what falls on each side, and
Images and platforms
draws the line in detail. Config and the inventory explains the config side.
chalkctl and chalklab run on the operator's machine¶
chalkctl reads the cluster from an inventory: the file --inventory names, else the file
$CHALKOSINVENTORY names, else chalkos.yaml in the current directory. The inventory names
each node, the address chalkctl reaches it at, the YAML files of its config and the image it
runs, by the image directory of its build. chalkctl reads the image directory when a command needs
the image. Without an inventory, a command that only reaches a node works with a client file and
--address. Config and the inventory
describes the files.
chalkctl authenticates with the secrets file, which holds every CA key of the cluster, or with a client file, which holds one certificate of the reader, operator or admin role. Installs, config changes and rotations need the secrets file. Upgrades, reboots, status and logs work with a client file, so the people who operate a cluster need not hold its CA keys. Clients authenticate with the secrets file or a client file explains both.
chalklab runs the nodes a lab file names as QEMU virtual machines on one machine, each with
Secure Boot and a TPM. It never builds images. It copies the kvm image the inventory names for
each node, signs the copy with lab keys it enrols in the VMs' firmware, boots each node and
installs it with chalkctl. A lab is how the quick start runs.
A node runs its image from a read-only store¶
A node runs its image from a read-only store checked by dm-verity, with its root file system in
memory. What it writes goes to two encrypted partitions. STATE, 128 MiB by default, holds the
node's config, certificates and Kubernetes secrets. VAR, mounted at /var, holds etcd,
containerd, the kubelet and logs. The image describes the disk, and
Storage and encryption describes STATE and VAR.
At every boot, two units in the initrd unlock STATE and mount VAR. A unit before networkd applies the config on STATE. Then chalkd starts and serves the node API, and on a counted boot a health check decides whether a new image is healthy. Boot and the image lists the units.
chalkd runs as root, because it partitions disks, enrols TPM keyslots, writes EFI variables, mounts volumes and sets the hostname. It restarts whenever it stops, with no limit, because it is the only way to reach the node. chalkd's sandbox lists what its unit allows.
The machine's screen shows chalkos-console, the console, in place of a login prompt. It reaches chalkd through a local socket and calls the node API there, for the reads and the few actions the console may call. It changes nothing on the node itself.
With the Kubernetes layer, chalkd also runs the control plane
On a control plane, chalkd renders the static pods of etcd, the API server, the controller-manager and the scheduler. It holds the VIP when elected, renews the node certificates of the cluster and drains nodes for upgrades. The control plane covers that side.
An image carries the base and at most one layer¶
Every image carries the base: chalkd, the config pipeline, storage, upgrades and the node API. A
layer is a predefined set of NixOS modules that chalkos ships, which an image lists in its modules
like any other module. It enables chalkos's modules of one role on top of the base. The
Kubernetes layer, chalkos.nixosModules.kubernetes, runs
containerd and the kubelet and makes a node a control plane or a worker. The router layer,
chalkos.nixosModules.router, forwards packets between the
node's interfaces. An image carries one layer at most, and one that lists two fails to build.
Each layer writes an empty marker, /etc/chalkos/layers/<layer>. chalkd reads the directory when
it starts and reports the layers in Info and Contract.
chalkd and chalkctl each carry a plugin of every layer, compiled in. chalkd starts a layer's
plugin only on an image that carries the layer. In chalkd, the plugin serves the layer's API beside
NodeService: KubernetesService for the Kubernetes layer, RouterService for the router layer. It
also runs the layer's loops, adds its certificates and health check to the node's, and takes its
part of a delivery, such as the Kubernetes share. On an image without the layer, a method of its
API fails with UNIMPLEMENTED and the message the kubernetes layer is not in this image, before
chalkd checks the mode or the role. In chalkctl, the plugin adds the layer's commands, such as
chalkctl etcd, its section of chalkctl status, and its steps of an upgrade or a rotation.
A layer's units run its chalkd subcommands as chalkd <layer> <command>, such as
chalkd kubernetes prepare. An effect may name an action of a layer as <layer>:<action>, such
as kubernetes:restart, which acts on what only the running chalkd holds;
React to changes
covers both. An image whose effects name a layer it does not carry fails to evaluate.
The installed marker decides chalkd's mode¶
chalkd runs in one of two modes, and the file installed on STATE decides which. The flowchart
shows how chalkd picks the mode when it starts.
flowchart TD
start["chalkd starts"] --> read{"Can it tell whether STATE holds installed?"}
read -->|no| fail["Fails, systemd restarts it"]
read -->|yes, installed| cert{"Node certificate on STATE loads?"}
cert -->|no| fail
cert -->|yes| normal["Normal mode: clients of the OS CAs on STATE"]
read -->|yes, not installed| ca{"Image carries an OS CA?"}
ca -->|yes| maintca["Maintenance mode: clients of that OS CA"]
ca -->|no| maintany["Maintenance mode: any client"]
Maintenance mode serves a node that is not installed¶
A machine reaches maintenance mode in one of two ways:
- It boots the installer, from a USB disk, an ISO or a network boot. The installer never touches the medium it boots from. It writes the node's image to another disk.
- It boots its image directly, written to its disk with
dd, used as a VM's disk or created by chalklab. On the first boot, systemd-repart adds slot B and an empty STATE behind the partitions the image ships with. STATE has noinstalledfile, so chalkd starts in maintenance mode.
In maintenance mode, chalkd serves a self-signed certificate and accepts only reads (node information, the image's contract, disks, logs and the drivers the firmware loaded), diagnostics, a reboot, a shutdown, the enrolment of Secure Boot keys and an install. How chalkctl trusts that certificate by the fingerprint on the console, and whom chalkd accepts, is explained in A node in maintenance mode is trusted by its fingerprint. Install a node covers the install as a task.
Normal mode serves an installed node¶
In normal mode, chalkd serves its node certificate and accepts only clients whose certificate chains to the OS CAs on STATE. Each method is open to the client roles allowed to call it, and the install method is closed.
When chalkd cannot read STATE well enough to tell whether the node is installed, it fails and restarts. It never falls back to maintenance mode, which would offer an installed node for a new install.
chalkctl and chalkd talk over the node API¶
They use the node API: Connect RPC over HTTPS with TLS 1.3 on TCP port 50000, with a client certificate on every call. chalkd takes the client's role from the certificate and checks the certificate again for every request, as chalkd checks the role of every request explains. Roles and credentials lists what each role may call.
Limits¶
- An image without an OS CA accepts any client in maintenance mode, so anyone on the machine's
network can install it. mkImages'
osCAcloses that. - How a node works lists what chalkos leaves to the operator, such as enrolling Secure Boot keys.
Related pages¶
- Images and platforms, Config and the inventory and The image for what Nix builds and what a node receives.
- Security model for the client roles and what Secure Boot and the TPM protect.
- Install a node for the install as a task.
chalkctl installand the node API for the details.- Architecture for developers for the code's layout, and The install in code for the steps of an install.