Skip to content

How a node works

chalkos is an operating system for servers, which Nix builds as read-only images. Each node boots a signed image, takes its settings from YAML documents and is reached only through its agent, chalkd. An upgrade replaces the whole image, and a node that does not come up healthy on the new image falls back to the old one.

Three machines take part

The build host builds and signs the images, the operator's workstation drives the nodes, and the node runs an image. The build host and the workstation can be one machine. In the diagram, a solid edge carries data and a dashed edge carries trust.

flowchart LR
    subgraph build["Build host"]
        nix["Nix and mkImages"] -->|image directory| chalkbuild
        dbkey["db key"] --> chalkbuild
    end
    subgraph workstation["Operator's workstation"]
        inventory["Inventory and YAML documents"] --> chalkctl
        secrets["Secrets file"] --> chalkctl
    end
    subgraph node["Node"]
        firmware["Firmware"] -->|boots| uki["UKI"]
        uki --> chalkd
        tpm["TPM"]
    end
    chalkbuild -->|signed image| chalkctl
    chalkctl -->|"image and config over the node API"| chalkd
    dbkey -.->|db certificate| firmware
    secrets -.->|OS CA| chalkd
    firmware -.->|PCR 7| tpm

The three trust edges are the anchors of a node:

  • The firmware boots only binaries that a certificate in its db signed. The operator enrols the db certificate before the first boot.
  • chalkd accepts only clients whose certificate chains to the OS CA. The image can carry the OS CA, and the install puts it on the node's disk.
  • The TPM seals the disk keys to PCR 7, the measurement of the Secure Boot state. The keys unseal only while the firmware boots under the same keys.

chalkctl reads only the inventory, the YAML documents, the secrets file and the image directories, so a machine without Nix, an air-gapped one included, can drive the nodes. Architecture names the parts of a node and where each one runs. Security model explains each trust anchor.

A node passes through stages

A machine starts blank and becomes a node at its install. From then on it stays in normal mode, through upgrades and recoveries, until a reinstall. The state diagram shows these stages.

stateDiagram-v2
    state "Blank machine" as Blank
    state "Maintenance mode" as Maintenance
    state "Normal mode" as Normal {
        state "Installed" as Installed
        state "Bootstrapped or joined" as Joined
        [*] --> Installed
        Installed --> Joined: Kubernetes layer only
    }
    state "Staged" as Staged
    state "Counted boot" as Counted
    state "Rolled back" as RolledBack
    state "Console asks for the recovery key" as Recovery
    [*] --> Blank
    Blank --> Maintenance: boots the installer or its image
    Maintenance --> Normal: chalkctl install
    Normal --> Staged: chalkctl upgrade
    Staged --> Counted: reboot
    Counted --> Normal: health check passes, blessed
    Counted --> RolledBack: no tries left
    RolledBack --> Normal: previous image runs
    Normal --> Recovery: TPM refuses to unseal
    Recovery --> Normal: recovery key typed
    Normal --> Maintenance: reinstall boots the installer

The sections below follow these stages in order. Each one links the page that explains it.

Nix builds an image and chalkbuild signs it

A flake calls chalkos.lib.mkImages, which builds one image directory for each image the flake lists, each with a name and a platform. The image directory holds an unsigned raw disk image and the JSON Schemas of the config the image accepts. chalkbuild then signs a copy of the image directory on the machine that holds the db key, so the key never enters /nix/store. Every file of the store is covered by the signature, because the signed UKI carries the store's root hash and dm-verity checks every read against it. The image describes what an image holds.

An install turns a machine in maintenance mode into a node

A machine reaches maintenance mode when it boots the installer or a node's image written straight to its disk. Architecture describes both ways and the two modes. chalkctl install checks the fingerprint chalkd prints on the console, as A node in maintenance mode is trusted by its fingerprint explains. It sends the node its config, its certificates and its share of the secrets. A machine that runs the installer also receives the image the inventory names for it. chalkd lays out the disk, seals STATE to the TPM and marks the node installed. The node then reboots into normal mode.

The firmware checks the image and the TPM unlocks the disks

At every boot, the firmware checks the signatures of systemd-boot and the UKI against its db and dbx. The initrd unseals STATE and VAR with the TPM and mounts them. The node's config on STATE is applied before the network comes up, and chalkd starts in normal mode with the node certificate from STATE. Boot, health and rollback follows the boot chain. Storage and encryption explains STATE, VAR and the unlock.

With the Kubernetes layer, the nodes form a Kubernetes cluster after their first boot

chalkctl bootstrap starts etcd and the control plane on the first control plane, once per cluster. Further control planes and workers join on their own. The control plane describes the bootstrap and the joins.

YAML documents configure the node without a reboot

Everything that differs between nodes is config: YAML documents of kinds such as Host, Network and SystemDisk. The inventory says which documents each node gets. chalkctl config apply delivers them to a running node. chalkd checks them against the schemas and change rules of the image the node runs, keeps them on STATE and restarts only the services whose config changed. The node refuses a change that the image's rules forbid, such as a storage change that would destroy data. Config and the inventory describes the documents, and The config pipeline follows a delivery through the node.

chalkd carries out every operation

A node has no login shell. Status, logs, reboots, config, upgrades, etcd membership and rotations all go through the node API that chalkd serves on port 50000. Every call carries a client certificate, and its role (reader, operator or admin) decides which methods it may call. chalkctl signs itself a short-lived admin certificate from the secrets file, or uses a client file that holds a certificate of one role. Security model explains the roles, and Roles and credentials lists who may call what.

An upgrade writes the inactive slot and boots it on trial

A disk holds two slots for the store, A and B. chalkctl upgrade streams a new image to the node, or has the node pull it from a registry, and sends the node's config with it. chalkd writes the image into the inactive slot and checks the config against the new image's schemas. It then installs the image's UKI with a count of boot tries, 3 by default. The node reboots into the new image, and a health check decides within 300 seconds by default whether the boot is good. A good boot blesses the new image, and the node keeps it from then on. Upgrades explains an upgrade from the image to the bless.

A node rolls back on its own and recovers at the console

A node that fails its health check reboots and tries again. Once its tries are used up, systemd-boot boots the previous image, whose slot the upgrade never touched. The node records why the boot failed, and chalkctl status reports it. chalkctl upgrade stops at the first node that rolls back.

A different failure stops the boot before chalkd runs. When PCR 7 changed, for example after a change of the Secure Boot keys, the TPM releases no key, and the console asks for the node's recovery key. chalkctl recovery-key derives it from the secrets file. A node that nothing recovers is reinstalled from the installer. Boot, health and rollback explains the rollback, and Storage and encryption explains the unlock with the recovery key.

Rotation replaces a CA across the cluster

A certificate stays valid until it expires or its CA is removed, so removing the CA is how access is revoked. chalkctl rotate replaces a CA or key across the cluster in four phases. Each node confirms a phase before the next one starts, so no node ever presents a certificate that another node does not trust yet. Renewal needs no rotation: it issues a node a new certificate from the same CA before the old one expires. Certificates explains renewal, rotation and revocation.

With the Kubernetes layer, chalkctl node revoke shuts out one node

A revoked node's certificates renew no more, and the API server denies its kubelet every write. Revoking a node cuts off its writes and renewals lists what revocation leaves open.

What chalkos does not do

  • A node has no SSH server, no passwords and no interactive accounts, as Nobody logs in to a node explains. Logs come through the node API.
  • A node has no Nix and no package manager. Its store is read-only, and new software means a new image.
  • There is no update server. An install or an upgrade starts only when chalkctl starts it, and a node pulls from a registry only when chalkctl tells it to.
  • chalkos enrols Secure Boot keys only in firmware in Setup Mode, before the install, and installing does not check that Secure Boot is on. The Security model's limits say who enrols them otherwise.
  • chalkctl never builds or signs images and never runs Nix.
  • Nothing revokes a single client certificate or kubeconfig. One works until it expires or its CA is rotated out, as Access is revoked by rotating its CA explains. With the Kubernetes layer, chalkctl node revoke shuts out a node's writes and renewals by name.