Skip to content

Boot, health and rollback

A node that rolled back from a new image reports it in the first lines of chalkctl status:

image <image>, booted from chalkos_<generation>.efi
upgrade to <new-image> failed: rolled back to <image>
  <the failed boot's last journal lines>

A node boots a chain of signed and hashed parts, from the firmware to chalkd, and each part checks the next. After an upgrade, systemd-boot gives the new image a fixed number of tries, and chalkd decides whether each try is healthy. A new image that never becomes healthy is replaced by the one the node ran before. A bad image therefore leaves a node running its previous image after a few reboots.

The boot chain checks each part before it starts the next

The boot chain runs from the firmware through systemd-boot and the UKI to the initrd. The initrd opens the store and the node's partitions before the system starts.

flowchart LR
    fw["Firmware"] -->|verifies and starts| sdboot["systemd-boot"]
    sdboot -->|picks; firmware verifies| uki["UKI"]
    uki -->|starts| initrd["initrd"]
    initrd -->|"usrhash="| store["Store on dm-verity"]
    initrd -->|TPM or console| state["STATE"]
    state -->|storage section| var["VAR"]
    store --> stage2["Stage 2"]
    var --> stage2
    stage2 --> chalkd
  1. The UEFI firmware starts systemd-boot from the ESP. With Secure Boot on, it first checks systemd-boot's signature against db and dbx.
  2. systemd-boot picks a UKI and asks the firmware to start it. The firmware checks the UKI's signature the same way, as The image describes.
  3. The UKI's stub starts the kernel with the initrd and command line the UKI carries, so its signature covers both.
  4. The initrd opens the store and the node's partitions.
  5. Stage 2 applies the node's config, starts the network and starts chalkd.

systemd-boot boots the first UKI in /EFI/Linux whose tries are not used up. Among a node's UKIs, the highest generation in the file name chalkos_<generation>.efi comes first, so the entry installed last boots first. Once its tries are used up, systemd-boot passes over it to the previous generation. chalkd sets no EFI variable to choose the entry.

The generation decides because every chalkos image sets IMAGE_VERSION in its os-release empty. systemd-boot orders the UKIs of one image ID by IMAGE_VERSION before their file names, and without the field by VERSION, which NixOS sets to its release. An empty IMAGE_VERSION leaves the order to the file name, so an older image installed by an upgrade still boots first. Boot lists the sort rules.

The initrd opens the store and the node's partitions

The initrd prepares the root file system in three steps:

  • systemd-repart compares the boot disk with the image's partition definitions. On the first boot of an image it adds slot B and an empty STATE. On later boots everything is there, and it changes nothing. It runs only on the disk systemd-boot was loaded from, which udev marks, so a second disk with the same image is never touched.
  • The kernel command line's usrhash= names the store's root hash. systemd finds the data and hash partitions whose UUIDs derive from it, opens them with dm-verity and mounts the store read-only.
  • chalkos-state.service unlocks and mounts STATE, and chalkos-storage.service then creates any missing volume and unlocks and mounts VAR. The TPM unseals their keys, and when it refuses, the console asks for the second key: the node's recovery key or a password. The initrd unlocks STATE first, then VAR describes both units.

Stage 2 applies the config before the network starts

Stage 2 starts the node's own services in order:

  1. chalkos-config.service applies the config that STATE keeps for the booted image, its config slot, before networkd starts. The config holds the hostname, the documents services read and the outputs rendered from them, networkd's units and chrony's time servers among them. A fallback boots the previous image with its own slot, so it runs the config it ran. An image without a slot fails here, and its boot is not found healthy. A config's trial that is still open ends first: the slot gets the config from before the trial back.
  2. systemd-networkd brings up the network.
  3. chalkd starts. It finds the installed marker on STATE and serves in normal mode, or finds none and serves in maintenance mode.
  4. On a boot that systemd-boot counts, chalkos-health.service decides whether the boot is healthy.

With the Kubernetes layer, Kubernetes starts once the network is online

On an image with Kubernetes, chalkos-kubernetes.service waits for chalkos-node-addresses.target, which follows network-online.target, and then prepares the node's certificates and configuration. containerd and the kubelet follow. chalkd does not wait for this preparation, so it answers while the node's addresses settle.

Boot lists the units with their commands.

A new image's entry boots a fixed number of times

An upgrade stages the new UKI as .staged-chalkos_<generation>.efi, which systemd-boot does not boot, and its activation renames it to chalkos_<generation>+<tries>.efi. <generation> is one above the highest generation on the ESP. <tries> is the new image's chalkos.upgrade.bootTries, 3 by default. The number travels in the image's os-release as CHALKOS_BOOT_TRIES, so the image being installed sets it, not the one running.

Before each boot of the entry, systemd-boot renames the file to count the try: +3 becomes +2-1, then +1-2, then +0-3. Once the system decides the boot is good, systemd-bless-boot.service renames the file to chalkos_<generation>.efi, without a counter. The entry is a normal, blessed entry from then on. An entry at +0 is used up, and systemd-boot skips it. The diagram shows the states of a new image's entry.

stateDiagram-v2
    state "Staged, not booted" as Staged
    state "Active with tries" as Active
    state "Counted boot" as Counted
    state "Tries used up" as UsedUp
    state "Previous image boots" as Fallback
    [*] --> Staged: upgrade writes the inactive slot
    Staged --> Active: activation names the entry
    Active --> Counted: reboot, systemd-boot takes a try
    Counted --> Blessed: health check passes
    Counted --> Counted: unhealthy, tries left, chalkd reboots
    Counted --> UsedUp: unhealthy on the last try
    UsedUp --> Fallback: chalkd reboots
    Blessed --> [*]
    Fallback --> [*]

The previous image's entry stays on the ESP throughout, and the upgrade never touches its slot, so the fallback boots exactly what the node ran before.

The health check decides whether a counted boot is good

chalkos-health.service runs chalkd health on every boot that systemd-boot counts, and only then. Only systemd-bless-boot.service pulls in boot-complete.target, which requires the health unit. The check repeats every 5 seconds until the node is healthy or chalkos.upgrade.healthTimeout passes, 300 seconds by default.

Every check starts with chalkd. It must complete a TLS handshake with the node's own certificate on port 50000, because a node that chalkd cannot serve cannot be managed and must fall back. On a node without the Kubernetes layer, and on a Kubernetes node without a Kubernetes share or whose control plane is not bootstrapped yet, two more criteria follow. The node's config must have been applied, and no unit of the boot may have failed. A unit of the boot is one that multi-user.target or sysinit.target pulls in. Jobs that timers and sockets start do not count, nor do the units listed in chalkos.upgrade.healthIgnoreUnits.

When the check passes, the unit succeeds, boot-complete.target is reached and systemd-bless-boot blesses the entry. Nothing rolls a blessed image back on its own.

With the Kubernetes layer, a cluster node is healthy when its Kubernetes parts are

A Kubernetes node that is part of a cluster is checked on its Kubernetes parts in place of its units, after the chalkd check. A Kubernetes node whose preparation failed is unhealthy. A worker is healthy when its kubelet's /healthz answers and its Node is registered. A control plane that is an etcd member is healthy when its local etcd member answers, its API server is ready and its kubelet's /healthz answers. A node that is not yet part of a cluster is checked like a node without Kubernetes.

Each kind of failure leaves the node in a known state

A boot can fail before chalkd runs, in the health check, or after the image was blessed. The diagram shows where each failure leaves the node.

flowchart TD
    boot["Boot"] --> tpm{"TPM unseals STATE?"}
    tpm -->|no| console["Console asks for the second key"]
    tpm -->|yes| hang{"Reaches the health check?"}
    hang -->|no| stuck["Stays in the boot until reset"]
    hang -->|yes| counted{"Boot counted?"}
    counted -->|"no, blessed"| runs["Runs; status lists failed units"]
    counted -->|yes| healthy{"Healthy within the timeout?"}
    healthy -->|yes| blessed["Blessed"]
    healthy -->|no| tries{"Tries left?"}
    tries -->|yes| retry["Record and reboot: next try"]
    tries -->|no| fallback["Record and reboot: previous image"]

The TPM refuses or chalkd cannot read STATE

When the TPM does not unseal STATE, for example after Secure Boot was turned off or db changed, the console asks for the recovery key or password. The node never drops to maintenance mode, which would let someone at the console install it anew. Storage and encryption explains the causes.

chalkd decides its mode from STATE alone. When it cannot read STATE to tell whether the node is installed, it fails and restarts, as The installed marker decides chalkd's mode describes.

A boot that hangs is never decided

The health check runs late in the boot, after chalkd starts. A counted boot that hangs before then is never decided by it, for example at the initrd's console prompt for a disk's second key, or in chalkos-config.service. The node stays in that boot until someone resets it. Only then does systemd-boot take the next try, or fall back once the tries are used up. A power reset from a BMC or a hypervisor is enough.

An unhealthy boot is recorded and rebooted

When the timeout passes, chalkd records the failure. It writes the last 30 journal lines of chalkd and the health check during this boot to /var/lib/chalkd/failed-boot on VAR, headed by the root hash of the store that booted and the generation of its entry. The record lives on VAR because the image the node falls back to must read it. The journal cannot serve: the read-only image gets a new machine ID at every boot, so each boot's journal stays apart.

Then chalkd reboots the node, but only when a reboot leads systemd-boot on. A reboot leads on to another try while the entry has tries left, and on the last try to an entry the node can fall back to. A reboot of a boot that is not counted, or of a used-up entry booted because nothing else is left, would boot the same image again in a loop. chalkd leaves such a node running and logs why. A health check that ends before it decides, stopped by its unit's timeout or crashed, is handled the same way.

After the last try, systemd-boot skips the used-up entry and boots the previous image, whose slot and UKI the upgrade never touched. Its entry carries no counter, so its boot is not counted, and no health check runs.

A blessed image is rolled back only on request

A blessed image that fails at a later boot stays. The health check does not run on its boots, so nothing rolls back on its own, and status lists the failed units. The check also judges only what it checks: a new image whose workloads fail while the parts the check covers are healthy is blessed.

chalkctl does not take a blessed entry for a healthy boot. When chalkctl node reboot or chalkctl upgrade waits for a node it rebooted, it asks the node's status for the health of the boot itself, and chalkd runs the health check's checks on the spot. While the boot is starting, chalkctl waits. Once systemd finished starting the boot and the image's health timeout passed, the time a counted boot has to become healthy, the boot has settled. A settled boot of a blessed image that fails the checks for a minute in a row stops the command, with what failed. A check that fails once, such as an etcd health check that answered late, does not stop it. Nothing rolls back.

chalkctl upgrade --rollback boots the image before again on request, while the inactive slot still holds it, as Upgrades describes. Whether its boot is counted depends on the boot the node runs:

  • After a boot that was found healthy, the image before boots counted, with the tries its UKI asks for. When its boots fail, systemd-boot falls back to the image the node ran.
  • After a boot that was not found healthy yet, the image before boots uncounted. Its boot was found healthy before, and it boots with the config it last ran, so it is the node's known-good image again and the node does not fall back from it. Until the running boot is found healthy, chalkd keeps that image's config as it last ran and refuses other config for it.

Recover a node and Troubleshooting cover the fixes.

Status reports the boot from the ESP

chalkd reads the boot from the ESP whenever it reports status: the entry it booted, whether that boot was blessed, an entry of the image with tries left that boots next, and a used-up entry it fell back from. The used-up UKI stays on the ESP until the next upgrade removes it. The record on VAR is removed once a counted boot is found healthy. Status, logs and the console explains each image line. chalkctl upgrade stops at a node that rolled back, as Upgrades describes.

Status also names the previous image, the one chalkctl upgrade --rollback boots. It is the newest entry below the booted one that is not used up and boots another store, while STATE keeps its config slot and the inactive slot holds no image staged or active. chalkd records on STATE the image the last rollback went to, and status reports that record while the node runs the image or boots it next.

Limits

  • Rollback protects the first boots of a new image only. Once blessed, an image stays, even if it fails later, until chalkctl upgrade --rollback boots the image before, which works only while the inactive slot still holds it.
  • The health check does not cover workloads. A regression that only workloads notice is blessed.
  • A hang before the health check, such as a console prompt, needs a reset from outside. The node cannot reboot itself from there.
  • Each try lasts up to the health timeout plus a reboot. With the defaults, a bad image runs for about 15 minutes before the node is back on its previous image.
  • An upgrade never updates or rolls back systemd-boot.