Upgrades¶
An upgrade installs a new image on a node. chalkctl upgrade prints each node's progress, here
for one node of a run, shortened:
cp1: installing 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6)
cp1: installed 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6), staged as .staged-chalkos_2.efi
cp1: activated 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6), which boots next as chalkos_2+3.efi
cp1: rebooting into 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6)
cp1: runs 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6), found healthy
chalkd writes the image into the node's inactive slot and stages it there once the image's own checker accepted the node's config. Activating the image gives its UKI a boot entry's name with a fixed number of tries. The node then boots the new image, and an image that does not become healthy is replaced by the one before. This page follows one node's upgrade.
With the Kubernetes layer, chalkctl rolls the upgrade through the cluster
chalkctl upgrades the control planes first, one at a time, while etcd keeps its quorum, and drains each node before it reboots. Kubernetes upgrades describes the roll.
For recovery while chalkd answers but the cluster does not, chalkctl upgrade --force --nodes
<node> skips cluster version checks and layer coordination. It still validates the image and
config and waits for a healthy new boot. The node's checks below, including protection of its
fallback image, are unchanged. Recover a node whose cluster is unavailable
shows the command and the checks it skips.
An upgrade replaces the whole image¶
An upgrade replaces the whole image with another, which its store's root hash identifies. Any change to an image's modules gives it a new root hash, so a new build needs nothing raised before it is installed. A node has no packages to update. A new release of any software on it is a new image, installed like any other.
The node receives three parts of the image, in this order:
- the store's data, the compressed erofs file system that the root hash covers;
- the store's dm-verity hash tree;
- the UKI, signed for Secure Boot.
Every build leaves room for the next image gives the size of each part.
An upgrade never sends a boot loader, and the node refuses one, so systemd-boot stays as it was installed. The parts are the ones an install sends, read from the same image, so a node installed from an image and one upgraded to it run the same bytes.
The node keeps no upgrade state of its own. The slots' partition UUIDs and labels and the UKIs on the ESP say what is installed. So any step can stop at any point, and running the same command again continues.
chalkctl takes each node's image from the inventory¶
Without --image, each node gets the image the inventory names for it. chalkctl asks every node
what it runs, refuses a node that runs another platform than that image, and opens each image
directory once for the nodes that share it. The image in the directory must be the build whose
root hash its metadata.json names, which chalkctl status compares the nodes with. A directory
whose image does not match stays unusable until the image is rebuilt or the directory is linked
again. The root hash
stays on the operator's machine, so a new build changes no node's config, and an upgrade needs no
chalkctl config apply after it.
With --image, chalkctl takes a prebuilt image: a raw image with repart-output.json beside it,
or an image directory. It installs the image on the nodes that run an image of its name, or on the
nodes --nodes names, whatever they run. Nodes on another platform than the image's are skipped
and named. chalkctl prints each node that gets an image of another name than the one it runs.
The check of the node's config against the image's schemas decides whether such an image suits
it. In both cases chalkctl sends the image as the directory holds it, a copy that
chalkbuild sign signed, so the signing key stays on the machine that ran chalkbuild. Without an
inventory, --address NODE=ADDR names the nodes, and --image is needed.
An image the inventory names in a registry, by source.oci, is pulled by each node. chalkctl
resolves the reference to a digest once and reads the manifest, the header and the contract. It
sends each node the reference by that digest, with the header's sizes and SHA-256 sums of the
store, hash tree and UKI. chalkd pulls those layers, through the node's
Mirror of the registry if it has one, and checks them against
what chalkctl sent. It authenticates to each host with the
Registry of that host, from the config the upgrade delivers
or else the config it runs. It pulls over HTTPS unless that Registry sets insecure.
Publish images to a registry covers the
steps.
chalkctl checks the image before it sends anything¶
Before it sends anything, chalkctl checks each image against the nodes:
- the UKI boots the store the image holds: its
usrhash=equals the store's root hash; - the UKI's os-release sets
IMAGE_VERSIONempty, so systemd-boot orders the node's entries by generation, and names the image and its platform; - the config the inventory gives each node that gets the image suits the
schemas/andfacts/in the image's directory, as an install checks the config it delivers.
chalkctl checks every node's config before it changes the first node, and a config the image does
not accept stops the run with the config of <node>: .... It needs the inventory for
that, whatever it authenticates with. Without one, the node's own check is all there is. For an
image in a registry, chalkctl checks the config against the artifact's contract and reads no UKI,
so the node alone checks the UKI's usrhash= and os-release.
chalkctl checks no signature. Each node checks the UKI against its firmware's db and dbx before it writes anything, as The image describes.
Operator upgrades carry no config¶
An upgrade with an admin client file or the secrets file carries config. chalkctl sends each node its config with the image: the set the inventory gives the node, which it checked against the image. The image checks the set again, with its own chalkd, and the node keeps it in the new image's config slot, beside the slot of the image it runs. The new image boots with its own config, and a fallback boots the previous image with the config it ran. Config that only the new image accepts, such as a document of a kind its extension declares, therefore goes with the upgrade.
The Upgrade method needs the operator role, and the admin
role when it carries config or a fallback secret, as config apply does. With an operator client
file, chalkctl sends no config. With an inventory it prints one line for the run, in place of a
line per node:
config not delivered: the operator role cannot deliver config; each node gives the image its pending config, else carries its own over
Without an inventory, chalkctl sends no config either. In both cases the node gives the new image its pending config, when it holds one, or else carries the documents of the running image's slot over, checked against the new image the same way.
Before it asks for any password or drains a node, chalkctl asks the first node it upgrades for its
config. A node that refuses to show its config to the credentials would refuse config with the
upgrade too. The run then delivers no config to any node and prints
<node> refuses to show its config to these credentials; config not delivered, each node gives the image its pending config, else carries its own over.
Without an inventory, as with --image and --address <node>=<address> outside the inventory's
directory, it prints for each node
<node>: no inventory to compare the config with; the node gives the image its pending config, else carries its own over.
The pending config is the set that chalkctl config apply --staged stores in
/state/config/pending on a node without a staged image. Every later config apply to the
running image replaces it with the config the running image applied, so the next upgrade boots the
config delivered last. chalkd only parses it there; the image
an upgrade installs checks it as it checks any config. chalkd removes the pending config once the
upgrade staged that image, and activated it unless the upgrade stops at staging, whether the image
got it or the upgrade carried config of its own. An upgrade that fails before, in the image's
check or while chalkd stages the image, keeps it. An activation that fails, in an upgrade that
activates the image as well, leaves the image staged while the node keeps the pending config.
Sending that image again without config then rewrites its slot with the pending config. After the
line of the install, the run prints
<node>: the image took the pending config <version>, which it checked first or
<node>: the inventory's config replaced the pending config <version>. An upgrade that stages the
known-good image a rollback left keeps the pending config, because that image keeps the config it
last ran. So does an upgrade to the image the node runs already. The run then prints
<node>: keeps the pending config <version> for a later upgrade. Should
the removal itself fail, chalkd logs it, the upgrade still succeeds, and status keeps showing the
pending config until config discard removes it. An upgrade that pulls its image
takes the Registry and Mirror documents from the same config: the upgrade's, else the pending
config, else the running one. A pending config holds no fallback secret, so an upgrade refuses
one that adds an encrypted volume, and the error says to upgrade with the config from an inventory.
The config meets the new image's change rules as config apply's
config meets the running image's. The rules compare it with the running slot's config and the
storage STATE records. A storage change that takes a reset or a reinstall is refused, and so is a
change of a setting the rules fix. A new volume is created before the image
boots, while the running image runs. chalkctl sends the fallback secret such a volume is enrolled
with: the recovery key from the secrets file, or a password from --password-file. Without
--password-file, chalkctl asks for each node's password on the terminal before it drains the
first node. The new image's checker fills in its own defaults on both sides of the comparison, so a
default that changed between the images is no change to a rule that keeps a field's value.
Config and the inventory lists what
each change takes.
Before it sends a node the image, chalkctl prints a line of what the config changes against the
running image's, the slot the new one replaces, such as w1: config 1 added, 0 removed, 1 changed.
Without an inventory, the line says there is nothing to compare with. chalkctl upgrade --diff
prints the whole comparison for every node and changes nothing.
The node checks the image against itself before it writes¶
Before it changes anything, chalkd needs normal mode and checks the image's header against the running image's os-release and the disk:
- The image ID and platform equal the running image's, and the architecture is the machine's.
- The root hash is not the running one. If it is, the node answers that it runs the image already and changes nothing.
- The entry the node booted, and any other the upgrade keeps, is named
chalkos_<generation>.efiwith a generation from 1 on and no leading zeros. No name on the ESP gives a generation too large for one to follow it in 64 bits. chalkd never writes other names. It refuses to upgrade past one placed by hand or corrupted until the node is reinstalled or the entry renamed. - The root hash is not that of the entry that boots next: the highest generation with tries left that the node did not boot. If it is, and the inactive slot holds its store, the image checks the config the upgrade carries, as step 4 below does. chalkd keeps the config for the image in place of the config it had, and answers that the image is staged already, without writing the image again. When the check refuses, it retires the slot and removes the entry and the image's config slot, so the running image boots next.
- The running boot was found healthy, when the upgrade would remove the image the node falls back to.
- The store and hash tree fit the inactive slot.
Any other root hash is installed as the next generation: a newer image, an older one, or one whose earlier entry used up its tries, which the retirement then removes.
The UKI is checked when it arrives, before anything can boot it. Its SHA-256 must match the
header. Its os-release must name the same image ID and platform as the header and set
IMAGE_VERSION empty. Its usrhash= must equal the header's root hash, and its boot tries must be
a positive number. With Secure Boot enforced, chalkd checks its signature as
The UKI is checked against db and dbx before anything runs it
describes.
chalkd stages the image in five steps and activates it in a sixth¶
chalkd stages an image in five steps and activates it in a sixth. At every point between them the disk boots the running image, or, after the last step, the new one with all its tries.
flowchart TD
start(["Upgrade starts"]) -->|"disk boots: running"| retire["1. Retire the inactive slot"]
retire -->|"disk boots: running"| write["2. Write and verify the store"]
write -->|"disk boots: running"| uki["3. Receive the UKI"]
uki -->|"disk boots: running"| config["4. Check and keep the config"]
config -->|"disk boots: running"| stage["5. Stage"]
stage -->|"disk boots: running"| activate["6. Activate"]
activate -->|"disk boots: new, all tries"| reboot(["Reboot"])
The Upgrade call runs all six steps, or the first five when its header asks to stop once the image
is staged, as chalkctl upgrade does. The ActivateImage call runs the sixth, which chalkctl
upgrade and chalkctl node activate call.
- Retire the inactive slot. chalkd removes every UKI of the image on the ESP except those that
boot the running store. This includes a UKI that a stopped upgrade left under its temporary
name, the UKI of an image staged before, and a file whose name gives a generation of the image
but that is no UKI. A UKI whose
os-release names no
IMAGE_ID, such as another system's, stays. A UKI the ESP fails to read stops the upgrade. Then chalkd gives the slot's partitions random UUIDs and the label_empty, so nothing can boot them. - Write and verify. chalkd writes the store and hash tree into the slot, checks their SHA-256 sums and verifies the store read back from the disk against the root hash. A slot that already holds this store is not written again.
- Receive the UKI. chalkd writes it to the ESP as
/EFI/Linux/.chalkos-<root-hash>.efi, a name systemd-boot ignores, and checks it. - Check and keep the config. chalkd opens the written store read-only through dm-verity with the
image's root hash and runs the image's own
chalkd check, from the system the UKI boots, in a sandbox whose/nix/storeis the image's store. The image checks the config the upgrade carries, or without one the node's pending config, or without that the running slot's documents, as it checks a delivery once it runs: the set rules, each document's schema, the cluster it gives on an image with Kubernetes, the image's change rules against the running slot's documents as they were delivered, with the image's own defaults filled in, the storage STATE records and the facts the image declares, and every output's renderer and check. The checker never sees a Secret's value. chalkd then makes the storage changes the image's rules accept, such as a new volume, and writes the image's config slot/state/config/<root-hash>, with the image's defaults and the delivered Secret values. An image withoutchalkd check, built before it existed, is refused. When the check refuses, chalkd answers with the reason, removes the UKI and leaves the slot retired. The running image stays the next boot, and the next upgrade reuses or overwrites the slot. A pending config stays when the image refuses it, and the error names it. A node whose running slot holds no documents and that gets none refuses the upgrade. - Stage. chalkd gives the slot the partition UUIDs derived from the root hash and the labels
store_<hash16>andstore-verity_<hash16>, from the root hash's first 16 hex characters. It renames the UKI to.staged-chalkos_<generation>.efi, of the generation one above the highest any name on the ESP gave before step 1, whatever its UKI holds. systemd-boot ignores the name, as it does the temporary one, and nothing boots the slot: only a UKI systemd-boot boots looks for a store by the slot's UUIDs. An upgrade's sweep of temporary UKIs leaves the staged one, so the image stays staged across a restart of chalkd and a reboot, which boots the running image. chalkd then records the image's revision and tag in/state/chalkd/images.json. It forgets the images that no UKI on the ESP boots any more, the staged one counting as a UKI that boots its image, with their config slots. A refused upgrade forgets the images it retired the same way. Last, chalkd removes the pending config that the image took or that the upgrade's config replaced. - Activate. chalkd checks the staged UKI again as step 3 checked it, Secure Boot included,
because db and dbx may have changed since. It does not check the config again: the image's
checker passed it in step 4. Then it renames the UKI to
chalkos_<generation>+<tries>.efi, of the staged generation, or one above the highest any name on the ESP gives when that is higher. The rename is the only change, so an activation that fails leaves the image staged. ActivateImage refuses when no image is staged, when the staged image has another root hash than the one the request names, and when STATE keeps no config slot of the image.
The running slot and its UKI are never written. When the client stops sending, chalkd stops between steps, never in the middle of a partition table change.
The node boots the new image on trial¶
Once the image is staged, chalkctl has the node's layers hold it, activates the image, reboots the
node and waits until it runs the new image's root hash in a new boot, as the boot ID in its status
tells, and its boot was found healthy: the entry is blessed, and the node's status, which runs the
health check's checks when chalkctl asks, finds the boot itself healthy. The status check also
covers the boot of an entry blessed before, which systemd-boot does not count.
chalkctl node reboot runs the same step. Once the boot has settled, after systemd finished
starting it and the image's health timeout passed, chalkctl stops when the boot stays unhealthy for
a minute. A check that fails once, such as an etcd health check that answered late, does not stop
it, and a healthy look starts the minute over. systemd-boot counts the new entry's tries, and the
health check blesses the entry once the boot is good.
Boot, health and rollback explains the counted tries, the health check,
the blessed boot and the fallback.
chalkctl upgrades the nodes that act for the others, such as Kubernetes control planes, one at a
time, and then the other nodes in batches of --max-unavailable, 1 by default. A node without
Kubernetes counts among the other nodes. Within each of the two groups the nodes go in the order
of their names. A batch that fails stops the run once the batch's other nodes finish. With
--until stage, it stages the image on every node and stops, and chalkctl node activate later
makes it boot next. With --until activate, it stages and activates the image on every node
without holding or rebooting it, and each node boots the new image at its next reboot. A node that
has the image staged or active already gets the image again only when the config kept for it is
not the inventory's. The node then replaces that config without writing the image again.
Staging another image replaces a staged one. An image the run leaves staged or active boots with
the config the upgrade gave it, unless a later config apply changes it.
A rolled-back node stops the run¶
A run stops at the first node that fails and leaves the rest as they are. The most important case is a rollback: a node that fell back from the new image. The error names the node and the image it runs again, and the journal lines its failed boot recorded follow it:
chalkctl: w2: upgrade to 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6) failed: rolled back to 81bd02c4a9e1 (v1.4.0, 0f1e2d3c4b5a); it stays cordoned; its boots logged:
A run also stops when a node does not come back healthy within --timeout, 30 minutes by
default.
Running the command again continues¶
Running the same command again continues where the last run stopped:
- A node that runs the new image with a blessed boot is skipped.
- A node that booted the new image and is still being counted is waited for.
- A node that has the image staged or active already is activated, where it is staged, and rebooted without the image being sent again.
- chalkctl uncordons only the nodes it cordoned itself.
An image whose transfer was interrupted leaves the inactive slot empty, and the node keeps booting its current image. The next run writes the slot in full.
A node that fell back from the image stops every later run too, so a known-bad image never boots
again by itself. --retry-failed installs it once more on such nodes, as the next generation, for
a failure that lay outside the image. Each node is upgraded once per run, so a second fallback
stops the run again. Usually the fix is a new build, which has a new root hash and is installed
like any other image.
A rollback boots the image before again¶
After an upgrade, the inactive slot holds the image the node ran before, and its UKI and config
slot stay too. An upgrade removes an image's UKIs before it writes over that image's slot, so a
UKI on the ESP of an image the node does not run means the slot still holds its store.
chalkctl upgrade --rollback uses that to boot the image again without sending it.
chalkd's RollbackImage refuses an image whose config slot STATE no longer keeps, since the image would boot without config. It checks that the inactive slot still holds the store and checks the UKI as an upgrade checks a new one, Secure Boot included, because db and dbx may have changed since the UKI was installed. Then it renames the UKI to the next generation. Nothing is written to the slot. While the running image's boot was found healthy the new name carries the UKI's tries, so the boot is counted and falls back to the running image. Before that it carries none, as Boot, health and rollback explains. An upgrade's staged or active image has written over the inactive slot, so a rollback is refused while the slot holds one. A node also keeps no previous image after an install, and after an upgrade or rollback it fell back from.
A counted rollback that fails its tries falls back to the image the node ran, and the run stops
with <node>: rollback to <old> failed: it fell back to <new>. A node whose running image was not
found healthy yet gets the image before without boot counting. That image was found healthy
before, so it is the node's known-good image again, and the node never falls back from it.
chalkctl says so on the boots next line, releases the node once its layers report it ready, and
ends the node's steps with <node>: runs <old> (its boots are not counted). Until the running boot
is found healthy, the node keeps the config that image last ran, and refuses config apply
--staged and an upgrade to that image with other config.
The run around it is an upgrade's: the same order, batches, layer hooks, reboot and wait for a
healthy boot. chalkctl picks one target image for the run at plan time, from the selected nodes'
status. chalkd records on STATE the image the last rollback went to, /state/chalkd/rollback,
and status reports that record while the node runs or stages the image. A node's vote is that
record, else its previous image. Nodes that already run the target are skipped, so a run started
again continues instead of booting the rolled-back nodes forward, and a second rollback after a
finished one changes nothing. Nodes whose votes differ are refused before anything changes. Votes
differ on a cluster whose control planes and workers run different images, and for a node that
went through other upgrades, such as one installed after the others' last upgrade: its previous
image is the one it was installed with. A rollback run with --until activate boots at the next
reboot.
An upgrade removes the record. chalkctl upgrade to the inventory's image therefore undoes a
rollback once the node runs the image before: the newer image is still in the inactive slot and
is reused without being written again. An upgrade to the image the node runs leaves a staged
rollback staged.
A downgrade is an upgrade to another root hash¶
chalkd has no notion of older or newer. A downgrade is an upgrade to another root hash, installed as the next generation, so systemd-boot boots it first like any upgrade. When its tries are used up, the node falls back to the image it ran. When the inactive slot still holds the older image, as after one upgrade, its store is not written again. Neither chalkctl nor chalkd warns of a downgrade. Whether the older image still suits the node's data, such as etcd's or a workload's, is for the operator to know.
Secure Boot decides which images a node trusts¶
chalkd checks an image's platform, the node's config and the store the UKI boots. Secure Boot decides whose images a node runs, as Secure Boot limits a node to images the key holder signed explains, together with what an operator can install without it.
An image a node pulls from a registry passes the same checks as a streamed one, against the digests and root hash chalkctl read. The registry needs no more trust than a directory an image is streamed from.
After a rotation of the OS CA, images rebuilt from the new secrets.pub.json and rolled out with
chalkctl upgrade bring the trust of maintenance mode up to date, for nodes that are installed
again later.
Limits¶
- Every node receives the whole compressed store with its hash tree and UKI. There are no delta updates, and nodes do not share a download.
- An image reaches a node only when chalkctl starts an install or an upgrade: streamed from a directory, or pulled from a registry.
- An upgrade never updates systemd-boot.
- The health check does not cover workloads. An image that breaks only workloads is blessed, and the run continues.
Related pages¶
- Boot, health and rollback for boot counting and the health check.
- The image for what an image holds and how its root hash identifies it.
- Kubernetes upgrades for the roll through a cluster.
- Upgrade nodes for the task, step by step.
chalkctl upgradeand the Upgrade method for the details.