Skip to content

Test and roll out an image

This page is for an image author whose changed image is ready to reach running nodes. At the end the image has passed its build checks and its tests, every node's config has passed chalkctl validate, the nodes run the image, and chalkctl status and chalkctl logs show that the outputs rendered and the services read them. The examples use the homelab with the node-bird-metal image and the BIRD extension added by Write a BGP extension.

A change reaches a node in one of two ways. A changed image reaches it with chalkctl upgrade, which brings the node's config along. Changed YAML alone reaches it with chalkctl config apply, and needs no new image.

Before you begin

You need:

  • The homelab flake and inventory with the BIRD image and node1's files set up as in Write a BGP extension, and a cluster whose nodes run an earlier image.
  • The secrets file to deliver config, here secrets.age. chalkctl upgrade also runs with an operator client file, and then sends no config.
  • For nodes that enforce Secure Boot, a signed copy of the image, as Sign images for Secure Boot makes it.

The image build checks the image

Build the image:

nix build .#node-bird-metal -o images/node-bird-metal

The build fails before any node gets the image when one of these checks fails:

  • Every rule of the image's kinds and effects compiles, stays within the cost limit, and reads only facts that a module declares.
  • Every effect acts on units, outputs and kinds that the image defines.
  • Every output and validator is a program of the image's store, and reads kinds the image declares.
  • The store, its hash tree and the UKIs fit their partitions with room to spare, as Add software, drivers and kernel modules describes.

The error names the rule, the effect, the output or the validator.

nix flake check runs the image's tests

The image's tests, chalkos.tests, run each test's config through the checks a node runs for a delivery: they render every output and run its check, run the validators, and compare the files, refusals and effects with what each test expects. The homelab's flake exports them as its checks, so one command runs every image's tests:

nix flake check

It fails on a test that fails, with a diff for each output whose files differ from the test's golden files. bird-conf renders bgp-examples/fabric.yaml in the test bgp-fabric, bird -p parses the result, and the test compares it with its golden bird.conf. The image's build runs the tests only with chalkos.testsOnBuild. Test an extension declares the tests and covers the documents nodes hold.

chalkctl validate checks each node's config

chalkctl validate checks each node's documents against the schemas of the image the inventory names for it. It also runs the image's rules that read no facts, and reaches no node:

$ chalkctl validate
node1: valid for the image node-bird-metal (node-bird-metal on metal, root hash ae887bd49725), config 03694425cf0db40d0fbbe558ec86f70411dd93bd465c4eed8a5c8342808bc09c
node2: valid for the image node-metal (node-metal on metal, root hash 3b80d0bf1243), config 10c35f7fe988679c3a54691ddf95e78267269d22297fb360ceae5527f5965d50
node3: valid for the image node-metal (node-metal on metal, root hash 3b80d0bf1243), config 55ed0b193224383f68d0bb4014e45885697279cd67dfa6fc9232627757cfbd44
router: valid for the image router-metal (router-metal on metal, root hash cd7ac91cb655), config dc6178bbae55b1fe0e6af7b10502a1099f708c995b7be242ee95bb92050f87e6

Each line names the image as the inventory names it, the name and platform it was built with, the start of its root hash, and the hash of the config the node would get. A Peer with asn: 0, below the kind's minimum of 1, fails on every node that gets the file, here node1 alone:

$ chalkctl validate
...
chalkctl: the config of 1 node is invalid:
node node1: Peer "router": - at '/spec/asn': minimum: got 0, want 1

Rules that compare with the config a node runs, such as the Peer rule that keeps asn, and rules that read facts run when the config reaches the node. Guard changes with rules and facts describes where each rule runs.

Roll the image out with chalkctl upgrade

chalkctl upgrade gives each node the image the inventory names for it:

chalkctl upgrade --secrets=secrets.age

Nodes that run the image already are skipped. Each other node gets its config with the image: the documents of the files the inventory gives it, checked against the new image before any node gets the image. The node checks them again and keeps them for that image, so a fallback boots the image before with its own config. Before it sends a node anything, chalkctl prints a line of what the node's config changes, such as node1: config unchanged.

The node has the new image check its config before it installs it, with that image's own renderers and their checks. A config one of them refuses refuses the upgrade of that node, and the run stops there. The image renders its outputs again at its first boot.

Upgrade nodes covers the run's order, downtime and rollbacks.

Deliver changed YAML with chalkctl config apply

A change to the documents alone, such as a new Peer, needs no upgrade. --diff shows what a delivery changes in the documents. --dry-run has the node check the delivery, its renderers and their checks included, and print the effects it would take. Neither changes anything:

chalkctl config apply node1 --secrets=secrets.age --diff
chalkctl config apply node1 --secrets=secrets.age --dry-run
chalkctl config apply node1 --secrets=secrets.age

The node renders and checks every output before it changes anything, so a document that gives bird a config bird -p rejects refuses the whole delivery. chalkctl prints each unit the effects restarted, as restarted <unit>, and then the config the node runs, as node1 runs config <hash>. Reloaded units are not listed.

After an upgrade with --until stage or --until activate, the image in the node's inactive slot keeps a config of its own, staged or active. chalkctl config apply delivers to that image as well as to the running one, once both checked the config, and chalkctl config apply --staged delivers to that image alone. With --dry-run, both check the config and nothing changes.

Config for an image that is not installed yet can wait on the node. Without a staged image, chalkctl config apply --staged holds the config pending, and the next upgrade gives it to the new image, which checks it before the node installs anything. An operator can then run that upgrade without the secrets file.

chalkctl status shows the outputs

chalkctl status shows the image a node booted and whether the boot was found healthy. It also lists each output the node serves:

chalkctl status node1

The outputs form a table with three columns:

  • OUTPUT is the output's name, such as bird-conf.
  • HASH is the start of the hash of the files the node serves, or none while it serves none.
  • STATE is rendered, failed: with the last failure, or not rendered yet.

A node whose store is not the image the inventory names gets a line that starts with its store's root hash is, and that names the command that fixes it. A unit that failed is named as failed unit <unit>.

chalkctl logs shows a unit's journal

chalkctl logs prints a node's journal of the current boot, for one unit with --unit:

chalkctl logs node1 --unit=bird.service

A render or a check runs as a transient unit, chalkos-render-<n>.service, which keeps the program's standard error. The journal of chalkd.service names that unit beside the failure:

chalkctl logs node1 --unit=chalkd.service
chalkctl logs node1 --unit=chalkos-render-<n>.service

--follow keeps printing new entries until you interrupt it.

What next