Skip to content

Publish images to a registry

This page is for an image author who ships images through an OCI registry instead of a directory on the operator's machine. At the end the image is in the registry, and the inventory names it by digest, so nodes pull it themselves on an install or an upgrade.

The registry holds the image as one OCI artifact, with the same parts as an image directory: the store, its hash tree, the UKI, the boot loader and the contract, which describes the config the image accepts. chalkctl validate, chalkctl config apply and chalkctl status read the image's header and contract from the registry. chalkctl install on a machine that runs the installer and chalkctl upgrade have the node pull the rest.

Before you begin

You need:

  • The flake of Your first image flake.
  • chalkbuild, from nix shell github:trevex/chalkos#chalkbuild or from chalkos.packages.${system}.chalkbuild in the cluster flake's dev shell.
  • Push access to a registry, with credentials that docker login stored in docker's config or a credential helper. chalkbuild reads them from there.
  • For machines that enforce Secure Boot, the db key and certificate of Sign images for Secure Boot.

Expose the layout

An OCI image layout is the directory form of an OCI image, which registries and tools such as skopeo copy. Each image mkImages returns carries its layout as oci, such as images.node-metal.oci. Each layout is built only when something asks for it. Expose the one to publish as a package of the cluster's flake, where images is the result of mkImages:

packages.x86_64-linux.node-metal-oci = images.node-metal.oci;

chalkos.lib.ociOf images returns every image's layout by the image's name.

Build the layout

Build the layout and link it next to the image directory:

nix build .#node-metal-oci -o oci/node-metal

The oci outputs run chalkbuild oci on the image directory. The same image directory always makes the same layout, so the command gives the same result outside Nix, from a directory that nix build already wrote:

chalkbuild oci images/node-metal oci/node-metal

The layout directory must not exist, as for every chalkbuild command that writes one.

Sign the layout

Skip this step when the nodes do not enforce Secure Boot. Otherwise sign a copy of the layout with the db key:

rm -rf oci/node-metal-signed
chalkbuild sign --key keys/db.key --cert keys/db.crt oci/node-metal oci/node-metal-signed

chalkbuild sign signs copies of the UKI and boot-loader layers, checks them against --cert and writes a new layout with them and a new manifest. The store, hash tree and contract layers are copied as they are. The output directory must not exist, so the rm -rf removes the signed layout of the previous build.

Push the layout

chalkbuild push pushes a layout to a reference that names the registry and a tag or a digest:

chalkbuild push oci/node-metal-signed ghcr.io/example/node-metal:v1.5.0

Push oci/node-metal instead when you skipped signing. The command prints the reference of the pushed manifest by digest, ghcr.io/example/node-metal@sha256:<digest>.

chalkbuild reaches a registry over plain HTTP at localhost with a port, at a name ending in .localhost, at 127.0.0.1, at ::1, or at a private (RFC 1918) address, and any other registry over HTTPS. Some registries, Quay among them, reject a manifest of an artifact type they do not recognise until an operator allow-lists application/vnd.chalkos.image.v1. The layout is a standard OCI image layout, so any tool that copies one, such as skopeo or oras, can push it too.

Name the digest in the inventory

The image's source in the inventory's spec.images takes the reference as oci, in place of directory. Use the reference by digest that chalkbuild push printed:

chalkos.yaml
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: homelab}
spec:
  images:
    router-metal: {source: {directory: images/router-metal}}
    node-metal: {source: {oci: "ghcr.io/example/node-metal@sha256:<digest>"}}
  groups:
    router: {imageName: router-metal, files: [router/*.yaml]}
    nodes: {imageName: node-metal, files: [cluster/*.yaml, bgp/fabric.yaml, bgp/secrets.yaml]}
  nodes:
    router: {address: 192.168.1.2, groups: [router]}
    node1: {address: "fdc8:a1b2:c3d4:ff::11", groups: [nodes], files: [nodes/node1.yaml]}
    node2: {address: "fdc8:a1b2:c3d4:ff::12", groups: [nodes], files: [nodes/node2.yaml]}
    node3: {address: "fdc8:a1b2:c3d4:ff::13", groups: [nodes], files: [nodes/node3.yaml]}

A reference by digest pins the image, so the inventory changes only when you push a new one. The inventory may name a tag instead, such as ghcr.io/example/node-metal:v1.5.0. chalkctl resolves a tag to a digest once per command, so one command reads one image even when the tag moves while it runs. Config and the inventory covers the rest of spec.images.

Record the release's tag in the image

Two tags are involved, and nothing compares them:

  • The tag argument of mkImages records where the image came from. metadata.json and the artifact's header carry it, and chalkctl shows it beside the root hash, as in 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6). It never reaches the store, so it changes no root hash.
  • The tag in the reference that chalkbuild push takes names the manifest in the registry. chalkctl resolves it to a digest once per command.

Give both the same value, the release's git tag. Nix cannot read git tags, so the build passes it in. Here the flake reads it from the environment variable RELEASE_TAG:

images = chalkos.lib.mkImages {
  revision = self.rev or self.dirtyRev or null;
  tag =
    let
      release = builtins.getEnv "RELEASE_TAG";
    in
    if release == "" then null else release;
  images = [
    {
      name = "node-metal";
      platform = "metal";
      modules = [ ./. ];
    }
  ];
};

builtins.getEnv reads the variable only with nix build --impure. Without it, the build reads an empty string and records no tag. Either way the store, and so the root hash, stays the same. The header changes with the tag, so the manifest's digest does too: two builds that differ only in their tag have one root hash and two digests. A node identifies an image by its root hash, so both install the same image. Signing writes a new manifest too, so pin the digest that the push of the signed layout prints.

Publish from a CI job

A release job runs the steps of this page in order: build the layout, sign it, push it and pin the pushed digest in the inventory. This script does it for one image, with the tag as its argument:

publish.sh
#!/usr/bin/env bash
# Builds node-metal as an OCI layout, signs it, pushes it under the release's tag and pins the
# pushed digest in chalkos.yaml. Run it in the flake's directory, on a checkout of the tag.
set -euo pipefail
tag=$1
repository=ghcr.io/example/node-metal

RELEASE_TAG=$tag nix build --impure .#node-metal-oci -o oci/node-metal
rm -rf oci/node-metal-signed
chalkbuild sign --key "$DB_KEY_FILE" --cert keys/db.crt oci/node-metal oci/node-metal-signed
reference=$(chalkbuild push oci/node-metal-signed "$repository:$tag")
sed -i "s|$repository@sha256:[0-9a-f]*|$reference|" chalkos.yaml
chalkctl validate

The job takes the db key from its secrets, as the file DB_KEY_FILE names, and the registry's credentials, in docker's config as docker login writes them. Only chalkbuild reads the key, so it never enters the Nix store. sed replaces the digest the inventory names, so the inventory must name the image by digest already. chalkctl validate then checks every node's files against the pushed image. Commit the changed chalkos.yaml, or open it for review, and roll the image out with chalkctl upgrade.

The artifact holds the image and its contract

The artifact is one manifest of the artifact type application/vnd.chalkos.image.v1. Its config is the image's header: its ID, root hash, architecture, platform, revision and tag. Its layers are the image's parts, uncompressed. Image and partition layout lists the media types and the header's fields.

Install from the registry

chalkctl install without --image, on a machine that runs the installer, sends the installer the image's reference by digest in place of the image:

chalkctl install node1 --secrets=secrets.age --address=<address> --fingerprint=<fingerprint>

It prints a line of this form, where <serial> is the system disk's serial, <root-hash> and <revision> the first 12 hex digits of the image's root hash and git revision, and <digest> the manifest's digest:

installing node1 onto serial "<serial>": chalkos <root-hash> (<revision>), which the installer pulls from ghcr.io/example/node-metal@sha256:<digest>

The installer pulls the store, the hash tree, the UKI and the boot loader, and takes the image's schemas, layer facts and partition definitions from the artifact's contract layer. The machine that runs the installer therefore needs a route and DNS to the registry or to an endpoint of its Mirror, as Ports and firewall lists.

The installer checks the pulled layers as it checks a streamed image: their digests, the store against the root hash, and the UKI and boot loader against the firmware's db and dbx when Secure Boot is enforced. It reads the manifest and the contract before it touches the disk. When no source serves them, because each is unreachable, refuses the credentials or lacks the digest, the install fails with an error that names the reference, and the disk stays as it was. An install that stops partway continues when the command runs again, by a pull or with --image.

Upgrade from the registry

chalkctl upgrade resolves the inventory's reference to a digest once and sends each node the reference by that digest, so every node pulls the same image even when the tag moves during the run:

chalkctl upgrade

chalkctl reads only the manifest, the header and the contract from the registry. Each node pulls the store, the hash tree and the UKI itself, so a node needs a connection to the registry or to an endpoint of its Mirror, as Ports and firewall lists. chalkd checks the pulled header and layer digests against the ones chalkctl sent. Then it checks the layers as it checks a streamed image: their SHA-256 sums, the store against the root hash, and the UKI against the firmware's db and dbx when Secure Boot is enforced. A layer whose source stops sending data for two minutes after it started stops that node's upgrade. A node that has the image staged already gets it without contacting the registry.

A private registry needs credentials on each node, in a Registry document of its config. Registries and mirrors sets them up, and pulls through a mirror.

Check that it worked

chalkctl validate reads the header and contract from the registry and checks each node's config against them:

chalkctl validate

For each node whose config is valid it prints a line of this form, where <root-hash> is the first 12 hex digits of the image's root hash and <hash> the hash of the node's config:

node1: valid for the image node-metal at ghcr.io/example/node-metal@sha256:<digest> (node-metal on metal, root hash <root-hash>), config <hash>
The digest is that of the manifest chalkctl read, so it names the image that was checked even when the inventory names a tag.

What next