Skip to content

Config and the inventory

The lab template's node cp1 gets a file of its own, with its system disk and its address on the lab network:

nodes/cp1.yaml
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
  type: controlplane
---
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
  disk: {path: /dev/vda}
---
# The card on the lab network, between the VMs. The other card reaches the internet and the
# ports chalklab forwards from 127.0.0.1, configured by DHCP.
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-lab}
spec:
  match: {macAddress: "52:54:00:7b:00:11"}
  network: {address: [192.168.123.11/24]}

A node's config is the set of YAML documents that makes it this node. It gives the host name, disks, network, time servers and the values of extensions, and with the Kubernetes layer the Kubernetes settings. The image is the same on many nodes, and the config is what differs. An inventory names the cluster's nodes and, for each, the address chalkctl reaches it at, the image it runs and the files whose documents form its config. chalkctl checks the config against the node's image and delivers it. chalkd checks it again, keeps it on STATE for the image it is for and applies it at every boot of that image.

A config document is an object of one kind

A config document is one object in the style of a Kubernetes object: an apiVersion of a group and a version, a kind, a metadata.name and a spec. The kind's JSON Schema, which the image carries, defines the spec, its defaults and what it refuses, so a misspelled field is an error and not a value that nothing reads. A file holds any number of documents, separated by ---.

The lab template's shared file holds the documents every node of the lab gets:

cluster/cluster.yaml
# Kubernetes with its defaults: IPv4, the pod range 10.244.0.0/16 and the service range
# 10.96.0.0/12.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
  # cp1 on the lab network: the API server of the only control plane.
  endpoint: https://192.168.123.11:6443
  # The images of the cluster's components, which containerd pulls as written. The tags of the API
  # server, the controller-manager, the scheduler and kube-proxy follow the kubelet of the image the
  # flake builds, the newest Kubernetes release chalkos pins; the image's checker holds them to
  # Kubernetes's version skew policy. The digests pin what containerd pulls. Read the digest of
  # another reference with `crane digest <ref>` or
  # `skopeo inspect --format '{{.Digest}}' docker://<ref>`.
  images:
    apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
    controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
    scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
    proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
    etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
    coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
    pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
    flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204

chalkos's kinds live in the group chalkos.dev

chalkos's own kinds live in the group chalkos.dev and its subdomains, all at version v1alpha1. Config kinds lists every field of each, generated from the schemas. Every image accepts these:

Kind Group Per node What it holds
Host chalkos.dev at most one The node's hostname, which defaults to the node's name
Secret chalkos.dev any Secret values by key, for the fields that reference them
SystemDisk storage.chalkos.dev exactly one The system disk, the size of VAR and the encryption of the node's partitions
Volume storage.chalkos.dev any A further volume: its disk, size, format, mount point and encryption
TimeSync time.chalkos.dev at most one Time servers, and whether those DHCP announces are used too
Network, NetDev, Link networkd.chalkos.dev any systemd-networkd's .network, .netdev and .link files, one document per file
Registry oci.chalkos.dev any Credentials of one registry or mirror host, and whether chalkd and containerd reach that host over plain HTTP
Mirror oci.chalkos.dev any The mirrors of one registry

The Per node column follows from each kind's multiple. A kind declared with multiple = true lets a node hold several documents of it, each with a name of its own, such as one Volume per volume. Without it, a node holds at most one, and chalkctl and chalkd refuse a second.

Registry and Mirror exist on every image, because chalkd reads them for the pull of an install's or an upgrade's image, and containerd's mirror configuration is written from them. A Registry gives a host's credentials and allows plain HTTP to it. A Mirror names the endpoints chalkd and containerd try before a registry itself. The Network, NetDev and Link documents mirror networkd's units one to one, as Network documents map one to one onto networkd units describes.

With the Kubernetes layer, four kinds describe the cluster

Images with Kubernetes also accept the kinds of the group kubernetes.chalkos.dev. A node holds exactly one Cluster, with the cluster's endpoint, families, ranges and VIP, and exactly one Node, with its type, controlplane or worker, and its own Kubernetes settings. Images built with flannel accept at most one Flannel, and any number of Addon documents hold objects the control planes apply. chalkctl delivers Addon documents to control planes alone; a shared file may hold them, and a worker's set leaves them out. On any other image these kinds are unknown, and a node without Kubernetes has no type. The kubelet also takes a Registry's credentials for every pod's pulls from its host. Four kinds of document describe the cluster explains them.

An image declares only the kinds its modules bring. An extension adds kinds of its own in a group outside chalkos.dev, declared in the NixOS modules of an image, as Declare a config kind shows. Their documents go into the same files as chalkos's. The Rack kind of the extensions example records where a node is mounted, and the example's node file holds one:

apiVersion: rack.example.org/v1alpha1
kind: Rack
metadata: {name: rack}
spec:
  location: rack-a/u12

The inventory gives each node a set of files

The inventory is a YAML document of its own kind, Inventory, that chalkctl reads from the file --inventory names, else the file $CHALKOSINVENTORY names, else chalkos.yaml in the current directory. It stays on the operator's machine and never reaches a node. The lab template's inventory reads:

chalkos.yaml
# The lab's nodes, the address chalkctl reaches each at on the lab network, the image they run
# and their config files: every node gets cluster/, each its own file in nodes/. chalklab reaches
# the nodes at ports it forwards instead, through the inventory it writes into the lab's state
# directory.
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: lab}
spec:
  images:
    # nix build .#node-kvm -o images/node-kvm puts the image here.
    node-kvm: {source: {directory: images/node-kvm}}
  all: {imageName: node-kvm, files: [cluster/*.yaml]}
  nodes:
    cp1: {address: 192.168.123.11, files: [nodes/cp1.yaml]}
    w1: {address: 192.168.123.12, files: [nodes/w1.yaml]}

Its spec.all is what every node gets, spec.groups what the nodes of each group get, and spec.nodes each node by its name with the address chalkctl reaches it at and its own files. Write a node's config walks through the fields, with a group, and Config kinds lists them.

A node's set is the documents of the files of spec.all, then of its groups' files in the order the node names the groups, then of its own files. Paths are relative to the inventory's directory, and a glob such as cluster/*.yaml takes its files in name order. Every path or glob must name at least one file, so a typo fails instead of leaving documents out. A file named twice counts once.

The order decides nothing about values, because documents never merge: a document is in a node's set or it is not, and two documents of the same kind and name in one node's set are an error that names both files. A value shared by every node goes into a shared file once. A node that needs another value gets its own document of that kind in its own file, and the shared file leaves the kind out. A group is the place for what a subset of nodes shares, such as the Node of GPU workers with their type and labels.

The inventory names no secrets file and no client file. chalkctl reads them from its flags and its environment, in the order Where chalkctl finds its credentials lists.

A node's name is its key in the inventory

A node's name is its key in the inventory's spec.nodes, such as cp1. At install, chalkctl issues the node's node certificate with the name as its subject's common name. chalkd reads the name from that certificate whenever it needs it and has no other source for it: no config document names the node, and a kind a node holds once takes any metadata.name, which the examples set to default.

The name feeds:

  • the hostname, unless a Host document's hostname gives another;
  • the seed of the node's partition UUIDs, as Storage and encryption describes;
  • the node's recovery key, derived from the secrets file and the name;
  • the node.name of the change rules, the effects and the renderers of outputs;
  • with the Kubernetes layer, the Kubernetes node name, by which the kubelet registers and the node's etcd member is called.

A node keeps its name for its whole life. Renaming one takes a reinstall, because its certificates and its recovery key carry the old name, and so does its etcd member.

The inventory's own metadata.name is a label and feeds nothing on a node. chalkctl names the cluster by it in its messages, and chalkctl kubeconfig names the kubeconfig's cluster, user and context after it unless --context gives another name. Two clusters may share it; their nodes still accept only their own cluster's certificates.

A valid set holds each kind as often as the kind allows

A node's set is valid when:

  • it holds exactly one SystemDisk, and on an image with Kubernetes one Cluster, which names the cluster's endpoint and whose defaults fill in its other fields, and one Node;
  • every other kind appears at most once, unless the kind allows several (Secret, Volume, Addon, Network, NetDev, Link and extension kinds declared with multiple = true);
  • every document's apiVersion and kind is one the node's image declares;
  • no two documents of a kind share a metadata.name;
  • every chalkosLib.types.ref field names a document of its kind in the set, unless it holds a reserved name; every secretRef names a Secret of the set and a key it holds; chalkctl and chalkd report each one that does not, one line each;
  • every secretRef can name the systemd credential a unit loads its value from: the key is printable ASCII without / or : and does not start with a dot, the Secret's name and the key together take at most 247 bytes, and no two references share the credential file <secret>.<key>, as Secret a.b with key c and Secret a with key b.c would;
  • no value repeats that a kind declares unique, among the kind's documents or within one list, as Declare a config kind describes; chalkctl and chalkd report each repeated value, one line each;
  • every document is valid against its kind's schema.

Beyond the schemas, chalkctl and chalkd translate the set as the node will use it, which checks what a schema cannot express: overlapping ranges, a VIP outside the node's networks, a volume that does not fit its disk, an extra flag that would override one chalkos sets.

With the Kubernetes layer, every node holds the same cluster documents

Each node builds the cluster from its own documents, so chalkctl refuses an inventory whose nodes hold different Cluster, Flannel or ClusterRegistry specs, naming the first field that differs. Every node holds the same cluster documents explains the comparison.

The inventory names the image each node runs

The inventory's spec.images names each image the nodes run and the image directory of its build in source.directory, relative to the inventory's directory. The directory is what nix build writes for one image, such as images/node-kvm in the lab template. Where nodes enforce Secure Boot, it is the copy of the build that chalkbuild sign signed. Its metadata.json names the image's name, platform and root hash, and its schemas/ holds the schemas of the kinds the image accepts.

An image published to a registry is named by its reference in source.oci instead, as chalkbuild push prints it. Each image sets exactly one of directory and oci. A reference by tag is resolved to a digest once per command, so one command reads one image. chalkctl validate, config apply and status read the image's header and contract from the registry. chalkctl install on a node that runs the installer and chalkctl upgrade each have the node pull the image by its digest, with the credentials of the node's Registry for the registry's host.

node-metal: {source: {oci: ghcr.io/example/node-metal@sha256:<digest>}}

Publish images to a registry builds, signs and pushes such an image.

A node runs the image of its own imageName, else that of its groups, else spec.all.imageName. chalkctl refuses an inventory that leaves a node without an image, or whose groups give a node two different images. Share files and an image through a group shows an inventory whose GPU workers run an image of their own.

The image stays out of the node's config. chalkctl reads the root hash from metadata.json when it needs it and never puts it, or the image's name, into the config it delivers. A new build of an image or a node's move to another image changes no node's config, and an upgrade needs no config apply after it.

chalkctl and chalkd both check the config

Config is checked twice, against the schemas of the image it is for, so an error shows before anything reaches a node and a node still never stores config its image does not accept.

chalkctl checks first. chalkctl validate builds the set of each node, or of the nodes it names, and checks it against the schemas in the directory of the node's image in the inventory. It reaches no node and needs neither a secrets file nor a client file, so it runs in CI or before a commit:

chalkctl validate

It prints a line for each node whose config is valid and names each error, and it warns of a node whose SystemDisk leaves STATE unencrypted. For the lab template:

cp1: valid for the image node-kvm (node-kvm on kvm, root hash 1044caccb099), config 7080d0996b95ef02999ede113b5fe02895fc1a3b87749fb5ad4a3ab28a105986
w1: valid for the image node-kvm (node-kvm on kvm, root hash 1044caccb099), config 0505d3467f8f64576cd78a10ed1c82f904e09f9229d588f402f630d2458f3861

Each line names the image's own name and platform, the start of its root hash, and the hash of the node's config that chalkctl status compares. The commands that deliver config run the same check first, each against the image the config is for:

  • chalkctl install against the image it installs: the directory of the node's image in the inventory, or the one --image gives;
  • chalkctl config apply against the image the node runs, whose schemas and contract the node returns through the Contract method, refusing an image of another platform than the node's image in the inventory. With --staged, against the node's image in the inventory, which the staged image is;
  • chalkctl upgrade against the image each node gets, before it drains any node.

chalkd checks again when the config arrives, against the schemas of the image the set is for, and refuses an invalid set as a whole. An image that does not run checks the set itself, with its own chalkd. chalkd, not chalkctl, fills in the schemas' defaults in the set it keeps, so the slot on STATE holds the complete set while the files stay as short as their authors wrote them.

Both also check the image's change rules. chalkd runs every rule. chalkctl runs those that read no facts about the node, validate without the node's config and config apply with the config the node serves, so a change a rule refuses fails before it is sent. Both run the router layer's validator router on a router image: chalkd runs the image's program, and chalkctl the same code, which it links, on the same documents.

Only chalkd runs the image's renderers and their checks, its other validators, and the rules that read facts, so a set chalkctl accepts can still be refused on delivery. A renderer repeats a check of the schemas, the rules or the validators only where a bad value would break its output, such as a line break in a networkd unit's value. Such a check fails with internal error: ..., which a checked config never causes. chalkctl config apply --dry-run has chalkd run every check of a delivery and changes nothing. chalkd answers with what the delivery would do: the config versions, the outputs that change, the effects with what fires each, and the storage changes.

chalkctl delivers a whole set at a time

chalkctl delivers a node's config at install, with each upgrade and with chalkctl config apply:

chalkctl config apply w1 --secrets secrets.age

Install and config apply need the secrets file, because the node's certificates and secrets travel in the same request. A set replaces the set the node keeps for the image as a whole: a document left out of the files is gone from the node after the next config apply.

Before it delivers a set, chalkctl asks the node for the set it keeps, through the GetConfig method, and prints one line of what the delivery changes:

w1: config 1 added, 1 removed (Volume "scratch"), 2 changed

The line names every removed document, so a document dropped from the files shows before the node loses it. chalkctl config apply --diff and chalkctl upgrade --diff print the whole comparison and deliver nothing: each document added (+), removed (-) or changed (~), and for a changed one each field by its path, with the old and the new value. A secret field shows only as changed. The node returns each value of a Secret as its SHA-256, and so each data and stringData value of a Secret object a document embeds, an object of apiVersion: v1 and kind: Secret such as the Kubernetes objects of an Addon. chalkctl compares the inventory's values by the same hashes.

A node keeps one config slot per image

A node keeps a config slot on STATE for each image it holds, /state/config/<root-hash>, named by the root hash of the image's store. A slot holds the documents as they were delivered, the same documents with the schemas' defaults filled in, the config's version and the node's identity, translated with that image's schemas. chalkd writes a slot to a new directory and renames a symbolic link to it over the old one, so a power cut leaves either the old slot or the new one, documents and identity together, never a mix. Root alone reads a slot: the node's services read the copy below /run/chalkos/config.

At every boot, chalkos-config.service loads the slot of the image the kernel command line's usrhash= names. The slots follow the images:

  • An install writes the slot of the image it installs.
  • An upgrade writes the slot of the new image. The image before keeps its own, so a fallback or upgrade --rollback boots it with the config it ran.
  • config apply writes the slot of the running image and then the slot of the image in the inactive slot, staged by upgrade --until stage or active after upgrade --until activate, unless that image is the known-good image a rollback staged. Both images check the config before chalkd writes either slot, as Every slot is checked by the image that boots it describes. config apply --staged writes the staged image's slot alone. Nothing restarts for the staged slot.
  • config apply --staged on a node without a staged image writes the pending slot /state/config/pending instead: the documents as delivered and their version, readable by root alone, and no defaults or identity. config apply without --staged replaces a pending slot the node holds, once the running image applied the config. No boot loads the pending slot, and nothing checks it until the next upgrade gives it to its image. That image checks it, and the upgrade writes it to the image's slot unless the upgrade carries config of its own. The upgrade removes the pending slot either way, once it staged the image. The pending slot's name is no root hash, so an upgrade's removal of other images' slots leaves it.
  • config apply keeps the config it replaces as the slot's previous config, /state/config/.<root-hash>.previous, one level deep. config rollback delivers that config again through the same checks, storage changes and effects as any delivery, and the config it replaces becomes the previous one. Status reports the previous config's version for the running image alone.
  • An install, an upgrade and config apply --staged write a slot without a previous config. An upgraded node therefore has no previous config until its first delivery.
  • An upgrade removes the slot of every image no UKI on the ESP boots any more, together with its previous config, so a node holds at most two image slots, one for each of its images, besides a pending slot.

An image whose slot is missing boots without config: chalkos-config fails, the health check fails with it, and a counted boot falls back.

Every slot is checked by the image that boots it

An install, an upgrade and every config apply write a config slot only after the image that boots it checked the config with its own checker: its schemas, its CEL rules, its facts, its validators and its renderers with their checks. The running image checks a delivery to its own slot in chalkd, and an install in place. An image that does not run checks config with its own chalkd check, which chalkd runs from that image's store in a sandbox, as An image that does not run checks its own config describes. That is how an upgrade checks the new image's config, how config apply --staged checks the staged image's, and how the installer checks the config of the image it writes, once the image's store is on the target disk and before STATE holds any of the node. A boot therefore renders the outputs of its slot and runs no validators.

config apply without --staged on a node with a staged image runs both checks before anything changes:

  1. The running image checks the config against its own slot and the machine, as for any delivery.
  2. The staged image checks the config against the running slot's documents as they were delivered, and against the storage that storage.json records once the running image applied the config.
  3. The running image applies the config: storage changes, identity, layers' config, its slot and the effects.
  4. chalkd makes the storage changes the staged image's config makes on top of those, if any, and writes the staged image's slot.

A refusal in step 1 or 2 changes nothing. A failure in step 4 leaves the staged slot with its earlier config and fails the delivery with an error that names the slot and lists what the running image's delivery did; the running image runs the new config, and the same delivery sent again writes the staged slot. Step 4 records a new volume in storage.json before it writes the slot, so a failure in between leaves a volume that every boot mounts, never a slot that names a volume the node lacks. A dry run runs steps 1 and 2 and plans both slots. On a node without a staged image, step 4 replaces the pending slot instead, when the node holds one, without a check.

A trial writes neither the staged slot nor the pending slot in step 4. It keeps the staged slot the staged image passed in /state/chalkd/trial-staged, next to the trial's record, and the record says what the confirmation writes. ConfirmConfig first removes the record, so no boot or rollback undoes the trial any more, and then writes the staged slot from that copy, or the pending slot from the running slot. The confirmation drops the copy and warns when the image is no longer staged in the inactive slot, is the running image after a boot into it, or is now the node's known-good image. A crash in between leaves both slots as they were, as a failed write does, and delivering the config again writes them. A confirmation sent again after such a crash finds the copy and warns that the staged slot was not written. Every other end of a trial, a rollback at the deadline or on request or a boot, removes the copy with the record, so the staged slot and the pending slot keep what they held before the trial. A copy without a record is never written; the next boot or chalkd start removes it. Nothing else writes either slot during a trial: chalkd refuses deliveries, upgrades, activations and image rollbacks until the trial ends.

The checker of another image compares the config with the running slot's documents as they were delivered and fills in its own schemas' defaults, as chalkctl's own check of the rules does. The running image's defaults are those of its own schemas, which the other image's rules do not read.

A staged image and the next upgrade take config of their own

chalkctl config apply to a node with a staged or active image in its inactive slot ends with <node> boots config <version> with its staged image too, as Deliver config to a staged image shows. A refusal by the staged image in step 2 starts with the image <hash> staged in the inactive slot:. A volume the config adds is created once and serves both images.

chalkctl sends the fallback secret when the running image's storage enrols it on a new volume. When only the staged image's storage needs it, because that image encrypts a volume the running image does not, the staged image refuses the config with ... no fallback secret was sent. The error names chalkctl config apply --staged <node> --secrets FILE. With --staged, chalkctl translates the config for the node's image in the inventory, which is the staged image, and sends the secret its storage needs.

The error of a failed step 4 lists in parentheses the volumes the node recorded for the delivery, the units it restarted and its warnings. A storage change that fails before the node records it lists no volume, although a partition it made may stay. The delivery sent again finds the volumes and partitions already made.

In a dry run, the staged slot's plan reads <node>: staged image's config <previous-version> → <version>. With --staged, that plan stands alone, with no outputs and no effects: the staged image renders its outputs and takes its effects when it boots.

After a rollback from a boot that was not found healthy, the image in the inactive slot is the node's known-good image. It keeps the config it last ran, and a delivery warns: the image <hash> staged to boot next is the one the node falls back to while the boot of <entry> is not found healthy; its config stays as it last ran.

A node without a staged image keeps --staged config as its pending config. chalkctl checks the files against the node's image in the inventory, the image the next upgrade installs. The node only parses them, because it has no image to check them against yet. Another --staged replaces the pending config, and its diff compares with the pending config. A dry run with --staged reports that the node would hold the config pending and stores nothing. A dry run without it names the pending config the delivery would replace. The delivery itself reports the replacement:

$ chalkctl config apply node1 --secrets secrets.age
node1: config 1 added, 0 removed, 0 changed
node1 runs config <version>
node1 holds config <version> pending, unchecked, in place of its pending config; the next upgrade checks it against its image

A later --staged then holds config for the next image alone again. Operator upgrades carry no config describes how the next upgrade takes the pending config.

A trial rolls a config back unless it is confirmed

chalkctl config apply --try delivers config on trial, as Try a config change shows. The trial's rollback is the slot's previous config, which every config apply keeps. A trial therefore delivers config documents alone and to the running image. chalkd refuses a trial that changes the node's storage, or that carries OS CAs, a node certificate or a layer's payload, because the previous config does not restore them.

Before the delivery changes anything, chalkd writes the trial's record to /state/chalkd/trial and syncs it. The record names the slot, the version on trial, the version from before it, the deadline and the units that the trial's health check leaves out. A crash after the record and before the slot was written leaves a slot that runs the version from before the trial, and chalkd ends that trial when it next reads the record.

Once the delivery is applied, chalkd arms a timer for the deadline. ConfirmConfig ends the trial: it runs the health check of a counted boot, leaving out the units the image ignores and the trial's own, and then removes the record and stops the timer. The request names the version it confirms, so a client cannot confirm a later trial that another client started. A confirmation sent again after its answer was lost succeeds when the node runs that version and no trial is open.

At the deadline, chalkd rolls the config back through the same steps as config rollback, then removes the record. A rollback that fails, as when the image's rules refuse it or an output does not render, leaves the record with the reason and whether the attempt changed the node. chalkd tries the rollback again each minute. ConfirmConfig still ends the trial and stops the retries, unless the failed attempt changed the node. Past the deadline, chalkctl config confirm tries for two minutes. After an attempt that changed the node, config rollback finishes the rollback at once. Status reports the trial with the reason of the last failure, in the lines that Read the config line lists.

While the record exists, chalkctl checks first, before it changes anything, and chalkd refuses every delivery, upgrade and image rollback, the certificates and shares that chalkctl node renew and chalkctl rotate deliver included, with config <version> is on trial until <deadline>: confirm it with chalkctl config confirm <node>, or roll it back with chalkctl config rollback <node>, before .... A rollback to the config from before the trial ends the trial. A volume reset, a reboot and the requests of the image's layers go ahead.

A record that chalkd cannot read refuses every change and every confirmation until a rollback removes it. Neither the deadline nor a reboot ends such a trial. config rollback delivers the previous config and removes the record, and chalkctl first warns which config it delivers. When a rollback at the deadline replaced the config already, the previous config is the one that was on trial, and a second rollback undoes it. A node without a previous config gets the config it runs delivered again, which removes the record as well.

A trial survives a restart of chalkd. When chalkd starts, it reads the record once it serves. It arms the timer again from the recorded deadline, or rolls the config back at once when the deadline has passed.

A boot ends an open trial. chalkos-config.service runs chalkd load-config, which reads the record before it applies the slot. When the slot runs the config on trial, chalkd exchanges the slot's two links, so the config from before the trial is the slot's config again and the config on trial its previous one. Then it removes the record. A slot that it cannot read or restore keeps the record, with the boot's time as the deadline, and chalkd rolls the trial back as it starts.

After every rollback of a trial, at the deadline, on request or at boot, the config that was on trial is the slot's previous config. config rollback then delivers it again, without a trial.

A rollback after a delivery that failed part way, the rollback of a trial included, takes the effects that delivery took, as The baseline survives a failed delivery describes.

chalkctl confirms a trial over a new connection

After a delivery on trial, chalkctl waits until the image's layers find the node serving with the config. On a Kubernetes node, that means its Node is Ready. On a control plane, its API server, controller-manager and scheduler must also run the images the config names and be Ready. A Node that is not Ready only because no CNI runs yet passes: its Ready condition names NetworkReady=false as its only cause. When the layers do not find the node serving by the deadline, chalkctl rolls the config back at once.

Then chalkctl closes the delivery's connection and asks the node to confirm over a new one, every 3 seconds until the deadline. The connection that delivered the change can survive a firewall that drops new connections, so only a new one proves that the node is still reachable. chalkctl config confirm confirms in the same way. The node confirms once it is healthy, as the health check of a new image's boot finds it: chalkd serves, no unit of the boot has failed, and the image's layers find their services healthy. The health check decides whether a counted boot is good describes the check.

A unit that failed before the trial keeps the node from confirming as well. The check leaves out the units the image lists in chalkos.upgrade.healthIgnoreUnits and those that --ignore-unit names. Before the delivery, chalkctl warns of every failed unit that the node's status reports and --ignore-unit does not name. The warning can therefore name a unit that the check leaves out anyway, such as a timer's job or a unit the image ignores. The record keeps the trial's ignored units, so config confirm and a restart of chalkd leave them out too, and status lists them on the trial's line.

When chalkctl cannot confirm the config by the deadline, it waits up to two minutes past it, over new connections, for the node to report how the trial ended, and exits non-zero. When the node does not answer in those two minutes either, the error adds the node did not answer after the deadline either: ...; chalkctl status <node> shows where it stands. A config that cuts chalkctl off usually also loses the answer to its delivery. chalkctl never confirms a config whose delivery it saw no answer to. Interrupting chalkctl, for example with Ctrl-C, leaves the trial open, and the error names the deadline and the two commands that end the trial earlier.

A delivery on trial can fail after it wrote the config, as when a restart fails. The config then stays on trial until the deadline, the error says config <version> stays on trial until <deadline>, and chalkctl does not try to confirm it. A delivery that fails before it wrote the config ends the trial: the node delivers the config from before it again, and the error says config <previous-version> is delivered again, and no trial is open.

chalkctl refuses --try with --staged, --diff and the layers' payload flags such as --kubernetes-share, and with the config that the node runs already.

Change rules decide which changes a node accepts

The config is the image's; the machine state is the node's. The machine state is what a node keeps outside the slots, once for all its images: storage.json, the storage as applied, with the disks and partitions it is pinned to, and the node certificate and the OS CAs it trusts. Every write of an image's slot (install, upgrade, config apply and config apply --staged) checks its config against the change rules of the image the slot is for. The rules compare it with the running image's config and read the machine state from facts. The built-in storage kinds' rules take:

Change Takes What carries it out
A new volume, a larger one, a larger VAR or another mount point nothing The write of the slot: the node makes the change at once.
A volume formatted or encrypted otherwise, or shrunk reset chalkctl storage reset, which recreates the volume and loses its data.
A volume removed or moved to another disk, or a change that would recreate VAR reinstall A reinstall of the node.

A node refuses the whole set when one of its rules fails, applies none of it, and names what the change needs, as Guard changes with rules and facts shows.

With the Kubernetes layer, the Kubernetes share fixes a node's type and the cluster's ranges

The Kubernetes share is machine state too. Once a node holds it, the rules refuse a Node.spec.type other than the one the node was installed with, which takes a reinstall as the new type. They also refuse a change of the cluster's IP families, pod or service ranges or DNS addresses, which stay for the cluster's life, as The ranges of a cluster are fixed once it exists explains. The cluster file /run/chalkos/kubernetes/cluster.json is not machine state: chalkos-config writes it from the slot of the image that boots, and chalkd writes it again when config apply changes the running image's config. A control plane's addresses are not refused, but they are pinned once it becomes an etcd member, as A control plane's addresses are pinned once it joins etcd explains.

The changes that take nothing are made whichever image the slot is for. An upgrade or config apply --staged that adds a volume creates it while the running image runs, enrolled with the fallback secret chalkctl sends, and storage.json gains it. Every boot sets up the volumes storage.json records, so the running image mounts the new volume as well, also after a fallback, although its config does not name it. Nothing at boot removes a volume. A slot whose config lacks the volume counts as removing it only when that slot is written again, so the next config for that image must name the volume too.

A delivery to the running image while an image is staged writes both slots, so both name the volumes it adds. A write of the staged image's slot alone, by an upgrade or config apply --staged, does not check the running image's slot: the running image mounts the new volume, and the next config apply to it must name the volume.

chalkos-config applies the slot at every boot

chalkos-config.service applies the running image's slot at every boot, before the network starts, as Stage 2 applies the config before the network starts describes. chalkd applies a new set the same way after every config apply to the running image, through the stages that The config pipeline follows. Each document except Secret documents and those of withheld kinds lands in /run/chalkos/config/<group>/<Kind>/<name>.json.

The storage section is machine state: only chalkd writes storage.json, when a config that changes the storage is delivered, and every boot sets up the volumes it records.

A service reads the node's config in one of two ways, which its image decides. A unit granted a kind reads the documents of that kind below /run/chalkos/config or through chalkcfg, as Read config in a service describes. An output renders the service's own config files from the documents instead. An effect restarts or reloads the service when its outputs or the documents of its kinds change, and config apply prints the units it restarted.

chalkcfg runs on the node, from a granted unit or a privileged pod. It prints no Secret and no document of a withheld kind.

Withheld kinds stay out of /run

A layer of chalkos can declare a kind withheld, for documents that chalkd alone reads. The kind's schema says so with withheld: true in its x-chalkos-kind, and chalkd, chalkcfg and the image's build read it there:

  • The documents stay in the node's config slot on STATE, which root alone reads, and never reach /run/chalkos/config.
  • chalkcfg prints none of them, and chalkcfg get lists no withheld kind.
  • The image's build refuses a unit granted the kind, an output or a validator that reads it, and an effect that fires on it. A when rule reads the documents, with the values of the Secret objects they embed hashed.

The Kubernetes layer withholds Addon. Its objects may include Secrets, and only a control plane's chalkd applies them.

Secrets reach only the units granted their kind

A Secret document holds secret values by key in spec.data:

nodes/cp1.secrets.yaml
apiVersion: chalkos.dev/v1alpha1
kind: Secret
metadata: {name: bgp}
spec:
  data:
    password: s3cret

A field of an extension kind declared with the type chalkosLib.types.secretRef references one of its keys instead of holding the value, written as password: {secretRef: {name: bgp, key: password}}. chalkctl and chalkd refuse a set in which a reference names a Secret or key the set lacks.

On the node, chalkd keeps Secret documents in the config slot on STATE, readable by root alone, and never writes them below /run/chalkos/config. A unit granted a kind with a reference gets the value as a systemd credential, and no other service sees it, as Pass secrets to a service describes. Status, chalkcfg and the config's hash never show a secret value.

With the Kubernetes layer, a Registry's password reaches the kubelet

chalkos-registry-credentials.service writes a Registry's password to a file only root reads, and the kubelet uses it for every pod's pulls from the Registry's host.

Status compares a node with the inventory

chalkctl status compares a node with what the inventory gives it, in two ways:

  • The config: chalkctl hashes the documents it would deliver with SHA-256, in a canonical form in which each Secret value is replaced by its own hash, and compares the result with the hash the node reports. A node that differs shows not the inventory's; chalkctl config apply <node> delivers it. config apply checks the config against the image the node runs, which need not accept the files of another image. A node that runs another image than the inventory's therefore shows chalkctl upgrade --nodes <node> delivers it with the image <image>, or it comes with the image <image> in the inactive slot when that image is staged or active. A known-good image keeps the config it last ran, so when it is the inventory's image, the advice is config apply once the node runs it. A delivery writes the config slot of a staged image too, which checks the config first. On a node that runs the inventory's image, a staged image that is not the inventory's adds that condition to the advice, as no command removes it. With an image staged, status compares the staged slot's config the same way. When it differs, the advice names chalkctl config apply <node>, which writes the config slots of both images, or chalkctl config apply --staged <node> when the running slot holds the inventory's config already, or when the staged image is the inventory's and the running image is not. When neither image is the inventory's, the advice is the upgrade. A rollback image that keeps the config it last ran gets no advice. The node tells that image in its boot status by the same test its deliveries take. A trial holds back every delivery, as chalkctl and the node test it: the advice is chalkctl config confirm <node> for the staged slot when the inventory's config is on trial, as the confirmation writes the staged slot, and the end of the trial otherwise. A trial record the node cannot read holds back every change but chalkctl config rollback.
  • The image: chalkctl compares the root hash of the store the node booted with the root hash in the metadata.json of the directory of its image in the inventory. It names both when they differ, or says when the image is installed and waits for the node's next boot.

The hash covers the documents as the files give them, before chalkd fills in defaults, so a new image with other defaults does not change it.

Limits

  • chalkctl reads every file of a node's set as plain YAML, Secret documents included. A file of Secrets must be in plain text where chalkctl runs: kept out of version control, or decrypted into place by a tool such as sops before the command and removed after.
  • Documents never merge and the inventory has no patches or templates, so two nodes whose Node differs in one label each hold a whole Node of their own.
  • Config carries data, not code. A new service, or a change to how a service reads its value, is an image change and an upgrade.
  • A kind has one version per image, and there is no conversion between versions. A new version of a kind is a new set of documents.
  • A rule compares a config with the running image's config alone, and chalkctl runs only the rules that read no facts; see Rules compare with the running config alone.