Config and the inventory¶
The lab template's node cp1 gets a file of its own, with its system disk and its address on the lab network:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
type: controlplane
---
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
disk: {path: /dev/vda}
---
# The card on the lab network, between the VMs. The other card reaches the internet and the
# ports chalklab forwards from 127.0.0.1, configured by DHCP.
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-lab}
spec:
match: {macAddress: "52:54:00:7b:00:11"}
network: {address: [192.168.123.11/24]}
A node's config is the set of YAML documents that makes it this node. It gives the host name, disks, network, time servers and the values of extensions, and with the Kubernetes layer the Kubernetes settings. The image is the same on many nodes, and the config is what differs. An inventory names the cluster's nodes and, for each, the address chalkctl reaches it at, the image it runs and the files whose documents form its config. chalkctl checks the config against the node's image and delivers it. chalkd checks it again, keeps it on STATE for the image it is for and applies it at every boot of that image.
A config document is an object of one kind¶
A config document is one object in the style of a Kubernetes object: an apiVersion of a group and
a version, a kind, a metadata.name and a spec. The kind's JSON Schema, which the image
carries, defines the spec, its defaults and what it refuses, so a misspelled field is an error
and not a value that nothing reads. A file holds any number of documents, separated by ---.
The lab template's shared file holds the documents every node of the lab gets:
# Kubernetes with its defaults: IPv4, the pod range 10.244.0.0/16 and the service range
# 10.96.0.0/12.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
# cp1 on the lab network: the API server of the only control plane.
endpoint: https://192.168.123.11:6443
# The images of the cluster's components, which containerd pulls as written. The tags of the API
# server, the controller-manager, the scheduler and kube-proxy follow the kubelet of the image the
# flake builds, the newest Kubernetes release chalkos pins; the image's checker holds them to
# Kubernetes's version skew policy. The digests pin what containerd pulls. Read the digest of
# another reference with `crane digest <ref>` or
# `skopeo inspect --format '{{.Digest}}' docker://<ref>`.
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
chalkos's kinds live in the group chalkos.dev¶
chalkos's own kinds live in the group chalkos.dev and its subdomains, all at version
v1alpha1. Config kinds lists every field of each, generated from the
schemas. Every image accepts these:
| Kind | Group | Per node | What it holds |
|---|---|---|---|
Host |
chalkos.dev |
at most one | The node's hostname, which defaults to the node's name |
Secret |
chalkos.dev |
any | Secret values by key, for the fields that reference them |
SystemDisk |
storage.chalkos.dev |
exactly one | The system disk, the size of VAR and the encryption of the node's partitions |
Volume |
storage.chalkos.dev |
any | A further volume: its disk, size, format, mount point and encryption |
TimeSync |
time.chalkos.dev |
at most one | Time servers, and whether those DHCP announces are used too |
Network, NetDev, Link |
networkd.chalkos.dev |
any | systemd-networkd's .network, .netdev and .link files, one document per file |
Registry |
oci.chalkos.dev |
any | Credentials of one registry or mirror host, and whether chalkd and containerd reach that host over plain HTTP |
Mirror |
oci.chalkos.dev |
any | The mirrors of one registry |
The Per node column follows from each kind's multiple. A kind declared with multiple = true
lets a node hold several documents of it, each with a name of its own, such as one Volume per
volume. Without it, a node holds at most one, and chalkctl and chalkd refuse a second.
Registry and Mirror exist on every image, because chalkd reads them for the pull of an
install's or an upgrade's image, and containerd's mirror configuration is written from them. A
Registry gives a host's credentials and allows plain HTTP to it. A Mirror names the endpoints
chalkd and containerd try before a registry itself. The Network, NetDev and Link documents
mirror networkd's units one to one, as
Network documents map one to one onto networkd units
describes.
With the Kubernetes layer, four kinds describe the cluster
Images with Kubernetes also accept the kinds of the group kubernetes.chalkos.dev. A node holds
exactly one Cluster, with the cluster's endpoint, families, ranges and VIP, and exactly one
Node, with its type, controlplane or worker, and its own Kubernetes settings. Images built
with flannel accept at most one Flannel, and any number of Addon documents hold objects the
control planes apply. chalkctl delivers Addon documents to control planes alone; a shared
file may hold them, and a worker's set leaves them out. On any other image these kinds are
unknown, and a node without Kubernetes has no type. The kubelet also takes a Registry's
credentials for every pod's pulls from its host.
Four kinds of document describe the cluster
explains them.
An image declares only the kinds its modules bring. An extension adds kinds of its own in a group
outside chalkos.dev, declared in the NixOS modules of an image, as
Declare a config kind shows. Their documents go into the same files
as chalkos's. The Rack kind of the
extensions example records
where a node is mounted, and the example's node file holds one:
The inventory gives each node a set of files¶
The inventory is a YAML document of its own kind, Inventory,
that chalkctl reads from the file --inventory names, else the file $CHALKOSINVENTORY names,
else chalkos.yaml in the current directory. It stays on the operator's machine and never reaches a node. The lab template's inventory
reads:
# The lab's nodes, the address chalkctl reaches each at on the lab network, the image they run
# and their config files: every node gets cluster/, each its own file in nodes/. chalklab reaches
# the nodes at ports it forwards instead, through the inventory it writes into the lab's state
# directory.
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: lab}
spec:
images:
# nix build .#node-kvm -o images/node-kvm puts the image here.
node-kvm: {source: {directory: images/node-kvm}}
all: {imageName: node-kvm, files: [cluster/*.yaml]}
nodes:
cp1: {address: 192.168.123.11, files: [nodes/cp1.yaml]}
w1: {address: 192.168.123.12, files: [nodes/w1.yaml]}
Its spec.all is what every node gets, spec.groups what the nodes of each group get, and
spec.nodes each node by its name with the address chalkctl reaches it at and its own files.
Write a node's config walks through the fields, with a group, and
Config kinds lists them.
A node's set is the documents of the files of spec.all, then of its groups' files in the order
the node names the groups, then of its own files. Paths are relative to the inventory's directory,
and a glob such as cluster/*.yaml takes its files in name order. Every path or glob must name at
least one file, so a typo fails instead of leaving documents out. A file named twice counts once.
The order decides nothing about values, because documents never merge: a document is in a node's
set or it is not, and two documents of the same kind and name in one node's set are an error that
names both files. A value shared by every node goes into a shared file once. A node that needs
another value gets its own document of that kind in its own file, and the shared file leaves the
kind out. A group is the place for what a subset of nodes shares, such as the Node of GPU workers
with their type and labels.
The inventory names no secrets file and no client file. chalkctl reads them from its flags and its environment, in the order Where chalkctl finds its credentials lists.
A node's name is its key in the inventory¶
A node's name is its key in the inventory's spec.nodes, such as cp1. At install, chalkctl issues
the node's node certificate with the name as its subject's common name. chalkd reads the name from
that certificate whenever it needs it and has no other source for it: no config document names the
node, and a kind a node holds once takes any metadata.name, which the examples set to default.
The name feeds:
- the hostname, unless a
Hostdocument'shostnamegives another; - the seed of the node's partition UUIDs, as Storage and encryption describes;
- the node's recovery key, derived from the secrets file and the name;
- the
node.nameof the change rules, the effects and the renderers of outputs; - with the Kubernetes layer, the Kubernetes node name, by which the kubelet registers and the node's etcd member is called.
A node keeps its name for its whole life. Renaming one takes a reinstall, because its certificates and its recovery key carry the old name, and so does its etcd member.
The inventory's own metadata.name is a label and feeds nothing on a node. chalkctl names the
cluster by it in its messages, and chalkctl kubeconfig
names the kubeconfig's cluster, user and context after it unless --context gives another name.
Two clusters may share it; their nodes still accept only their own cluster's certificates.
A valid set holds each kind as often as the kind allows¶
A node's set is valid when:
- it holds exactly one
SystemDisk, and on an image with Kubernetes oneCluster, which names the cluster'sendpointand whose defaults fill in its other fields, and oneNode; - every other kind appears at most once, unless the kind allows several (
Secret,Volume,Addon,Network,NetDev,Linkand extension kinds declared withmultiple = true); - every document's
apiVersionandkindis one the node's image declares; - no two documents of a kind share a
metadata.name; - every
chalkosLib.types.reffield names a document of its kind in the set, unless it holds a reserved name; everysecretRefnames aSecretof the set and a key it holds; chalkctl and chalkd report each one that does not, one line each; - every
secretRefcan name the systemd credential a unit loads its value from: the key is printable ASCII without/or:and does not start with a dot, theSecret's name and the key together take at most 247 bytes, and no two references share the credential file<secret>.<key>, asSecreta.bwith keycandSecretawith keyb.cwould; - no value repeats that a kind declares unique, among the kind's documents or within one list, as Declare a config kind describes; chalkctl and chalkd report each repeated value, one line each;
- every document is valid against its kind's schema.
Beyond the schemas, chalkctl and chalkd translate the set as the node will use it, which checks what a schema cannot express: overlapping ranges, a VIP outside the node's networks, a volume that does not fit its disk, an extra flag that would override one chalkos sets.
With the Kubernetes layer, every node holds the same cluster documents
Each node builds the cluster from its own documents, so chalkctl refuses an inventory whose
nodes hold different Cluster, Flannel or ClusterRegistry specs, naming the first field
that differs.
Every node holds the same cluster documents
explains the comparison.
The inventory names the image each node runs¶
The inventory's spec.images names each image the nodes run and the image directory of its build
in source.directory, relative to the inventory's directory. The directory is what nix build
writes for one image, such as images/node-kvm in the lab template. Where nodes
enforce Secure Boot, it is the copy of the build that
chalkbuild sign signed. Its metadata.json names the
image's name, platform and root hash, and its schemas/ holds the schemas of the kinds the
image accepts.
An image published to a registry is named by its reference in source.oci instead, as
chalkbuild push prints it. Each image sets exactly one of
directory and oci. A reference by tag is resolved to a digest once per command, so one command
reads one image. chalkctl validate, config apply and status read the image's header and
contract from the registry. chalkctl install on a node that runs the installer and
chalkctl upgrade each have the node pull the image by its digest, with the credentials of the
node's Registry for the registry's host.
Publish images to a registry builds, signs and pushes such an image.
A node runs the image of its own imageName, else that of its groups, else spec.all.imageName.
chalkctl refuses an inventory that leaves a node without an image, or whose groups give a node two
different images.
Share files and an image through a group
shows an inventory whose GPU workers run an image of their own.
The image stays out of the node's config. chalkctl reads the root hash from metadata.json when it
needs it and never puts it, or the image's name, into the config it delivers. A new build of an
image or a node's move to another image changes no node's config, and an upgrade needs
no config apply after it.
chalkctl and chalkd both check the config¶
Config is checked twice, against the schemas of the image it is for, so an error shows before anything reaches a node and a node still never stores config its image does not accept.
chalkctl checks first. chalkctl validate builds the set
of each node, or of the nodes it names, and checks it against the schemas in the directory of the
node's image in the inventory. It reaches no node and needs neither a secrets file nor a client
file, so it runs in CI or before a commit:
It prints a line for each node whose config is valid and names each error, and it warns of a node
whose SystemDisk leaves STATE unencrypted. For the lab template:
cp1: valid for the image node-kvm (node-kvm on kvm, root hash 1044caccb099), config 7080d0996b95ef02999ede113b5fe02895fc1a3b87749fb5ad4a3ab28a105986
w1: valid for the image node-kvm (node-kvm on kvm, root hash 1044caccb099), config 0505d3467f8f64576cd78a10ed1c82f904e09f9229d588f402f630d2458f3861
Each line names the image's own name and platform, the start of its root hash, and the hash of the
node's config that chalkctl status compares. The commands
that deliver config run the same check first, each against the image the config is for:
chalkctl installagainst the image it installs: the directory of the node's image in the inventory, or the one--imagegives;chalkctl config applyagainst the image the node runs, whose schemas and contract the node returns through the Contract method, refusing an image of another platform than the node's image in the inventory. With--staged, against the node's image in the inventory, which the staged image is;chalkctl upgradeagainst the image each node gets, before it drains any node.
chalkd checks again when the config arrives, against the schemas of the image the set is for, and refuses an invalid set as a whole. An image that does not run checks the set itself, with its own chalkd. chalkd, not chalkctl, fills in the schemas' defaults in the set it keeps, so the slot on STATE holds the complete set while the files stay as short as their authors wrote them.
Both also check the image's change rules. chalkd runs every rule.
chalkctl runs those that read no facts about the node, validate without the node's config and
config apply with the config the node serves, so a change a rule refuses fails before it is sent.
Both run the router layer's validator router on a router image:
chalkd runs the image's program, and chalkctl the same code, which it links, on the same
documents.
Only chalkd runs the image's renderers and their checks, its other validators, and the rules that
read facts, so a set chalkctl accepts can still be refused on delivery. A renderer repeats a check
of the schemas, the rules or the validators only where a bad value would break its output, such as
a line break in a networkd unit's value. Such a check fails with internal error: ..., which a
checked config never causes.
chalkctl config apply --dry-run has chalkd run
every check of a delivery and changes nothing. chalkd answers with what the delivery would do: the
config versions, the outputs that change, the effects with what fires each, and the storage
changes.
chalkctl delivers a whole set at a time¶
chalkctl delivers a node's config at install, with each upgrade and with
chalkctl config apply:
Install and config apply need the secrets file, because the node's certificates and secrets
travel in the same request. A set replaces the set the node keeps for the image as a whole: a
document left out of the files is gone from the node after the next config apply.
Before it delivers a set, chalkctl asks the node for the set it keeps, through the GetConfig method, and prints one line of what the delivery changes:
The line names every removed document, so a document dropped from the files shows before the node
loses it. chalkctl config apply --diff and chalkctl upgrade --diff print the whole comparison
and deliver nothing: each document added (+), removed (-) or changed (~), and for a changed
one each field by its path, with the old and the new value. A secret field shows only as
changed. The node returns each value of a Secret as its SHA-256, and so each data and
stringData value of a Secret object a document embeds, an object of apiVersion: v1 and kind:
Secret such as the Kubernetes objects of an Addon. chalkctl compares the inventory's values by
the same hashes.
A node keeps one config slot per image¶
A node keeps a config slot on STATE for each image it holds, /state/config/<root-hash>, named by
the root hash of the image's store. A slot holds the documents as they were delivered, the same
documents with the schemas' defaults filled in, the config's version and the node's identity,
translated with that image's schemas. chalkd writes a slot to a new directory and renames a
symbolic link to it over the old one, so a power cut leaves either the old slot or the new one,
documents and identity together, never a mix. Root alone reads a slot: the node's services read
the copy below /run/chalkos/config.
At every boot, chalkos-config.service loads the slot of the image the kernel command line's
usrhash= names. The slots follow the images:
- An install writes the slot of the image it installs.
- An upgrade writes the slot of the new image. The image before keeps its own, so a
fallback or
upgrade --rollbackboots it with the config it ran. config applywrites the slot of the running image and then the slot of the image in the inactive slot, staged byupgrade --until stageor active afterupgrade --until activate, unless that image is the known-good image a rollback staged. Both images check the config before chalkd writes either slot, as Every slot is checked by the image that boots it describes.config apply --stagedwrites the staged image's slot alone. Nothing restarts for the staged slot.config apply --stagedon a node without a staged image writes the pending slot/state/config/pendinginstead: the documents as delivered and their version, readable by root alone, and no defaults or identity.config applywithout--stagedreplaces a pending slot the node holds, once the running image applied the config. No boot loads the pending slot, and nothing checks it until the next upgrade gives it to its image. That image checks it, and the upgrade writes it to the image's slot unless the upgrade carries config of its own. The upgrade removes the pending slot either way, once it staged the image. The pending slot's name is no root hash, so an upgrade's removal of other images' slots leaves it.config applykeeps the config it replaces as the slot's previous config,/state/config/.<root-hash>.previous, one level deep.config rollbackdelivers that config again through the same checks, storage changes and effects as any delivery, and the config it replaces becomes the previous one. Status reports the previous config's version for the running image alone.- An install, an upgrade and
config apply --stagedwrite a slot without a previous config. An upgraded node therefore has no previous config until its first delivery. - An upgrade removes the slot of every image no UKI on the ESP boots any more, together with its previous config, so a node holds at most two image slots, one for each of its images, besides a pending slot.
An image whose slot is missing boots without config: chalkos-config fails, the health check
fails with it, and a counted boot falls back.
Every slot is checked by the image that boots it¶
An install, an upgrade and every config apply write a config slot only after the image that
boots it checked the config with its own checker: its schemas, its CEL rules, its facts, its
validators and its renderers with their checks. The running image checks a delivery to its own
slot in chalkd, and an install in place. An image that does not run checks config with its own
chalkd check, which chalkd runs from that image's store in a sandbox, as
An image that does not run checks its own config
describes. That is how an upgrade checks the new image's config, how config apply --staged
checks the staged image's, and how the installer checks the config of the image it writes, once
the image's store is on the target disk and before STATE holds any of the node. A boot therefore
renders the outputs of its slot and runs no validators.
config apply without --staged on a node with a staged image runs both checks before anything
changes:
- The running image checks the config against its own slot and the machine, as for any delivery.
- The staged image checks the config against the running slot's documents as they were
delivered, and against the storage that
storage.jsonrecords once the running image applied the config. - The running image applies the config: storage changes, identity, layers' config, its slot and the effects.
- chalkd makes the storage changes the staged image's config makes on top of those, if any, and writes the staged image's slot.
A refusal in step 1 or 2 changes nothing. A failure in step 4 leaves the staged slot with its
earlier config and fails the delivery with an error that names the slot and lists what the running
image's delivery did; the running image runs the new config, and the same delivery sent again
writes the staged slot. Step 4 records a new volume in storage.json before it writes the slot,
so a failure in between leaves a volume that every boot mounts, never a slot that names a volume
the node lacks. A dry run runs steps 1 and 2 and plans both slots. On a node without a staged
image, step 4 replaces the pending slot instead, when the node holds one, without a check.
A trial writes neither the staged slot nor the pending slot in step 4. It keeps the staged slot
the staged image passed in /state/chalkd/trial-staged, next to the trial's record, and the
record says what the confirmation writes. ConfirmConfig first removes the record, so no boot or
rollback undoes the trial any more, and then writes the staged slot from that copy, or the pending
slot from the running slot. The confirmation drops the copy and warns when the image is no longer
staged in the inactive slot, is the running image after a boot into it, or is now the node's
known-good image. A crash in between leaves both slots as they were, as a failed write does, and
delivering the config again writes them. A confirmation sent again after such a crash finds the
copy and warns that the staged slot was not written. Every other end of a trial, a rollback at the
deadline or on request or a boot, removes the copy with the record, so the staged slot and the
pending slot keep what they held before the trial. A copy without a record is never written; the
next boot or chalkd start removes it. Nothing else writes either slot
during a trial: chalkd refuses deliveries, upgrades, activations and image rollbacks until the
trial ends.
The checker of another image compares the config with the running slot's documents as they were delivered and fills in its own schemas' defaults, as chalkctl's own check of the rules does. The running image's defaults are those of its own schemas, which the other image's rules do not read.
A staged image and the next upgrade take config of their own¶
chalkctl config apply to a node with a staged or active image in its inactive slot ends with
<node> boots config <version> with its staged image too, as
Deliver config to a staged image
shows. A refusal by the staged image in step 2 starts with
the image <hash> staged in the inactive slot:. A volume the config adds is created once and
serves both images.
chalkctl sends the fallback secret when the running image's storage enrols it on a new volume. When
only the staged image's storage needs it, because that image encrypts a volume the running image
does not, the staged image refuses the config with ... no fallback secret was sent. The
error names chalkctl config apply --staged <node> --secrets FILE. With --staged, chalkctl
translates the config for the node's image in the inventory, which is the staged image, and sends
the secret its storage needs.
The error of a failed step 4 lists in parentheses the volumes the node recorded for the delivery, the units it restarted and its warnings. A storage change that fails before the node records it lists no volume, although a partition it made may stay. The delivery sent again finds the volumes and partitions already made.
In a dry run, the staged slot's plan reads <node>: staged image's config <previous-version> →
<version>. With --staged, that plan stands alone, with no outputs and no effects: the staged
image renders its outputs and takes its effects when it boots.
After a rollback from a boot that was not found healthy, the image in the inactive slot is the
node's known-good image. It keeps the config it last ran, and a delivery warns:
the image <hash> staged to boot next is the one the node falls back to while the boot of <entry> is not found healthy; its config stays as it last ran.
A node without a staged image keeps --staged config as its pending config. chalkctl checks the
files against the node's image in the inventory, the image the next upgrade installs. The node only
parses them, because it has no image to check them against yet. Another --staged replaces the
pending config, and its diff compares with the pending config. A dry run with --staged reports
that the node would hold the config pending and stores nothing. A dry run without it names the
pending config the delivery would replace. The delivery itself reports the replacement:
$ chalkctl config apply node1 --secrets secrets.age
node1: config 1 added, 0 removed, 0 changed
node1 runs config <version>
node1 holds config <version> pending, unchecked, in place of its pending config; the next upgrade checks it against its image
A later --staged then holds config for the next image alone again.
Operator upgrades carry no config describes how
the next upgrade takes the pending config.
A trial rolls a config back unless it is confirmed¶
chalkctl config apply --try delivers config on trial, as
Try a config change shows. The trial's
rollback is the slot's previous config, which every config apply keeps. A trial therefore
delivers config documents alone and to the running image. chalkd refuses a trial that changes the
node's storage, or that carries OS CAs, a node certificate or a layer's payload, because the
previous config does not restore them.
Before the delivery changes anything, chalkd writes the trial's record to /state/chalkd/trial
and syncs it. The record names the slot, the version on trial, the version from before it, the
deadline and the units that the trial's health check leaves out. A crash after the record and
before the slot was written leaves a slot that runs the version from before the trial, and chalkd
ends that trial when it next reads the record.
Once the delivery is applied, chalkd arms a timer for the deadline. ConfirmConfig ends the trial: it runs the health check of a counted boot, leaving out the units the image ignores and the trial's own, and then removes the record and stops the timer. The request names the version it confirms, so a client cannot confirm a later trial that another client started. A confirmation sent again after its answer was lost succeeds when the node runs that version and no trial is open.
At the deadline, chalkd rolls the config back through the same steps as
config rollback, then removes the record. A rollback that fails, as when the image's
rules refuse it or an output does not render, leaves the record with the reason and whether the
attempt changed the node. chalkd tries the rollback again each minute. ConfirmConfig still ends
the trial and stops the retries, unless the failed attempt changed the node. Past the deadline,
chalkctl config confirm tries for two minutes. After an attempt that changed the node, config rollback finishes the
rollback at once. Status reports the trial with the reason of the last failure, in the lines that
Read the config line
lists.
While the record exists, chalkctl checks first, before it changes anything, and chalkd refuses
every delivery, upgrade and image rollback, the certificates and shares that chalkctl node renew
and chalkctl rotate deliver included, with
config <version> is on trial until <deadline>: confirm it with chalkctl config confirm
<node>, or roll it back with chalkctl config rollback <node>, before .... A rollback to
the config from before the trial ends the trial. A volume reset, a reboot and the requests of the
image's layers go ahead.
A record that chalkd cannot read refuses every change and every confirmation until a rollback
removes it. Neither the deadline nor a reboot ends such a trial. config rollback delivers the
previous config and removes the record, and chalkctl first warns which config it delivers. When a
rollback at the deadline replaced the config already, the previous config is the one that was on
trial, and a second rollback undoes it. A node without a previous config gets the config it runs
delivered again, which removes the record as well.
A trial survives a restart of chalkd. When chalkd starts, it reads the record once it serves. It arms the timer again from the recorded deadline, or rolls the config back at once when the deadline has passed.
A boot ends an open trial. chalkos-config.service runs chalkd load-config, which reads the
record before it applies the slot. When the slot runs the config on trial, chalkd exchanges the
slot's two links, so the config from before the trial is the slot's config again and the config on
trial its previous one. Then it removes the record. A slot that it cannot read or restore keeps the
record, with the boot's time as the deadline, and chalkd rolls the trial back as it starts.
After every rollback of a trial, at the deadline, on request or at boot, the config that was on
trial is the slot's previous config. config rollback then delivers it again, without a trial.
A rollback after a delivery that failed part way, the rollback of a trial included, takes the effects that delivery took, as The baseline survives a failed delivery describes.
chalkctl confirms a trial over a new connection¶
After a delivery on trial, chalkctl waits until the image's layers find the node serving with the
config. On a Kubernetes node, that means its Node is Ready. On a control plane, its API server,
controller-manager and scheduler must also run the images the config names and be Ready. A Node
that is not Ready only because no CNI runs yet passes: its Ready condition names
NetworkReady=false as its only cause. When the layers do not find the node serving by the
deadline, chalkctl rolls the config back at once.
Then chalkctl closes the delivery's connection and asks the node to confirm over a new one, every 3
seconds until the deadline. The connection that delivered the change can survive a firewall that
drops new connections, so only a new one proves that the node is still reachable. chalkctl config
confirm confirms in the same way. The node confirms once it is healthy, as the health check of a
new image's boot finds it: chalkd serves, no unit of the boot has failed, and the image's layers
find their services healthy.
The health check decides whether a counted boot is good
describes the check.
A unit that failed before the trial keeps the node from confirming as well. The check leaves out
the units the image lists in
chalkos.upgrade.healthIgnoreUnits
and those that --ignore-unit names. Before the delivery, chalkctl warns of every failed unit
that the node's status reports and --ignore-unit does not name. The warning can therefore name a
unit that the check leaves out anyway, such as a timer's job or a unit the image ignores. The
record keeps the trial's ignored units, so config confirm and a restart of chalkd leave them out
too, and status lists them on the trial's line.
When chalkctl cannot confirm the config by the deadline, it waits up to two minutes past it, over
new connections, for the node to report how the trial ended, and exits non-zero. When the node
does not answer in those two minutes either, the error adds the node did not answer after the
deadline either: ...; chalkctl status <node> shows where it stands. A config that cuts chalkctl
off usually also loses the answer to its delivery. chalkctl never confirms a config whose delivery
it saw no answer to. Interrupting chalkctl, for example with Ctrl-C, leaves the trial open, and the
error names the deadline and the two commands that end the trial earlier.
A delivery on trial can fail after it wrote the config, as when a restart fails. The config then
stays on trial until the deadline, the error says config <version> stays on trial until
<deadline>, and chalkctl does not try to confirm it. A delivery that fails before it wrote the
config ends the trial: the node delivers the config from before it again, and the error says
config <previous-version> is delivered again, and no trial is open.
chalkctl refuses --try with --staged, --diff and the layers' payload flags such as
--kubernetes-share, and with the config that the node runs already.
Change rules decide which changes a node accepts¶
The config is the image's; the machine state is the node's. The machine state is what a node keeps
outside the slots, once for all its images: storage.json, the storage as applied, with the disks
and partitions it is pinned to, and the node certificate and the OS CAs it trusts. Every write of
an image's slot (install, upgrade, config apply and config apply --staged) checks its config
against the change rules of the image the slot is for. The rules
compare it with the running image's config and read the machine state from facts. The built-in
storage kinds' rules take:
| Change | Takes | What carries it out |
|---|---|---|
| A new volume, a larger one, a larger VAR or another mount point | nothing | The write of the slot: the node makes the change at once. |
| A volume formatted or encrypted otherwise, or shrunk | reset | chalkctl storage reset, which recreates the volume and loses its data. |
| A volume removed or moved to another disk, or a change that would recreate VAR | reinstall | A reinstall of the node. |
A node refuses the whole set when one of its rules fails, applies none of it, and names what the change needs, as Guard changes with rules and facts shows.
With the Kubernetes layer, the Kubernetes share fixes a node's type and the cluster's ranges
The Kubernetes share is machine state too. Once a node holds it, the rules refuse a
Node.spec.type other than the one the node was installed with, which takes a reinstall as the
new type. They also refuse a change of the cluster's IP families, pod or service ranges or DNS
addresses, which stay for the cluster's life, as
The ranges of a cluster are fixed once it exists
explains. The cluster file
/run/chalkos/kubernetes/cluster.json is not machine state: chalkos-config writes it from the
slot of the image that boots, and chalkd writes it again when config apply changes the running
image's config. A control plane's addresses are not refused, but they are pinned once it
becomes an etcd member, as
A control plane's addresses are pinned once it joins etcd
explains.
The changes that take nothing are made whichever image the slot is for. An upgrade or
config apply --staged that adds a volume creates it while the running image runs, enrolled with
the fallback secret chalkctl sends, and storage.json gains it. Every boot sets up the volumes
storage.json records, so the running image mounts the new volume as well, also after a fallback,
although its config does not name it. Nothing at boot removes a volume. A slot whose config lacks
the volume counts as removing it only when that slot is written again, so the next config for that
image must name the volume too.
A delivery to the running image while an image is staged writes both slots, so both name the
volumes it adds. A write of the staged image's slot alone, by an upgrade or
config apply --staged, does not check the running image's slot: the running image mounts the
new volume, and the next config apply to it must name the volume.
chalkos-config applies the slot at every boot¶
chalkos-config.service applies the running image's slot at every boot, before the network
starts, as
Stage 2 applies the config before the network starts
describes. chalkd applies a new set the same way after every config apply to the running image,
through the stages that The config pipeline follows. Each document except
Secret documents and those of withheld kinds lands in
/run/chalkos/config/<group>/<Kind>/<name>.json.
The storage section is machine state: only chalkd writes storage.json, when a config that changes
the storage is delivered, and every boot sets up the volumes it records.
A service reads the node's config in one of two ways, which its image decides. A unit granted a
kind reads the documents of that kind below /run/chalkos/config or through chalkcfg, as
Read config in a service describes. An output renders the service's
own config files from the documents instead. An effect restarts or reloads the service when its
outputs or the documents of its kinds change, and config apply prints the units it restarted.
chalkcfg runs on the node, from a granted unit or a privileged pod. It prints no Secret and no
document of a withheld kind.
Withheld kinds stay out of /run¶
A layer of chalkos can declare a kind withheld, for documents that chalkd alone reads. The kind's
schema says so with withheld: true in its x-chalkos-kind, and chalkd, chalkcfg and the image's
build read it there:
- The documents stay in the node's config slot on STATE, which root alone reads, and never reach
/run/chalkos/config. - chalkcfg prints none of them, and
chalkcfg getlists no withheld kind. - The image's build refuses a unit granted the kind, an output or a validator that reads it, and
an effect that fires on it. A
whenrule reads the documents, with the values of the Secret objects they embed hashed.
The Kubernetes layer withholds Addon. Its objects may include Secrets, and only a control
plane's chalkd applies them.
Secrets reach only the units granted their kind¶
A Secret document holds secret values by key in spec.data:
apiVersion: chalkos.dev/v1alpha1
kind: Secret
metadata: {name: bgp}
spec:
data:
password: s3cret
A field of an extension kind declared with the type chalkosLib.types.secretRef references one of
its keys instead of holding the value, written as
password: {secretRef: {name: bgp, key: password}}. chalkctl and chalkd refuse a set in which a
reference names a Secret or key the set lacks.
On the node, chalkd keeps Secret documents in the config slot on STATE, readable by root alone,
and never writes them below /run/chalkos/config. A unit granted a kind with a reference gets the
value as a systemd credential, and no other service sees it, as
Pass secrets to a service describes. Status, chalkcfg and the
config's hash never show a secret value.
With the Kubernetes layer, a Registry's password reaches the kubelet
chalkos-registry-credentials.service writes a Registry's password to a file only root
reads, and the kubelet uses it for every pod's pulls from the Registry's host.
Status compares a node with the inventory¶
chalkctl status compares a node with what the inventory
gives it, in two ways:
- The config: chalkctl hashes the documents it would deliver with SHA-256, in a canonical form in
which each
Secretvalue is replaced by its own hash, and compares the result with the hash the node reports. A node that differs showsnot the inventory's; chalkctl config apply <node> delivers it.config applychecks the config against the image the node runs, which need not accept the files of another image. A node that runs another image than the inventory's therefore showschalkctl upgrade --nodes <node> delivers it with the image <image>, orit comes with the image <image> in the inactive slotwhen that image is staged or active. A known-good image keeps the config it last ran, so when it is the inventory's image, the advice isconfig applyonce the node runs it. A delivery writes the config slot of a staged image too, which checks the config first. On a node that runs the inventory's image, a staged image that is not the inventory's adds that condition to the advice, as no command removes it. With an image staged, status compares the staged slot's config the same way. When it differs, the advice nameschalkctl config apply <node>, which writes the config slots of both images, orchalkctl config apply --staged <node>when the running slot holds the inventory's config already, or when the staged image is the inventory's and the running image is not. When neither image is the inventory's, the advice is the upgrade. A rollback image that keeps the config it last ran gets no advice. The node tells that image in its boot status by the same test its deliveries take. A trial holds back every delivery, as chalkctl and the node test it: the advice ischalkctl config confirm <node>for the staged slot when the inventory's config is on trial, as the confirmation writes the staged slot, and the end of the trial otherwise. A trial record the node cannot read holds back every change butchalkctl config rollback. - The image: chalkctl compares the root hash of the store the node booted with the root hash in
the
metadata.jsonof the directory of its image in the inventory. It names both when they differ, or says when the image is installed and waits for the node's next boot.
The hash covers the documents as the files give them, before chalkd fills in defaults, so a new image with other defaults does not change it.
Limits¶
- chalkctl reads every file of a node's set as plain YAML,
Secretdocuments included. A file ofSecrets must be in plain text where chalkctl runs: kept out of version control, or decrypted into place by a tool such as sops before the command and removed after. - Documents never merge and the inventory has no patches or templates, so two nodes whose
Nodediffers in one label each hold a wholeNodeof their own. - Config carries data, not code. A new service, or a change to how a service reads its value, is an image change and an upgrade.
- A kind has one version per image, and there is no conversion between versions. A new version of a kind is a new set of documents.
- A rule compares a config with the running image's config alone, and chalkctl runs only the rules that read no facts; see Rules compare with the running config alone.
Related pages¶
- The config pipeline for how a delivery passes rules, outputs and effects.
- Declare a config kind for how an extension declares a kind, and Images and platforms for the images that accept the config.
- Architecture for where the inventory and the image directories meet chalkctl.
- Guard changes with rules and facts for how an image decides which changes a node accepts.
- Storage and encryption and Networking for what the storage and network kinds do on the node.
- Config kinds for every field of every kind, the inventory's included.
chalkctl validateandchalkctl config applyfor the commands.- Write a node's config for writing the inventory and the documents, and Set up your workstation for the secrets file and the client files.
- The quick start and Install a node for config files in use.