A toolkit for immutable operating systems
Build your own API-driven, immutable Linux from NixOS modules
OS
Every node
- Secure Boot, signed UKI
- dm-verity store
- TPM-sealed LUKS2
- A/B slots, fallback
- mTLS node API
SSH, shell, logins
I love Talos
Talos Linux showed how a Kubernetes node can work: a read-only image, no logins, and every change through an API.
- API-drivenevery change through one API
- mTLS rolesclient certificates of a role: reader, operator, admin
- Immutablethe node boots a read-only image
- No SSH, no shellnothing to log in to, nothing drifts
- A/B upgradesthe whole image, with rollback
My worry
One vendor decides the licence of the base my machines run on.
The worry is about who holds that decision. A fleet runs for years, and I would rather build its operating system from parts I can rebuild myself.
Licence changes elsewhere in infrastructure software
| Year | Project | From | To |
|---|---|---|---|
| 2021 | Elasticsearch | Apache 2.0 | SSPL / Elastic Licence |
| 2023 | Terraform | MPL 2.0 | BUSL 1.1 |
| 2024 | Redis | BSD 3-clause | RSALv2 / SSPLv1 |
I love Talos, but it runs Kubernetes only
My other machines deserve the same model. Today each is installed and patched its own way.
Firewall, BGP, DNS, DHCP and VPN in front of the cluster.
One service on one machine, with nobody at the console.
Remote machines that must survive a bad upgrade alone.
So build the distribution
A toolkit that builds API-driven, immutable Linux distributions from NixOS modules.
Much like a Talos node.
- kubelet, etcd, control plane
- VIP, BGP speaker
- cluster registry
Between Talos and VyOS.
- firewall, NAT, BGP
- VRFs, NAT64, DNS
- DHCP, WireGuard, NTP
No layer: a minimal appliance.
- network, disks, time
- node API
- your modules
Compared with Talos, from the docs
| Topic | Talos | chalkos |
|---|---|---|
| Purpose | A distribution for Kubernetes nodes | Builds distributions: Kubernetes, routers, other appliances |
| Config | One machine config per node, Talos's own schema | Kubernetes-style YAML, one kind per concern; kinds and schemas ship in the image |
| Upgrades | Node keeps one config across upgrades | One config slot per image; a rollback boots its own config |
| Images | One image, signed system extensions | One image per kind of node, from NixOS modules |
| Node CA | Control planes hold the root CA's key | Control planes hold a node CA; the OS CA key stays offline |
| Maturity | Released, many platforms | Unreleased; x86-64 bare metal and KVM |
Anatomy of a node
systemd-boot
chalkos_<gen>.efi
hash tree
erofs, zstd · 3 GiB
an upgrade
LUKS2, TPM
LUKS2, TPM
from config
systemd-boot starts one UKI per slot. Kernel, initrd and command line sit under one signature.
An upgrade writes the inactive slot. The store's dm-verity root hash names the image; there is no version number.
LUKS2 with keys sealed by the TPM. STATE holds config and certificates, VAR what services write; root is a tmpfs.
Trust from firmware to API call
chalkctl securebootchalkbuild signSTATE, VAR and volumes are LUKS2, keys sealed by the TPM to the Secure Boot state.
- Same db key: upgrades unlock unattended
- Disk moved or Secure Boot changed: no unseal
- Fallback: the node's recovery key
Node API on TCP 50000, mutual TLS, client chain checked on every request.
- No SSH, no login
- Client files of one role each
No revocation list on nodes: access ends with the certificate, or a rotation.
- chalkctl rotate os-ca cuts old certificates off
- chalkctl node revoke on Kubernetes
- The OS CA key stays in the secrets file
Customise with plain NixOS modules
images = chalkos.lib.mkImages { revision = self.rev or self.dirtyRev or null; images = [ { name = "router-metal"; platform = "metal"; modules = [ chalkos.nixosModules.router ]; } { name = "node-metal"; platform = "metal"; modules = [ chalkos.nixosModules.kubernetes ./extensions # your kinds { chalkos.kubernetes.bgp.enable = false; } ]; } ]; };
One layer per image, plus a platform: metal or kvm.
An extension is a NixOS module that sets these options
| Option | What it does | Where it runs |
|---|---|---|
| chalkos.kinds | A typed config kind. Rejects invalid documents. | chalkctl and the node |
| …kinds.<Kind>.rules | CEL against the running config. Rejects forbidden changes. | chalkctl and the node |
| chalkos.validators | Checks all documents together. Rejects what it reports. | The node |
| chalkos.outputs | Renders config files. Rejects files that fail their check. | The node, at each change and boot |
| chalkos.effects | Reloads or restarts units after a change | The node |
| chalkos.diagnostics | Read-only commands for chalkctl node diag | The node, on request |
| chalkos.tests | Expected files, rejections and effects | Build time, Nix sandbox |
Config kinds ship in the image
A module declares a kind like a CRD: typed fields and CEL rules. The build ships it as JSON Schema, so config is checked against the image that will run it.
chalkos.kinds.Peer = { group = "bgp.example.org"; version = "v1alpha1"; multiple = true; spec.asn = chalkosLib.mkOption { type = lib.types.ints.between 1 4294967295; description = "AS number of the neighbour."; rules = [{ rule = "self == oldSelf"; messageExpression = ''"the AS number stays " + string(oldSelf) + ...''; }]; }; # address, interfaces, password, bfd ... };
{
"type": "integer",
"minimum": 1,
"maximum": 4294967295,
"x-chalkos-rules": [{
"rule": "self == oldSelf",
"takes": "refuse"
}]
}
# In the image directory and in
# /etc/chalkos/schemas on the node.
# Rules compile at build time,
# within a cost limit.
- chalkctl runs the rules that read no facts
- The node runs every rule again
Documents in, services out
text/template: bird.conf.tmplbird.conf in the output's directorybird -p parses it without running itbird.service when bird.conf changed{{- range $.documents.Peer }}
{{- $peer := . }}
{{- range .spec.interfaces }}
protocol bgp '{{ $peer.metadata.name }}_{{ . }}' {
local as {{ $asn }};
neighbor {{ $peer.spec.address }}
as {{ $peer.spec.asn }};
interface {{ quote . }};
{{- if $peer.spec.bfd }}
bfd on;
{{- end }}
...
protocol bgp 'router_enp1s0' { local as 65001; neighbor fdc8:a1b2:c3d4:1::1 as 65000; interface "enp1s0"; direct; connect retry time 5; bfd on; authentication md5; password 'secret_bgp.password'; ...
chalkos.outputs.bird-conf = { inputs = [ "Speaker" "Peer" ]; render = chalkosLib.render .template ./bird.conf.tmpl; check = '' ... bird -p -c "$conf" ''; }; chalkos.effects.bird = chalkosLib.reloadOn { outputs = [ "bird-conf" ]; } [ "bird.service" ];
Apply config with a way back
k1: config 2 added, 0 removed, 1 changed ~ BGPSpeaker "bgp" spec.export.interfaces[1]: (none) → "lb0" + NetDev "20-lb0" + Network "20-lb0" k1: config dd1465db… → 9a461558… outputs: bgp, networkd effects: bgp: reload frr.service (output bgp) kubernetes: restart chalkos-kubernetes.service kubelet.service (rule) networkd: reload systemd-networkd.service storage: none
A trial: chalkctl config apply --try 5m
| chalkctl | What happens |
|---|---|
| config apply | Checks and applies; a staged image checks it too |
| … --dry-run | Prints the diff, outputs and effects; changes nothing |
| … --try 5m | A trial that rolls back at the deadline unless confirmed |
| config confirm | Ends a trial early, after a health check |
| config rollback | Delivers the previous config again |
| … --staged | Writes only the staged image's slot, or pending config |
| config discard | Drops the pending config |
Breaking config changes, safely
upgrade --rollback boots A with this configchalkd check, run from B's store in a sandbox, before anything is writtenIllustration: Backend is the example kind of the docs' “Declare a config kind”; the hashes are made up.
- 1Change the kind in the image's moduleA new required field or API version is fine
- 2Write config of the new form
chalkctl validatechecks it against the new image - 3Upgrade with the secrets fileThe config travels with the image; B checks it
- 4B boots with its own slotUnhealthy: back to A and A's config
config apply checks with both images and writes both slots; --staged writes the staged image's slot alone. An upgrade drops slots no UKI boots any more.
Upgrades replace the whole image
cp1: installing 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6) cp1: installed 3fa9c1e2b7d0 (…), staged as .staged-chalkos_2.efi cp1: activated 3fa9c1e2b7d0 (…), which boots next as chalkos_2+3.efi cp1: rebooting into 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6) cp1: runs 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6), found healthy
The new entry counts its tries
Kubernetes clusters roll control planes one at a time, workers in batches, and stop at the first rollback.
The upgrade carries each node's config; the new image checks it before it is written.
- The old image keeps its own config for a fallback
- chalkctl upgrade --rollback on request
- No deltas: about 300 MiB per node
- A downgrade is an upgrade to another root hash
Layers, and what you can build
Much like a Talos node.
- One chalkctl bootstrap; other nodes join on their own
- VIP held through an etcd election
- FRR speaker announces node address and VIP
- ClusterRegistry for air gaps
- A Kubernetes upgrade is an image upgrade
The same API, between Talos and VyOS.
| Firewall | nftables zones, policies, NAT |
| Routing | FRR BGP with BFD, VRFs, policy routing |
| NAT64, DNS | Jool, Unbound with DNS64 |
| DHCP | Kea DHCPv4 and v6, relays |
| VPN | WireGuard through networkd |
| Time | chrony with a TimeServer |
The base OS and API, then add the rest as modules.
- Any NixOS module or package
- Kinds, outputs, effects of your own
- Repo examples: Hardware, Rack, BIRD BGP
{
name = "appliance-kvm";
platform = "kvm";
modules = [ ./my-service.nix ];
}
Test it before a node sees it
| # | Step | Command | Catches |
|---|---|---|---|
| 01 | Schemas | nix build …chalkosSchemas | The kinds, in seconds |
| 02 | Image tests | nix flake check | Files, rejections, effects; no node |
| 03 | Image build | nix build .#<image> | Broken rules, no room for the next image |
| 04 | Validate | chalkctl validate | Every node's files, no node reached |
| 05 | Lab | chalklab create | QEMU VMs with Secure Boot and a TPM |
| 06 | Roll out | config apply --dry-run / --try | The plan, then a trial that rolls back |
| 07 | Diagnose | chalkctl node diag | The image's diagnostics |
chalkos.tests.bgp-new-asn = { before = ./bgp-examples/fabric.yaml; config = ./tests/new-asn.yaml; refusals = [{ kind = "Peer"; name = "router"; path = "spec.asn"; message = "the AS number stays 65000; add a Peer of …"; }]; };
nix flake check: unsigned and tampered images refused, upgrades that fall back, HA failover, BGP between routers.
Demo: a fresh air-gapped cluster
Three fresh control planes on an IPv6-only fabric; node addresses fdc8:a1b2:c3d4:ff::11 to ::13, API VIP ff::10, both announced to the router over BGP.
- 1Prepare on a connected machineImages, installer and container images, copied with crane
- 2Cut the fabric offRouter config without NAT64, with a TimeServer:
config apply router --try 5m - 3Install three fresh nodes
chalkctl install; their config brings the ClusterRegistry. No bootstrap yet - 4Push the images
chalkctl registry pushto every control plane - 5Check the registry
chalkctl registry list: all on every line - 6Bootstrap
chalkctl bootstrap node1; control planes pull from the registry - 7Prove itPods pull, clocks follow the router, and tcp-connect to the internet fails
Guides: guides/air-gapped, guides/home-lab
Thank you
| Code | github.com/trevex/chalkos |
| Docs | trevex.github.io/chalkos |
| Try it | nix flake init -t github:trevex/chalkos#lab |
| Home lab | trevex.github.io/chalkos/guides/home-lab/ |