Edit mode: click text to change it · Ctrl+S saves a copy · E or Esc leaves

A toolkit for immutable operating systems

chalkos

Build your own API-driven, immutable Linux from NixOS modules

Speaker Niklas Voss

Every node

  • Secure Boot, signed UKI
  • dm-verity store
  • TPM-sealed LUKS2
  • A/B slots, fallback
  • mTLS node API
  • SSH, shell, logins
Unreleased
github.com/trevex/chalkos trevex.github.io/chalkos

I love Talos

Motivation · 1 of 2
What its model gets right

Talos Linux showed how a Kubernetes node can work: a read-only image, no logins, and every change through an API.

  • API-drivenevery change through one API
  • mTLS rolesclient certificates of a role: reader, operator, admin
  • Immutablethe node boots a read-only image
  • No SSH, no shellnothing to log in to, nothing drifts
  • A/B upgradesthe whole image, with rollback

My worry

One vendor decides the licence of the base my machines run on.

The worry is about who holds that decision. A fleet runs for years, and I would rather build its operating system from parts I can rebuild myself.

Licence changes elsewhere in infrastructure software

YearProjectFromTo
2021ElasticsearchApache 2.0SSPL / Elastic Licence
2023TerraformMPL 2.0BUSL 1.1
2024RedisBSD 3-clauseRSALv2 / SSPLv1
The question Can I get the same model from parts I already trust, such as NixOS modules and nixpkgs?
chalkos · Niklas Voss
02 / 16

I love Talos, but it runs Kubernetes only

Motivation · 2 of 2

My other machines deserve the same model. Today each is installed and patched its own way.

Routers

Firewall, BGP, DNS, DHCP and VPN in front of the cluster.

Appliances

One service on one machine, with nobody at the console.

Edge boxes

Remote machines that must survive a bad upgrade alone.

The wish One model for all of them
Read-only image
Node API over mTLS
A/B upgrade with fallback
Config checked before it lands
No logins
chalkos · Niklas Voss
03 / 16

So build the distribution

What chalkos is

A toolkit that builds API-driven, immutable Linux distributions from NixOS modules.

Nix buildsmkImages + modules
→
Image dirdisk + schemas
→
chalkctl deliversnode API, mTLS
Kubernetes layer

Much like a Talos node.

  • kubelet, etcd, control plane
  • VIP, BGP speaker
  • cluster registry
nixosModules.kubernetes
Router layer

Between Talos and VyOS.

  • firewall, NAT, BGP
  • VRFs, NAT64, DNS
  • DHCP, WireGuard, NTP
nixosModules.router
Your own

No layer: a minimal appliance.

  • network, disks, time
  • node API
  • your modules
your NixOS modules

Compared with Talos, from the docs

TopicTaloschalkos
PurposeA distribution for Kubernetes nodesBuilds distributions: Kubernetes, routers, other appliances
ConfigOne machine config per node, Talos's own schemaKubernetes-style YAML, one kind per concern; kinds and schemas ship in the image
UpgradesNode keeps one config across upgradesOne config slot per image; a rollback boots its own config
ImagesOne image, signed system extensionsOne image per kind of node, from NixOS modules
Node CAControl planes hold the root CA's keyControl planes hold a node CA; the OS CA key stays offline
MaturityReleased, many platformsUnreleased; x86-64 bare metal and KVM
Compared with NixOS No Nix daemon, no rebuild on the node: a software change is a new image.
chalkos · Niklas Voss
04 / 16

Anatomy of a node

EFI · systemd-boot · UKI · A/B slots
ESP vfat · 1 GiB
systemd-boot
chalkos_<gen>.efi
verity128 MiB
hash tree
Slot Astore_<hash16>
erofs, zstd · 3 GiB
verity_empty
Slot B_empty until
an upgrade
STATE ext4 · 128 MiB
LUKS2, TPM
VAR /var · rest of disk
LUKS2, TPM
Volumes 60-<name>
from config
System region: fixed by the image, the same on each of its nodes
From config: SystemDisk, Volume
1 EFI boot of a signed UKI

systemd-boot starts one UKI per slot. Kernel, initrd and command line sit under one signature.

2 A/B slots, root hash as identity

An upgrade writes the inactive slot. The store's dm-verity root hash names the image; there is no version number.

3 Encrypted STATE and VAR

LUKS2 with keys sealed by the TPM. STATE holds config and certificates, VAR what services write; root is a tmpfs.

The build checks the fit · Kubernetes image, metal
246MiBstore data, erofs
2MiBdm-verity hash tree
44MiBUKI, mostly initrd
80%most a part may fill of its partition
~300MiBsent per node by an upgrade
chalkos · Niklas Voss
05 / 16

Trust from firmware to API call

Secure Boot · dm-verity · TPM2 · mTLS
Firmwaredb certificateEnrolled by chalkctl secureboot
→
Verifiessystemd-boot + UKISigned by chalkbuild sign
→
UKI cmdlineusrhash= root hashUnder the same signature
→
dm-verityhash treeA changed block fails to read
→
Resultevery store byteOne signature covers the whole OS
Disks tied to the machine

STATE, VAR and volumes are LUKS2, keys sealed by the TPM to the Secure Boot state.

  • Same db key: upgrades unlock unattended
  • Disk moved or Secure Boot changed: no unseal
  • Fallback: the node's recovery key
chalkd is the only way in

Node API on TCP 50000, mutual TLS, client chain checked on every request.

reader<operator<admin
  • No SSH, no login
  • Client files of one role each
Ending access

No revocation list on nodes: access ends with the certificate, or a rotation.

  • chalkctl rotate os-ca cuts old certificates off
  • chalkctl node revoke on Kubernetes
  • The OS CA key stays in the secrets file
chalkos · Niklas Voss
06 / 16

Customise with plain NixOS modules

mkImages · layers · extensions · toggles
flake.nixafter the homelab example
images = chalkos.lib.mkImages {
  revision = self.rev or self.dirtyRev or null;
  images = [
    {
      name = "router-metal";
      platform = "metal";
      modules = [ chalkos.nixosModules.router ];
    }
    {
      name = "node-metal";
      platform = "metal";
      modules = [
        chalkos.nixosModules.kubernetes
        ./extensions      # your kinds
        { chalkos.kubernetes.bgp.enable = false; }
      ];
    }
  ];
};

One layer per image, plus a platform: metal or kvm.

An extension is a NixOS module that sets these options

OptionWhat it doesWhere it runs
chalkos.kindsA typed config kind. Rejects invalid documents.chalkctl and the node
…kinds.<Kind>.rulesCEL against the running config. Rejects forbidden changes.chalkctl and the node
chalkos.validatorsChecks all documents together. Rejects what it reports.The node
chalkos.outputsRenders config files. Rejects files that fail their check.The node, at each change and boot
chalkos.effectsReloads or restarts units after a changeThe node
chalkos.diagnosticsRead-only commands for chalkctl node diagThe node, on request
chalkos.testsExpected files, rejections and effectsBuild time, Nix sandbox
Feature toggles: on by default, off per image
kubernetes.bgpkubernetes.registry router.bgprouter.nat64router.resolver router.dhcprouter.dhcpRelayrouter.wireguardrouter.ntp
Off removes the feature: its packages, kinds and outputs.
chalkos · Niklas Voss
07 / 16

Config kinds ship in the image

The headline idea

A module declares a kind like a CRD: typed fields and CEL rules. The build ships it as JSON Schema, so config is checked against the image that will run it.

1 · Declarebgp.nix of the BIRD example, shortened
chalkos.kinds.Peer = {
  group = "bgp.example.org";
  version = "v1alpha1";
  multiple = true;
  spec.asn = chalkosLib.mkOption {
    type = lib.types.ints.between 1 4294967295;
    description = "AS number of the neighbour.";
    rules = [{
      rule = "self == oldSelf";
      messageExpression = ''"the AS number
        stays " + string(oldSelf) + ...'';
    }];
  };
  # address, interfaces, password, bfd ...
};
→
2 · Buildschemas/…_Peer.json
{
  "type": "integer",
  "minimum": 1,
  "maximum": 4294967295,
  "x-chalkos-rules": [{
    "rule": "self == oldSelf",
    "takes": "refuse"
  }]
}

# In the image directory and in
# /etc/chalkos/schemas on the node.
# Rules compile at build time,
# within a cost limit.
→
3 · Check, before and on the node
chalkctl validate · no node needed node node1: Peer "router": - at '/spec': additional properties 'asm' not allowed
config apply · asn 65000 → 65002 spec.asn: the AS number stays 65000; add a Peer of another name for 65002
  • chalkctl runs the rules that read no facts
  • The node runs every rule again
Why per image A new image may change a kind incompatibly: its own schemas check the config it boots.
chalkos · Niklas Voss
08 / 16

Documents in, services out

The BIRD BGP extension example
1DocumentsThe node's Speaker and Peer YAML
→
2RenderA Go text/template: bird.conf.tmpl
→
3Output filesbird.conf in the output's directory
→
4Checkbird -p parses it without running it
→
5EffectReload bird.service when bird.conf changed
bird.conf.tmplexcerpt, lines wrapped
{{- range $.documents.Peer }}
{{- $peer := . }}
{{- range .spec.interfaces }}

protocol bgp '{{ $peer.metadata.name }}_{{ . }}' {
  local as {{ $asn }};
  neighbor {{ $peer.spec.address }}
    as {{ $peer.spec.asn }};
  interface {{ quote . }};
{{- if $peer.spec.bfd }}
  bfd on;
{{- end }}
  ...
bird.conf, renderedgolden file, excerpt, wrapped
protocol bgp 'router_enp1s0' {
  local as 65001;
  neighbor fdc8:a1b2:c3d4:1::1
    as 65000;
  interface "enp1s0";
  direct;
  connect retry time 5;
  bfd on;
  authentication md5;
  password 'secret_bgp.password';
  ...
bgp.nixsimplified
chalkos.outputs.bird-conf = {
  inputs = [ "Speaker" "Peer" ];
  render = chalkosLib.render
    .template ./bird.conf.tmpl;
  check = ''
    ... bird -p -c "$conf"
  '';
};

chalkos.effects.bird =
  chalkosLib.reloadOn
    { outputs = [ "bird-conf" ]; }
    [ "bird.service" ];
If it fails A template that refuses, or a check that fails, stops the delivery before anything changes: the node keeps its old documents, its old bird.conf and a bird that was not reloaded.
chalkos · Niklas Voss
09 / 16

Apply config with a way back

Dry run · trial · rollback
$ chalkctl config apply k1 --secrets=secrets.age --dry-run
k1: config 2 added, 0 removed, 1 changed
~ BGPSpeaker "bgp"
    spec.export.interfaces[1]: (none) → "lb0"
+ NetDev "20-lb0"
+ Network "20-lb0"
k1: config dd1465db… → 9a461558…
  outputs:  bgp, networkd
  effects:  bgp: reload frr.service (output bgp)
            kubernetes: restart chalkos-kubernetes.service
                        kubelet.service (rule)
            networkd: reload systemd-networkd.service
  storage:  none

A trial: chalkctl config apply --try 5m

1Node applies the config and starts a deadline
→
2chalkctl confirms over a new connection
→
3No confirmation: chalkd rolls back by itself
chalkctlWhat happens
config applyChecks and applies; a staged image checks it too
… --dry-runPrints the diff, outputs and effects; changes nothing
… --try 5mA trial that rolls back at the deadline unless confirmed
config confirmEnds a trial early, after a health check
config rollbackDelivers the previous config again
… --stagedWrites only the staged image's slot, or pending config
config discardDrops the pending config
chalkos · Niklas Voss
10 / 16

Breaking config changes, safely

One config slot per image
Slot A · running, known-goodImage Aroot hash 3fa9c1e2b7d0…
Its kinds and schemasBackend · v1alpha1
Config slot/state/config/<hash A>kind: Backend · v1alpha1address: …
Stays as it ran: a fallback or upgrade --rollback boots A with this config
→ upgrade carries the new config
Slot B · staged, boots nextImage Broot hash 9d02a7c4e1f8…
Its kinds and schemas changedBackend · v1, a new required field
Config slot/state/config/<hash B>kind: Backend · v1address: … port: 8080
Checked by image B's own chalkd check, run from B's store in a sandbox, before anything is written

Illustration: Backend is the example kind of the docs' “Declare a config kind”; the hashes are made up.

  1. 1
    Change the kind in the image's module
    A new required field or API version is fine
  2. 2
    Write config of the new form
    chalkctl validate checks it against the new image
  3. 3
    Upgrade with the secrets file
    The config travels with the image; B checks it
  4. 4
    B boots with its own slot
    Unhealthy: back to A and A's config
Also With an image staged, config apply checks with both images and writes both slots; --staged writes the staged image's slot alone. An upgrade drops slots no UKI boots any more.
chalkos · Niklas Voss
11 / 16

Upgrades replace the whole image

Inactive slot · boot counting · fallback
SourceImage directory, or OCI pulled by digest
CheckHash, config, Secure Boot db
WriteInto the inactive slot
ActivateNext boot entry, 3 tries
HealthWithin 300 s by default
Bless or fall backTries used up: the previous slot boots
$ chalkctl upgradeone node, shortened
cp1: installing 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6)
cp1: installed 3fa9c1e2b7d0 (…), staged as .staged-chalkos_2.efi
cp1: activated 3fa9c1e2b7d0 (…), which boots next as chalkos_2+3.efi
cp1: rebooting into 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6)
cp1: runs 3fa9c1e2b7d0 (v1.5.0, a1b2c3d4e5f6), found healthy

The new entry counts its tries

chalkos_2+3.efi→+2-1→+1-2→+0-3 skippedorchalkos_2.efi blessed

Kubernetes clusters roll control planes one at a time, workers in batches, and stop at the first rollback.

Config travels with the image

The upgrade carries each node's config; the new image checks it before it is written.

  • The old image keeps its own config for a fallback
  • chalkctl upgrade --rollback on request
  • No deltas: about 300 MiB per node
  • A downgrade is an upgrade to another root hash
chalkos · Niklas Voss
12 / 16

Layers, and what you can build

One layer per image at most
Kubernetes nixosModules.kubernetes

Much like a Talos node.

  • One chalkctl bootstrap; other nodes join on their own
  • VIP held through an etcd election
  • FRR speaker announces node address and VIP
  • ClusterRegistry for air gaps
  • A Kubernetes upgrade is an image upgrade
Router nixosModules.router

The same API, between Talos and VyOS.

Firewallnftables zones, policies, NAT
RoutingFRR BGP with BFD, VRFs, policy routing
NAT64, DNSJool, Unbound with DNS64
DHCPKea DHCPv4 and v6, relays
VPNWireGuard through networkd
Timechrony with a TimeServer
Diagnostics
traceroutebgp-summarybfd-peersnat64-sessionsdhcp-leasesntp-clients
Your own

The base OS and API, then add the rest as modules.

  • Any NixOS module or package
  • Kinds, outputs, effects of your own
  • Repo examples: Hardware, Rack, BIRD BGP
{
  name = "appliance-kvm";
  platform = "kvm";
  modules = [ ./my-service.nix ];
}
chalkos · Niklas Voss
13 / 16

Test it before a node sees it

From seconds to VMs
#StepCommandCatches
01Schemasnix build …chalkosSchemasThe kinds, in seconds
02Image testsnix flake checkFiles, rejections, effects; no node
03Image buildnix build .#<image>Broken rules, no room for the next image
04Validatechalkctl validateEvery node's files, no node reached
05Labchalklab createQEMU VMs with Secure Boot and a TPM
06Roll outconfig apply --dry-run / --tryThe plan, then a trial that rolls back
07Diagnosechalkctl node diagThe image's diagnostics
A test in the modulebgp.nix, shortened
chalkos.tests.bgp-new-asn = {
  before = ./bgp-examples/fabric.yaml;
  config = ./tests/new-asn.yaml;
  refusals = [{
    kind = "Peer"; name = "router";
    path = "spec.asn";
    message = "the AS number stays
      65000; add a Peer of …";
  }];
};
chalkos's own end-to-end tests QEMU VMs under nix flake check: unsigned and tampered images refused, upgrades that fall back, HA failover, BGP between routers.
chalkos · Niklas Voss
14 / 16

Demo: a fresh air-gapped cluster

Live · home lab · from scratch
IPv4 internet NO NAT64 home network · IPv4 by DHCP workstation chalkctl · crane · kubectl router AS 65000 · FRR · Unbound WireGuard · no NAT64 TimeServer (chrony) wan WireGuard to the router fdc8:a1b2:c3d4:f0::/64 fabric · ::1 and ::2 switch · fabric fdc8:a1b2:c3d4:1::/64 · router advertisements node1 ff::11 · AS 65001 registry :5000 node2 ff::12 · AS 65001 registry :5000 node3 ff::13 · AS 65001 registry :5000 each node: 2 ports, a BGP session with BFD on each

Three fresh control planes on an IPv6-only fabric; node addresses fdc8:a1b2:c3d4:ff::11 to ::13, API VIP ff::10, both announced to the router over BGP.

  1. 1
    Prepare on a connected machine
    Images, installer and container images, copied with crane
  2. 2
    Cut the fabric off
    Router config without NAT64, with a TimeServer: config apply router --try 5m
  3. 3
    Install three fresh nodes
    chalkctl install; their config brings the ClusterRegistry. No bootstrap yet
  4. 4
    Push the images
    chalkctl registry push to every control plane
  5. 5
    Check the registry
    chalkctl registry list: all on every line
  6. 6
    Bootstrap
    chalkctl bootstrap node1; control planes pull from the registry
  7. 7
    Prove it
    Pods pull, clocks follow the router, and tcp-connect to the internet fails

Guides: guides/air-gapped, guides/home-lab

chalkos · Niklas Voss
15 / 16

Thank you

16 / 16