Skip to content

Build an air-gapped cluster

This guide runs a chalkos cluster on a network without internet access. Everything the nodes would fetch from the internet comes in from a connected machine instead: the OS images, chalkctl, the container images and the time. At the end the cluster runs, every node pulls its container images from the control planes, and its clocks follow a time server inside the air gap. The last sections cover upgrades, new images and the limits.

The guide uses the cluster of Build a cluster on real machines: the cluster-metal template's three control planes cp1, cp2 and cp3 at 192.168.1.11 to 192.168.1.13, with the network's router at 192.168.1.1. Read that guide first; this one lists what changes. The home lab has a variant that cuts a running lab off the internet.

What comes in from outside

A chalkos cluster takes four things from the internet by default. In an air gap, each comes from somewhere else:

What Default source In the air gap
OS images and the installer built with Nix, which downloads the sources built on a connected machine and carried in as image directories
chalkctl the chalkos flake its static program, copied from a connected machine
Container images the registries the Cluster document names the cluster registry on the control planes, filled with chalkctl registry push
Time ptbtime1.ptb.de to ptbtime3.ptb.de over NTS a time server inside the air gap, named in a TimeSync document

chalkctl needs no internet. It reaches the nodes at the addresses of the inventory, reads the image directories the inventory names, and generates and holds the cluster's secrets on the workstation. The installer holds no config and fetches nothing: chalkctl install sends the image to it.

The nodes need no DNS names either, as long as every address chalkos uses is an IP address: the inventory's, the ClusterRegistry's endpoints and the TimeSync's servers. Names outside the cluster do not resolve, in pods or on the nodes. CoreDNS forwards every name outside the cluster to the servers of the node's resolv.conf, and refuses to start when the file names none, so keep a resolver in the air gap that the nodes' Network documents name, such as the network's router.

Before you begin

  • You followed Build a cluster on real machines up to its install, or know its steps.
  • A connected machine with Nix builds the images, as the build host describes, and has crane and yq.
  • The workstation inside the air gap has chalkctl, mtools, kubectl and yq. Copy chalkctl's program as Requirements describes.
  • The workstation inside the air gap holds the age identity or SSH key that secrets.age is encrypted to, as Requirements lists.
  • A time server answers NTP inside the air gap, such as a clock with a GPS receiver or a chalkos router with a TimeServer, as Serve time with NTP and the home lab's router show. A server without a reference clock answers as unsynchronised, at stratum 16, and its clients refuse it, unless it serves its own clock, as a TimeServer with localStratum: 10 does. Other chalkos nodes run no NTP server for other machines.
  • A way to carry files into the air gap, such as a USB disk.

Prepare on a connected machine

In the repository of the cluster, build the image and the installer as Build the image and the installer describes, with the cluster's OS CA in osCA:

nix build .#node-metal -o images/node-metal
nix build .#installer -o installer

Then copy the container images. The Cluster document's images block names every image the control plane, CoreDNS, the pod sandbox and flannel run, each pinned by digest. Copy each into an OCI image layout of its own with crane, as Bring the images into the air gap explains:

mkdir -p images/oci
for ref in $(yq 'select(.kind == "Cluster") | .spec.images[]' cluster/cluster.yaml); do
  repo=${ref%%@*}; name=${repo##*/}; name=${name%%:*}
  rm -rf "images/oci/$name"
  crane pull --format=oci "$ref" "images/oci/$name"
done

Copy the images of your workloads, and those that the objects of your Addon documents name, the same way, each into a directory of its own.

Copy the repository to the disk you carry in, <carry>, with the builds as directories rather than the links nix build makes, and chalkctl's program beside it. Leave out keys/, which holds the Secure Boot keys of Sign the image and the installer: whoever holds the db key decides what the machines boot, and the air gap needs only the signed builds, so the keys stay offline with the connected machine:

rsync -aL --exclude keys/ ./ <carry>/cluster/
cp "$(nix build --no-link --print-out-paths github:trevex/chalkos/<rev>#chalkctl)/bin/.chalkctl-wrapped" \
  <carry>/chalkctl

<rev> is the chalkos revision that flake.lock pins, so chalkctl and the image match. The copy holds the inventory, the config files, secrets.age, secrets.pub.json, the image, the installer and the container images.

Give the nodes a time server

A node checks every certificate against its clock, and the default time servers are on the internet. Name the time server inside the air gap in a TimeSync document that every node gets:

cluster/time.yaml
apiVersion: time.chalkos.dev/v1alpha1
kind: TimeSync
metadata: {name: default}
spec:
  servers:
    - {host: 192.168.1.1, nts: false}

Replace 192.168.1.1 with your time server's address. nts: false takes plain NTP, which most local time servers serve; leave it out for a server with NTS. A node without a reachable server keeps its clock running from where it stands, and its clock drifts.

Turn the cluster registry on

The cluster registry runs on the control planes and serves the container images to every node. Add a ClusterRegistry that every Kubernetes node gets, with an address of every control plane and every host the cluster's images come from:

cluster/cluster-registry.yaml
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: ClusterRegistry
metadata: {name: cluster}
spec:
  mirrors: [docker.io, ghcr.io, registry.k8s.io]
  endpoints: [192.168.1.11, 192.168.1.12, 192.168.1.13]

Write the ClusterRegistry explains the fields. Check the files against the image:

chalkctl validate

Install the nodes

Install the three nodes as Build a cluster on real machines describes. The inventory's image is a directory, images/node-metal, which chalkctl reads without a network. Each control plane starts the cluster registry at its first boot, and every node follows the time server of the TimeSync. Do not bootstrap yet: the control plane's images are not in the registry.

Push the images and bootstrap

Push every image of the Cluster document to every endpoint, from the layouts you carried in:

for ref in $(yq 'select(.kind == "Cluster") | .spec.images[]' cluster/cluster.yaml); do
  repo=${ref%%@*}; name=${repo##*/}; name=${name%%:*}
  chalkctl registry push "images/oci/$name" "$ref" --secrets secrets.age
done

chalkctl refuses a layout whose digest is not the one the reference pins, so a changed copy never reaches the registry. Check that every endpoint holds every image:

chalkctl registry list --secrets secrets.age

Every line ends in all. Then bootstrap the cluster and write a kubeconfig, as Bootstrap Kubernetes describes. The control planes pull their images from the cluster registry.

Cut the connection

A cluster prepared on a connected network may have reached the internet while you set it up. Until the cut, containerd falls back to an image's host when the cluster registry lacks the image, so a missing image goes unnoticed. chalkctl registry list showing all for every image before the bootstrap shows that the cluster does not depend on the registries outside. Cut that path now, on the network's router or firewall, and leave the workstation's path to the nodes. The cluster needs nothing from outside afterwards: its images are in the registry, and its time and names come from inside the air gap.

Check that it worked

The nodes are Ready and the cluster's pods run:

kubectl get nodes
kubectl get pods -A

chalkctl registry list shows all for every image and exits 0. chalkctl status cp1 prints a line time: that names the air gap's time server as the one the clock follows. A pod whose image is not in the registry stays in ImagePullBackOff, because containerd finds it neither in the cluster registry nor at its host.

Upgrade and change the cluster later

Every change that brings something new starts on the connected machine.

Upgrade the OS image

Build the new image on the connected machine, copy its directory into the air gap, and point the inventory's image at it. Then roll it out with chalkctl upgrade, as Upgrade nodes describes. chalkctl reads the image from the directory and sends it to each node.

Move to a new Kubernetes release

Copy the new release's images with crane, carry them in, and push them before you deliver the Cluster document that names them. A control plane cannot start an image the registry lacks, so its trial ends in a rollback. Upgrade Kubernetes describes both steps of a release: the control plane's images through config, then the kubelets through a new OS image.

Add and remove images

Copy a workload's image, carry it in and push it before you deploy the pods that run it, as Push the workloads' images describes. Delete images you no longer run with chalkctl registry delete, and free their space with chalkctl registry gc, as Delete images and free space describes. Keep the images of the previous release until the new one has proven itself: a rollback of the Cluster document names them again, and a node that lacks them pulls them.

Add a control plane

Add its address to the ClusterRegistry's endpoints, install it, and push every image again, as Add or remove a control plane describes.

Limits

  • Nothing outside the cluster registry's mirrors reaches the nodes. A pod that names an image of another host fails to pull it.
  • chalkd does not pull OS images from the cluster registry. An inventory image with an oci source needs a registry that the workstation and the nodes reach; the air gap takes image directories instead.
  • The copies take more space than the nodes run: crane copies every platform of an index. The cluster-metal template's eight images take about 1.2 GiB.
  • A time server without a reference clock serves only its own clock, as a router's TimeServer does with localStratum, or chrony's local stratum 10 orphan on another machine. It keeps the nodes' clocks together, but they drift from real time with its clock. Certificates that other systems issue, and those chalkos issues for them, are checked against these clocks, so a production air gap takes its time from a reference clock.
  • Names outside the cluster do not resolve. Anything that needs one, such as a webhook of an external service, needs that service and its name inside the air gap.

What next