Run an air-gapped cluster with the cluster registry¶
This page is for an operator whose Kubernetes cluster has no connection to the registries its
container images come from. The cluster registry is a registry on the control planes:
chalkos-registry runs on each node whose address the ClusterRegistry document lists, and
containerd on every node looks there first for the images of the hosts the document mirrors. At
the end every node pulls the control plane's images and the workloads' images from the control
planes, and chalkctl registry list shows the same images on every endpoint.
A cluster that reaches the registries, or a mirror on its network that holds every image, does not need the cluster registry; Registries and mirrors covers those. The cluster registry suits a cluster with nothing else to pull from, such as one on an isolated site.
The configuration on this page adds a ClusterRegistry to the homelab, whose control planes
node1, node2 and node3 have the addresses fdc8:a1b2:c3d4:ff::11 to
fdc8:a1b2:c3d4:ff::13. The output comes from a lab cluster whose control plane cp1 at
192.168.100.11 serves the registry and whose worker w1 pulls from it. The lab's
ClusterRegistry mirrors cluster-registry.chalkos.test, a host the lab cannot reach.
Before you begin¶
- You have the workstation that Set up your workstation describes, with the inventory of a Kubernetes cluster.
- You have the cluster's secrets, from any source
--secretstakes, as Manage secrets describes, or an admin client file. Pushing, deleting and collecting garbage need the admin role. Listing needs a client file of any role, because chalkctl verifies the endpoints by the OS CA it holds. - The workstation reaches every endpoint on the registry's port, TCP 5000 by default.
- You have the images to push: as OCI image layouts or archives carried into the air gap, or in a registry the workstation reaches.
- The nodes' images carry the cluster registry. Kubernetes images do by default. An image built
with
chalkos.kubernetes.registry.enableset to false does not, and refuses theClusterRegistrykind.
Write the ClusterRegistry¶
One ClusterRegistry document turns the cluster registry on. Put it in a file that every
Kubernetes node gets, such as one in cluster/:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: ClusterRegistry
metadata: {name: cluster}
spec:
mirrors: [docker.io, ghcr.io, registry.k8s.io]
endpoints: ["fdc8:a1b2:c3d4:ff::11", "fdc8:a1b2:c3d4:ff::12", "fdc8:a1b2:c3d4:ff::13"]
port: 5000
mirrors lists the registry hosts whose images come from the cluster registry first, as image
references name them and without a port. docker.io is Docker Hub. The registry holds each image
below its host's name: docker.io/library/nginx:1.27 is
<endpoint>:5000/docker.io/library/nginx:1.27. containerd mirrors whole hosts, so it looks for
every image of a mirrored host in the cluster registry first. In an air-gapped cluster, mirrors
holds every host the Cluster document's images come from: registry.k8s.io for the control
plane, CoreDNS and the pod sandbox, and ghcr.io for flannel.
endpoints lists IP addresses, without names or ports. A node serves the cluster registry when one
of its static addresses, the Address= values of its Network documents, is an endpoint. List an
address of every control plane. The image refuses a control plane whose addresses are missing,
with this message for chalkctl validate, chalkctl config apply and the node itself:
the control plane <node> serves the cluster registry, but the ClusterRegistry's endpoints name none of its static addresses (<addresses>); add one of them to the endpoints and apply the config to every node
<node> is the node's name and <addresses> its static addresses.
port is 5000 by default. It must be 1024 or above, because chalkos-registry runs without
privileges.
Every Kubernetes node holds the same document. chalkctl refuses an inventory whose nodes hold
different ClusterRegistry specs, or one on some nodes and none on others, as it does for
Cluster. A router image does not know the kind, so the file goes to the Kubernetes nodes alone.
Config kinds lists the fields.
A serving node runs the registry under a user of its own, keeps its images in
/var/lib/chalkos-registry and opens the port in its firewall. containerd on it pulls from its own
endpoint first. Every node then tries the other endpoints in the order of endpoints, then the
host's Mirror and the host itself.
Pull from the cluster registry lists the
order, and
How Kubernetes runs on a node
the mechanism.
Apply the config to every node¶
A cluster that is not installed yet gets the document with its install, and each control plane
starts the registry at its first boot. A running cluster gets it with chalkctl config apply to
every Kubernetes node:
The delivery rewrites containerd's mirrors on every node and starts the registry on serving nodes. containerd reads the mirrors on every pull, so the kubelet keeps running. A kubelet that the same delivery restarts for another change waits for the node's registry, so it pulls from it at once.
The lab's delivery also added a Registry with credentials for cluster-registry.chalkos.test.
That Registry restarts chalkos-registry-credentials.service and kubelet.service on its own,
as the credentials
need, so the kubelet restarts after the registry. On cp1 the run printed:
cp1: config 2 added, 0 removed, 0 changed
restarted chalkos-registry-credentials.service
restarted chalkos-registry-mirrors.service
restarted chalkos-registry.service
restarted kubelet.service
cp1 runs config 68029e1e09940d0aecf780223d9af91ca4dd1e8da3072d8054995cca55de0feb
A worker serves no endpoint, so a ClusterRegistry restarts chalkos-registry-mirrors.service
alone there. In the lab, w1 restarted the Registry's two units as well:
w1: config 2 added, 0 removed, 0 changed
restarted chalkos-registry-credentials.service
restarted chalkos-registry-mirrors.service
restarted kubelet.service
w1 runs config 84a6e74a9aa45837cf76805559c058caf3233c9aca73904d1f0c11a9fb4c2f8d
Bring the images into the air gap¶
The images block of the Cluster document names the images
of the API server, the controller-manager, the scheduler, kube-proxy, etcd, CoreDNS, the pod
sandbox and, on an image with flannel, flannel, each pinned by digest.
Name the images of the components explains the
references and their digests.
On a machine connected to the registries, copy each image with all its platforms into an OCI image layout, keeping its digest. An OCI layout or archive holds one image or index for chalkctl, so give each image a directory of its own. With crane and yq:
mkdir -p images/oci
for ref in $(yq 'select(.kind == "Cluster") | .spec.images[]' cluster/cluster.yaml); do
repo=${ref%%@*}; name=${repo##*/}; name=${name%%:*}
rm -rf "images/oci/$name"
crane pull --format=oci "$ref" "images/oci/$name"
done
crane pull --format=oci writes the index with every platform it names, with the manifests'
bytes and media types as the registry serves them, so the layout serves the digest the Cluster
document pins. A pull into a layout that exists adds a second entry to it, which chalkctl refuses,
so the loop removes the directory first. Carry images/oci/ into the air gap. A tar of a layout,
tar -cf <name>.tar -C images/oci/<name> ., is an OCI archive, which chalkctl reads too.
The indexes are larger than the images a node runs. The homelab's eight images take about 1.2 GiB, of which the pause image's index takes about 400 MiB, most of it its Windows images.
skopeo keeps no Docker manifest list in an OCI layout
registry.k8s.io serves its images as Docker manifest lists. skopeo's oci: and
oci-archive: transports write OCI indexes alone, so
skopeo copy --all --preserve-digests stops on such an image:
Manifest list must be converted to type "application/vnd.oci.image.index.v1+json" to be written to destination, but we cannot modify it: "Instructed to preserve digests"
Without --preserve-digests, skopeo converts the list, which gives it a digest other than the
one the Cluster document pins. skopeo copies an image whose index is an OCI index already,
such as flannel's, with its digest kept.
Push the control plane's images before bootstrap¶
containerd pulls the Cluster document's images once the kubelet starts the static pods, at
bootstrap and when a node joins. On a new cluster, install every control plane with the
ClusterRegistry in its config, push the images, and then run
chalkctl bootstrap.
Name each image as the Cluster document names it, tag and digest together. This loop pushes
every image of spec.images from the directories above:
for ref in $(yq 'select(.kind == "Cluster") | .spec.images[]' cluster/cluster.yaml); do
repo=${ref%%@*}; name=${repo##*/}; name=${name%%:*}
chalkctl registry push "images/oci/$name" "$ref" --secrets secrets.age
done
chalkctl registry push pushes each image under its
tag. It refuses a source of another digest before it reaches an endpoint, so a copy that changed
the image never lands in the registry.
Check that every endpoint holds every image of spec.images, as
Check the endpoints describes. Then bootstrap the cluster as
Bootstrap and get a kubeconfig describes.
Choose the source¶
The first argument of chalkctl registry push is one of these:
- An OCI image layout directory.
- An OCI archive, a tar of a layout. chalkctl unpacks it below
$TMPDIR. - A docker archive, as
docker savewrites it. It holds one platform's image, whose digest differs from the one its registry serves. Pods and documents that pin the upstream digest do not find it, so it does not serve the images of aClusterdocument. - An image in a registry the workstation reaches, read with the workstation's docker config.
chalkctl reads a local source through before it pushes to the first endpoint, so a missing blob
fails at once. A digest that a Cluster document pins is usually that of an index, which names an
image per platform. The push carries the index with every image it names, attestations included,
so it takes several times the space of one image.
Push the workloads' images¶
Name a workload's image as the pod names it, with its registry host:
A pod that runs busybox:1.37.0 or docker.io/library/busybox:1.37.0 then pulls it from the
cluster registry. The host must be one of mirrors; chalkctl refuses any other, because containerd
never looks for it in the cluster registry. A reference by digest alone pushes the manifest
without a tag.
The push prints the image's digest, then a line for each endpoint. The lab's push of an OCI archive printed:
pushing cluster-registry.chalkos.test/e2e/busybox:1.37.0 as cluster-registry.chalkos.test/e2e/busybox@sha256:66a6306db78bf2dbf3487f293aa8d6990d8e506fdffab9cc43fe422becf886e4
192.168.100.11: pushed 192.168.100.11:5000/cluster-registry.chalkos.test/e2e/busybox:1.37.0 as sha256:66a6306db78bf2dbf3487f293aa8d6990d8e506fdffab9cc43fe422becf886e4
The lab reaches cp1's registry through a forwarded port, with
--connect-to 192.168.100.11=127.0.0.1:<port>. The flag reaches an endpoint at another address
and still verifies its certificate for the endpoint.
Push every image to every endpoint¶
Nothing copies images between the endpoints. Each chalkctl registry command acts on every
endpoint, one after the other, and reports each. A push fails unless every endpoint serves the
image's digest afterwards:
the push failed on <failed> of <total> endpoints of the cluster registry: <endpoints>; nothing copies images between them, so push again once they are reachable
Push the image again once the endpoints it names answer. A control plane that was down while you pushed misses those images until you push them again, and its kubelet pulls them from the next endpoint meanwhile.
Check the endpoints¶
chalkctl registry list lists the tags of every
endpoint and the endpoints that serve each. In the lab:
$ chalkctl registry list
REPOSITORY TAG DIGEST ENDPOINTS
cluster-registry.chalkos.test/e2e/busybox 1.37.0 sha256:66a6306db78b all
all means every endpoint serves the tag with the same digest. A tag's digest is the one most
endpoints serve. Otherwise the column lists the endpoints that serve it, followed by a mark for
each other endpoint:
| Mark | Means |
|---|---|
missing on <endpoint> |
The endpoint lacks the tag. |
differs on <endpoint> (<digest>) |
The endpoint serves the tag with another digest. |
broken on <endpoint> (lacks <n> images of the index) |
The endpoint lacks images the tag's index names, so a pull of those platforms fails there. |
The command exits 1 when it marks any image or cannot list an endpoint. Push the marked images again to make the endpoints alike. Images pushed by digest alone, without a tag, are not listed.
Delete images and free space¶
Nothing deletes images on its own: no scheduled collection, no expiry and no removal of untagged manifests. The registry's directory grows until you delete images and collect the garbage.
chalkctl registry delete deletes a tag or a
digest on every endpoint. An endpoint that does not hold it counts as done.
- A tag goes alone. The manifest it named stays, and with it its layers, and chalkctl prints the digest and the tags that still name it.
- A digest takes the manifest. chalkctl refuses a digest that tags name unless
--with-tagsis given, and then deletes those tags with it. - chalkctl always refuses a digest that a tagged index names, because the index would break. Delete the index first.
In the lab:
$ chalkctl registry delete cluster-registry.chalkos.test/e2e/busybox@sha256:66a6306db78bf2dbf3487f293aa8d6990d8e506fdffab9cc43fe422becf886e4 --with-tags
192.168.100.11: deleted 192.168.100.11:5000/cluster-registry.chalkos.test/e2e/busybox@sha256:66a6306db78bf2dbf3487f293aa8d6990d8e506fdffab9cc43fe422becf886e4 and its tags 1.37.0
The layers stay on disk until
chalkctl registry gc collects the garbage on each
endpoint, one after the other:
A collection removes the layers and configs that no manifest references, and keeps every manifest, tagged or not. Deleting a tag alone frees nothing.
While an endpoint collects, it serves pulls and refuses pushes and deletes with
the registry is collecting garbage; write again once it finished. A push whose layers went up
before the collection started may fail at its manifest, because the collection removed them. Run
the command when no push is in progress, and push again what failed. An endpoint that collects
already fails a second request. Once begun, a collection runs to its end: Ctrl-C stops chalkctl,
not the collection.
Give the registry a disk of its own¶
The registry keeps its images on VAR by default. A Volume
mounted at /var/lib/chalkos-registry gives it a disk of its own. Put it in the file of each
serving node:
apiVersion: storage.chalkos.dev/v1alpha1
kind: Volume
metadata: {name: registry}
spec:
disk: {model: "Samsung SSD 870*", type: ssd}
format: xfs
mountPoint: /var/lib/chalkos-registry
The registry starts after the mount, and systemd makes the mounted file system the registry
user's before it starts. Add the volume before you install the node, so the registry stores on it
from the start. A volume added later with chalkctl config apply restarts the registry, which then
stores on the volume. Images pushed before stay hidden on VAR below the mount, so push them again.
The unit requires the mount, as RequiresMountsFor= implies Requires=. When the volume's disk is
missing or its mount fails, the registry does not start on that node, and containerd moves on to
the next endpoint. Add storage volumes covers disks and formats.
Add or remove a control plane¶
Every control plane serves the cluster registry, so a change of the control planes changes
endpoints, and every node needs the new document.
To add a control plane:
- Add its address to
endpoints. Its config is refused until the list names it. - Apply the config to every node already installed.
- Install the new node, as Add a control plane describes. It pulls the control plane's images from the other endpoints while its own registry is empty.
- Push every image again, so the new endpoint holds them, and check with
chalkctl registry list.
To remove a control plane, take it out of etcd and the inventory as
Remove a control plane describes. Then take its
address out of endpoints and apply the config to every remaining node. Until then, containerd
passes over the missing endpoint to the next source.
Turn the cluster registry off¶
Delete the ClusterRegistry document and apply the config to every Kubernetes node. Each node
stops the registry, closes its port and drops the endpoints from containerd's mirrors. The images
stay in /var/lib/chalkos-registry.
An image author leaves the feature out of an image with
chalkos.kubernetes.registry.enable
set to false. Such an image carries no chalkos-registry, no user chalkos-registry, no
ClusterRegistry kind and none of its outputs, effects, firewall rules and diagnostics, and it
refuses a config that holds a ClusterRegistry.
Plan around the limits¶
- Nothing copies images between the endpoints. Each push and delete goes to every endpoint, and
chalkctl registry listshows where they differ. - containerd mirrors per host. Every image of a mirrored host is looked for in the cluster registry first; a host cannot be mirrored for some repositories alone.
- A push that spans a garbage collection may fail and needs to run again.
chalkctl registry listdoes not show manifests without a tag.- An endpoint is a node's static address, which its node certificate names, so a VIP cannot be
one.
chalkctl validatedoes not notice an endpoint that no node owns;chalkctl registry pushfails on it, and containerd passes over it. - chalkd does not pull OS images from the cluster registry. Air-gapped upgrades take the images from a directory, as Upgrade nodes describes.
Check that it worked¶
chalkctl registry list shows all for every image, and exits 0. The diagnostic
cluster-registry shows the repositories one node's registry holds, as its catalog lists them:
$ chalkctl node diag cp1 cluster-registry
{"repositories":["cluster-registry.chalkos.test/e2e/busybox"]}
port=<port> passes a port other than 5000. A pod that runs a pushed image on a node that has no
route to the image's host shows that the node pulled it from the cluster registry.
If something goes wrong¶
- The registry does not run on a node that should serve it. Read its journal with
chalkctl logs <node> --unit chalkos-registry.service. When the last line of the unit isServe the cluster registry skipped, unmet condition check ConditionPathExists=/run/chalkos/outputs/cluster-registry/registry.env, the node does not serve an endpoint: none of its static addresses is inendpoints. A delivery whose restart ofchalkos-registry.servicefails names the unit. A volume mounted at/var/lib/chalkos-registrythat is missing keeps the registry from starting, as Give the registry a disk of its own describes. - The workstation or the nodes cannot reach the port.
chalkctl node diag <node> nft-rulesetshows the tableinet chalkos-registry, which opens it. A network firewall in front of the nodes must let TCP to the port through, as Ports and firewall lists. registry push needs the admin role; the reader role may not: chalkctl authenticated with a reader client file. Use an admin client file or the secrets.the ClusterRegistry mirrors <hosts>, not <host>, so containerd never pulls <reference> from it; add <host> to its mirrors: the reference's host is not mirrored. Add it tomirrorsand apply the config to every node.<reference> names the digest <digest>, but the source's image is <other>; push a source of that digest, such as the image in its registry by that reference, since docker save and rebuilt images get other digests: the source is not the image the reference pins, often a docker archive. Push the upstream image or an OCI layout copied from it.- Pods stay in
ErrImagePullorImagePullBackOff. Check withchalkctl registry listthat the endpoints hold the image under the reference the pod names, then readchalkctl logs <node> --unit containerd.servicefor the URLs containerd tried. Troubleshooting covers the mirrors' units.
What next¶
- Registries and mirrors combines the cluster registry with
MirrorandRegistrydocuments. - How Kubernetes runs on a node explains the registry's TLS, roles and garbage collection.
chalkctl registrylists the commands and their flags.