Registries and mirrors¶
This page is for an operator whose inventory names an image in a registry, as
Publish images to a registry pushes one. At the end each node holds
the credentials of a private registry in its config, and pulls through a mirror before it asks the
registry itself. The examples on this page are for nodes that pull from ghcr.io through a mirror
on the local network. Their documents go in a file such as cluster/registry.yaml, which the
homelab example does not have.
A node pulls its own image at an install and at an upgrade. chalkctl sends it the image's
reference by digest, and chalkd pulls the image with the Registry and Mirror documents of its
config.
Before you begin¶
You need the workstation that Set up your workstation describes, with the secrets
file secrets.age, and an inventory that names an image by source.oci.
chalkctl reads the image's manifest, header and contract on your workstation, with your docker
config. The workstation needs a connection to the registry itself, because chalkctl uses no
Mirror. Each node needs a connection to the registry or to a mirror's endpoint.
Give the nodes the registry's credentials¶
A Registry document gives the credentials for one registry host. Its password comes from a
Secret:
apiVersion: oci.chalkos.dev/v1alpha1
kind: Registry
metadata: {name: ghcr}
spec:
host: ghcr.io
username: example-bot
password: {secretRef: {name: registry, key: token}}
---
apiVersion: chalkos.dev/v1alpha1
kind: Secret
metadata: {name: registry}
spec:
data: {token: <token>}
<token> stands for the registry's access token. Put both documents in a file that every node
gets, such as a file in cluster/.
The config holds one Registry per host. host is the registry's host as the image reference
names it. chalkd matches it in any case and as image references name registries, so docker.io
matches index.docker.io. A Registry gives username and password together, or neither for
anonymous pulls. A public registry that serves the image anonymously needs no Registry.
chalkd reads the credentials from one config:
- At an upgrade, from the config the upgrade delivers. An upgrade without config, such as one with an operator client file, uses the config the node runs.
- At an install, from the config the install delivers, because the installer has no config of its own.
Allow a registry that serves plain HTTP¶
chalkd pulls over HTTPS only and trusts the certificate authorities of the image's system trust
store. A Registry adds no CA of its own. A registry that serves only plain HTTP, such as one in a
lab, needs insecure: true:
apiVersion: oci.chalkos.dev/v1alpha1
kind: Registry
metadata: {name: lab}
spec:
host: registry.lab:5000
insecure: true
chalkd never falls back to plain HTTP by itself, also not for a registry at a private or loopback
address. Without insecure, a request to such a registry over plain HTTP fails with
plain HTTP to <host> is refused: set insecure: true in the Registry of the host to allow it.
With insecure, the credentials and the image travel unencrypted, so anyone on the network path
can read the credentials. The image stays protected, because the node checks every byte against
the digests chalkctl read and against the root hash. Set insecure only for a registry on a
trusted network.
Pull through a mirror¶
A Mirror document puts endpoints in front of one registry host:
apiVersion: oci.chalkos.dev/v1alpha1
kind: Mirror
metadata: {name: ghcr}
spec:
registry: ghcr.io
endpoints: [https://mirror.lan/ghcr]
chalkd tries each endpoint in order, then the registry. On an endpoint it asks for the same
repository and digest below the endpoint's path, as containerd does. The image
ghcr.io/example/node-metal comes from https://mirror.lan/ghcr/v2/example/node-metal/.... An
endpoint whose path ends in /v2 serves the API at that path.
Each endpoint authenticates with the Registry of its own host, here mirror.lan, or
anonymously without one. The credentials of ghcr.io go to ghcr.io alone. An https://
endpoint is reached over HTTPS only. An http:// endpoint needs a Registry of its host with
insecure: true, and chalkctl and chalkd refuse a config without it.
chalkd reads the Mirror documents from the same config as the Registry documents.
Expect a failing endpoint to pass the pull on¶
chalkd passes over an endpoint and asks the next source when the endpoint:
- cannot be reached, refuses the connection or fails the TLS handshake;
- gives no manifest and header within 30 seconds;
- answers 401, 403, 404, 429 or a 5xx status, as a pull-through cache can when it lacks a layer;
- sends no data for two minutes after a layer starts.
The manifest comes from the first source that serves it. Each layer starts at that source and moves on under the same rules. A source that serves bytes other than those of the pinned digest, or a header or layer sizes other than those chalkctl sent, fails the pull at once, without asking the next source.
When the last source fails too, the error names the source it ended at, followed by
after the mirror <endpoint>: <error> for each endpoint it passed over.
Pull container images with the same documents¶
With the Kubernetes layer, containerd pulls the pods' container images through the endpoints of the
same Mirror documents. The same Registry gives every pod on the node the images of its host,
without imagePullSecrets.
Pods pull private images with node credentials
describes how the kubelet gets the credentials.
A cluster without internet access pulls every container image from a mirror, or from the
cluster registry, which must hold them all. Besides the
workloads' images, these are the images the images block of the
Cluster document names: the API server, the controller-manager,
the scheduler, kube-proxy, etcd, CoreDNS, the pod sandbox and, on an image with flannel, flannel.
The mirror serves each under the repository and the tag or digest the reference names.
With the Kubernetes layer, a mirror receives the registry's credentials
The kubelet gives containerd the credentials for the image's registry without the host they belong to. containerd sends them to every endpoint that asks for credentials, and to the token service such an endpoint names. A mirror of an authenticated registry therefore receives that registry's credentials on container pulls. Give its endpoints the trust you give the registry.
Pull from the cluster registry¶
With the Kubernetes layer, a ClusterRegistry document runs a registry on the control planes, and
containerd on every node looks there first for the images of the hosts its mirrors names.
Run an air-gapped cluster with the cluster registry sets it up.
For a mirrored host, containerd tries these sources in turn:
- the node's own endpoint of the cluster registry, when the node serves one;
- the other endpoints of the cluster registry, in the order of
endpoints; - the endpoints of the host's
Mirror, in order; - the host itself, over plain HTTP when its
Registrysetsinsecure.
A host without a Mirror goes from the cluster registry straight to the host. containerd verifies
the cluster registry's endpoints by the OS CA, because they serve the node certificate.
chalkd does not use the cluster registry. It pulls OS images through the Mirror and Registry
documents alone, as the sections above describe.
containerd sends a host's credentials to an endpoint only when the endpoint asks for them. The
cluster registry serves pulls to anyone and never asks, so the credentials of a mirrored host's
Registry do not reach it, and it ignores any that a client sends with a pull. The warning above
still holds for the host's Mirror endpoints.
Check that it worked¶
chalkctl validate checks the Registry and Mirror documents with the rest of each node's
config:
Then upgrade one node, as Upgrade nodes describes:
The node pulls the image with the delivered config. A run that ends with
upgraded 1 node to <new> pulled it. <new> stands for the image's name.
If something goes wrong¶
the mirror <endpoint> is plain HTTP; set insecure: true in the Registry of <host> to allow it: add aRegistryof the endpoint's host withinsecure: true.plain HTTP to <host> is refused: set insecure: true in the Registry of the host to allow it: the registry or an endpoint redirected to plain HTTP.- The registry refuses the pull with 401 or 403: compare the
Registry'shostwith the host as the image reference names it, port included. - With the Kubernetes layer, pods stay in
ErrImagePullorImagePullBackOff. Troubleshooting reads the units that write the credentials and the mirrors.
What next¶
- Publish images to a registry pushes the image and names it in the inventory.
- Ports and firewall lists the connections a node makes.
- Config kinds lists every field of
RegistryandMirror.