How Kubernetes runs on a node¶
chalkos runs upstream Kubernetes in the layout kubeadm uses, without kubeadm, on images with the
Kubernetes layer. The layer is a predefined set of NixOS modules that chalkos ships,
chalkos.nixosModules.kubernetes, which an image lists in its modules like any other module. The
Node document of a node gives its type, and the type decides which components run. The diagram
shows the components of each type of node.
flowchart TB
subgraph cp["Control plane"]
cpchalkd["chalkd"]
cpkubelet["containerd and the kubelet"]
pods["Static pods: etcd, API server, controller-manager, scheduler"]
cpchalkd -->|renders| pods
cpkubelet -->|runs| pods
end
subgraph worker["Worker"]
wkubelet["containerd and the kubelet"]
end
subgraph plain["Node of an image without the Kubernetes layer"]
pchalkd["chalkd, no Kubernetes"]
end
wkubelet -->|registers at the cluster endpoint| pods
addons["Cluster objects: kube-proxy, flannel, CoreDNS"]
cpchalkd -->|applies through the API server| addons
containerd and the kubelet are systemd units of the image. etcd, the API server, the
controller-manager and the scheduler run as static pods from the images the config names. chalkd
renders them on every control plane. Certificates come from the cluster's secrets file instead of
kubeadm init. Control planes join etcd on their own, and chalkd applies the cluster's add-ons at
every boot. The image fixes the kubelet's release. Every other image and every setting of the
cluster and of each node is in a config document. The control plane explains
the static pods, the bootstrap, the joins and the add-ons.
The image fixes the kubelet, the config names the other images¶
Each image fixes the parts of Kubernetes that run on the host: containerd, the CNI plugins and
the kubelet of the release
chalkos.kubernetes.version selects,
the newest one chalkos pins by default. The images block of the
Cluster document names every other image: the API server,
the controller-manager, the scheduler, kube-proxy, etcd, CoreDNS, the pod sandbox and, on an image
with flannel, flannel. containerd pulls them through the Registry and Mirror documents, which
Registries and mirrors describes.
The tags of the API server, the controller-manager, the scheduler and kube-proxy name their Kubernetes version. The image's checker holds these versions to Kubernetes's version skew policy against the image's kubelet, so a config whose control plane is older than the kubelet is refused. Configure Kubernetes lists the rules. An upgrade of the kubelet is an image upgrade, which Kubernetes upgrades describes.
The type of a node decides what runs¶
spec.type of the node's Node document decides what the node is
in Kubernetes, so control planes and workers boot the same image. A controlplane runs
containerd, the kubelet, and etcd and the control plane as static pods. A worker runs containerd
and the kubelet. Images whose modules do not list the
Kubernetes layer, chalkos.nixosModules.kubernetes, lack it.
They run no Kubernetes and have no Node kind, so their nodes have no type. Their chalkd answers
the methods of KubernetesService, the Kubernetes layer's API, such as Bootstrap, with
UNIMPLEMENTED. Components by node type
lists them in one table.
A node keeps its type once it holds its Kubernetes share. chalkd refuses config that changes the type, because the share holds certificates for that type.
The Kubernetes share holds the secrets of a node¶
The Kubernetes share is the set of Kubernetes secrets a node receives at install, which chalkd
keeps on STATE. On a control plane, it holds the cluster, front-proxy and etcd CAs with their keys,
the service-account key, the encryption key and the node CA. On a worker, it holds the cluster CA
certificate and a kubelet client certificate. A node without a Kubernetes share, which is not
installed yet, runs no kubelet. chalkctl config apply --kubernetes-share gives an installed node
a share. Kubernetes certificates describes what the control plane issues from
it.
Four kinds of document describe the cluster¶
The image fixes what it ships: the kubelet's release and the pod network. It records them in
/etc/chalkos/kubernetes/image.json. Everything else is config, in documents of the kinds of the
group kubernetes.chalkos.dev:
- One
Clusterdocument, which is required. It holds the URL of the cluster's API server, the address families and ranges, the DNS domain and address, how nodes pick their addresses, the VIP, scheduling on control planes, the images of the cluster's components and extra flags of the control plane's components. - One
Nodedocument, which is required. It holds the node's type and its own labels, taints, addresses, extra flags of its kubelet and the interface it holds the VIP on. - At most one
Flanneldocument, on images with flannel. It holds the MTU flannel assumes and the ranges its VXLAN must come from. - Any number of
Addondocuments, which hold Kubernetes objects the control planes apply.
Two more kinds of the group configure the node's BGP speaker on images that carry it: one
BGPSpeaker and any number of
BGPPeer documents. They describe the node's
routing, and a node without them runs no BGP.
The cluster's documents go into the files the inventory gives every node, so all nodes hold the
same ones. A node's Node goes into its own file, as
Config and the inventory
describes.
On an image with Kubernetes, the node's config must hold one Cluster and one Node document.
chalkctl validate and chalkd refuse a set without
one, with one of these messages:
no Cluster: the config of a node whose image runs Kubernetes holds one Cluster document
no Node: the config of a node whose image runs Kubernetes holds one Node document
The Cluster document names the cluster endpoint, the URL of the API server, which every node
names alike, and the images of the cluster's components. Every other field has a default. A
cluster on IPv4 with the default ranges needs no more than the lab template's document:
# Kubernetes with its defaults: IPv4, the pod range 10.244.0.0/16 and the service range
# 10.96.0.0/12.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
# cp1 on the lab network: the API server of the only control plane.
endpoint: https://192.168.123.11:6443
# The images of the cluster's components, which containerd pulls as written. The tags of the API
# server, the controller-manager, the scheduler and kube-proxy follow the kubelet of the image the
# flake builds, the newest Kubernetes release chalkos pins; the image's checker holds them to
# Kubernetes's version skew policy. The digests pin what containerd pulls. Read the digest of
# another reference with `crane digest <ref>` or
# `skopeo inspect --format '{{.Digest}}' docker://<ref>`.
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
chalkctl delivers the documents at install and with
chalkctl config apply. At boot and on every
apply, chalkd builds the node's cluster from its Node, Cluster and Flannel documents and
image.json. It writes the result to /run/chalkos/kubernetes/cluster.json. The file carries the
release of the image's kubelet as version and the Kubernetes version of images.apiServer's tag
as apiServerVersion, such as 1.37.2 for both; the image's rules read them as
facts.cluster.version and facts.cluster.apiServerVersion.
Every node holds the same cluster documents¶
Every node of an inventory whose set holds a Cluster document must hold the same Cluster spec,
the same Flannel spec, or no Flannel document on any of them, and likewise the same
ClusterRegistry spec or none, because each node builds the cluster from its own documents. The
specs compare as written, whatever the documents are called: a
field left to its default differs from one set to the default value. Only the order of the
Flannel document's vxlanSourceSubnets makes no difference. chalkctl validate and every
chalkctl command that reads the inventory refuse an inventory whose nodes differ. The error names
the first field that differs, in the order of the fields' names, and the nodes of each of its
values:
chalkos.yaml: the nodes hold different Flannel specs: spec.mtu is 1480 on node1 and node2, 1450 on node3; every node of the cluster holds the same Cluster, Flannel and ClusterRegistry specs
A node without the Flannel document is named as holding none, as in none on node3. A node whose
files chalkctl cannot read, such as one whose files hold a document twice, is left out of the
comparison, and every command that reads its config refuses it. chalkctl kubeconfig points
clients at the endpoint of the nodes' Cluster documents, so it refuses an inventory without
nodes.
So the cluster's documents belong in the files every node gets. What differs from node to
node goes into each node's Node document instead, which chalkctl does not compare:
type,controlplaneorworker;labelsandtaints;nodeIP.addressesandnodeIP.validSubnets, the node's own addresses or the subnets it picks them from;kubeletExtraArgs, the extra flags of the node's kubelet;vip.interface, the interface a control plane adds the VIP to, which a worker leaves null.
The ranges of a cluster are fixed once it exists¶
A changed cluster restarts chalkos-kubernetes and the kubelet, as their
effect says. chalkd on a control plane then
starts its Kubernetes work again, so the static pods and add-ons follow the new settings.
Some settings are fixed once the cluster exists: ipFamilies, the pod ranges of podCIDRs, the
service ranges of serviceCIDRs and the addresses of dnsIPs. The certificates, etcd and the
ranges the control plane allocated were made for them. On a node that holds a Kubernetes share,
chalkctl config apply is refused when it changes them:
the config changes the cluster's podCIDRs from ipv4 10.244.0.0/16 to ipv4 10.245.0.0/16; they are fixed once the cluster exists, so keep them, or reinstall the node into a new cluster
The change rules of the Cluster kind decide this.
chalkos-kubernetes prepares a node before its kubelet starts¶
containerd and the kubelet run as systemd units from the image. Before the kubelet starts,
chalkos-kubernetes.service prepares the node. It picks the node's addresses, as
Kubernetes networking describes, and issues a control plane's certificates. It
writes the kubelet's kubeconfig and per-node flags below /run/chalkos/kubernetes.
chalkd renders the kubelet's configuration from the node's cluster into
/run/chalkos/kubernetes/kubelet/config.json. The configuration is the same on every node of the
cluster:
- webhook authentication and authorisation;
- no anonymous access and no read-only port;
- the systemd cgroup driver;
- the cluster's DNS address and domain;
rotateCertificatesandserverTLSBootstrap.
The kubelet gets what differs per node as flags in $KUBELET_ARGS. These are the node name, which
is the node's name in the inventory,
and the addresses, labels, taints and extra flags of its Node document. This worker carries fast
storage and keeps other workloads off:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
type: worker
labels: {node.kubernetes.io/storage: ssd}
taints: [{key: dedicated, value: storage, effect: NoSchedule}]
A control plane is tainted node-role.kubernetes.io/control-plane:NoSchedule unless the Cluster
document sets allowSchedulingOnControlPlanes: true. The kubelet applies the labels and taints,
this taint among them, only when it registers the node. So labels, taints and
allowSchedulingOnControlPlanes are fixed once the node holds a Kubernetes share, and kubectl
changes them on the Kubernetes node. kubeletExtraArgs of a node's Node document adds flags to
its kubelet. Protected kubelet flags lists
the flags chalkos refuses there.
containerd pulls through the mirrors of the node¶
containerd pulls through the endpoints of the Mirror documents. It tries them in order before
the registry itself. Each document names one registry, as image references name it, in
spec.registry:
apiVersion: oci.chalkos.dev/v1alpha1
kind: Mirror
metadata: {name: docker.io}
spec:
registry: docker.io
endpoints: [https://mirror.example.com]
chalkos-registry-mirrors.service writes containerd's
/run/containerd/certs.d/<registry>/hosts.toml from the Mirror and
Registry documents. It runs before containerd starts and
whenever the documents change. containerd reads those files on every pull. Each file lists the
endpoints as the Mirror names them, then the registry itself. A host that a ClusterRegistry
mirrors gets the cluster registry's endpoints before them, as
the next section describes.
A registry whose Registry sets insecure is reached over HTTP. It gets a file even without a
Mirror. An http:// endpoint needs a Registry of its host with insecure: true, because
nothing authenticates what a plain HTTP endpoint serves.
The cluster registry serves images from the control planes¶
A ClusterRegistry document runs a registry on the
control planes, for clusters that cannot reach the registries their images come from. A node
whose static address is 192.0.2.12, with a ClusterRegistry of the endpoints fd00::11 and
192.0.2.12 that mirrors docker.io, and a Mirror of docker.io at https://mirror.example,
gets this docker.io/hosts.toml:
server = "https://registry-1.docker.io"
[host."https://192.0.2.12:5000/v2/docker.io"]
capabilities = ["pull", "resolve"]
override_path = true
ca = "/state/chalkd/ca.crt"
[host."https://[fd00::11]:5000/v2/docker.io"]
capabilities = ["pull", "resolve"]
override_path = true
ca = "/state/chalkd/ca.crt"
[host."https://mirror.example"]
capabilities = ["pull", "resolve"]
The node serves 192.0.2.12, so that endpoint comes first, then the others in the order of
endpoints, then the Mirror's, then the host. containerd verifies the cluster registry by the OS
CAs in STATE's chalkd/ca.crt. A node serves the cluster registry when one of its static
addresses is an endpoint, and the image's rule on endpoints refuses a control plane that does
not.
chalkos-registry is the registry: distribution's registry handlers with its filesystem storage,
in a binary of chalkos's own. It runs as chalkos-registry.service under its own user, ID 333,
without capabilities, and keeps the images in /var/lib/chalkos-registry. The output
cluster-registry renders its port and firewall table on serving nodes alone, and the unit does
not start without them. The unit is ready once the registry listens, and the kubernetes effect
runs after the registry's effects, so a kubelet restarted by the same delivery finds its node's
registry answering. At boot the registry starts before the kubelet. The unit requires the mount
of its directory, and its effect also restarts it when a Volume mounted at or above
/var/lib/chalkos-registry changes, since systemd gives a new file system to the registry's user
only as the unit starts.
The registry holds each image below its host's name: docker.io/library/nginx lives at
<endpoint>:5000/docker.io/library/nginx. containerd's override_path makes the endpoint's path,
/v2/docker.io, the root of the registry API it asks, so one registry serves every mirrored
host.
Kubernetes certificates
describes who may read and write. Nothing copies images between the endpoints:
chalkctl registry push and chalkctl registry delete act on each endpoint in turn.
Nothing deletes images on its own. A tag deleted leaves its manifest. A manifest deleted leaves its layers until an admin asks for garbage collection, which runs distribution's mark and sweep and keeps untagged manifests. While it runs, the registry serves reads and refuses writes with 503, so no upload lands in the middle of a sweep. A push whose layers went up before the collection and whose manifest comes after it fails, and is pushed again. A second collection asked for while one runs gets 409. A collection runs to its end even when its client goes away.
Pods pull private images with node credentials¶
A Registry with username and password gives every pod on
the node the images of its host, without imagePullSecrets:
apiVersion: oci.chalkos.dev/v1alpha1
kind: Registry
metadata: {name: ghcr}
spec:
host: ghcr.io
username: robot
password: {secretRef: {name: registry, key: token}}
The kubelet runs chalkd as its image credential provider. It asks chalkd for the credentials of an image's host, and hands them to containerd as it hands over those of a pull secret.
The kubelet matches the host, in lower case and port included, against the image's host as the
pod spec writes it. It names Docker Hub docker.io. The comparison is case-sensitive, so
GHCR.io/org/app gets no credentials. An IPv6 host needs its port, 443 included, in both the
Registry and the image. The kubelet reads a bracketed address without a port as a glob character
class, so chalkd refuses such a Registry with credentials.
chalkos-registry-credentials.service writes the provider's configuration and the credentials to
/run/chalkos/kubernetes/credential-provider/. Only root reads its credentials.json. A node
without a Registry that has credentials runs the kubelet without a credential provider. When the
unit fails, the kubelet still starts and pulls without the node's credentials.
The kubelet reads the provider's configuration when it starts, and keeps each answer for 5
minutes. So a change of a Registry, or of a password it references, restarts both
chalkos-registry-credentials.service and kubelet.service. The restart empties that cache, and
running pods keep running.
A mirror receives the credentials of its registry¶
containerd sends the credentials the kubelet gives it for the image's registry to every endpoint of
a Mirror that asks for credentials, and to the token service such an endpoint names. This holds
for credentials from the node's Registry and from an imagePullSecret alike. The kubelet passes
them without the host they belong to, so containerd 2.4 cannot tell a mirror from the registry. In
other words, a mirror of an authenticated registry receives that registry's credentials. An
http:// endpoint, allowed by an insecure Registry of its host, receives them over plain HTTP.
The credentials of an endpoint's own Registry do not reach containerd.
chalkd pulls OS images through the same Mirror documents. It authenticates each endpoint with the
Registry of the endpoint's host instead, and sends each host only its own credentials, as
Pull through a mirror describes. The
e2e-kubernetes check pins this behaviour. Its
worker pulls a private image through a mirror endpoint that accepts only the credentials of the
image's registry.
Limits¶
- A node keeps its type, labels and taints once it holds a Kubernetes share. A new type needs a reinstall, and kubectl changes labels and taints.
- Node credentials hold for every pod on the node, in every namespace. Any pod scheduled there can
pull every image those credentials read. Images that only some workloads may read need an
imagePullSecretin their namespace. - A mirror of an authenticated registry receives that registry's credentials on container pulls, so it needs the trust the registry has.
- Image hosts in a pod spec match a
Registryonly in lower case. - The endpoints of the cluster registry do not copy images to each other. An endpoint that missed a push lacks the image until it is pushed again.
- containerd mirrors whole hosts, so the cluster registry cannot mirror some repositories of a host alone.
- An endpoint of the cluster registry is a node's static address, which its node certificate names, so a VIP cannot be one.
- Anyone who reaches the cluster registry's port can pull every image it holds.
Related pages¶
- The control plane: the static pods, the bootstrap, the joins and the add-ons.
- Run an air-gapped cluster with the cluster registry.
- Kubernetes networking, Kubernetes certificates and Kubernetes upgrades.
- Config and the inventory: how the Kubernetes documents reach the nodes.
- Configure Kubernetes and Registries and mirrors for the tasks.
- Config kinds for every field of
Cluster,Node,FlannelandAddon, and Kubernetes for the components, flags, add-ons and status states.