Skip to content

How Kubernetes runs on a node

chalkos runs upstream Kubernetes in the layout kubeadm uses, without kubeadm, on images with the Kubernetes layer. The layer is a predefined set of NixOS modules that chalkos ships, chalkos.nixosModules.kubernetes, which an image lists in its modules like any other module. The Node document of a node gives its type, and the type decides which components run. The diagram shows the components of each type of node.

flowchart TB
    subgraph cp["Control plane"]
        cpchalkd["chalkd"]
        cpkubelet["containerd and the kubelet"]
        pods["Static pods: etcd, API server, controller-manager, scheduler"]
        cpchalkd -->|renders| pods
        cpkubelet -->|runs| pods
    end
    subgraph worker["Worker"]
        wkubelet["containerd and the kubelet"]
    end
    subgraph plain["Node of an image without the Kubernetes layer"]
        pchalkd["chalkd, no Kubernetes"]
    end
    wkubelet -->|registers at the cluster endpoint| pods
    addons["Cluster objects: kube-proxy, flannel, CoreDNS"]
    cpchalkd -->|applies through the API server| addons

containerd and the kubelet are systemd units of the image. etcd, the API server, the controller-manager and the scheduler run as static pods from the images the config names. chalkd renders them on every control plane. Certificates come from the cluster's secrets file instead of kubeadm init. Control planes join etcd on their own, and chalkd applies the cluster's add-ons at every boot. The image fixes the kubelet's release. Every other image and every setting of the cluster and of each node is in a config document. The control plane explains the static pods, the bootstrap, the joins and the add-ons.

The image fixes the kubelet, the config names the other images

Each image fixes the parts of Kubernetes that run on the host: containerd, the CNI plugins and the kubelet of the release chalkos.kubernetes.version selects, the newest one chalkos pins by default. The images block of the Cluster document names every other image: the API server, the controller-manager, the scheduler, kube-proxy, etcd, CoreDNS, the pod sandbox and, on an image with flannel, flannel. containerd pulls them through the Registry and Mirror documents, which Registries and mirrors describes.

The tags of the API server, the controller-manager, the scheduler and kube-proxy name their Kubernetes version. The image's checker holds these versions to Kubernetes's version skew policy against the image's kubelet, so a config whose control plane is older than the kubelet is refused. Configure Kubernetes lists the rules. An upgrade of the kubelet is an image upgrade, which Kubernetes upgrades describes.

The type of a node decides what runs

spec.type of the node's Node document decides what the node is in Kubernetes, so control planes and workers boot the same image. A controlplane runs containerd, the kubelet, and etcd and the control plane as static pods. A worker runs containerd and the kubelet. Images whose modules do not list the Kubernetes layer, chalkos.nixosModules.kubernetes, lack it. They run no Kubernetes and have no Node kind, so their nodes have no type. Their chalkd answers the methods of KubernetesService, the Kubernetes layer's API, such as Bootstrap, with UNIMPLEMENTED. Components by node type lists them in one table.

A node keeps its type once it holds its Kubernetes share. chalkd refuses config that changes the type, because the share holds certificates for that type.

The Kubernetes share holds the secrets of a node

The Kubernetes share is the set of Kubernetes secrets a node receives at install, which chalkd keeps on STATE. On a control plane, it holds the cluster, front-proxy and etcd CAs with their keys, the service-account key, the encryption key and the node CA. On a worker, it holds the cluster CA certificate and a kubelet client certificate. A node without a Kubernetes share, which is not installed yet, runs no kubelet. chalkctl config apply --kubernetes-share gives an installed node a share. Kubernetes certificates describes what the control plane issues from it.

Four kinds of document describe the cluster

The image fixes what it ships: the kubelet's release and the pod network. It records them in /etc/chalkos/kubernetes/image.json. Everything else is config, in documents of the kinds of the group kubernetes.chalkos.dev:

  • One Cluster document, which is required. It holds the URL of the cluster's API server, the address families and ranges, the DNS domain and address, how nodes pick their addresses, the VIP, scheduling on control planes, the images of the cluster's components and extra flags of the control plane's components.
  • One Node document, which is required. It holds the node's type and its own labels, taints, addresses, extra flags of its kubelet and the interface it holds the VIP on.
  • At most one Flannel document, on images with flannel. It holds the MTU flannel assumes and the ranges its VXLAN must come from.
  • Any number of Addon documents, which hold Kubernetes objects the control planes apply.

Two more kinds of the group configure the node's BGP speaker on images that carry it: one BGPSpeaker and any number of BGPPeer documents. They describe the node's routing, and a node without them runs no BGP.

The cluster's documents go into the files the inventory gives every node, so all nodes hold the same ones. A node's Node goes into its own file, as Config and the inventory describes.

On an image with Kubernetes, the node's config must hold one Cluster and one Node document. chalkctl validate and chalkd refuse a set without one, with one of these messages:

no Cluster: the config of a node whose image runs Kubernetes holds one Cluster document
no Node: the config of a node whose image runs Kubernetes holds one Node document

The Cluster document names the cluster endpoint, the URL of the API server, which every node names alike, and the images of the cluster's components. Every other field has a default. A cluster on IPv4 with the default ranges needs no more than the lab template's document:

# Kubernetes with its defaults: IPv4, the pod range 10.244.0.0/16 and the service range
# 10.96.0.0/12.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
  # cp1 on the lab network: the API server of the only control plane.
  endpoint: https://192.168.123.11:6443
  # The images of the cluster's components, which containerd pulls as written. The tags of the API
  # server, the controller-manager, the scheduler and kube-proxy follow the kubelet of the image the
  # flake builds, the newest Kubernetes release chalkos pins; the image's checker holds them to
  # Kubernetes's version skew policy. The digests pin what containerd pulls. Read the digest of
  # another reference with `crane digest <ref>` or
  # `skopeo inspect --format '{{.Digest}}' docker://<ref>`.
  images:
    apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
    controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
    scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
    proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
    etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
    coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
    pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
    flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204

chalkctl delivers the documents at install and with chalkctl config apply. At boot and on every apply, chalkd builds the node's cluster from its Node, Cluster and Flannel documents and image.json. It writes the result to /run/chalkos/kubernetes/cluster.json. The file carries the release of the image's kubelet as version and the Kubernetes version of images.apiServer's tag as apiServerVersion, such as 1.37.2 for both; the image's rules read them as facts.cluster.version and facts.cluster.apiServerVersion.

Every node holds the same cluster documents

Every node of an inventory whose set holds a Cluster document must hold the same Cluster spec, the same Flannel spec, or no Flannel document on any of them, and likewise the same ClusterRegistry spec or none, because each node builds the cluster from its own documents. The specs compare as written, whatever the documents are called: a field left to its default differs from one set to the default value. Only the order of the Flannel document's vxlanSourceSubnets makes no difference. chalkctl validate and every chalkctl command that reads the inventory refuse an inventory whose nodes differ. The error names the first field that differs, in the order of the fields' names, and the nodes of each of its values:

chalkos.yaml: the nodes hold different Flannel specs: spec.mtu is 1480 on node1 and node2, 1450 on node3; every node of the cluster holds the same Cluster, Flannel and ClusterRegistry specs

A node without the Flannel document is named as holding none, as in none on node3. A node whose files chalkctl cannot read, such as one whose files hold a document twice, is left out of the comparison, and every command that reads its config refuses it. chalkctl kubeconfig points clients at the endpoint of the nodes' Cluster documents, so it refuses an inventory without nodes.

So the cluster's documents belong in the files every node gets. What differs from node to node goes into each node's Node document instead, which chalkctl does not compare:

  • type, controlplane or worker;
  • labels and taints;
  • nodeIP.addresses and nodeIP.validSubnets, the node's own addresses or the subnets it picks them from;
  • kubeletExtraArgs, the extra flags of the node's kubelet;
  • vip.interface, the interface a control plane adds the VIP to, which a worker leaves null.

The ranges of a cluster are fixed once it exists

A changed cluster restarts chalkos-kubernetes and the kubelet, as their effect says. chalkd on a control plane then starts its Kubernetes work again, so the static pods and add-ons follow the new settings.

Some settings are fixed once the cluster exists: ipFamilies, the pod ranges of podCIDRs, the service ranges of serviceCIDRs and the addresses of dnsIPs. The certificates, etcd and the ranges the control plane allocated were made for them. On a node that holds a Kubernetes share, chalkctl config apply is refused when it changes them:

the config changes the cluster's podCIDRs from ipv4 10.244.0.0/16 to ipv4 10.245.0.0/16; they are fixed once the cluster exists, so keep them, or reinstall the node into a new cluster

The change rules of the Cluster kind decide this.

chalkos-kubernetes prepares a node before its kubelet starts

containerd and the kubelet run as systemd units from the image. Before the kubelet starts, chalkos-kubernetes.service prepares the node. It picks the node's addresses, as Kubernetes networking describes, and issues a control plane's certificates. It writes the kubelet's kubeconfig and per-node flags below /run/chalkos/kubernetes.

chalkd renders the kubelet's configuration from the node's cluster into /run/chalkos/kubernetes/kubelet/config.json. The configuration is the same on every node of the cluster:

  • webhook authentication and authorisation;
  • no anonymous access and no read-only port;
  • the systemd cgroup driver;
  • the cluster's DNS address and domain;
  • rotateCertificates and serverTLSBootstrap.

The kubelet gets what differs per node as flags in $KUBELET_ARGS. These are the node name, which is the node's name in the inventory, and the addresses, labels, taints and extra flags of its Node document. This worker carries fast storage and keeps other workloads off:

apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
  type: worker
  labels: {node.kubernetes.io/storage: ssd}
  taints: [{key: dedicated, value: storage, effect: NoSchedule}]

A control plane is tainted node-role.kubernetes.io/control-plane:NoSchedule unless the Cluster document sets allowSchedulingOnControlPlanes: true. The kubelet applies the labels and taints, this taint among them, only when it registers the node. So labels, taints and allowSchedulingOnControlPlanes are fixed once the node holds a Kubernetes share, and kubectl changes them on the Kubernetes node. kubeletExtraArgs of a node's Node document adds flags to its kubelet. Protected kubelet flags lists the flags chalkos refuses there.

containerd pulls through the mirrors of the node

containerd pulls through the endpoints of the Mirror documents. It tries them in order before the registry itself. Each document names one registry, as image references name it, in spec.registry:

apiVersion: oci.chalkos.dev/v1alpha1
kind: Mirror
metadata: {name: docker.io}
spec:
  registry: docker.io
  endpoints: [https://mirror.example.com]

chalkos-registry-mirrors.service writes containerd's /run/containerd/certs.d/<registry>/hosts.toml from the Mirror and Registry documents. It runs before containerd starts and whenever the documents change. containerd reads those files on every pull. Each file lists the endpoints as the Mirror names them, then the registry itself. A host that a ClusterRegistry mirrors gets the cluster registry's endpoints before them, as the next section describes.

A registry whose Registry sets insecure is reached over HTTP. It gets a file even without a Mirror. An http:// endpoint needs a Registry of its host with insecure: true, because nothing authenticates what a plain HTTP endpoint serves.

The cluster registry serves images from the control planes

A ClusterRegistry document runs a registry on the control planes, for clusters that cannot reach the registries their images come from. A node whose static address is 192.0.2.12, with a ClusterRegistry of the endpoints fd00::11 and 192.0.2.12 that mirrors docker.io, and a Mirror of docker.io at https://mirror.example, gets this docker.io/hosts.toml:

server = "https://registry-1.docker.io"

[host."https://192.0.2.12:5000/v2/docker.io"]
  capabilities = ["pull", "resolve"]
  override_path = true
  ca = "/state/chalkd/ca.crt"

[host."https://[fd00::11]:5000/v2/docker.io"]
  capabilities = ["pull", "resolve"]
  override_path = true
  ca = "/state/chalkd/ca.crt"

[host."https://mirror.example"]
  capabilities = ["pull", "resolve"]

The node serves 192.0.2.12, so that endpoint comes first, then the others in the order of endpoints, then the Mirror's, then the host. containerd verifies the cluster registry by the OS CAs in STATE's chalkd/ca.crt. A node serves the cluster registry when one of its static addresses is an endpoint, and the image's rule on endpoints refuses a control plane that does not.

chalkos-registry is the registry: distribution's registry handlers with its filesystem storage, in a binary of chalkos's own. It runs as chalkos-registry.service under its own user, ID 333, without capabilities, and keeps the images in /var/lib/chalkos-registry. The output cluster-registry renders its port and firewall table on serving nodes alone, and the unit does not start without them. The unit is ready once the registry listens, and the kubernetes effect runs after the registry's effects, so a kubelet restarted by the same delivery finds its node's registry answering. At boot the registry starts before the kubelet. The unit requires the mount of its directory, and its effect also restarts it when a Volume mounted at or above /var/lib/chalkos-registry changes, since systemd gives a new file system to the registry's user only as the unit starts.

The registry holds each image below its host's name: docker.io/library/nginx lives at <endpoint>:5000/docker.io/library/nginx. containerd's override_path makes the endpoint's path, /v2/docker.io, the root of the registry API it asks, so one registry serves every mirrored host. Kubernetes certificates describes who may read and write. Nothing copies images between the endpoints: chalkctl registry push and chalkctl registry delete act on each endpoint in turn.

Nothing deletes images on its own. A tag deleted leaves its manifest. A manifest deleted leaves its layers until an admin asks for garbage collection, which runs distribution's mark and sweep and keeps untagged manifests. While it runs, the registry serves reads and refuses writes with 503, so no upload lands in the middle of a sweep. A push whose layers went up before the collection and whose manifest comes after it fails, and is pushed again. A second collection asked for while one runs gets 409. A collection runs to its end even when its client goes away.

Pods pull private images with node credentials

A Registry with username and password gives every pod on the node the images of its host, without imagePullSecrets:

apiVersion: oci.chalkos.dev/v1alpha1
kind: Registry
metadata: {name: ghcr}
spec:
  host: ghcr.io
  username: robot
  password: {secretRef: {name: registry, key: token}}

The kubelet runs chalkd as its image credential provider. It asks chalkd for the credentials of an image's host, and hands them to containerd as it hands over those of a pull secret.

The kubelet matches the host, in lower case and port included, against the image's host as the pod spec writes it. It names Docker Hub docker.io. The comparison is case-sensitive, so GHCR.io/org/app gets no credentials. An IPv6 host needs its port, 443 included, in both the Registry and the image. The kubelet reads a bracketed address without a port as a glob character class, so chalkd refuses such a Registry with credentials.

chalkos-registry-credentials.service writes the provider's configuration and the credentials to /run/chalkos/kubernetes/credential-provider/. Only root reads its credentials.json. A node without a Registry that has credentials runs the kubelet without a credential provider. When the unit fails, the kubelet still starts and pulls without the node's credentials.

The kubelet reads the provider's configuration when it starts, and keeps each answer for 5 minutes. So a change of a Registry, or of a password it references, restarts both chalkos-registry-credentials.service and kubelet.service. The restart empties that cache, and running pods keep running.

A mirror receives the credentials of its registry

containerd sends the credentials the kubelet gives it for the image's registry to every endpoint of a Mirror that asks for credentials, and to the token service such an endpoint names. This holds for credentials from the node's Registry and from an imagePullSecret alike. The kubelet passes them without the host they belong to, so containerd 2.4 cannot tell a mirror from the registry. In other words, a mirror of an authenticated registry receives that registry's credentials. An http:// endpoint, allowed by an insecure Registry of its host, receives them over plain HTTP. The credentials of an endpoint's own Registry do not reach containerd.

chalkd pulls OS images through the same Mirror documents. It authenticates each endpoint with the Registry of the endpoint's host instead, and sends each host only its own credentials, as Pull through a mirror describes. The e2e-kubernetes check pins this behaviour. Its worker pulls a private image through a mirror endpoint that accepts only the credentials of the image's registry.

Limits

  • A node keeps its type, labels and taints once it holds a Kubernetes share. A new type needs a reinstall, and kubectl changes labels and taints.
  • Node credentials hold for every pod on the node, in every namespace. Any pod scheduled there can pull every image those credentials read. Images that only some workloads may read need an imagePullSecret in their namespace.
  • A mirror of an authenticated registry receives that registry's credentials on container pulls, so it needs the trust the registry has.
  • Image hosts in a pod spec match a Registry only in lower case.
  • The endpoints of the cluster registry do not copy images to each other. An endpoint that missed a push lacks the image until it is pushed again.
  • containerd mirrors whole hosts, so the cluster registry cannot mirror some repositories of a host alone.
  • An endpoint of the cluster registry is a node's static address, which its node certificate names, so a VIP cannot be one.
  • Anyone who reaches the cluster registry's port can pull every image it holds.