Configure Kubernetes¶
This page is for an operator who sets how Kubernetes runs on the cluster's nodes. Four kinds of
documents hold the settings: Cluster for what every node shares, Node for one node, Flannel
for the pod network and Addon for objects the control planes apply. At the end the nodes run
with your settings, delivered with the rest of their config. The page builds on the homelab, whose
cluster/cluster.yaml sets the endpoint and the images:
# Kubernetes on IPv6 alone, with the pod range fd00:10:244::/56 and the service range
# fd00:10:96::/112, its defaults. All three nodes are control planes and run workloads too.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
ipFamilies: [ipv6]
allowSchedulingOnControlPlanes: true
# The API server's VIP, which one control plane holds at a time on its interface node0. FRR on
# that node announces it with the node's address, so the VIP's route follows its holder.
endpoint: "https://[fdc8:a1b2:c3d4:ff::10]:6443"
vip: {addresses: ["fdc8:a1b2:c3d4:ff::10"], mode: routed}
# The images of the cluster's components, which containerd pulls as written. The tags of the API
# server, the controller-manager, the scheduler and kube-proxy follow the kubelet of the image the
# flake builds, the newest Kubernetes release chalkos pins; the image's checker holds them to
# Kubernetes's version skew policy. The digests pin what containerd pulls. Read the digest of
# another reference with `crane digest <ref>` or
# `skopeo inspect --format '{{.Digest}}' docker://<ref>`. The documentation includes the
# section the comments --8<-- mark.
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
---
# flannel would take the MTU of node0, the dummy interface that holds the node's address, which
# says nothing of the fabric's. Its VXLAN devices and the pods get spec.mtu less 50 bytes, and
# VXLAN over IPv6 takes 70, so spec.mtu is the fabric's 1500 less 20.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Flannel
metadata: {name: default}
spec:
mtu: 1480
Every other field takes its default. Config kinds lists every field of the four kinds.
The images of the Cluster document must suit the kubelet that the node's image carries. The image
directory names that kubelet's release in facts/kubernetes.json:
On a running node, chalkctl status shows it in the line kubernetes versions: API server 1.37.2,
kubelet 1.37.2. The tags of apiServer, controllerManager, scheduler and proxy must stay
within the skew policy of that release. The lab
template's
cluster/cluster.yaml
holds a set of references, pinned by digest, that the template-lab check validates against the
template's image. Start from it when the image comes from the same chalkos revision.
Before you begin¶
You need the workstation that Set up your workstation describes, and
the decisions of Plan a cluster. Changes reach a running node with
chalkctl config apply, as Change a node's config describes, which
needs the secrets file.
Write the Cluster document¶
Put the Cluster document into the files every node gets. chalkctl refuses an inventory whose
nodes hold different Cluster or Flannel specs, and a node whose config holds no Cluster
document. Its fields fall into groups:
| Fields | Set | Default |
|---|---|---|
endpoint |
The URL of the API server, the cluster endpoint | required |
images |
The images of the control plane, kube-proxy, CoreDNS, the pod sandbox and flannel | required |
domain |
The cluster's DNS domain | cluster.local |
ipFamilies, podCIDRs, serviceCIDRs, dnsIPs |
The address families, their pod and service ranges and the cluster DNS address of each; fixed once the cluster exists | IPv4, 10.244.0.0/16, 10.96.0.0/12, the 10th address of the service range |
nodeCIDRMaskSizes |
The prefix length of each node's part of the pod range | 24 for IPv4, 64 for IPv6 |
nodeIP |
The subnets nodes without a fixed address pick theirs from, and how long a node waits for it at boot | any global unicast address, 300 seconds |
vip |
The VIPs the control planes hold | none |
allowSchedulingOnControlPlanes |
Whether workloads may run on control planes; fixed once the cluster exists | false |
extraArgs |
Extra flags of etcd, the API server, the controller-manager and the scheduler | none |
Plan a cluster covers the endpoint, the VIP and the address families. The sections below cover the images, scheduling and the extra flags.
Name the images of the components¶
The node's image fixes the kubelet, containerd and the CNI plugins. The images block of the
Cluster document names every other image the cluster runs, so a control-plane release changes
with the config. Every field is required:
| Field | Image | Tag |
|---|---|---|
apiServer |
kube-apiserver |
a Kubernetes version |
controllerManager |
kube-controller-manager |
a Kubernetes version |
scheduler |
kube-scheduler |
a Kubernetes version |
proxy |
kube-proxy |
a Kubernetes version |
etcd |
etcd, which control planes run | any |
coredns |
CoreDNS, the cluster DNS | any |
pause |
the pod sandbox, which containerd runs for every pod | any |
flannel |
flannel, on an image whose CNI is flannel; another image refuses it | any |
Each field is one image reference, used as written. A reference names its registry, its
repository, and a tag, a digest or both, such as
registry.k8s.io/kube-apiserver:v1.37.2@sha256:<digest>. chalkos adds no default registry, so
kube-apiserver:v1.37.2 is refused. With a digest, containerd pulls the image by its digest, and
the tag labels it. This Cluster of a single control plane at 192.0.2.11 takes the images block
of the homelab's, which pins every image this way:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
endpoint: https://192.0.2.11:6443
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
crane digest <ref> or skopeo inspect --format '{{.Digest}}' docker://<ref> prints the digest of
a reference. Nothing offline checks that a tag and a digest agree.
A reference with a tag and no digest is valid, but chalkctl validate, config apply, install
and upgrade warn of it, because each node pulls whatever the tag points to when it pulls, so two
nodes may run different images under one reference:
chalkctl: warning: node1: Cluster "default": images.proxy: registry.k8s.io/kube-proxy:v1.37.2 names a tag but no digest, so nodes pull whatever the tag points to when they pull; pin it as registry.k8s.io/kube-proxy:v1.37.2@sha256:<digest>
containerd pulls the images when the static pods, the add-ons and the pods need them, through the
Registry and Mirror documents. A cluster without internet access serves them from a mirror, as
Registries and mirrors
describes, or from its control planes, as
Run an air-gapped cluster with the cluster registry describes.
Keep the versions within the skew policy¶
The tags of apiServer, controllerManager, scheduler and proxy name their Kubernetes
version, v<major>.<minor>.<patch> with an optional suffix such as -rc.1. The image's checker
compares them with each other and with the release of the image's kubelet, by the
Kubernetes version skew policy. It compares
minor versions alone, so patch releases may differ:
| Image | Allowed versions |
|---|---|
apiServer |
no older than the kubelet, at most three minors newer |
controllerManager, scheduler |
no newer than apiServer, at most one minor older |
proxy |
no newer than apiServer, at most three minors older than it, and at most three minors older or newer than the kubelet |
Each image's checker checks the config against its own kubelet, so a config delivered while a new image is staged must suit the staged image's kubelet too. The refusals name the field:
Cluster "default": images.apiServer: v1.36.0 is older than the image's kubelet 1.37; a kubelet must not be newer than the API server, so upgrade the control plane first
Cluster "default": images.apiServer: v1.41.0 is 4 minors newer than the image's kubelet 1.37; a kubelet may be at most three minors older than the API server, so upgrade the nodes' images first
Cluster "default": images.scheduler: v1.38.0 is newer than images.apiServer v1.37.2; it must not be newer than the API server
Cluster "default": images.proxy: v1.36.0 is 4 minors older than images.apiServer v1.40.0; kube-proxy may be at most three minors older
The checker sees the config and its own image alone. chalkctl config apply also checks the API
server against the versions every node of the cluster runs, and chalkctl upgrade checks a new
image's kubelet against the oldest API server, as
Upgrade Kubernetes describes.
A reference of these four fields without a version tag, such as one by digest alone, fails the schema:
'registry.k8s.io/kube-apiserver@sha256:<digest>' is not valid versioned-image-reference: this image names its version in its tag, as registry.example/app:v1.2.3, optionally followed by @sha256:<digest>; a reference by digest alone or with another tag has no version
A change of any image restarts the kubelet and the preparation, which writes the static pods, and
chalkd applies the add-ons again. A change of pause also restarts containerd, whose running
containers keep running; pods started after it get the new sandbox.
Give a node its type, labels and taints¶
Each node's config holds one Node document, usually in the node's own file. A worker node4,
which the homelab could add as Add and remove nodes does, here
serves storage: its label attracts the storage system's replicas, and its taint keeps other
workloads off.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
type: worker
labels: {node.kubernetes.io/storage: ssd}
taints: [{key: dedicated, value: storage, effect: NoSchedule}]
type is controlplane or worker, and it is fixed once the node holds a Kubernetes share. A
taint's effect is NoSchedule, PreferNoSchedule or NoExecute, and its value may be left
out.
The kubelet sets the labels on its own node, so the labels it may set limit yours. In the
kubernetes.io and k8s.io namespaces it sets only labels under kubelet.kubernetes.io/ and
node.kubernetes.io/, and a short list of well-known labels. chalkctl validate refuses any other,
such as a node-role.kubernetes.io/ label:
node node4: invalid Kubernetes settings:
- Node: label "node-role.kubernetes.io/storage": the kubelet sets labels in the kubernetes.io and k8s.io namespaces on its own node only under kubelet.kubernetes.io/ and node.kubernetes.io/ or from the list kubernetes.io/hostname, topology.kubernetes.io/zone, topology.kubernetes.io/region, failure-domain.beta.kubernetes.io/zone, failure-domain.beta.kubernetes.io/region, beta.kubernetes.io/instance-type, node.kubernetes.io/instance-type, kubernetes.io/os, kubernetes.io/arch, beta.kubernetes.io/os, beta.kubernetes.io/arch
The kubelet applies the labels and taints when it registers its node, the first time the node
joins, and chalkd never changes the Kubernetes node after that. So labels and taints are fixed
once the node holds a Kubernetes share, and a node refuses a delivery that changes them:
the config is refused: Node/default: spec.labels: the config changes the node's labels; the value applies when the node registers with the cluster, so changing it on a running node changes nothing; change the Node object with kubectl label instead, or reinstall the node
Change the labels and taints of a registered node with kubectl, and keep its Node document as it
is:
kubectl label node node4 node.kubernetes.io/storage=nvme --overwrite
kubectl taint node node4 dedicated=storage:NoSchedule-
A reinstall registers the node again with the labels and taints of its Node document.
The Node document also holds the node's own address settings, nodeIP, and on a control plane
the interface it holds the VIP on, vip.interface.
Kubernetes networking
explains both.
Run workloads on control planes¶
A control plane registers with the taint node-role.kubernetes.io/control-plane:NoSchedule, so
only pods that tolerate it run there. Set allowSchedulingOnControlPlanes before the first node
installs to leave the taint off the control planes. The homelab sets it, because its three nodes are
all control planes:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
endpoint: https://192.0.2.11:6443
allowSchedulingOnControlPlanes: true
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
The taint is part of the control plane's registration, so allowSchedulingOnControlPlanes is fixed
once a node holds a Kubernetes share. Every node gets the same Cluster document, so every
running node refuses a delivery that changes it:
the config is refused: Cluster/default: spec.allowSchedulingOnControlPlanes: the config changes allowSchedulingOnControlPlanes; the value applies when the node registers with the cluster, so changing it on a running node changes nothing; add or remove the node-role.kubernetes.io/control-plane:NoSchedule taint with kubectl taint instead, or reinstall the node
On a running cluster, keep the setting and remove or add the taint with kubectl:
kubectl taint node node1 node-role.kubernetes.io/control-plane:NoSchedule-
kubectl taint node node1 node-role.kubernetes.io/control-plane:NoSchedule
Add flags to the control plane's components¶
extraArgs of the Cluster document adds flags to etcd, the API server, the controller-manager
and the scheduler, each a map of flags named without their leading dashes:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
endpoint: https://192.0.2.11:6443
extraArgs:
# NodePort services take ports from a smaller range.
kube-apiserver: {service-node-port-range: 30000-31000}
# etcd may grow to 8 GiB.
etcd: {quota-backend-bytes: "8589934592"}
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
The components take the flags on every control plane. A flag chalkos also sets takes your value.
A flag's name holds letters, digits, _, . and -, and starts with a letter or digit.
The control plane
explains the flags chalkos sets.
The API server's flags that decide who may do what are refused, also when spelt with _ for -:
anonymous-auth, authentication-config, authorization-mode and
enable-bootstrap-token-auth. chalkctl validate names the flag:
node node1: the Kubernetes cluster the documents give: Cluster "default": extraArgs: kube-apiserver --authentication-config is set by chalkos and cannot be overridden
The API server refuses to start with some flags next to flags chalkos sets, so chalkctl validate
refuses them too. Conflicting API server flags
lists them.
The error names the flag, the flag it conflicts with and what to do instead:
node node1: the Kubernetes cluster the documents give: Cluster "default": extraArgs: kube-apiserver --admission-control conflicts with --enable-admission-plugins: chalkos sets --enable-admission-plugins, and the API server refuses this flag next to it; give the plugins in enable-admission-plugins and disable-admission-plugins
From the node, the control plane's pods mount the certificates and kubeconfigs chalkd writes, and etcd's pod its data directory. A flag that names a file of your own, such as an audit policy or a webhook's configuration, finds no file to read.
Add flags to a node's kubelet¶
kubeletExtraArgs of a node's Node document adds flags to that node's kubelet:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
type: worker
labels: {node.kubernetes.io/storage: ssd}
taints: [{key: dedicated, value: storage, effect: NoSchedule}]
kubeletExtraArgs: {max-pods: "200"}
The kubelet's configuration is the same on every node and comes from the cluster. Its flags come last on the kubelet's command line, so they override that configuration. chalkos refuses the flags it sets itself. They give the kubelet its configuration, credentials, certificates and static pods. They also set the node's name, addresses, labels and taints, and who may use the kubelet's API. Protected kubelet flags lists them.
A refused flag names its replacement where there is one:
node node4: the Kubernetes cluster the documents give: Node "default": kubeletExtraArgs: --register-with-taints is set by chalkos and cannot be overridden; give the node's labels and taints in the Node document's labels and taints
A kubelet flag's name and value hold no whitespace and no backslash, because the flags reach the kubelet through an environment file that systemd splits at whitespace. A flag that needs a list separated by spaces is refused:
node node4: the Kubernetes cluster the documents give: Node "default": kubeletExtraArgs: --eviction-hard="memory.available<500Mi nodefs.available<10%": the kubelet's flags cannot hold whitespace or backslashes
Choose how clients authenticate to the API server¶
The API server accepts client certificates of the cluster's Kubernetes CA and service-account
tokens. Anonymous requests reach only /livez, /readyz and /healthz. chalkos writes the API
server's authentication configuration and protects the authentication-config flag that names it.
chalkos has no way to add an OIDC issuer or another authenticator today. The API server refuses
the oidc-* flags next to an authentication configuration, so chalkctl validate refuses them in
extraArgs.kube-apiserver:
node node1: the Kubernetes cluster the documents give: Cluster "default": extraArgs: kube-apiserver --oidc-issuer-url conflicts with --authentication-config: chalkos configures authentication through --authentication-config, and the API server refuses oidc-* flags next to it; chalkos sets no other authenticator; grant narrower access with RBAC
For access narrower than cluster-admin, bind users and groups to roles with RBAC.
Bootstrap and get a kubeconfig describes
the kubeconfigs chalkctl issues.
Set the pod network's MTU and VXLAN sources¶
On images that run flannel, a Flannel document in the files every node gets sets two things:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Flannel
metadata: {name: default}
spec:
# Nodes accept the pod network's VXLAN traffic only from the nodes' own addresses.
vxlanSourceSubnets: ["fdc8:a1b2:c3d4:ff::/64"]
mtu: 1480
vxlanSourceSubnetslists the ranges flannel's VXLAN traffic must come from, at least one of each family. Without it, nodes accept VXLAN from any source. Set it to the ranges your nodes live in.mtuis the MTU of the network between the nodes. Flannel's VXLAN devices and the pods get 50 less. Without it, flannel takes the MTU of the interface that holds the node's address. The homelab sets it, because that interface is the dummynode0, whose MTU says nothing of the fabric's.
chalkctl validate refuses a node whose address lies outside the VXLAN sources, since the other
nodes would drop its traffic. With vxlanSourceSubnets: ["fdc8:a1b2:c3d4:fe::/64"], a typo for
the nodes' ff::/64, it refuses every node, such as node4:
node node4: invalid Kubernetes settings:
- Node: nodeIP.addresses: the node's address fdc8:a1b2:c3d4:ff::14 is outside vxlanSourceSubnets fdc8:a1b2:c3d4:fe::/64, from which alone the other nodes take VXLAN
It also refuses a source range that overlaps the pod or service ranges, and an MTU that leaves pods less than IPv4's minimum of 576, or IPv6's 1280 when the cluster uses IPv6.
Apply your own objects with Addon documents¶
An Addon document holds Kubernetes objects that the control planes apply after chalkos's own
add-ons. This one creates a namespace and lets the group team-a read what runs in it:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Addon
metadata: {name: apps}
spec:
objects:
- apiVersion: v1
kind: Namespace
metadata: {name: apps}
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: team-a-view, namespace: apps}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: view}
subjects: [{apiGroup: rbac.authorization.k8s.io, kind: Group, name: team-a}]
A node's config may hold several Addon documents, each with its own name. The control planes
apply the objects with server-side apply, each document's objects in their order and the documents
in the order of their names. They apply them at bootstrap, at every boot and after every delivery
that changes them.
- chalkctl delivers
Addondocuments to control planes alone, so the file may sit among the files every node gets. A control plane keeps them on STATE, readable by root alone, so the objects may include Secrets. - An object you remove from an
Addondocument stays in the cluster. Delete it with kubectl. - chalkd applies the objects as field manager
chalkdand takes over fields that others changed, so a change made with kubectl to such an object is undone at the next apply.
Pull private images with the nodes' credentials¶
A Registry document gives every node the credentials of a private registry. The kubelet then
pulls that registry's images for every pod on the node without imagePullSecrets.
Registries and mirrors
describes the document. Keep credentials that only some workloads may use in their namespaces' pull
secrets instead.
Deliver the change¶
Check the config of every node, then deliver it to each node whose files changed:
A changed Cluster, Flannel or Node document restarts chalkos-kubernetes.service and
kubelet.service on the node, and chalkd on a control plane renders the control plane's pods again.
A changed Node document also restarts chalkos-controlplane.service, which opens the control
plane's ports by the node's type.
A changed Addon document restarts nothing: the control planes apply its objects again.
Built-ins lists these effects.
A node that holds a Kubernetes share refuses a config that changes ipFamilies, podCIDRs,
serviceCIDRs or dnsIPs, or the node's type, and names what the change takes.
Check that it worked¶
chalkctl status shows each control plane's Kubernetes lines. control plane current means its API
server and etcd run with the certificates the node holds and the API server answers ready.
control-plane pods current means its API server, controller-manager and scheduler run the images
the Cluster document names and are Ready:
kubernetes controlplane: bootstrapped, node ready: True, control plane current
kubernetes versions: API server 1.37.2, kubelet 1.37.2, control-plane pods current
kubectl shows what reached Kubernetes. The labels and taints of a node, and the objects of the
Addon above:
kubectl --kubeconfig kubeconfig get node node4 --show-labels
kubectl --kubeconfig kubeconfig describe node node4
kubectl --kubeconfig kubeconfig get rolebinding -n apps
A component that does not start with an extra flag shows in its control plane's status and in the
kubelet's journal, chalkctl logs node1 --unit=kubelet.service.
Troubleshoot Kubernetes covers further symptoms.
What next¶
- Run a highly available control plane.
- Upgrade Kubernetes to a new release.
- Change a node's config covers previews, refusals and the services a change restarts.
- How Kubernetes runs on a node explains what runs on each node, and The control plane what the control plane applies.