Plan a cluster¶
This page is for an operator who is about to install the nodes of a new Kubernetes cluster. It
takes the decisions that shape the Cluster document every node gets and the Node document of
each control plane. A few of them are fixed once the cluster runs, so they are made here, before
the first install. At the end you have a Cluster document and a list of control planes with
their addresses.
The decisions lead to a document such as this one, for three control planes behind a VIP:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
# The VIP the control planes hold, so the endpoint outlives any one of them.
endpoint: https://192.0.2.10:6443
vip: {addresses: [192.0.2.10]}
# Fixed once the cluster exists.
ipFamilies: [ipv4]
podCIDRs: {ipv4: 10.244.0.0/16}
serviceCIDRs: {ipv4: 10.96.0.0/12}
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
The references of images must suit the kubelet release of the nodes' image, which the image
directory names in facts/kubernetes.json. Configure Kubernetes shows near its top
how to read it, and where to find a set of references that matches.
Before you begin¶
You need the workstation that Set up your workstation describes: the repository with the inventory, the secrets file and the image directories. The nodes' files are laid out as Write a node's config describes. Who may use the nodes and the Kubernetes API is on Manage access and certificates.
Know the machines: how many there are, which network segments they sit on and which addresses they can keep.
Choose one or three control planes¶
A control plane runs etcd, the API server, the controller-manager and the scheduler. Every control plane is an etcd member, and etcd needs a majority of its voters to accept a write. The number of control planes therefore decides how many machines the cluster can lose:
| Control planes | Survives the loss of | Upgrades and reboots |
|---|---|---|
| 1 | none | The API server is down while the node reboots, so chalkctl upgrade needs --allow-downtime. |
| 3 | one | One at a time, with no API downtime. |
| 5 | two | One at a time. Each member adds etcd write latency. |
Use three. One control plane suits a lab, or a cluster whose API may go away while the node reboots. An even number adds no tolerance: two control planes need both for a majority, and four tolerate the loss of one, like three. Two are less available than one, because etcd stops when either of two machines fails.
A node is a control plane when its Node document has type: controlplane, and a worker with
type: worker. The type is fixed once the node holds a Kubernetes share; a node of the other type
is a reinstall. Run a highly available control plane installs three.
Choose the cluster endpoint¶
The cluster endpoint, spec.endpoint of the Cluster document, is the URL that kubelets,
kubeconfigs and joining control planes use to reach the API server on port 6443. Workers also
renew their node certificates through it: chalkd on a worker connects to the endpoint's host on
port 50000, where a control plane's chalkd answers. Whatever the endpoint points at must
reach both ports of a healthy control plane.
The diagram shows the two ways to keep the endpoint reachable when a control plane fails.
flowchart LR
clients["kubectl, kubelets, workers' chalkd"]
clients -->|"6443 and 50000"| vip(["VIP"])
vip --- holder["the control plane holding the VIP"]
clients -->|"6443 and 50000"| lb["your load balancer"]
lb --> node1["node1"]
lb --> node2["node2"]
lb --> node3["node3"]
Choose one:
- A VIP,
spec.vip.addresses, which chalkd holds on one healthy control plane at a time. By default the holder announces it with gratuitous ARP and unsolicited neighbour advertisements, so the control planes must share one layer-2 segment. Use it for machines in one rack or VLAN. Withspec.vip.mode: routed, a routing daemon on each control plane announces the VIP instead, such as the Kubernetes layer's BGP speaker, as in the homelab, and the control planes may sit on different links. The endpoint is the VIP or a name that resolves to it;chalkctl validaterefuses an endpoint address that is none of the VIPs, withvip: the endpoint https://192.0.2.11:6443 is none of the virtual addresses. - A load balancer you run, for control planes in different segments. It forwards TCP ports 6443
and 50000 to the control planes and passes TLS through, because clients authenticate with
certificates. A balancer that forwards only 6443 leaves workers unable to renew their node
certificates. Leave
spec.vip.addressesempty, and make the endpoint the balancer's address or name.
When the holder crashes or loses its network, another control plane takes the VIP once the holder's etcd lease of 10 seconds expires. Kubernetes networking describes the election.
A single control plane can take the endpoint of its own address. Every kubeconfig names the endpoint, so moving to a VIP later means issuing every kubeconfig again. A VIP from the start, even on one control plane, keeps the endpoint when control planes join.
Keep the control planes' addresses¶
Give every control plane a static address in a Network document of its own file, and the same
address as its address in the inventory. A control plane's addresses are pinned on STATE when it
becomes an etcd member, and every later boot waits for exactly those addresses before it starts
etcd. Changing a control plane's address means taking it out of etcd and reinstalling
it. The homelab's control planes hold their addresses on a dummy interface, node0, which keeps
them whichever port or cable fails.
Workers may take their address from DHCP. spec.nodeIP.validSubnets of the Cluster document
then picks it from the subnets you list. The inventory names such a worker by an address or name
that stays.
Choose the IP families and ranges¶
spec.ipFamilies lists the cluster's address families, the primary one first: [ipv4] by
default, [ipv6] for IPv6 alone, as in the homelab, or [ipv4, ipv6] for dual stack. Each family
has a pod range, podCIDRs, and a service range, serviceCIDRs. The families, the ranges and the
cluster DNS addresses of dnsIPs are fixed once the cluster exists, because the certificates, etcd
and the ranges the control plane allocated were made for them. A node that holds a Kubernetes share
refuses a config that changes them, with a message such as:
the config changes the cluster's podCIDRs from ipv4 10.244.0.0/16 to ipv4 10.245.0.0/16; they are fixed once the cluster exists, so keep them, or reinstall the node into a new cluster
A different choice later means a new cluster. Pick ranges that overlap none of your networks, including the networks pods must reach. A dual-stack cluster names both families and a range of each:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
endpoint: https://192.0.2.10:6443
vip: {addresses: [192.0.2.10]}
ipFamilies: [ipv4, ipv6]
podCIDRs:
ipv4: 10.244.0.0/16
ipv6: fd00:10:244::/56
serviceCIDRs:
ipv4: 10.96.0.0/12
ipv6: fd00:10:96::/112
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
Kubernetes networking explains how nodes pick an address of each family.
Decide where workloads run¶
Each control plane registers with a NoSchedule taint, so only pods that tolerate it run there. A
cluster of one to three machines without workers leaves the taint off. Decide before the control
planes register, because the setting reaches only control planes that register later.
Run workloads on control planes shows the setting.
Check that it worked¶
Put the Cluster document into the files every node gets. Give each control plane a Node
document with type: controlplane, and each worker one with type: worker; type has no
default. Then check every node's config against its image. chalkctl validate reaches no node and
needs no secrets file:
$ chalkctl validate
node1: valid for the image node-metal (node-metal on metal, root hash 3b80d0bf1243), config 451b547e8373fe70683c9dbf95f7cc08916ae0e920702da6c1a2c3e85c6f4080
node2: valid for the image node-metal (node-metal on metal, root hash 3b80d0bf1243), config 10c35f7fe988679c3a54691ddf95e78267269d22297fb360ceae5527f5965d50
node3: valid for the image node-metal (node-metal on metal, root hash 3b80d0bf1243), config 55ed0b193224383f68d0bb4014e45885697279cd67dfa6fc9232627757cfbd44
router: valid for the image router-metal (router-metal on metal, root hash cd7ac91cb655), config dc6178bbae55b1fe0e6af7b10502a1099f708c995b7be242ee95bb92050f87e6
The output is the homelab's, whose router runs no Kubernetes. Each node of your inventory gets a line. A refused config names the node and the field, as in the VIP message above.
What next¶
- Install a node on each machine.
- Bootstrap and get a kubeconfig once the nodes are installed.
- Configure Kubernetes for labels, taints, extra flags and add-ons.
- Run a highly available control plane for three control planes behind a VIP.
- Add and remove nodes to add workers to a running cluster.