Run a highly available control plane¶
This page is for an operator who runs a cluster with three control planes behind a VIP. Three control planes keep the Kubernetes API and etcd available while one of them reboots or fails. At the end three control planes are healthy etcd voters, and you know how to add, remove and replace one without losing etcd's quorum.
A healthy control plane is three voters, each healthy:
$ chalkctl etcd members
NAME ID PEER URLS MEMBER HEALTH
cp1 d9fb26556fe7b4a5 https://192.168.123.11:2380 voter healthy
cp2 7cd467b152867819 https://192.168.123.12:2380 voter healthy
cp3 af880a99b80b32b3 https://192.168.123.13:2380 voter healthy
The output on this page comes from a lab cluster whose control planes cp1, cp2 and cp3 sit at
192.168.123.11 to 192.168.123.13, behind the VIP 192.168.123.10.
Before you begin¶
- You have the workstation that Set up your workstation describes,
with the inventory, the secrets file
secrets.ageand the signed image directories. - You made the decisions of Plan a cluster: three control planes and a VIP as the cluster endpoint.
- The three machines share one layer-2 segment. Each has a static address, and one free address on the segment is left for the VIP. Control planes on different links announce the VIP over BGP instead, as Announce the VIP over BGP describes.
- Each machine is booted into the installer, as Install a node describes.
Define three control planes and the VIP¶
Every node gets the Cluster document, which puts the VIP in spec.vip.addresses and makes it the
cluster endpoint:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
endpoint: https://192.168.123.10:6443
vip: {addresses: [192.168.123.10]}
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
Each control plane has a file of its own with type: controlplane, its system disk and its static
address:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
type: controlplane
---
apiVersion: storage.chalkos.dev/v1alpha1
kind: SystemDisk
metadata: {name: system}
spec:
disk: {path: /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_2TB_S7KHNJ0W100001}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-uplink}
spec:
match: {name: enp1s0}
network: {address: [192.168.123.11/24], gateway: 192.168.123.1}
nodes/cp2.yaml and nodes/cp3.yaml differ only in their disk and address. The inventory names
the three nodes at the same addresses:
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: prod}
spec:
images:
node-metal: {source: {directory: images/node-metal-signed}}
all: {imageName: node-metal, files: [cluster/*.yaml]}
nodes:
cp1: {address: 192.168.123.11, files: [nodes/cp1.yaml]}
cp2: {address: 192.168.123.12, files: [nodes/cp2.yaml]}
cp3: {address: 192.168.123.13, files: [nodes/cp3.yaml]}
chalkd adds the VIP to the interface that holds the node's address of the VIP's family. A Node
document can name another interface in spec.vip.interface. A cluster takes at most one VIP per
family, and an endpoint that is an IP address must be one of the VIPs. Check the files with
chalkctl validate.
Announce the VIP over BGP¶
The default mode of the VIP, l2, needs the control planes on one layer-2 segment. Its holder
announces the VIP there with gratuitous ARP and unsolicited neighbour advertisements. When the
control planes sit on different links, or the nodes announce their addresses to a fabric over BGP,
set spec.vip.mode to routed. The holder then adds the VIP to an interface as a /32 or /128 host
address and announces nothing on the link. A routing daemon on each control plane announces the
addresses of that interface to its peers. By default this is the Kubernetes layer's BGP speaker,
FRR, which Announce addresses with BGP sets up. An image that leaves the speaker out
can carry another daemon from an extension, such as bird, which chalkos does not configure.
The Cluster document names the mode beside the addresses:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
endpoint: https://10.0.100.10:6443
vip: {addresses: [10.0.100.10], mode: routed}
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
Each control plane names the interface it adds the VIP to in spec.vip.interface, usually a dummy
interface that holds only the VIPs. The node's file creates it with a NetDev document and brings
it up with a Network document:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Node
metadata: {name: default}
spec:
type: controlplane
vip: {interface: node0}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: NetDev
metadata: {name: 10-node0}
spec:
netDev: {name: node0, kind: dummy}
---
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-node0}
spec:
match: {name: node0}
network: {keepConfiguration: "yes"}
keepConfiguration: "yes" keeps the VIP when networkd configures node0 again: chalkd adds an
IPv6 VIP with a preferred lifetime of 0, which networkd would otherwise remove as a dynamic address
it did not configure. The rest of the file stays as above. chalkctl validate refuses a control plane without
vip.interface in a cluster with a routed VIP. A worker needs none.
Name node0 in the BGPSpeaker's export interfaces, as
Announce the VIP shows, or configure the extension's daemon to announce
the addresses of node0. The VIP's route then follows the holder: when a new control plane wins
the election, the old holder's daemon withdraws the route and the new holder's announces it. A failover takes the election's time plus the time the daemons and
their peers need to move the route. chalkctl status shows vip holder and vip standby as with
an l2 VIP. The homelab announces its VIP this way, with the Kubernetes layer's BGP speaker over
both ports of each node, as Build a home lab sets up.
chalkd adds an IPv6 VIP with a preferred lifetime of 0, so the node never uses it as a source
address. ip -6 addr shows it as deprecated, and routing daemons announce it all the same.
Install the three, then bootstrap one¶
Install each control plane at the address and with the fingerprint its console shows:
chalkctl install cp1 --secrets=secrets.age --address=<address> --fingerprint=<fingerprint>
chalkctl install cp2 --secrets=secrets.age --address=<address> --fingerprint=<fingerprint>
chalkctl install cp3 --secrets=secrets.age --address=<address> --fingerprint=<fingerprint>
Each node reboots into its image and waits, because it cannot tell whether it should start the
cluster or join one. chalkctl status shows it:
Bootstrap exactly one of them:
$ chalkctl bootstrap cp1 --secrets=secrets.age
bootstrapping cp1; the control plane pulls its images and starts
cp1 is bootstrapped; applied 19 objects
Bootstrap and get a kubeconfig covers the bootstrap, the kubeconfig and why a second bootstrap is refused. Once cp1's API server is ready, cp1 holds the VIP.
Watch the other control planes join etcd¶
cp2 and cp3 find the cluster at the endpoint and join on their own. etcd accepts one learner at a time, so they join one after the other. A joining node adds itself as a learner, a member that receives etcd's data but has no vote. It waits until its member has caught up with the leader, has it promoted to a voter and then starts the rest of its control plane. Its status names the step:
kubernetes controlplane: joining the cluster at https://192.168.123.10:6443: checking etcd's members
kubernetes controlplane: joining the cluster at https://192.168.123.10:6443: etcd member af880a99b80b32b3 catches up
Before the node adds itself as a learner, it pins its addresses on STATE. From then on, every boot waits for exactly these addresses before it starts etcd. A joined control plane reports:
kubernetes controlplane: bootstrapped, node ready: True, vip standby, control plane current
kubernetes versions: API server 1.37.2, kubelet 1.37.2, control-plane pods current
vip standby means another control plane holds the VIP, and the holder shows vip holder. A
standby node takes part in the VIP's election only while its API server answers ready.
control plane current means the node's API server and etcd serve the certificates the node
holds, and its API server answers ready. Right after the join, the line can read
control plane the API server is not ready for a few seconds while the API server starts.
Check that the VIP moves¶
When the holder's API server stops answering, or the holder fails, another healthy control plane takes the VIP through an election in etcd. The holder's lease lasts 10 seconds. The HA end-to-end test requires the VIP to move within 30 seconds of its holder's network link going down. When the old holder comes back, the VIP stays with its new holder. Kubernetes networking explains the election. Clients with an open connection to the old holder notice only when the connection times out. A cluster without etcd quorum has no holder, because no node can win the election.
Change the control planes through etcd¶
etcd's membership decides which control planes count towards the quorum. Every change therefore updates etcd before or after the machine changes:
flowchart TD
change{"Change"}
change -->|add| file["Add the node's file and inventory entry"]
file --> install["chalkctl install"]
install --> join["The node joins etcd as a learner, then as a voter"]
change -->|remove| leave["chalkctl etcd leave"]
leave --> off["kubectl delete node, power off, remove from the inventory"]
change -->|replace a failed machine| remove["chalkctl etcd remove-member"]
remove --> install
change -->|reinstall| leave2["chalkctl etcd leave"]
leave2 --> installer["Boot the installer"]
installer --> install
The diagram shows the four changes this page covers. Each etcd command refuses a change that would leave fewer healthy voters than etcd's quorum, the majority of voters that etcd needs to accept a write.
Add a control plane¶
Write the new node's file with type: controlplane, add the node to the inventory with its address
and file, and install it. It joins as cp2 and cp3 did, with no further command.
Grow to five control planes, not four. Four voters need three for a quorum, so they tolerate one failure, the same as three.
A cluster with a ClusterRegistry needs the new control plane's address in its endpoints first,
on every node, as
Add or remove a control plane describes.
Remove a control plane¶
Take the node out of etcd before you switch it off.
chalkctl etcd leave makes the node release the VIP,
remove its own member, stop its control plane, delete its etcd data and unpin its addresses:
The node's status now reports:
It does not join again on its own, also after a reboot. Delete its Node with
kubectl delete node cp2, power it off, and remove it from the inventory and its file from the
repository. In a cluster with a ClusterRegistry, take its address out of endpoints and apply
the config to every remaining node.
The leave is refused when the voters left would have fewer healthy members than their quorum, because etcd would then stop accepting writes. With cp3 down, the leave of cp2 is refused, and the message names the counts:
chalkctl: cp2: failed_precondition: without cp2 etcd has 2 voters, 1 healthy, fewer than the 2 a quorum needs
Bring the unhealthy member back first. chalkctl etcd members shows which one it is:
$ chalkctl etcd members
NAME ID PEER URLS MEMBER HEALTH
cp1 d9fb26556fe7b4a5 https://192.168.123.11:2380 voter healthy
cp2 7cd467b152867819 https://192.168.123.12:2380 voter healthy
cp3 af880a99b80b32b3 https://192.168.123.13:2380 voter unhealthy: context deadline exceeded
Replace a failed machine¶
A control plane whose machine is gone cannot leave by itself. Remove its member through the other
control planes with
chalkctl etcd remove-member, which takes the
node's name or the member's ID:
The same quorum guard applies. --force skips it, and etcd then accepts no writes until enough of
the remaining members are healthy again.
Then install the replacement machine under the same name. Give it the addresses of the node's
Network document, so the inventory's address stays right. Change its SystemDisk if the new disk
has another serial number or path. Boot it into the installer and install it, with --wipe-disk
when its disk holds data:
chalkctl install cp3 --secrets=secrets.age --address=<address> --fingerprint=<fingerprint> \
--wipe-disk
The node joins etcd as a new member.
Reinstall a control plane¶
A control plane that keeps its machine but starts over, for a new disk layout or a lost STATE, leaves etcd first and is reinstalled after:
chalkctl etcd leave cp3
chalkctl install cp3 --secrets=secrets.age --address=<address> --fingerprint=<fingerprint> \
--wipe-disk
Between the two commands, boot the machine into the installer through its BMC or boot menu. An
installed node runs no maintenance mode, so chalkctl install cannot reach it before.
Plan for an etcd without backups¶
chalkos has no command that snapshots etcd or restores it. Three control planes survive the loss of one machine, but they do not bring back an object someone deleted. A cluster that loses two of its three control planes at once loses etcd's quorum, and chalkos has no command that rebuilds etcd from the one member left. Back up what the workloads keep with a tool that works at the Kubernetes level. Back up what you cannot rebuild lists the files to keep, and the roadmap lists etcd backup and restore.
Check that it worked¶
Run chalkctl etcd members, which lists etcd's members as one control plane's member sees them.
It needs a reader client file or the secrets file. Expect three voters, each healthy, as at the
top of this page. A member that joins shows learner, and an unhealthy one shows
unhealthy: and the reason. --via=<node> asks a particular control plane, which helps when two of
them disagree. Then check that the API server answers through the VIP:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
cp1 Ready <none> 41s v1.37.2
cp2 Ready <none> 39s v1.37.2
cp3 Ready <none> 38s v1.37.2
w1 Ready <none> 37s v1.37.2
If something goes wrong¶
These failures are specific to control planes.
A control plane cannot rejoin etcd¶
A control plane's Kubernetes line in chalkctl status names what holds it back, and the fix:
| Status | Cause | Fix |
|---|---|---|
preparation failed: pinned address ... is not present |
The node booted without one of the addresses it was pinned to when it became an etcd member. | Restore the address and reboot. Or remove its member with chalkctl etcd remove-member <node>, as the status suggests, or with chalkctl etcd leave <node> --force, which also cleans up the node; then reinstall it. |
joining the cluster at ...: etcd has a member <node> already; remove it with chalkctl etcd remove-member <node> |
The node was reinstalled without leaving etcd, and its old member is still there. | Run the command the status names. The node tries again every 10 seconds and joins. |
left etcd; reinstall the node to join the cluster again |
The node left etcd with chalkctl etcd leave. |
Reinstall it. |
etcd data missing: restore etcd or reinstall the node |
The node is an etcd member, but VAR lost its etcd data. | chalkos cannot restore etcd. Run chalkctl etcd remove-member <node>, then reinstall it. |
joining the cluster at ...: the node's etcd member was removed while joining; run chalkctl etcd leave and reinstall the node |
An operator removed the member while the node was joining. | Run chalkctl etcd leave <node>, then reinstall it. |
A reinstalled node finds its old member¶
A control plane reinstalled without chalkctl etcd leave finds a member of its own name in etcd
and stops. Joining would hand it the old member's identity without its data:
kubernetes controlplane: joining the cluster at https://192.168.123.10:6443: etcd has a member cp3 already; remove it with chalkctl etcd remove-member cp3
Run the command the status names. The node joins on its next attempt.
A control plane lost its pinned address¶
A pinned control plane that boots without one of its addresses, after a cabling or switch change
for example, waits for spec.nodeIP.timeout seconds of the Cluster document. Then it runs
neither etcd nor the kubelet:
kubernetes controlplane: preparation failed: pinned address 192.168.123.13 is not present, 5m0s after chalkos-node-addresses.target; restore it, or remove the node's etcd member with chalkctl etcd remove-member cp3 and reinstall the node
The other two keep the cluster running. Restore the address and reboot the node, or take it out.
Its own etcd member does not answer, so chalkctl etcd leave cp3 refuses without --force. With
--force it removes the member through the others and cleans up the node. Then reinstall it.
Recover a node covers nodes that do not unlock or boot. Troubleshoot Kubernetes starts from symptoms.
What next¶
- Upgrade Kubernetes rolls a new image through the control planes one at a time.
- Add and remove nodes covers workers and nodes without Kubernetes.
- The control plane explains bootstrap, join and the static pods.
chalkctl etcdlists the etcd commands and their flags.