Skip to content

Upgrade Kubernetes

This page is for an operator who moves a cluster to a new Kubernetes release. The release takes two steps. First the control plane moves through config: chalkctl config apply delivers new image references to the control planes one at a time, then to the other nodes. Then the kubelets move through new OS images: chalkctl upgrade installs them, keeps etcd's quorum and drains each node within its PodDisruptionBudgets. Both commands keep the Kubernetes version skew policy across the nodes. At the end every node runs the new release, and its Node is Ready and schedulable again.

The second step builds on Upgrade nodes, which covers the image, the config and rollback. A run of chalkctl upgrade on three control planes and a worker reads:

upgrading node-kvm on kvm to 23112865089f (v2): cp1, cp2, cp3, w1
cp1: installing 23112865089f (v2)
cp1: config unchanged
cp1: installed 23112865089f (v2), staged as .staged-chalkos_2.efi
cp1: cordoned, evicted 2 pods, kept 6
cp1: activated 23112865089f (v2), which boots next as chalkos_2+3.efi
cp1: rebooting into 23112865089f (v2)
cp1: runs 23112865089f (v2), found healthy
cp1: uncordoned
cp2: installing 23112865089f (v2)
...
cp3: uncordoned
w1: installing 23112865089f (v2)
w1: config unchanged
w1: installed 23112865089f (v2), staged as .staged-chalkos_2.efi
w1: cordoned, evicted 1 pod, kept 2
w1: activated 23112865089f (v2), which boots next as chalkos_2+3.efi
w1: rebooting into 23112865089f (v2)
w1: runs 23112865089f (v2), found healthy
w1: uncordoned
upgraded 4 nodes to 23112865089f (v2)

The output comes from a lab cluster. ... stands for cp2's and cp3's lines, which repeat cp1's.

Before you begin

  • You have run an upgrade as Upgrade nodes describes, and hold the credentials it lists.
  • The nodes reach the new release's images, in their registry or through a mirror, as Registries and mirrors describes.
  • The image author has built the new image, and it is signed into the directory the inventory names.
  • The cluster has three control planes, for an upgrade without API downtime. A cluster with one or two needs --allow-downtime, described below.
  • kubectl is on your workstation, for the checks.

Plan the two steps of a release

The image fixes the kubelet's release. The image author sets it in the image's modules, as Choose the kubelet's release describes, and every image with the Kubernetes layer is built again. The control plane, kube-proxy, CoreDNS and the pod sandbox run the images the Cluster document names, as Name the images of the components describes.

A kubelet must not be newer than the API server it talks to, so the control plane moves first:

  1. Change the references of images in the Cluster document to the new release and deliver the config to every node with chalkctl config apply, as Roll the control plane out through config describes.
  2. Upgrade the nodes to the new image with chalkctl upgrade, as the sections from Choose the order and the batch size on describe.

A patch release of the control plane is the first step alone, and a patch release of the kubelet is the second step alone. Move one minor release at a time, as the Kubernetes version skew policy requires.

The two steps run apart. chalkctl upgrade delivers each node's config with its image, so it refuses a config whose API server differs from the one the node runs, before it sends any image:

chalkctl: w1: the inventory's config changes the API server from 1.37.2 to 1.37.3, which an upgrade does not deliver; deliver the config with chalkctl config apply first, which orders the nodes for it, then upgrade

Deliver the new Cluster document with chalkctl config apply to every node first. The upgrade that follows delivers the same config, which changes no API server.

Roll the control plane out through config

Change the four versioned references of the Cluster document to the new release, and the other references when the release names new ones. Pin each by its digest, as Name the images of the components describes:

cluster/cluster.yaml
images:
  apiServer: registry.k8s.io/kube-apiserver:v1.37.3@sha256:<digest>
  controllerManager: registry.k8s.io/kube-controller-manager:v1.37.3@sha256:<digest>
  scheduler: registry.k8s.io/kube-scheduler:v1.37.3@sha256:<digest>
  proxy: registry.k8s.io/kube-proxy:v1.37.3@sha256:<digest>
  # etcd, coredns, pause and flannel as before

Then deliver the config to every node of the cluster in one command:

chalkctl config apply cp1 cp2 cp3 w1 --secrets secrets.age

The config changes the API server's version, so chalkctl orders the run. It checks the versions every node runs first, as Check the versions every node runs describes. Then the control planes get the config one at a time, in the order of their names, whatever order the command names them in:

  1. chalkctl checks that etcd keeps its quorum without the control plane, as before a reboot.
  2. It delivers the config on trial, for --try or else 10 minutes. The kubelet restarts and replaces the static pods of the API server, the controller-manager and the scheduler.
  3. It waits until the node's Node is Ready and its three static pods run the images the config names and are Ready. A Node that is not Ready only because no CNI runs yet passes, as in a cluster created without one.
  4. It confirms the trial once the node is healthy, as any trial.

The other nodes get the config after the last control plane, without a trial unless --try asks for one. No node reboots and no node drains. A run on three control planes and a worker reads:

cp1: config 0 added, 0 removed, 1 changed
cp1: delivering config <version> on trial until <deadline>
restarted chalkos-kubernetes.service
restarted kubelet.service
cp1 runs config <version> on trial; it rolls back to config <previous-version> by itself unless confirmed within 10m0s
cp1: waiting for its layers to serve with the config
cp1: waiting for a new connection and the node's health to confirm config <version>
cp1 keeps config <version>: the trial is confirmed
cp2: config 0 added, 0 removed, 1 changed
...
cp3 keeps config <version>: the trial is confirmed
w1: config 0 added, 0 removed, 1 changed
restarted chalkos-kubernetes.service
restarted kubelet.service
w1 runs config <version>

<version> stands for each node's new config version, and ... for cp2's and cp3's lines, which repeat cp1's. Each control plane takes a few minutes: its kubelet pulls the new images and starts the static pods again. The first control plane that gets the config applies the add-ons too, so kube-proxy's DaemonSet moves to the new image while the other control planes still run the old API server. Within a patch release the skew policy allows that. In a minor upgrade, kube-proxy runs one minor ahead of the API servers not yet rolled until the run finishes, so run it to the end without a pause.

A command that names one node takes the same path when the config changes its API server: the version check over every node, etcd's quorum check, the trial and the wait. Without such a change, chalkctl config apply delivers node by node, as for any other config.

Act on a node that fails its trial

The first node that fails stops the run. A control plane whose static pods do not run the new images by the trial's deadline is rolled back at once, and the nodes after it get nothing. chalkctl prints which nodes run which API server:

cp2 runs config <previous-version> again: config <version> was rolled back
API server 1.37.2: cp2, cp3, w1
API server 1.37.3: cp1
chalkctl: cp2: config <version> was not confirmed within its trial: its control plane: kube-apiserver is not Ready

Nodes that run the new version keep it, which the skew policy allows. The error names what the node's static pods lacked. A reference the registry does not serve, or a pull the mirror refuses, shows as a pod that is not Ready, as kube-apiserver has no mirror pod yet, or as its static pods are unknown while the node's API server is down. chalkctl logs cp2 --unit kubelet.service names the image the kubelet could not pull, and chalkctl status prints each node's versions in its kubernetes versions line. Fix the reference or the registry, then run the same command again. It skips the nodes that run the config already, checks that each such control plane still serves with it, and continues with the first that does not:

cp1 runs config <version> already
cp1: waiting for its layers to serve with the config
cp2: config 0 added, 0 removed, 1 changed
...

Walk the control planes by hand

--until deliver delivers the config on trial to the first node that does not run it yet, and leaves the trial open. chalkctl config confirm confirms it once its static pods run the new images, and the same --until deliver command then goes on with the next control plane:

chalkctl config apply cp1 cp2 cp3 w1 --secrets secrets.age --until deliver
kubectl get pods -n kube-system -o wide
chalkctl config confirm cp1 --secrets secrets.age
chalkctl config apply cp1 cp2 cp3 w1 --secrets secrets.age --until deliver

While a trial is open, the run refuses to go on, and chalkctl config rollback cp1 rolls the open trial back. Once every control plane runs the config, the last command delivers it to the other nodes and finishes.

Check the versions every node runs

Each node reports, in its status, the Kubernetes version of the API server its config names and the release of its image's kubelet. chalkctl reads them from the inventory's nodes with a Node document and holds them to the Kubernetes version skew policy, minor versions compared:

Rule Checked by
The API servers of a cluster are at most one minor apart. config apply
A kubelet is no newer than the API server. the image's checker, config apply, upgrade
A kubelet is at most three minors older than the API server. the image's checker, config apply
The controller-manager and the scheduler are no newer than the API server and at most one minor older. the image's checker
kube-proxy is no newer than the API server and at most three minors older. the image's checker

The image's checker compares the config with the image's own kubelet, as Keep the versions within the skew policy describes. chalkctl config apply compares the config's API server with every node's API server and kubelet before it delivers anything to a node, and chalkctl upgrade compares each new image's kubelet with the oldest API server the control planes run before it sends any image. The upgrade asks the control planes only when it gives an image with a kubelet to a node with a Node document, so an upgrade of nodes outside the cluster reaches none of the cluster's nodes. A refusal names the node:

chalkctl: images.apiServer: v1.39.0 is 2 minors newer than the API server 1.37 that cp1 runs; a cluster's API servers may be at most one minor apart, so upgrade one minor at a time
chalkctl: w1: the image's kubelet 1.38.0 would be newer than the API server 1.37.3 that cp2 runs, the oldest of the cluster; upgrade the control plane first: name the new release's images in the Cluster document and deliver it with chalkctl config apply

config apply also refuses a config that names two API servers for the nodes of one run, and a worker's new API server while a control plane outside the run does not run it yet. A config older than a minor the control planes run is a downgrade, which --allow-downgrade allows by one minor.

The checks need the versions of every node. A node that does not answer stops a config apply that changes the API server, and a control plane that does not answer stops an upgrade, with the versions <node> runs are unknown: <error>. A node whose inventory entry holds no Kubernetes Node document does not count.

Choose the order and the batch size

chalkctl upgrades the control planes first, one at a time, each only while etcd keeps its quorum without it. The workers follow in batches of --max-unavailable, one node by default:

flowchart TD
    stagecp["Stage the control plane's image"] --> guard["Check etcd's quorum without the control plane"]
    guard -->|quorum kept| cp["Drain, activate and reboot the control plane"]
    cp --> healthy{"Found healthy and Ready?"}
    healthy -->|yes| uncordon["Uncordon it"]
    uncordon -->|next control plane| stagecp
    uncordon -->|last control plane| batch["Stage a batch of workers' images"]
    batch --> install["Drain, activate and reboot the batch"]
    install --> ready{"Found healthy and Ready?"}
    ready -->|yes| next["Uncordon the batch"]
    next -->|next batch| batch
    guard -->|no quorum| stop["Stop the run"]
    healthy -->|no| stop
    ready -->|no| stop

The diagram shows the order. Within each group the nodes go in the order of their names. A node is drained once its image is staged, and the image is activated right after the drain, before the node reboots. A quorum check or a drain that fails stops the run with the image staged and not active, so the node keeps booting its current image. An activation that fails stops the run too, and chalkctl uncordons the node, which does not reboot. After each reboot, chalkctl waits until the node runs the new image, its boot was found healthy and its Node is Ready. Then it uncordons the node. chalkctl node reboot runs the same quorum check, drain, reboot, wait and uncordon for one node.

Keep etcd's quorum while a control plane reboots

Before chalkctl touches a control plane, it checks that etcd keeps its quorum without it. The other voters must be healthy and enough for a quorum, the majority of voters etcd needs to accept a write. Right after the previous control plane came back, etcd may need a moment, so chalkctl tries the check again for up to a minute. With cp3 down, an upgrade of cp1 stops there:

upgrading node-kvm on kvm to c3a9674a8c68: cp1
chalkctl: without cp1 etcd has 1 healthy voter of 3, fewer than the 2 its quorum needs

This output is from another lab run, which upgraded to the image c3a9674a8c68. chalkctl etcd members shows which voter is unhealthy. Bring it back, then run the same command again.

A cluster of one or two control planes loses etcd's quorum while any of them reboots, so etcd and the API server are down until it is back. chalkctl refuses such a run:

chalkctl: etcd has 2 voters, and without cp1 fewer than the 2 its quorum needs: it and the API server are down while cp1 reboots; pass --allow-downtime to accept that

Pass --allow-downtime when that downtime is acceptable. With three or more control planes, the flag changes nothing: chalkctl still waits for the other voters to be healthy.

chalkctl config apply runs the same check before each control plane gets a new API server. The delivery restarts that control plane's API server, controller-manager and scheduler, not etcd, so with too few other control planes the cluster's API may be unavailable meanwhile. Its message reads the API server of cp1 restarts, and with no other control plane the cluster's API may be unavailable meanwhile; pass --allow-downtime to accept that, and its --allow-downtime accepts it the same way.

A control plane that is joining blocks the run too, because its etcd member is a learner. The message reads etcd member cp3 is a learner yet; upgrade once it joined. Run the command again once the node has joined.

Drain within the PodDisruptionBudgets

The drain cordons the node and evicts its pods through the eviction API, so each pod's PodDisruptionBudget applies. An eviction that the budget does not allow yet is tried again until --timeout passes. The drain keeps these pods on the node:

  • DaemonSet pods, which run on the node again once it is back;
  • static pods, such as the control plane's own;
  • finished pods;
  • pods without a controller, which nothing starts elsewhere.

chalkctl names the last kind, because those pods are gone for good if the node stays down longer than their tolerations allow:

w1: keeps pods without a controller: <pods>; nothing starts them elsewhere, and they are deleted for good if the node stays down longer than their tolerations allow

<pods> stands for the pods' namespaces and names. A pod with an emptyDir volume stops the drain unless --delete-emptydir-data allows the eviction, because the eviction deletes the volume's data:

pods <pods> have emptyDir volumes, whose data an eviction deletes; pass --delete-emptydir-data to evict them

A node that was cordoned before the run, by kubectl cordon for example, stays cordoned after it. The drain line then reads was cordoned before and stays so. chalkctl uncordons only the nodes it cordoned itself.

Choose the flags for Kubernetes

These flags of chalkctl upgrade change how it treats the cluster. chalkctl upgrade lists the others.

Flag Default Effect
--allow-downtime off Lets a cluster of one or two control planes upgrade, although etcd and the API server are down while one of them reboots.
--delete-emptydir-data off Also evicts pods with emptyDir volumes, whose data is lost.
--timeout=<duration> 30 minutes How long each drain may take, and how long each node may take to come back healthy and Ready.
--max-unavailable=<n> 1 How many workers go at once. Control planes always go one at a time.

Raise --max-unavailable only when the other nodes can run the evicted pods. Pods that find no room stay pending, and their budgets then stop the next drain.

These flags of chalkctl config apply change how it rolls the control plane out:

Flag Default Effect
--allow-downtime off Lets a cluster of one or two control planes take a new API server, although the cluster's API may be unavailable while a control plane's API server restarts.
--allow-downgrade off Lets the config name an API server one minor older than the newest the control planes run.
--try=<duration> 10 minutes for control planes How long each node's trial lasts. Its static pods must run the new images within it.
--until=deliver confirm Leaves the first node's trial open, for chalkctl config confirm.

Run the same command again to continue

A run that stopped continues where it stopped. When a budget allows no eviction within --timeout, the drain fails and the node stays cordoned:

chalkctl: drain w1: cp1: failed_precondition: drain w1: pods default/web-64f96cd96-xzlzx still run: default/web-64f96cd96-xzlzx: Cannot evict pod as it would violate the pod's disruption budget.; it stays cordoned

kubectl get pdb -A shows the budget that allows no disruption:

$ kubectl get pdb -A
NAMESPACE   NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
default     web    1               N/A               0                     11m

Here both pods of web ran on w1, the only worker, so no evicted pod could start elsewhere. Make room for the pods, or change the budget, then run the same command again. It skips the nodes that run the new image and drains w1 again:

upgrading node-kvm on kvm to 23112865089f (v2): cp1, cp2, cp3, w1
cp1: runs 23112865089f (v2), found healthy
cp2: runs 23112865089f (v2), found healthy
cp3: runs 23112865089f (v2), found healthy
w1: cordoned, evicted 1 pod, kept 2
w1: installing 23112865089f (v2)
...

Uncordon a node that rolled back

A node of the cluster whose new image is not found healthy boots its previous image and stays cordoned, so you can look at it before workloads return. The run stops with rolled back to <old>; it stays cordoned. A node rolled back during the upgrade explains how to find the cause.

The next run that installs an image on the node uncordons it once the node is healthy, with a fixed image or with --retry-failed. To give the node back to the scheduler before that, uncordon it yourself:

kubectl uncordon w1

Check that it worked

kubectl get nodes shows each node's kubelet release in the VERSION column, and no node SchedulingDisabled:

$ kubectl get nodes
NAME   STATUS   ROLES    AGE   VERSION
cp1    Ready    <none>   19m   v1.37.2
cp2    Ready    <none>   19m   v1.37.2
cp3    Ready    <none>   19m   v1.37.2
w1     Ready    <none>   19m   v1.37.2

chalkctl etcd members shows every control plane as a healthy voter again.

chalkctl status shows each node's versions after its Kubernetes line. On a control plane the line ends control-plane pods current once its static pods run the config's images:

kubernetes controlplane: bootstrapped, node ready: True, vip holder, control plane current
kubernetes versions: API server 1.37.3, kubelet 1.37.2, control-plane pods current

kubectl get pods -n kube-system -o wide lists the static pods, kube-proxy and CoreDNS, and kubectl describe pod -n kube-system kube-apiserver-cp1 shows the image the API server runs.

If something goes wrong

  • without cp1 etcd has 1 healthy voter of 3, fewer than the 2 its quorum needs: another control plane is down or unhealthy. chalkctl does not reboot cp1 until enough of the others are healthy. chalkctl etcd members shows which one is not.
  • etcd has 2 voters, and without cp1 fewer than the 2 its quorum needs: ...; pass --allow-downtime to accept that: the cluster has one or two control planes, so etcd and the API server are down while one reboots. Pass --allow-downtime when that is acceptable.
  • pods ... have emptyDir volumes, whose data an eviction deletes; pass --delete-emptydir-data to evict them: the drain refuses to delete those pods' scratch data. Pass the flag when the data may be lost.
  • etcd member cp3 is a learner yet; upgrade once it joined: a control plane is still joining. Run the command again once it has joined.
  • drain w1: ... Cannot evict pod as it would violate the pod's disruption budget.; it stays cordoned: a PodDisruptionBudget allowed no eviction within --timeout. kubectl get pdb -A shows it.
  • images.apiServer: v1.39.0 is 2 minors newer than the API server 1.37 that cp1 runs; ...: the config skips a minor release. Name the next minor release first, and roll the control plane out once per minor.
  • images.apiServer: v1.37.5 is older than the API server 1.38 that cp1 runs; pass --allow-downgrade to downgrade the control plane: the config names an older minor than a control plane runs. Pass the flag when the downgrade is meant.
  • images.apiServer: v1.38.0 is 4 minors newer than the kubelet 1.34 that w1 runs; ...: a node runs a kubelet too old for the new API server. Upgrade that node's image first.
  • w1 is no control plane, and the control planes cp2 do not run the API server 1.37.3 yet; deliver the config to them first: the command named a worker before every control plane runs the new API server. Name the control planes too, or deliver to them first.
  • cp2: config <version> was not confirmed within its trial: its control plane: kube-apiserver is not Ready: the control plane's static pods did not run the new images within the trial, and chalkctl rolled its config back. Check that the registry or the mirror serves the references, then run the same command again.
  • the versions w1 runs are unknown: ...; the version skew check needs every node of the cluster: chalkctl could not read a node's versions. Bring the node back, or remove it from the inventory.
  • w1: the image's kubelet 1.38.0 would be newer than the API server 1.37.3 that cp2 runs, ...: chalkctl upgrade refuses an image whose kubelet is ahead of the control plane. Roll the control plane out through config first.

What next