Recover a node¶
This page is for an operator whose node is stuck: its disks do not unlock, its new image fell
back, its certificate expired, or it sits in maintenance mode. Each section starts from what the
console or chalkctl status shows, names the cause and gives the fix. Reinstalling the node is
the last resort, and the loss of the secrets file closes the page.
Before you begin¶
You need the workstation that Set up your workstation describes, with the secrets
file secrets.age. Most fixes here need it.
You also need the node's console, through its BMC or in person, for the sections that start there. When you do not know yet what is wrong, start at Troubleshooting.
The console asks for a passphrase at boot¶
The node stops early in the boot. systemd-cryptsetup asks on the console for the passphrase of
state, then of var and of each encrypted volume. The TPM refused to unseal their keys, because
PCR 7 changed since the keys were sealed to it:
flowchart LR
keys["Secure Boot keys or state change"] --> pcr["PCR 7 holds a new value"]
firmware["Firmware update brings a new dbx"] --> pcr
pcr --> tpm["TPM refuses to unseal the keys"]
cleared["TPM cleared, or disk in another machine"] --> tpm
tpm --> prompt["Console asks for a passphrase"]
prompt --> unlock["Recovery key or password unlocks"]
The diagram shows the chain from a change to the prompt. PCR 7 measures the Secure Boot state and the PK, KEK, db and dbx. Any of these causes changes it:
- Secure Boot was turned off, or its keys changed: a new PK, KEK or db.
- A firmware update brought a new dbx.
- The image was signed with another db key than the one the node was installed with.
A cleared TPM, or a disk moved to another machine, has no sealed keys at all, with the same result.
systemd-cryptsetup asks on every kernel console at once, the screen and each serial port, and the
first answer counts. A machine without a serial port shows the prompt on its screen, and its
keyboard answers it. The boot's status around the prompt, such as the unit that waits, shows only
on the last console of chalkos.consoles, which
is the screen unless the image puts a serial port last;
Images and platforms
describes the choice. A volume that asks after the boot shows its prompt in the console on the
screen, as The console answers password requests
describes.
The node stays at the prompt. It does not fall back to maintenance mode, because that would let anyone with physical access give it a new config. A boot that waits at the prompt never reaches the health check either. After an upgrade, the node stays in that boot until you unlock it or reset the machine. Only then does the boot loader try the new image again or fall back.
Unlock the node with its recovery key¶
Print the node's recovery key on the workstation:
The key is eight groups of eight letters, separated by dashes. Type it at each prompt, and the node boots on. A node whose fallback is a password takes that password instead.
chalkctl derives the key from the secrets file and the node's name. It stays the same for the node's whole life, also across reinstalls. Whoever reads it can unlock that node's disks with physical access for good, so keep the console session private.
Make the prompt go away¶
The prompt comes back at every boot until PCR 7 matches the sealed keys again. chalkos has no command that seals the keys to the new PCR 7. The prompt stops after one of these:
- Undo the change: turn Secure Boot back on, restore the previous keys, or boot an image signed with the original db key. The keys unseal again at the next boot.
- Accept the change and reinstall the node, as Reinstall the node as the last resort describes. The installer seals the new keys to the PCR 7 the node has now.
A node whose fallback is none has no second key. STATE and VAR stay locked, so the only ways
back are to undo the change or to reinstall. An extra volume that does not unlock stays unmounted
while the rest of the node runs. Reset a volume creates it
again, empty.
The node fell back after an upgrade¶
chalkctl status shows the node running its previous image, with the failed boot's log:
image <old>, booted from chalkos_1.efi
upgrade to <new> failed: rolled back to <old>
<journal lines>
<old> and <new> stand for the images' names, and <journal lines> for the last 30 lines that
chalkd and the health check logged in the failed boot. The new image did not become healthy within
its tries, and systemd-boot booted the previous one. The node runs healthy on the old image.
Find the cause in the journal lines, fix it in a new build and upgrade again, as
A node rolled back during the upgrade
describes.
A change misbehaves after the node was found healthy¶
A boot that was found healthy is kept, so a problem that shows only later needs a rollback on request. Each one undoes one kind of change:
- A config delivery that broke a service:
chalkctl config rollback w1 --secrets secrets.agedelivers the config the image ran before. Roll back a config change describes it. - An image that misbehaves after its boot was found healthy:
chalkctl upgrade --rollback --nodes w1boots the image the node ran before, which its inactive slot still holds. Go back to the previous image describes it.
The node's OS CAs, node certificate, storage and layer payloads stay as they are with either rollback.
A config change that might cut chalkctl off from the node, such as a firewall change, goes on trial
with chalkctl config apply w1 --secrets secrets.age --try 5m. The node rolls it back by itself
unless chalkctl confirms it over a new connection, and a reboot during the trial boots the config
from before it. Try a config change describes it.
The node certificate expired¶
chalkctl refuses the node's certificate as expired, so every command against the node fails. The node itself still runs. A node of a Kubernetes cluster renews its node certificate once two thirds of its year have passed, so its certificate expires only when it could not renew for four months: it was switched off or could not reach a control plane. Before the expiry, its status showed why:
A node whose image runs no Kubernetes cannot renew on its own, and its certificate expires when
nobody ran chalkctl node renew for it. Its status said so from the renewal point on:
certificates:
node expires <date> the node certificate expires on <date>; this image cannot renew it on its own; run chalkctl node renew w1
chalkctl node renew issues and delivers a new certificate, also to a node whose certificate
expired:
<time> stands for the new expiry, such as 2027-10-10T09:12:44Z. For this command alone,
chalkctl verifies the expired certificate as of its start date, so it trusts the node's old key.
A node renews its certificate through a control plane
explains the risk. Fix the reason the renewal failed too, or the certificate expires again next
year.
With the Kubernetes layer, two more recoveries concern Kubernetes
A worker whose VAR lost its data may hold an expired kubelet certificate. The kubelet certificate was lost with VAR delivers a new one. A control plane that cannot rejoin etcd is covered by Run a highly available control plane.
The node is in maintenance mode¶
The console shows chalkd's certificate fingerprint and the node's addresses, and
chalkctl status fails to verify the node. The node boots an image with no installation to load:
- The installer medium is still attached and comes first in the boot order. Remove the medium or fix the boot order, and reboot.
- The install did not finish. Run
chalkctl installagain. It continues an unfinished install of the node on the same disk.
A node in maintenance mode answers chalkctl logs and chalkctl node reboot with
--fingerprint or --insecure, so you can read why an install failed:
<address> is the address the console shows. Without --address, chalkctl reaches the address
the inventory names, which a node in maintenance mode may not have, because it runs without its
config.
Reinstall the node as the last resort¶
A reinstall gives the node a new disk layout, new keys sealed to its TPM and a fresh VAR, from its config in the inventory's files. Everything on its disks is lost. The node keeps its name and its recovery key, and gets new certificates. Reinstall a node on its machine gives the steps, including the steps of the Kubernetes layer.
The secrets file is lost¶
Without the secrets file or a backup of it, the nodes keep running, but much of their management
is gone for good. Commands with existing client files still work within their roles: status,
logs, disks, node activate, node reboot and upgrade. storage reset still works where it
creates no volume whose fallback is the recovery key. Control planes keep renewing the nodes'
certificates until the node CA expires.
These are gone:
- installing or reinstalling a node, and
config apply; chalkctl node renewand everyrotate;- new client files;
- recovery keys, so a node whose TPM stops unsealing cannot be unlocked.
The client files expire within their validity, a year by default, and nothing can issue new ones. Plan a move to a new cluster before then. Generate new secrets, build new images, install new nodes and move the workloads across. Back up what you cannot rebuild lists the secrets file among the files to keep in a second place.
With the Kubernetes layer, Kubernetes access runs out too
bootstrap and the etcd commands keep working with client files, and the kubelets keep
renewing their certificates. Existing kubeconfigs work until they expire, and no new ones can
be issued.
What next¶
- Troubleshooting starts from a symptom.
- Status, logs and the console reads what the node reports.
- Storage and encryption explains the sealed keys and the fallback.