Troubleshoot Kubernetes¶
This page is for an operator whose cluster misbehaves on chalkos nodes. Each section starts from a
symptom, names the likely cause and links the page with the fix. Most symptoms show in the node's
Kubernetes line of chalkctl status, the others in kubectl.
Troubleshooting covers the node itself: its boot, disks, upgrades
and node certificates.
Before you begin¶
You need the secrets file or a reader client file for chalkctl status and chalkctl logs, and a
kubeconfig for kubectl, as Bootstrap and get a kubeconfig
describes.
Run chalkctl status on the node first, and read its Kubernetes line:
The line starts with kubernetes worker: or kubernetes controlplane:, followed by the node's
state. Read the Kubernetes
line
explains the rest of it. Then find the symptom below.
The node has no address¶
The Kubernetes line reads:
kubernetes worker: preparation failed: no ipv4 node address matches <selector> (the node has <addresses>), 5m0s after chalkos-node-addresses.target
<selector> names what the node looked for: validSubnets with the subnets, nodeIP with a
fixed address, or the default filter. <addresses> lists the node's addresses of that family,
each with its interface. The line names ipv6 when the IPv6 address is missing. The node waited
spec.nodeIP.timeout of the Cluster document, 300 seconds by default and shown as 5m0s. It
looked for an address that matches nodeIP.addresses or nodeIP.validSubnets of its Node
document, or the cluster's nodeIP.validSubnets. It found none, so it runs neither the kubelet
nor static pods.
Check the node's Network documents and its DHCP lease, and that the subnets include the network
the node is on. Configure the
interfaces shows the
documents, and Configure Kubernetes shows nodeIP. Deliver the corrected
documents, or reboot the node once the network is right:
Two other preparation failures name the address too:
flannel needs the node's IPv4 and IPv6 addresses on one interface; ...appears on a dual-stack node whose two addresses sit on different interfaces. Move both onto one interface in theNetworkdocuments.pinned address <address> is not presentappears on a control plane that booted without an address it was pinned to when it joined etcd. A control plane lost its pinned address covers the fix.
A control plane waits for bootstrap¶
The Kubernetes line reads:
Every control plane reads this before the cluster exists. Run
chalkctl bootstrap on exactly one of them, as
Bootstrap and get a kubeconfig describes.
After the bootstrap, the line means the node finds no API server of its cluster at the endpoint. Check that:
- the endpoint's name resolves from the node;
- the VIP is held, when the endpoint is a VIP: another control plane's line shows
vip holder; - the node reaches port 6443 at the endpoint, through any load balancer in between.
Choose the cluster endpoint lists what the endpoint must reach.
A control plane does not finish joining¶
A joining control plane names its current step, or its last error, after
joining the cluster at <endpoint>:. Two states point to a cause:
etcd member <id> catches upfor a long time: the node cannot reach the other members on port 2380, or its etcd is still copying a large database. Check the network between the control planes.etcd has a member cp3 already; remove it with chalkctl etcd remove-member cp3: a member is left over from an earlier install of the node. Run the command the line names. The node tries again every 10 seconds and joins.
A control plane cannot rejoin etcd lists every state that holds a control plane back, with its fix.
kubectl logs and exec fail on a node¶
kubectl logs, kubectl exec and kubectl port-forward fail for pods on one node, with a TLS
error from that node's kubelet. Other commands work. The API server checks the kubelet's serving
certificate when it connects, and that node has none yet. Its request for one is pending, as the
next section describes.
Kubelet certificate requests stay pending¶
Each kubelet requests a serving certificate through a CertificateSigningRequest. chalkd on every control plane approves a request that meets three conditions:
- The node asks for its own name.
- The request is for server use only.
- It names only addresses and names that the node's Node object reports.
chalkd leaves every other request pending. List the requests:
A request of signer kubernetes.io/kubelet-serving whose condition stays Pending was refused.
The chalkd journal of a control plane names the reason once per request:
The line reads leaving certificate request <name> pending: <reason>. The most common reasons:
node w1 does not report the address <address>: the kubelet asks for an address the Node lacks. Comparekubectl get node w1 -o widewith the node'sNetworkandNodedocuments.node w1 has no Node object: the kubelet has not registered its Node yet.node w1 is revoked since <time>: the node is revoked, as A node's writes are denied as revoked describes.
chalkd looks at the pending requests every 10 seconds and checks each one again, so it approves a request once the Node reports what the request asks for. chalkd approves no client certificate requests. The controller-manager approves those when a kubelet renews its own client certificate. Kubelets renew their own certificates explains both certificates.
Pods on different nodes cannot reach each other¶
Pod traffic between nodes travels in flannel's VXLAN, on UDP port 8472. Three causes stop it:
- A node whose address lies outside
vxlanSourceSubnetsof theFlanneldocument refuses to prepare, and its line saysthe node's address ... is outside vxlanSourceSubnets. The other nodes would drop its VXLAN. Widen the ranges in theFlanneldocument. chalkctl refuses an inventory whose nodes hold differentFlannelspecs, so every node gets the same ranges. - Small packets pass and large ones hang, as with TLS handshakes or large responses. The VXLAN MTU
is too large for the network. Set
spec.mtuof theFlanneldocument to the network's MTU, minus 20 when IPv6 is in use. IPv6 VXLAN takes 70 bytes of each packet, and IPv4 VXLAN 50. - A firewall between the nodes drops UDP port 8472.
Set the pod network's MTU and VXLAN sources shows the document.
A Node has no pod range¶
A worker's Kubernetes line reads:
kubernetes worker: joined: the Node has no ipv6 pod range; the control plane allocates ranges of the families the cluster was created with, node ready: <ready>
<ready> is the Node's Ready condition, as in any worker's line. The controller-manager
allocates pod ranges of the families the control planes run with. This worker's Cluster
document names a family the cluster was not created with. That happens when a new worker is
installed after ipFamilies changed in the inventory. Running nodes refuse that change, but an
install has no running config to compare with. Restore the original ipFamilies and reinstall
the worker, as Reinstall a node on its
machine describes. Choose the IP families
and ranges explains
why.
A control plane refuses the cluster's families¶
A control plane's Kubernetes line reads:
kubernetes controlplane: preparation failed: the cluster was created with families [ipv4]; ipFamilies is [ipv4 ipv6]; changing the families of a running cluster is not supported
When a control plane joins etcd, it pins an address of each family the cluster was created with,
the primary one first. At every boot it compares them with ipFamilies of its Cluster document.
This node's document lists other families, or the same ones in another order, so it runs neither
etcd nor the kubelet.
Set ipFamilies back to the families after created with, in that order, and deliver the config
with chalkctl config apply cp1 --secrets=secrets.age. A node whose running config holds the
changed families already refuses that change too. Then take it out of etcd with
chalkctl etcd leave cp1 --force and reinstall it, as A control plane lost its pinned
address describes.
The kubelet certificate was lost with VAR¶
A worker's kubelet renews its client certificate itself and keeps it on VAR. A worker whose VAR
lost its data falls back to the kubelet certificate in its Kubernetes share on STATE, which it
received at install. When that one expired too, the kubelet cannot join, and the certificates
section of chalkctl status w1 says:
certificates:
...
kubelet client expires 2026-09-14 expired; deliver a new one with chalkctl config apply w1 --kubernetes-share
Deliver a new share with a new kubelet certificate:
chalkctl issues the certificate from the Kubernetes CA in the secrets file. Certificates lists every certificate and its lifetime.
A node's writes are denied as revoked¶
The node's Node is gone, and its status shows a failing renewal that names a revocation:
kubernetes worker: joined, node ready: unknown: nodes "w1" not found
certificates:
node expires 2027-10-23 renewal failing: the control plane at 192.168.100.11:50000: permission_denied: node w1 is revoked since 2026-10-23T22:42:25Z; chalkctl node unrevoke w1 lets it renew again; expires 2027-10-23T22:37:07Z
The API server answers every write by the node's kubelet, its Node registration included, with:
nodes "w1" is forbidden: ValidatingAdmissionPolicy 'chalkos-revoked-nodes' with binding 'chalkos-revoked-nodes' denied request: node w1 is revoked since 2026-10-23T22:42:25Z
The name w1 is in the cluster's revoked nodes, which chalkctl status cp1 lists under
revoked nodes:. Every machine of that name is shut out, a node installed afresh included.
- A new node that reuses the name joins once the name is unrevoked. Rotate
os-caandkubernetes-ca, then runchalkctl node unrevoke w1, as Reuse a removed node's name describes. - A node revoked by mistake comes back with
chalkctl node unrevoke w1 --force, which lets its old certificates work again. Then reboot it withchalkctl node reboot w1 --no-drain, because its kubelet registers its Node only as it starts.
What next¶
- Troubleshooting covers the node: boot, disks, upgrades and node certificates.
- Status, logs and the console explains each line of
chalkctl statusand the units to read. - Run a highly available control plane covers etcd's members and how to bring a control plane back.
- The control plane explains bootstrap, join and the certificates.