Skip to content

Bootstrap and get a kubeconfig

This page is for an operator whose nodes are installed and waiting for their cluster. One command starts the cluster on the first control plane, and every other node joins it on its own. At the end every node is Ready in Kubernetes, and you hold a kubeconfig for the API server. The page uses the homelab's control planes node1, node2 and node3, behind the VIP fdc8:a1b2:c3d4:ff::10.

In short:

chalkctl status node1 --secrets=secrets.age
chalkctl bootstrap node1 --secrets=secrets.age
chalkctl kubeconfig --secrets=secrets.age
kubectl --kubeconfig kubeconfig get nodes

Before you begin

  • Every node is installed and back up in normal mode, as Install a node describes.
  • The cluster's Cluster document names the endpoint you chose in Plan a cluster.
  • You hold the secrets file secrets.age. chalkctl bootstrap also works with an admin client file; chalkctl kubeconfig needs the secrets file.
  • kubectl is on your workstation, for the check at the end.

A cluster starts once, on one control plane. After that, nothing runs a join command: each node finds the cluster at the endpoint by itself, as chalkctl bootstrap starts the cluster once explains.

Check that the control planes wait

An installed control plane cannot tell whether it should start the cluster or join one, so it waits. chalkctl status shows it in the Kubernetes line:

kubernetes controlplane: waiting for bootstrap or for the cluster at https://[fdc8:a1b2:c3d4:ff::10]:6443

The URL is the cluster endpoint, here the VIP. A worker reads joined once it has prepared its Kubernetes files, since it has no etcd member to add. Its node ready reads unknown, followed by the error, until its kubelet has registered its node with the API server.

Bootstrap the first control plane

Run chalkctl bootstrap on one control plane:

$ chalkctl bootstrap node1 --secrets=secrets.age
bootstrapping node1; the control plane pulls its images and starts
node1 is bootstrapped; applied 19 objects

chalkd on node1 records the bootstrap on STATE, starts etcd as the cluster's first member and starts the API server, the controller-manager and the scheduler. It then applies chalkos's add-ons: RBAC, kube-proxy, flannel and CoreDNS. Last come the objects of the Addon documents. applied 19 objects counts all of them; the homelab has no Addon documents, so all 19 are chalkos's.

chalkctl waits until the objects are applied, up to --timeout, 20 minutes by default. Most of that time goes into pulling the control plane's images. When --timeout passes first, chalkctl stops with a deadline_exceeded error. The bootstrap stays in place and chalkd keeps applying, so wait and watch chalkctl status node1. Do not bootstrap again: chalkd refuses the second run.

Bootstrap one control plane only. A second bootstrap would start a second etcd cluster, so chalkos refuses it in these cases:

  • On a node that is bootstrapped already, chalkd answers the node is bootstrapped already.
  • On a node whose cluster answers at the endpoint, chalkd answers the cluster's API server answers at https://[fdc8:a1b2:c3d4:ff::10]:6443; this node joins it on its own.
  • On a worker, chalkctl refuses before it sends anything: node4 is not a control-plane node; bootstrap one of the cluster's control-plane nodes.

Watch the other nodes join

Each other control plane finds the cluster at the endpoint, joins etcd and starts its control plane. etcd accepts one learner at a time, so control planes join one after another. While a node joins, its status names the step after joining the cluster at <endpoint>:. In the homelab it reads:

kubernetes controlplane: joining the cluster at https://[fdc8:a1b2:c3d4:ff::10]:6443: checking etcd's members
kubernetes controlplane: joining the cluster at https://[fdc8:a1b2:c3d4:ff::10]:6443: etcd member <id> catches up

<id> is the new member's etcd ID in hexadecimal. A joined control plane reads bootstrapped, the same as the first one:

kubernetes controlplane: bootstrapped, node ready: True, vip standby, control plane current
kubernetes versions: API server 1.37.2, kubelet 1.37.2, control-plane pods current

vip standby appears when the cluster has a VIP and another control plane holds it; the holder reads vip holder. Run a highly available control plane covers the joins of three control planes and what stops one.

A worker's kubelet registers its node at the endpoint with the certificate the worker received at install. Once flannel runs on the node, the node is ready:

kubernetes worker: joined, node ready: True
kubernetes versions: API server 1.37.2, kubelet 1.37.2

Get a kubeconfig

chalkctl kubeconfig writes a kubeconfig for the cluster endpoint, with a client certificate it issues from the secrets file:

$ chalkctl kubeconfig --secrets=secrets.age
wrote kubeconfig; its certificate is valid for 8760h0m0s

The defaults write ./kubeconfig, valid for a year, for the user admin. The file holds the certificate's private key and is written with mode 0600. chalkctl refuses to replace an existing file unless --force is given. Other flags change what it writes:

Flag Default Sets
--out kubeconfig The file to write; - writes to standard output.
--ttl 8760h How long the certificate is valid.
--name admin The user name in the certificate, which the API server's audit log records.
--context the inventory's metadata.name The name of the kubeconfig's cluster, user and context.
--server the cluster endpoint Another URL to reach the API server at, such as a forwarded port. The certificate is still verified for the endpoint.

--context keeps the names apart when you merge the file into a kubeconfig that holds other clusters.

Give someone access to the Kubernetes API

Issue each person a kubeconfig of their own, with their name and the shortest validity your routine tolerates:

$ chalkctl kubeconfig --name=alice --ttl=2160h --out=alice.kubeconfig --secrets=secrets.age
wrote alice.kubeconfig; its certificate is valid for 2160h0m0s

Every kubeconfig chalkctl issues puts its holder in the group chalkos:cluster-admins, which is bound to cluster-admin, so each one grants full access to the cluster.

Nothing revokes a single kubeconfig. It works until its certificate expires or the Kubernetes CA is rotated. Rotate the Kubernetes CAs and keys takes access away from everyone at once, so a short --ttl lets a kubeconfig lapse without a rotation.

Narrower access uses Kubernetes RBAC. The API server accepts only client certificates and service-account tokens, as Configure Kubernetes explains. A cluster admin can issue a client certificate for a user and group of their choice through the Kubernetes certificate signing request API, and bind that user or group to a narrower role.

Check that it worked

List the nodes with the kubeconfig:

$ kubectl --kubeconfig kubeconfig get nodes
NAME    STATUS   ROLES    AGE   VERSION
node1   Ready    <none>   4m    v1.37.2
node2   Ready    <none>   2m    v1.37.2
node3   Ready    <none>   1m    v1.37.2

Every node of the inventory appears, with the Kubernetes release of its image. ROLES is empty because chalkos sets no node-role label. A node may show NotReady for the first minute, until flannel runs on it.

chalkctl status shows each node's Kubernetes line, as in the sections above.

If something goes wrong

Symptom Cause Fix
A control plane stays at waiting for bootstrap or for the cluster at ... after the bootstrap It cannot reach the API server at the endpoint. Check that the endpoint reaches a bootstrapped control plane on port 6443.
A worker's node ready stays unknown Its kubelet cannot register at the endpoint. Check chalkctl logs <node> --unit=kubelet.service.
The bootstrap times out The control plane is still pulling its images. Wait, and watch chalkctl status node1.
chalkctl kubeconfig fails with no Cluster document of the cluster ... names an endpoint The inventory's files give no node a Cluster document. Add the Cluster document to the files every node gets.

Troubleshoot Kubernetes covers further symptoms.

What next