Kubernetes¶
This page is for operators of a cluster with the Kubernetes layer who read a node's units and pods,
a refusal of chalkctl validate or the Kubernetes line of chalkctl status. It lists what runs
on each type of node, the flags chalkos refuses in extraArgs and kubeletExtraArgs, the
objects the control planes apply and the states of the Kubernetes line.
How Kubernetes runs on a node explains the mechanism.
Components by node type¶
spec.type of a node's Node document sets its type.
| Type | Runs |
|---|---|
controlplane |
containerd, the kubelet, and etcd, the API server, the controller-manager and the scheduler as static pods |
worker |
containerd and the kubelet |
| none, on an image without Kubernetes | No Kubernetes at all; the image leaves it out |
Protected API server flags¶
extraArgs.kube-apiserver of the Cluster document refuses these flags, also spelt with _ for
-. They decide who may do what.
| Flag | chalkos sets |
|---|---|
anonymous-auth |
Anonymous requests reach only /livez, /readyz and /healthz, through the authentication configuration |
authentication-config |
The API server's authentication configuration, which chalkos writes |
authorization-mode |
Node,RBAC |
enable-bootstrap-token-auth |
false |
Protected kubelet flags¶
kubeletExtraArgs of a Node document refuses these flags, also spelt with _ for -. They set
where the kubelet finds its configuration, credentials, credential provider and static pods, the
node's name, addresses, labels and taints, and who may use the kubelet's API. A refused
node-labels or register-with-taints names the Node document's labels and taints instead.
anonymous-authauthentication-token-webhookauthorization-modebootstrap-kubeconfigcert-dirclient-ca-fileconfigconfig-dirhostname-overrideimage-credential-provider-bin-dirimage-credential-provider-configkubeconfignode-ipnode-labelspod-manifest-pathread-only-portregister-with-taintsrotate-certificatestls-cert-filetls-private-key-file
A kubelet flag's name and value also hold no whitespace and no backslash, because the flags reach the kubelet through an environment file that systemd splits at whitespace.
Conflicting API server flags¶
The API server refuses to start with these flags next to flags chalkos sets, so
extraArgs.kube-apiserver refuses them too.
| Flags | Conflict with |
|---|---|
oidc-issuer-url, oidc-client-id, oidc-ca-file, oidc-username-claim, oidc-username-prefix, oidc-groups-claim, oidc-groups-prefix, oidc-signing-algs, oidc-required-claim |
authentication-config |
authorization-config, authorization-policy-file, authorization-webhook-config-file |
authorization-mode=Node,RBAC |
admission-control |
enable-admission-plugins |
service-account-signing-endpoint |
service-account-signing-key-file and service-account-key-file |
Add-ons¶
A bootstrapped control plane applies these objects in this order, with server-side apply as field
manager chalkd. chalkd renders the first four from the node's cluster.
| Order | Objects |
|---|---|
| 1 | RBAC: chalkos:cluster-admins binds the group chalkos:cluster-admins of admin kubeconfigs to cluster-admin; chalkos:selfnodeclient lets kubelets renew their own client certificates; chalkos:apiserver-kubelet gives the API server access to the kubelets |
| 2 | kube-proxy, as a DaemonSet in nftables mode |
| 3 | The pod network: flannel, unless chalkos.cni.provider is none |
| 4 | CoreDNS, at the primary family's address of dnsIPs |
| 5 | The objects of the Addon documents: each document's spec.objects in their order, and the documents in the order of their names |
The admission policy chalkos-revoked-nodes is not among these objects. A control plane applies it
whenever the list of revoked nodes changes.
Kubernetes status states¶
The Kubernetes line of chalkctl status names the node's kind and one of these states.
| State | Meaning |
|---|---|
no share |
The node holds no Kubernetes share yet; it gets one at install or with chalkctl config apply --kubernetes-share |
preparing |
The preparation of the node's addresses, certificates and kubelet flags runs |
waiting for bootstrap or for the cluster at <endpoint> |
A control plane that found no cluster to join yet |
joining the cluster at <endpoint>: <step> |
A join that runs, with its step or its last error |
bootstrapped |
A control plane that is an etcd voter, by bootstrap or join |
bootstrapped: <problem> |
A control plane whose etcd member belongs to another cluster than the other control planes' members |
joined |
A worker |
joined: <problem> |
A worker whose Node lacks a pod range of one of the cluster's families |
preparation failed: <reason> |
The preparation stopped, for example because no address matched |
left etcd; reinstall the node to join the cluster again |
A control plane after chalkctl etcd leave |
etcd data missing: restore etcd or reinstall the node |
A bootstrapped control plane whose etcd data is gone, as after VAR was lost |
After the state, the line shows whether the node's Node is Ready. A control plane adds its VIP
role, vip holder or vip standby, and control plane current when its API server and etcd
serve the current certificates and the API server answers ready.
Sources: pkg/kubernetes/cluster.go, pkg/kubernetes/manifests/manifests.go,
pkg/kubernetes/addons/addons.go, pkg/chalkd/layers/kubernetes/status.go,
pkg/chalkd/layers/kubernetes/join.go and modules/node/kubernetes.