Roadmap and known limits¶
This page lists the work planned for chalkos and the limits of the current code. Nothing here has a date; the plans are ordered roughly by how much else depends on them. Each limit says what happens today, so a cluster can be planned around it. Contribute says what a change that takes one up needs.
Good first changes¶
These changes are small, and a check proves each of them. The code or the docs need each one today:
- Carry the kernel modules of every
NetDevkind. ANetDevdocument's schema acceptsspec.l2tp,spec.macsec,spec.batmanAdvancedandspec.bareUDP, but no group ofchalkos.kernel.groupscarries their modules, so networkd cannot create the device. Add the directoriesnet/l2tpandnet/batman-advand the modulesmacsecandbareudpto thenetworkgroup inmodules/node/kernel.nix, and buildkernel-modulesandimage-size. - Open the NodePort range for UDP.
modules/node/kubernetes/firewall.nixopens ports 30000 to 32767 for TCP alone, as Known limits says. Add the same range toallowedUDPPortRanges, and update Ports and firewall and this page.
Planned work¶
Each item says what chalkos does today and what the plan changes.
Secure Boot key enrolment¶
Today chalkctl secureboot new creates an
operator's PK, KEK and db keys, and
chalkctl secureboot enroll enrols them in
firmware that is in Setup Mode, from the installer or a node's image in maintenance mode. chalkctl
signs the variable updates, so a node never holds a key that could sign for the firmware. Firmware
that is not in Setup Mode gets its db certificate out of band, in the firmware's setup or through a
BMC. chalklab enrols its own lab keys into its VMs' firmware.
chalkos writes no Secure Boot variables on an installed node: it applies no dbx updates and changes no keys there. Such a change alters PCR 7, so the node's next boot asks for its recovery key. No plan covers that yet, and no setting makes a node refuse to install without Secure Boot.
OCI publishing and pulls¶
Today chalkbuild publishes an image as an OCI artefact, and chalkctl install on the installer
and chalkctl upgrade both have the node pull an image the inventory names in a registry,
through the endpoints of a Mirror when the node has one, and
authenticated by a Registry document for each host. The same
Registry documents give the kubelet the credentials of pods' container images, without
imagePullSecrets. An air-gapped Kubernetes cluster pulls its container images from the
cluster registry on its control planes, which
chalkctl registry push fills endpoint by endpoint. The plan adds a check of a signature of the
artefact itself.
An in-cluster upgrade controller¶
Today chalkctl upgrade rolls an upgrade through a
cluster from an operator's machine. The plan adds a controller in the cluster, driven by a
NodeUpgrade resource, that upgrades nodes through chalkd with an
operator certificate, using the OCI pulls above.
Boot loader updates¶
Today an install writes systemd-boot, and an upgrade never changes it: chalkd refuses an upgrade
image that brings a boot loader. A node keeps the systemd-boot it was installed with until it is
reinstalled. The plan adds a flag to chalkctl upgrade that updates the boot loader with an
A/B scheme of its own, so a boot loader that does not start leaves
the old one to fall back to.
More platforms¶
chalkos defines the platforms metal and kvm. Planned are ESXi and cloud platforms, PXE and
HTTP boot, and compressed images for distribution.
More router protocols¶
Today the router layer routes dynamically with BGP alone, through FRR, and routers share no address. The plan adds these protocols, each as a feature an image can leave out:
| Item | What it adds |
|---|---|
| OSPF | OSPFv2 and OSPFv3 through FRR, beside BGP. |
| VRRP | Addresses that move to another router when one fails, and later the sync of connection tracking state between the routers. |
| LLDP | Discovery of the neighbours on the router's links, and announcement of the router to them. |
Smaller items¶
| Item | What it adds |
|---|---|
| Delta transfers | Upgrades that send only the changed parts of the store, instead of the whole store to every node. |
| Signed PCR 11 policies | Keys sealed to the signed image as well as to PCR 7. |
| KMS and SSH unlock | Unlocking encrypted partitions in the initrd through a key management service, or over SSH, besides the TPM and the recovery key. |
| Adding an IP family | Turning a single-stack cluster into a dual-stack one while it runs. |
| etcd backup and restore | Snapshots of etcd and a restore into a new control plane. |
| Replication of the cluster registry | Images pushed to one endpoint of the cluster registry copied to the others, so an endpoint that missed a push, or a new control plane, catches up without another push. |
| Cilium | A CNI besides flannel. |
| Bootstrap through TPM attestation | Nodes that install themselves from a management server that verifies their TPM. |
| External secret sources | The secrets file's contents in Vault or OpenBao. |
Known limits¶
These limits hold in the current code, grouped by area.
Security¶
- Keys of encrypted partitions are sealed to PCR 7 alone. Without Secure Boot enforced, an operator can install any image and it unseals STATE and VAR; Security model explains the consequence.
- Single certificates are not revoked. Access is revoked by rotating the CA that issued it with
chalkctl rotate, and rotations run when an admin starts them, not on a schedule.
Upgrades¶
- An upgrade sends the whole store, its hash tree and the UKI to each node, about 300 MiB for an image with Kubernetes as The image measures it.
- The boot loader is never updated after install, as above.
Networking¶
- A control plane's addresses are pinned when it becomes an etcd
member, so its addresses cannot change while it runs.
Changing them takes
chalkctl etcd leaveand a reinstall. - A cluster's IP families are fixed when it is created.
- The firewall opens etcd's ports 2379 and 2380 on control planes to every source, not only to the other control planes; etcd accepts only certificates of its CA there.
- The firewall opens the NodePort range for TCP only.
- Packets a node receives lose the packet mark bit 0x02000000, which the firewall's rule for the
control plane's ports uses, and on a node that serves the cluster registry the bit 0x04000000
of its port's rule. A network plugin of the cluster's own (
chalkos.cni.provider = "none") must not use them. - The endpoints of the cluster registry do not replicate. Each holds what was pushed to it, and
chalkctl registry listshows where they differ. - The Kubernetes layer's BGP speaker announces the addresses of interfaces alone, such as the node's address and a routed VIP: no pod ranges and no LoadBalancer addresses.
Ports and firewall lists the rules and Kubernetes networking the design behind them.
Storage¶
- Volumes other than VAR are mounted with
nofail, solocal-fs.targetdoes not wait for them, and a service that writes below a volume's mount point needsRequiresMountsFor=on that path, or it writes to the directory underneath while the volume is missing. - chalkos-storage finds every disk and creates and grows its volumes in the initrd at boot. A disk behind a storage controller whose driver the initrd does not carry is not found there, and the node's storage status reports it as failed for that boot.
Storage and encryption explains volumes and their encryption.
Platforms and development¶
- The flake builds for x86-64 Linux alone, so images, chalkctl, chalklab and labs are x86-64 only. The code reads arm64's partition types and boot loader name, but no arm64 image is built or tested.
- Changing a node's platform is a reinstall: chalkd refuses an image of another platform than the one it runs, and translates config for its own.
- CI runs on GitHub-hosted x86-64 runners alone. The end-to-end checks run on a pull request only when a label asks for them; Continuous integration lists the tiers.
- The documentation is not versioned; it describes
main.