Set up your workstation¶
This page is for an operator who prepares the machine they run chalkctl on. At the end you have
chalkctl, a repository laid out like the
cluster-metal template, the
encrypted cluster secrets, and a client file for everyday commands. You also know which
credential each command needs.
Before you begin¶
You need an x86-64 Linux machine. To get chalkctl from the chalkos flake, the machine needs Nix with flakes enabled. A machine without Nix can run a copy of the program with mtools installed.
Manage secrets covers the keys you need to encrypt and back up the cluster's secrets.
The images come from an image author, who builds them with Nix as
Your first image flake shows. chalkctl needs the image directories the
builds write, and no Nix. Without an image author, take the generic images each chalkos release
publishes, as Releases describes, or build images yourself from a
template or an example with nix build.
If you are both operator and image author
The operator pages hand these edits of the cluster's flake.nix to the image author. Each is a
line or two of Nix:
osCA = ./secrets.pub.json;in the arguments ofmkImagesties the images and the installer to the cluster, as Tie the images to one cluster shows. The templates set it already.- An image with
platform = "kvm";runs the nodes as virtual machines, as List the images shows. chalkos.debug.tools = true;in an image's modules adds the toolskubectl debuguses, as Add the debug tools shows.- The options
chalkos.router.<feature>.enableleave router features out of an image, as Leave router features out lists.
Get chalkctl from the chalkos flake¶
chalkctl is the package chalkctl of the chalkos flake, built for x86_64-linux. Take it from the
chalkos release your images are built from, because before 1.0 its commands and node API change
between releases. Run it from a shell that has it on the PATH:
<tag> is the release that the image author's flake.nix names in its chalkos input, such as
v0.1.0. A cluster's flake can add chalkctl to its development shell instead, as the lab
template does, so that nix develop gives the chalkctl of the pinned release.
To run chalkctl on a machine without Nix, download the program from the release, as Download chalkctl and the installer shows, and install mtools, as Requirements describes.
Lay out the repository¶
A cluster lives in one repository. That of the cluster-metal template looks like this once its
image and installer are built and its secrets are generated:
cluster/
├── .gitignore keeps images/, the installer and nix build's result links out of git
├── flake.nix the image author's flake, which builds the image and the installer
├── flake.lock the chalkos revision the flake pins
├── chalkos.yaml the inventory
├── cluster/ config files every node gets
│ └── cluster.yaml
├── nodes/ each node's own config file
│ ├── cp1.yaml
│ ├── cp2.yaml
│ └── cp3.yaml
├── images/ image directories, outside git
│ └── node-metal the build, which nix build links here
├── installer the installer's build, outside git
├── secrets.age the secrets file, encrypted with age
└── secrets.pub.json the public half of the secrets file
A larger cluster adds directories of its own. The
homelab example gives its router
and its Kubernetes nodes separate directories and images, and keeps its NixOS extensions in
extensions/.
The inventory, chalkos.yaml, names the cluster's nodes, the address chalkctl reaches each at,
the image each runs and the config files each gets:
# The cluster's three nodes, the address chalkctl reaches each at, the image they run and their
# config files: every node gets cluster/, each its own file in nodes/. Replace the addresses with
# free ones of your network, here and in the nodes' files.
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: cluster}
spec:
images:
# nix build .#node-metal -o images/node-metal puts the image here.
node-metal: {source: {directory: images/node-metal}}
all: {imageName: node-metal, files: [cluster/*.yaml]}
nodes:
cp1: {address: 192.168.1.11, files: [nodes/cp1.yaml]}
cp2: {address: 192.168.1.12, files: [nodes/cp2.yaml]}
cp3: {address: 192.168.1.13, files: [nodes/cp3.yaml]}
chalkctl reads chalkos.yaml in the current directory, so you run chalkctl from the repository's
root. --inventory or $CHALKOSINVENTORY names an inventory elsewhere, and either takes
precedence over ./chalkos.yaml. Paths in the inventory are relative to its directory.
Write a node's config covers the inventory and the files it names.
images/ holds the image directories. nix build links a build there, and
chalkbuild sign writes a signed copy beside it, such as images/node-metal-signed, for nodes
that enforce Secure Boot. Both are outside git: the template's .gitignore lists images/.
Commit the inventory, the config files and secrets.pub.json. A flake that names
secrets.pub.json in osCA reads it, and a flake sees only the files git tracks. secrets.age
may be committed too, as the next section explains.
Generate the cluster's secrets with age¶
Manage secrets shows how to create the bundle and its public half. It also covers generation with SOPS.
Build the images with the cluster's OS CA¶
Give secrets.pub.json to the image author, who names it in the osCA argument of mkImages, as
osCA = ./secrets.pub.json;. The image author then builds the images and the installer again.
Their chalkd in maintenance mode accepts only clients with a certificate from this cluster's OS
CA, so nobody else can install an image on a machine that waits for its install.
Tie the images to one cluster
describes the argument, and Your first image flake shows the flake that
takes it.
Images built without osCA accept any client until they are installed, as an admin. Anyone on
the machine's network may then install it, reboot it and run every diagnostic on it, those for
operators such as nft-ruleset included. The templates name osCA already. The homelab's flake
leaves it out, so that it builds before any cluster has secrets, and
Build a home lab adds it.
Keep the services' secrets in Secret documents¶
Manage secrets shows how to
write Secret documents and source their values.
Choose a role for each client file¶
A client file holds a certificate of one client role, its key and the OS CA. It lets one person
run the everyday commands without the secrets file. chalkctl client new issues one from the
secrets file:
$ chalkctl client new --name alice --role operator --secrets secrets.age
wrote /home/alice/.config/chalkos/config for alice with the operator role; its certificate expires on 2027-10-23
The certificate is valid for a year unless --ttl gives another validity, such as --ttl 720h.
Without --out, the file goes to ~/.config/chalkos/config with mode 0600. Give each person a
file of their own, with the least role their work needs:
- reader reads node information, disks, status and logs.
- operator may also reboot and upgrade nodes.
- admin may call every method chalkd serves, but for the one nodes renew their own certificates with.
Nothing revokes a single client file. Take access away describes how to take access away.
Pass the secrets file or a client file¶
Pass the secrets file with --secrets to the commands that issue certificates or deliver keys and
config: install, config apply, client new, recovery-key, node renew and rotate. Run the
others with a client file of a high enough role, or with the secrets file, which counts as admin.
Commands lists the credential each command needs. A command
that needs the secrets file says so when --secrets is missing.
Manage secrets lists the supported file formats and provider URIs.
chalkctl upgrade with an operator client file sends no config, and each node gives the new image
its pending config, else carries its own config over, as
Operator upgrades carry no config
explains.
chalkd on each node enforces the roles. A call that the role does not allow fails
with a message that names the role it needs, such as
/chalkos.node.v1.NodeService/Reboot needs the operator role.
Roles and credentials lists the role each method needs.
Choose which client file chalkctl uses¶
The commands that need the secrets file read it only from --secrets. chalkctl never looks for a
secrets file on its own. The commands that accept a client file take the first credential they
find, in the order
Where chalkctl finds its credentials
lists. With none of them, the command stops and says so.
$CHALKOSCONFIG and $CHALKOSINVENTORY let you keep the files of several clusters apart:
export CHALKOSCONFIG=~/clusters/cluster/alice.json
export CHALKOSINVENTORY=~/clusters/cluster/chalkos.yaml
A client file names no cluster. Its OS CA ties it to one: a node of another cluster fails the TLS handshake, and chalkctl says the node does not accept this client file and asks whether it is another cluster's.
The secrets file, the client files and the images relate as follows:
flowchart LR
keys["Age identities or SOPS keys"] -->|decrypt| secrets["Secrets file"]
gen["chalkctl gen secrets"] -->|writes| secrets
gen -->|writes| pub["secrets.pub.json"]
pub -->|osCA| images["Images and installer"]
secrets -->|chalkctl client new| client["Client file of one role"]
secrets -->|admin certificate for one run| chalkctl["chalkctl"]
client -->|certificate of its role| chalkctl
chalkctl -->|mutual TLS| chalkd["chalkd on each node"]
The diagram shows the secrets file at the centre. The keys of its recipients decrypt it. It issues client files, and chalkctl issues itself a one-hour admin certificate from it for each command. Its public half reaches the images, whose chalkd accepts certificates of the OS CA.
Back up what you cannot rebuild¶
Some files exist only where you keep them: the secrets file and its recipients' keys, the Secure Boot keys, the repository and the previous image. Back up what you cannot rebuild lists them.
Check that it worked¶
chalkctl validate reads the inventory, the config files and the image directories without
connecting to the nodes. Provider-backed service secrets require their provider's keys. Build
the image named by the inventory, then run the command in the repository's root.
It prints a line per node, as
Write a node's config shows. The lines show that chalkctl
runs, finds the inventory and reads the image directory. The repository then holds your encrypted cluster secrets
file and secrets.pub.json. Your client file lives in ~/.config/chalkos/config.
If validate cannot open chalkos.yaml, run it in the repository's root, or name the inventory
with --inventory or $CHALKOSINVENTORY. If a command says that no usable age identity was
found, pass the key the secrets file or the age source was encrypted to with --identity.
What next¶
- Write a node's config describes the inventory and the config documents of each node.
- Install a node installs the first node.
- Security model explains the CAs, the roles and how a node in maintenance mode is trusted.