Skip to content

Set up your workstation

This page is for an operator who prepares the machine they run chalkctl on. At the end you have chalkctl, a repository laid out like the cluster-metal template, the encrypted cluster secrets, and a client file for everyday commands. You also know which credential each command needs.

Before you begin

You need an x86-64 Linux machine. To get chalkctl from the chalkos flake, the machine needs Nix with flakes enabled. A machine without Nix can run a copy of the program with mtools installed.

Manage secrets covers the keys you need to encrypt and back up the cluster's secrets.

The images come from an image author, who builds them with Nix as Your first image flake shows. chalkctl needs the image directories the builds write, and no Nix. Without an image author, take the generic images each chalkos release publishes, as Releases describes, or build images yourself from a template or an example with nix build.

If you are both operator and image author

The operator pages hand these edits of the cluster's flake.nix to the image author. Each is a line or two of Nix:

  • osCA = ./secrets.pub.json; in the arguments of mkImages ties the images and the installer to the cluster, as Tie the images to one cluster shows. The templates set it already.
  • An image with platform = "kvm"; runs the nodes as virtual machines, as List the images shows.
  • chalkos.debug.tools = true; in an image's modules adds the tools kubectl debug uses, as Add the debug tools shows.
  • The options chalkos.router.<feature>.enable leave router features out of an image, as Leave router features out lists.

Get chalkctl from the chalkos flake

chalkctl is the package chalkctl of the chalkos flake, built for x86_64-linux. Take it from the chalkos release your images are built from, because before 1.0 its commands and node API change between releases. Run it from a shell that has it on the PATH:

nix shell github:trevex/chalkos/<tag>#chalkctl
chalkctl --help

<tag> is the release that the image author's flake.nix names in its chalkos input, such as v0.1.0. A cluster's flake can add chalkctl to its development shell instead, as the lab template does, so that nix develop gives the chalkctl of the pinned release.

To run chalkctl on a machine without Nix, download the program from the release, as Download chalkctl and the installer shows, and install mtools, as Requirements describes.

Lay out the repository

A cluster lives in one repository. That of the cluster-metal template looks like this once its image and installer are built and its secrets are generated:

cluster/
├── .gitignore         keeps images/, the installer and nix build's result links out of git
├── flake.nix          the image author's flake, which builds the image and the installer
├── flake.lock         the chalkos revision the flake pins
├── chalkos.yaml       the inventory
├── cluster/           config files every node gets
│   └── cluster.yaml
├── nodes/             each node's own config file
│   ├── cp1.yaml
│   ├── cp2.yaml
│   └── cp3.yaml
├── images/            image directories, outside git
│   └── node-metal     the build, which nix build links here
├── installer          the installer's build, outside git
├── secrets.age        the secrets file, encrypted with age
└── secrets.pub.json   the public half of the secrets file

A larger cluster adds directories of its own. The homelab example gives its router and its Kubernetes nodes separate directories and images, and keeps its NixOS extensions in extensions/.

The inventory, chalkos.yaml, names the cluster's nodes, the address chalkctl reaches each at, the image each runs and the config files each gets:

chalkos.yaml
# The cluster's three nodes, the address chalkctl reaches each at, the image they run and their
# config files: every node gets cluster/, each its own file in nodes/. Replace the addresses with
# free ones of your network, here and in the nodes' files.
apiVersion: chalkctl.chalkos.dev/v1alpha1
kind: Inventory
metadata: {name: cluster}
spec:
  images:
    # nix build .#node-metal -o images/node-metal puts the image here.
    node-metal: {source: {directory: images/node-metal}}
  all: {imageName: node-metal, files: [cluster/*.yaml]}
  nodes:
    cp1: {address: 192.168.1.11, files: [nodes/cp1.yaml]}
    cp2: {address: 192.168.1.12, files: [nodes/cp2.yaml]}
    cp3: {address: 192.168.1.13, files: [nodes/cp3.yaml]}

chalkctl reads chalkos.yaml in the current directory, so you run chalkctl from the repository's root. --inventory or $CHALKOSINVENTORY names an inventory elsewhere, and either takes precedence over ./chalkos.yaml. Paths in the inventory are relative to its directory. Write a node's config covers the inventory and the files it names.

images/ holds the image directories. nix build links a build there, and chalkbuild sign writes a signed copy beside it, such as images/node-metal-signed, for nodes that enforce Secure Boot. Both are outside git: the template's .gitignore lists images/.

Commit the inventory, the config files and secrets.pub.json. A flake that names secrets.pub.json in osCA reads it, and a flake sees only the files git tracks. secrets.age may be committed too, as the next section explains.

Generate the cluster's secrets with age

Manage secrets shows how to create the bundle and its public half. It also covers generation with SOPS.

Build the images with the cluster's OS CA

Give secrets.pub.json to the image author, who names it in the osCA argument of mkImages, as osCA = ./secrets.pub.json;. The image author then builds the images and the installer again. Their chalkd in maintenance mode accepts only clients with a certificate from this cluster's OS CA, so nobody else can install an image on a machine that waits for its install. Tie the images to one cluster describes the argument, and Your first image flake shows the flake that takes it.

Images built without osCA accept any client until they are installed, as an admin. Anyone on the machine's network may then install it, reboot it and run every diagnostic on it, those for operators such as nft-ruleset included. The templates name osCA already. The homelab's flake leaves it out, so that it builds before any cluster has secrets, and Build a home lab adds it.

Keep the services' secrets in Secret documents

Manage secrets shows how to write Secret documents and source their values.

Choose a role for each client file

A client file holds a certificate of one client role, its key and the OS CA. It lets one person run the everyday commands without the secrets file. chalkctl client new issues one from the secrets file:

$ chalkctl client new --name alice --role operator --secrets secrets.age
wrote /home/alice/.config/chalkos/config for alice with the operator role; its certificate expires on 2027-10-23

The certificate is valid for a year unless --ttl gives another validity, such as --ttl 720h. Without --out, the file goes to ~/.config/chalkos/config with mode 0600. Give each person a file of their own, with the least role their work needs:

  • reader reads node information, disks, status and logs.
  • operator may also reboot and upgrade nodes.
  • admin may call every method chalkd serves, but for the one nodes renew their own certificates with.

Nothing revokes a single client file. Take access away describes how to take access away.

Pass the secrets file or a client file

Pass the secrets file with --secrets to the commands that issue certificates or deliver keys and config: install, config apply, client new, recovery-key, node renew and rotate. Run the others with a client file of a high enough role, or with the secrets file, which counts as admin. Commands lists the credential each command needs. A command that needs the secrets file says so when --secrets is missing.

Manage secrets lists the supported file formats and provider URIs.

chalkctl upgrade with an operator client file sends no config, and each node gives the new image its pending config, else carries its own config over, as Operator upgrades carry no config explains.

chalkd on each node enforces the roles. A call that the role does not allow fails with a message that names the role it needs, such as /chalkos.node.v1.NodeService/Reboot needs the operator role. Roles and credentials lists the role each method needs.

Choose which client file chalkctl uses

The commands that need the secrets file read it only from --secrets. chalkctl never looks for a secrets file on its own. The commands that accept a client file take the first credential they find, in the order Where chalkctl finds its credentials lists. With none of them, the command stops and says so.

$CHALKOSCONFIG and $CHALKOSINVENTORY let you keep the files of several clusters apart:

export CHALKOSCONFIG=~/clusters/cluster/alice.json
export CHALKOSINVENTORY=~/clusters/cluster/chalkos.yaml

A client file names no cluster. Its OS CA ties it to one: a node of another cluster fails the TLS handshake, and chalkctl says the node does not accept this client file and asks whether it is another cluster's.

The secrets file, the client files and the images relate as follows:

flowchart LR
    keys["Age identities or SOPS keys"] -->|decrypt| secrets["Secrets file"]
    gen["chalkctl gen secrets"] -->|writes| secrets
    gen -->|writes| pub["secrets.pub.json"]
    pub -->|osCA| images["Images and installer"]
    secrets -->|chalkctl client new| client["Client file of one role"]
    secrets -->|admin certificate for one run| chalkctl["chalkctl"]
    client -->|certificate of its role| chalkctl
    chalkctl -->|mutual TLS| chalkd["chalkd on each node"]

The diagram shows the secrets file at the centre. The keys of its recipients decrypt it. It issues client files, and chalkctl issues itself a one-hour admin certificate from it for each command. Its public half reaches the images, whose chalkd accepts certificates of the OS CA.

Back up what you cannot rebuild

Some files exist only where you keep them: the secrets file and its recipients' keys, the Secure Boot keys, the repository and the previous image. Back up what you cannot rebuild lists them.

Check that it worked

chalkctl validate reads the inventory, the config files and the image directories without connecting to the nodes. Provider-backed service secrets require their provider's keys. Build the image named by the inventory, then run the command in the repository's root.

chalkctl validate

It prints a line per node, as Write a node's config shows. The lines show that chalkctl runs, finds the inventory and reads the image directory. The repository then holds your encrypted cluster secrets file and secrets.pub.json. Your client file lives in ~/.config/chalkos/config.

If validate cannot open chalkos.yaml, run it in the repository's root, or name the inventory with --inventory or $CHALKOSINVENTORY. If a command says that no usable age identity was found, pass the key the secrets file or the age source was encrypted to with --identity.

What next