Skip to content

Development setup

This page sets up a chalkos checkout for development and lists the commands of a development loop: building the tools and images, running the Go tests, regenerating code and pages, formatting and linting, and running a lab from the checkout. Everything runs through Nix, so the only requirements are an x86-64 Linux machine with Nix and flakes enabled, and /dev/kvm for anything that boots a virtual machine.

First change in 10 minutes

These steps take a small Go change from a fresh clone to a commit. The rest of this page explains each command.

  1. Clone the repository and enter its development shell. The first run downloads the toolchain:

    git clone https://github.com/trevex/chalkos
    cd chalkos
    nix develop
    
  2. Run the tests of the package you change, here pkg/inventory:

    go test ./pkg/inventory/...
    
  3. Edit the code and its test, then run the same tests again.

  4. Format the Go code and the Nix files:

    gofmt -l -w cmd pkg test
    nix fmt
    
  5. Build the flake check of what you touched, with its log. For Go code, that is the linter:

    nix build -L .#checks.x86_64-linux.go-lint
    

    Testing maps each part of the repository to its checks. Leave nix flake check for later: it runs every check, the end-to-end tests that boot VMs included.

  6. Commit in the form of Conventional Commits, with a type, a scope and a summary in the imperative:

    git commit -am "fix(inventory): <summary>"
    

What a change needs lists what a pull request must carry.

Enter the development shell

Clone the repository and enter its development shell:

git clone https://github.com/trevex/chalkos
cd chalkos
nix develop

The shell, defined in nix/outputs/shell.nix, provides:

  • Go, gopls, staticcheck, buf with protoc-gen-go, protoc-gen-connect-go and protoc-gen-doc, openssl, jq, nixfmt and virt-firmware;
  • the tools the tests run: QEMU, swtpm, mtools, sbsigntool, dosfstools, cryptsetup, util-linux, e2fsprogs, vde2, socat and iproute2;
  • the variables the tests read: CHALKLAB_OVMF_CODE, CHALKLAB_OVMF_VARS and CHALKLAB_OVMF_VARS_ENROLLED (OVMF firmware, blank and with test Secure Boot keys enrolled), CHALKLAB_SB_KEYS (those keys), CHALKOS_TEST_EFI, CHALKOS_TEST_REPART and CHALKOS_TEST_IMAGE_DEFINITIONS.

Tests that need a tool or variable skip without it, so run them in this shell. The flake checks use the same tools and variables from nix/testing/env.nix.

The Go toolchain

The go directive of go.mod names the Go release that builds chalkos, such as go 1.27.4. The flake input go-overlay provides that release as the Go project publishes it, independent of the Go version nixpkgs ships. nix/overlay/default.nix selects it from go.mod with go-overlay's fromGoModStrict, which fails when go-overlay lacks that exact release, and adds it to nixpkgs as pkgs.chalkos.go, in the scope of chalkos's packages. The development shell, every Go package, the go-unit and go-lint checks and the programs in the images use it, and the shell's gopls and staticcheck, which go-lint runs too, are built with it. mkImage applies the same overlay to the images' nixpkgs, so images built from another nixpkgs still get chalkos's programs built with this Go.

To move to another Go release:

  1. Update go-overlay if it predates the release: nix flake update go-overlay.
  2. Change the go directive in go.mod.
  3. Run nix run .#update-vendor-hash, which sets vendorHash in nix/overlay/build-go-program.nix again if the Go modules' download changed.
  4. Run the go-lint and go-unit checks and fix their findings.

nix run github:purpleclay/go-overlay#goversions lists the releases go-overlay provides. Renovate proposes each new Go release in a pull request of its own, and its fixup workflow does the first and third steps, as Dependency updates describes.

Repository layout

Directory What lives there
api/ The protobuf definitions of the node API, api/chalkos/node/v1/node.proto, and of each layer's API, api/chalkos/kubernetes/v1/kubernetes.proto and api/chalkos/router/v1/router.proto, and of what chalkd and an image's checker exchange, api/chalkos/check/v1/check.proto, and the template of their reference page.
cmd/ Thin mains: chalkctl, chalkd, chalklab, chalkbuild, chalkcfg, chalkos-console, chalkos-networkd-schema, chalkos-render, chalkos-rules, chalkos-storage and docgen.
pkg/ The Go packages, including pkg/api, the code buf generate writes.
modules/node/ The NixOS modules every image is built from.
modules/platforms/ The metal and kvm platforms.
modules/installer/ The installer image and its ISO.
modules/testing/ The test probe and the test image's small partitions.
lib/ mkImage, mkInstaller, mkImages and the module of their arguments, and the layers, the flake's nixosModules.
nix/ Tooling: packages, apps, checks, the development shell, the docs build, and the test images and their secrets in nix/testing/.
templates/lab/ The lab template the quick start uses.
templates/router-lab/ The router lab template the router quick start uses.
templates/cluster-metal/ The template of a cluster on real machines, which the guide Build a cluster on real machines uses.
examples/homelab/ The router and Kubernetes node images with built-in BGP, the installer, and the inventory and YAML for Build a home lab. The example-homelab check builds the images and validates the inventory.
examples/bgp-extension/ The BIRD extension, its example config and golden files, and the node-bird-metal image for Write a BGP extension.
examples/extensions/ An example of extension kinds: a flake that builds an image with the Rack and Hardware extensions and Kubernetes off, an inventory and its node's YAML, which the example-extensions check builds and validates.
test/e2e/ The end-to-end tests, which boot virtual machines.
test/nix/ The Nix evaluation tests.
test/veritysize/ A helper of the image-size check that reads an image's store and hash tree sizes with pkg/verity.
test/fixtures/ Static test data: the e2e clusters' YAML, kind schemas and config documents.
docs/ The documentation site's pages.

Packages and programs describes the Go packages and programs one by one, and Nix code the modules.

Build the tools

The flake's packages build the programs with Nix, as a user gets them:

nix build .#chalkctl
nix build .#chalklab
nix build .#chalkbuild
nix build .#chalkd

chalklab is wrapped with QEMU, OVMF, swtpm and the other tools a lab runs, and with chalkctl, so its closure is larger than chalkctl's. chalkbuild is wrapped with mtools and sbsigntool, which it signs images with. During development, run the commands from the working tree instead:

go run ./cmd/chalkctl --help
go run ./cmd/chalklab --help
go run ./cmd/chalkbuild --help

chalkd only does useful work on a node; the end-to-end tests and a lab run it there.

Build images

A cluster's flake builds its images with chalkos.lib.mkImages, which returns each image by its name, and exposes them as packages. The repository's own flake exposes its test images, which nix/testing/images.nix builds, as packages:

Package What it is
test-image The image test-metal, with small partitions and the test probe; most VM tests boot it.
test-installer The test images' installer.
test-kubernetes-image The image k8s-kvm, with the Kubernetes layer, which the Kubernetes test's control plane and worker boot.
test-kubernetes-ha-image The image k8s-metal, without BGP, which the HA test's control planes boot.
test-router-image The image router-kvm, with the router layer, which the router test's routers boot.
test-upgrade-image, test-unhealthy-image, test-kubernetes-ha-upgrade-image Images the upgrade tests install.
installer An installer of a cluster without an OS CA, which accepts any client until it installs a node.
nix build .#test-image

The result is a directory with the raw image, repart-output.json, repart.d, metadata.json and schemas, as Image and partition layout lists.

Run the Go tests

Run the unit tests in the development shell:

go test ./pkg/... ./cmd/...

None of them needs root. The tests of pkg/kubernetes/vip and pkg/kubernetes/nodeip run themselves again in a new user and network namespace, and a test of pkg/kubernetes/node in a new user and mount namespace, so the kernel must allow unprivileged user namespaces. The install and upgrade tests lay out disk image files with the real sfdisk and systemd-repart and emulate mounts, LUKS, the TPM and the firmware, so they need no loop devices.

go test ./... also runs the end-to-end tests in test/e2e, which boot VMs as soon as the variables they need are set; the development shell sets those of the firmware test. Run them through their flake checks instead, as Testing describes.

Regenerate code and pages

The Go code of the node API and of each layer's API, and their reference page, come from the proto files below api/chalkos/:

buf generate

The reference pages for the commands, the options and the API come from the code:

nix run .#docgen

Both write into the working tree, and the results are committed. The api-generated and docs-generated checks fail when they are out of date.

Format and lint

nix fmt
gofmt -l -w cmd pkg test

nix fmt runs nixfmt over the Nix files; nix fmt -- --ci checks them without changing anything.

Lint the Nix code with statix and with deadnix, which finds unused bindings and arguments:

deadnix .
statix check .

statix.toml lists the statix lints chalkos leaves out, each with its reason. The nix-lint check runs nixfmt, deadnix and statix in the Nix sandbox and fails on any finding.

Lint the Go code, the end-to-end tests included, with go vet and staticcheck:

go vet ./pkg/... ./cmd/... ./test/...
staticcheck ./pkg/... ./cmd/... ./test/...

staticcheck reads staticcheck.conf at the root of the repository, which enables all its checks but a few and gives the reason for each one it leaves out. A finding that is wrong for one line takes a //lint:ignore <check> <reason> comment above that line. The go-lint check runs gofmt, go vet and staticcheck in the Nix sandbox and fails on any finding.

Run a lab from the checkout

A lab runs a cluster's nodes as QEMU VMs with Secure Boot and a TPM, which is the quickest way to see a change on a booted cluster. Create one from the template, pointing its chalkos input at the checkout:

mkdir ~/lab && cd ~/lab
nix flake init -t <checkout>#lab

<checkout> is the absolute path of the repository. In the lab's flake.nix, replace the input's URL:

inputs.chalkos.url = "git+file://<checkout>";

Then follow the template's README, with chalklab and chalkctl now built from the checkout:

git init && git add .
nix develop
chalkctl gen secrets --plaintext
git add . && git commit -m "A chalkos lab"
chalklab create

A git+file input reads the files git tracks, so add new files in the checkout before the lab sees them. After changing the checkout, run nix flake update chalkos in the lab to lock the new state, then upgrade the lab's nodes as the template's README shows, or recreate the lab with chalklab destroy and chalklab create. The quick start walks through a lab step by step.

Serve the documentation

nix run .#docs-serve

This serves the site from the working tree and rebuilds it as pages change. Writing documentation covers the rest of the documentation's tooling.