Continuous integration¶
Every change reaches main through a pull request, and CI builds the flake checks of
Testing on GitHub-hosted runners, which have /dev/kvm. The checks are the same
ones a developer builds with nix build; CI only decides which of them run for an event. This
page explains the tiers, the labels that run the end-to-end checks on a pull request, the nightly
run and the binary cache.
Pull requests¶
A pull request targets main, and each of its commits is a
Conventional Commit: a rebase merge puts the commits on
main as they are, and the changelog comes from them. Branch protection requires the result
check, which passes when every check of the run passed, and a review from the code owners of
.github/CODEOWNERS. The pull request template asks for the end-to-end labels the change needs.
Tiers¶
nix/checks/tiers.nix puts every flake check in one of three tiers, and the workflow
.github/workflows/ci.yml reads them from the flake output ciTiers:
| Tier | Checks | Runs on | Jobs |
|---|---|---|---|
fast |
the evaluation tests, linters, generated files, the docs, the Go unit tests and the namespace checks | every pull request, every push to main and the nightly run |
one job, fast, of at most 90 minutes |
images |
the image builds and their checks, the templates, fixtures, and the NixOS VM checks frr-reload and image-checker |
every pull request, every push to main and the nightly run |
one job per check, of at most 120 minutes |
e2e |
every e2e-* check, the example-* checks, which build the examples' images that no other check shares, and the release-image-* checks, which build the release images |
every push to main, the nightly run, and a pull request by its labels |
one job per check, of at most 180 minutes |
A job whose build fails builds once more, which rebuilds only what failed, because a download from a module proxy or a registry fails now and then. The job's log warns of the retry.
A check in no tier, in two tiers, or a name in a tier that is no check fails the evaluation of
the checks, so a new check cannot be left out of CI. The e2e tier takes the checks of the
labels in e2eLabels of tiers.nix: every check whose name starts with e2e- or
release-image-, and the examples' checks. A new example check goes in the examples list there; a new check of another
kind goes in the fast or images list. A check goes in fast when it builds no image and boots
no VM, and in images when it builds or boots images that other checks share.
The ci-tiers check also wants each label of e2eLabels in .github/labels.yml, so a new
end-to-end check needs its label there too, and the label ci-failure.
End-to-end labels¶
A pull request runs no end-to-end check unless a label asks for it:
| Label | Runs |
|---|---|
e2e |
every end-to-end check |
e2e:<name> |
the check e2e-<name>, such as e2e:upgrade for e2e-upgrade |
e2e:examples |
the checks of the examples: example-homelab, example-extensions, example-bgp-extension and example-bgp-extension-validator-docs |
e2e:release |
the release-image-* checks, which build the release images |
Add e2e when a change touches boot, the install, upgrades or a layer, e2e:<name> when one
check covers the change, e2e:examples when it touches examples/, and e2e:release when it
touches nix/release.nix or what the release images record. Adding any label, an end-to-end one or another, runs the workflow again:
the fast and image tiers, and the end-to-end checks the pull request's labels name. Only
people with triage access to the repository can add labels, so a pull request from a fork boots
no end-to-end VM before a maintainer has read it. The release pull request always carries e2e,
as Release chalkos describes.
Renovate's pull requests and the weekly update of flake.lock carry their labels from the
start, as Dependency updates describes.
The workflow .github/workflows/labels.yml creates or updates the repository's labels from
.github/labels.yml when that file changes on main.
Nightly run and failures¶
The workflow .github/workflows/nightly.yml runs every tier on main each night, unless a run
of a push already passed on main's latest commit. A run started by hand always builds. When a check fails on a push to main or in the nightly run, CI opens an issue
labelled ci-failure, or comments on the open one, with a link to the run.
Binary cache¶
CI caches with no setup. Every job runs Hestia, a Nix binary
cache kept in the repository's GitHub Actions cache: Nix substitutes from it what an earlier run
built, and at the end of the job Hestia uploads what the job built, in deduplicated chunks. Paths
that cache.nixos.org serves stay out of it.
The Actions cache holds 10 GB for the whole repository. GitHub evicts the entries used least
recently once it is full, and any entry unused for 7 days; a job rebuilds what was evicted. The
workflow .github/workflows/cache-gc.yml deletes each day the paths that no branch's latest run
used. A run on main writes to main's scope, which every pull request reads; a pull request's
run writes to a scope of its own, which only that pull request reads and which goes when its
branch does, so a pull request cannot change what main substitutes. The cache's token lasts
about six hours, longer than any job's timeout. The cache serves CI alone; it is not reachable
from a developer's machine.
A Cachix binary cache can serve local builds too. Every job reads it
once the repository names it in the Actions variable CACHIX_NAME, and runs on main push to
it with the secret CACHIX_AUTH_TOKEN. Pull requests never push to it. A developer reads it
with cachix use <name>.
The jobs of the release workflow run on main and read both caches, which only
runs on main write to, but write to neither. They rebuild whatever the caches miss, so a release
needs neither cache. The composite action .github/actions/nix installs Nix and
sets up the caches for the jobs of both workflows.
Release images¶
nix/release.nix builds the images a release of chalkos publishes, the flake output
releaseImages. Each published Kubernetes minor of modules/node/kubernetes/releases.nix, the
minors with publish = true, and the router get one image on metal and one on KVM. The images
carry no OS CA and no Secure Boot signature, record the tag v<version> of version.nix and the
commit's revision, and annotate their manifests as
Image layout describes.
Each image is named after the repository below ghcr.io/trevex/chalkos/ that a release pushes
it to, and carries that repository and its tag:
nix eval --json .#releaseImages.x86_64-linux \
--apply 'builtins.mapAttrs (_: image: { inherit (image) repository tag; })'
{
"kubernetes-1.37-kvm": { "repository": "ghcr.io/trevex/chalkos/kubernetes-1.37-kvm", "tag": "v0.1.0" },
"kubernetes-1.37-metal": { "repository": "ghcr.io/trevex/chalkos/kubernetes-1.37-metal", "tag": "v0.1.0" },
"router-kvm": { "repository": "ghcr.io/trevex/chalkos/router-kvm", "tag": "v0.1.0" },
"router-metal": { "repository": "ghcr.io/trevex/chalkos/router-metal", "tag": "v0.1.0" }
}
A Kubernetes image's own name, which its os-release names, leaves out the minor:
kubernetes-metal and kubernetes-kvm. chalkctl upgrade --image then finds the nodes of a
previous minor by the image's name. Since the names hold dots, a build quotes them:
The fast check release-eval evaluates every release image and holds it to its repository, tag,
provenance and annotations without building it. Each release-image-<name> check, with the dots
of the name as dashes, builds one image and its OCI image layout and compares the manifest's
annotations with the image's. They run in the e2e tier, one job each, as four full images are
too heavy for every pull request.
Reproduce a job¶
Each job runs nix build of its checks, so a failure reproduces locally with the same command.
Build one check with its log:
The fast job builds every check of its tier in one nix build --keep-going; the log names the
check that failed.