Skip to content

Networking

This node has one static address on its uplink, and a route through a second router:

apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-uplink}
spec:
  match: {name: enp1s0}
  network: {address: [192.0.2.11/24], gateway: 192.0.2.1}
  route:
    - {destination: 198.51.100.0/24, gateway: 192.0.2.254}

A chalkos node configures its interfaces with systemd-networkd from documents like this one, filters traffic with nftables, and keeps its clock with chrony. All three come from the node's config and the image, and none of them needs Kubernetes.

With the Kubernetes layer, Kubernetes picks its addresses from these interfaces

A node with Kubernetes picks one address per IP family of the cluster from the addresses its interfaces hold, and control planes can share a VIP. flannel connects pods across nodes with VXLAN. Kubernetes networking describes all three.

Network documents map one to one onto networkd units

A node's network is a set of config documents of three kinds, Network, NetDev and Link. They mirror systemd-networkd's .network, .netdev and .link files one to one:

  • A document's metadata.name is the unit file's name without its suffix, so 10-uplink becomes /run/systemd/network/10-uplink.network and networkd's order by file name holds.
  • Its spec holds networkd's sections and keys in camelCase: [Match] is match, MACAddress= is macAddress, DHCP= is dhcp.
  • Any section may be a list, for a section a unit repeats, such as route above. Any key may be a list, written once per element.

The schemas of these kinds are generated from the gperf tables of the image's own systemd. They accept exactly the sections and keys its networkd accepts, so chalkctl validate refuses a misspelled key, or a value that spans lines, before the node sees it. Config kinds lists every section with its networkd name.

Each key records its networkd spelling. From it, the built-in output networkd renders the units into /run/systemd/network, where they take precedence over the image's. chalkd renders them for every delivery before it changes anything, and chalkos-config.service renders them at every boot before networkd starts. A changed network reaches a running node with chalkctl config apply, which makes networkd reload its .network and .netdev units.

udev applies the .link units, and it applies one to a device only while it handles the device's add event. At boot the kernel announces the devices before chalkos-config.service renders the units, so chalkos-links.service has udev handle the add event of every physical port again once they are rendered, before networkd starts. The unit leaves out virtual devices, such as a bond, a VLAN or a bridge. networkd creates them after the unit has run, and udev applies their .link units then.

A delivery that changes a Link document runs chalkos-links.service again, through the effect links. The kernel renames a port that is up, so a new name takes effect at once, and networkd then matches the port to the units of its new name. An added alternative name or setting takes effect at once too, where the driver accepts it on a port that is up. udev only adds alternative names and leaves out settings a unit no longer names, and the unit does not run once the last Link document is gone, so a removal takes effect at the next boot. A Link document of a virtual device applies when networkd next creates the device.

Each add event also has systemd-sysctl apply the sysctl.d settings of the port again, such as systemd's default rp_filter. A per-link setting of a Network document, such as IPv4ReversePathFilter=, is then overwritten until networkd next configures the link: when a delivery changes the port's Network document, or at the next boot.

Ethernet interfaces without a unit get DHCP

The image brings one unit of its own, NixOS's 99-ethernet-default-dhcp. It runs DHCP on every physical Ethernet interface that no earlier unit matches. A node whose config holds no Network document still comes up with DHCP, and a node that configures one interface keeps DHCP on the others.

The firewall drops what it does not accept

Everything on a node runs on nftables. The NixOS firewall is configured with flushRuleset = false, so reloading it replaces its own table and leaves the tables of other services alone. Its input chain drops what it does not accept, whatever other tables do. On every node it accepts chalkd's port 50000, established and related connections, the loopback interface, ICMP echo requests, ICMPv6 other than redirects and node information queries, and DHCPv6 replies to link-local addresses. A reverse-path check drops packets from a source address the node has no route back to. Ports and firewall lists each port and who needs it.

A router image has no NixOS firewall. The router layer's own table, rendered from the node's router documents, takes its place, as The router's firewall describes.

With the Kubernetes layer, Kubernetes opens more ports

A node with Kubernetes also accepts the kubelet's port, the NodePort range and VXLAN sent to its own addresses. A control plane accepts the API server's and etcd's ports. Kubernetes networking explains how.

chrony keeps the clock with authenticated servers

A node keeps its clock with chrony. By default it synchronises with the Physikalisch-Technische Bundesanstalt's servers ptbtime1.ptb.de to ptbtime3.ptb.de, authenticated with NTS. A TimeSync document names other servers, each with NTS unless nts: false:

apiVersion: time.chalkos.dev/v1alpha1
kind: TimeSync
metadata: {name: default}
spec:
  servers:
    - {host: nts.example.org}
    - {host: ntp.example.org, nts: false}

The built-in output chrony renders the servers into a sources file that chrony reads, so one image serves nodes with different servers. dhcp: true adds the servers that DHCP announces, which are not authenticated. chrony keeps its NTS cookies on VAR, so after a reboot it needs no new key exchange with each server. A node runs no NTP server for other machines, except a router with a TimeServer, as Serve time with NTP describes.

The clock decides which certificates a node accepts, as The clock decides whether a certificate is valid explains.

Limits

  • The network kinds accept the keys of the image's own networkd. A key that only a newer systemd knows is refused until the image carries that systemd.
  • The firewall opens only the ports chalkos's modules open. A service an image adds opens its ports in the image's NixOS modules. On a router image, a FirewallPolicy to local opens them.
  • Time servers that DHCP announces are not authenticated.