Networking¶
This node has one static address on its uplink, and a route through a second router:
apiVersion: networkd.chalkos.dev/v1alpha1
kind: Network
metadata: {name: 10-uplink}
spec:
match: {name: enp1s0}
network: {address: [192.0.2.11/24], gateway: 192.0.2.1}
route:
- {destination: 198.51.100.0/24, gateway: 192.0.2.254}
A chalkos node configures its interfaces with systemd-networkd from documents like this one, filters traffic with nftables, and keeps its clock with chrony. All three come from the node's config and the image, and none of them needs Kubernetes.
With the Kubernetes layer, Kubernetes picks its addresses from these interfaces
A node with Kubernetes picks one address per IP family of the cluster from the addresses its interfaces hold, and control planes can share a VIP. flannel connects pods across nodes with VXLAN. Kubernetes networking describes all three.
Network documents map one to one onto networkd units¶
A node's network is a set of config documents of three kinds,
Network, NetDev and
Link. They mirror systemd-networkd's .network, .netdev and
.link files one to one:
- A document's
metadata.nameis the unit file's name without its suffix, so10-uplinkbecomes/run/systemd/network/10-uplink.networkand networkd's order by file name holds. - Its
specholds networkd's sections and keys in camelCase:[Match]ismatch,MACAddress=ismacAddress,DHCP=isdhcp. - Any section may be a list, for a section a unit repeats, such as
routeabove. Any key may be a list, written once per element.
The schemas of these kinds are generated from the gperf tables of the image's own systemd. They
accept exactly the sections and keys its networkd accepts, so
chalkctl validate refuses a misspelled key, or a value
that spans lines, before the node sees it. Config kinds lists
every section with its networkd name.
Each key records its networkd spelling. From it, the built-in
output networkd renders the units into
/run/systemd/network, where they take precedence over the image's. chalkd renders them for every
delivery before it changes anything, and chalkos-config.service renders them at every boot before
networkd starts. A changed network reaches a running node with
chalkctl config apply, which makes networkd reload
its .network and .netdev units.
udev applies the .link units, and it applies one to a device only while it handles the device's
add event. At boot the kernel announces the devices before chalkos-config.service renders the
units, so chalkos-links.service has udev handle the add event of every physical port again once
they are rendered, before networkd starts. The unit leaves out virtual devices, such as a bond, a
VLAN or a bridge. networkd creates them after the unit has run, and udev applies their .link
units then.
A delivery that changes a Link document runs chalkos-links.service again, through the
effect links. The kernel renames a port that is up,
so a new name takes effect at once, and networkd then matches the port to the units of its new
name. An added alternative name or setting takes effect at once too, where the driver accepts it
on a port that is up. udev only adds alternative names and leaves out settings a unit no longer
names, and the unit does not run once the last Link document is gone, so a removal takes effect
at the next boot. A Link document of a virtual device applies when networkd next creates the
device.
Each add event also has systemd-sysctl apply the sysctl.d settings of the port again, such as
systemd's default rp_filter. A per-link setting of a Network document, such as
IPv4ReversePathFilter=, is then overwritten until networkd next configures the link: when a
delivery changes the port's Network document, or at the next boot.
Ethernet interfaces without a unit get DHCP¶
The image brings one unit of its own, NixOS's 99-ethernet-default-dhcp. It runs DHCP on every
physical Ethernet interface that no earlier unit matches. A node whose config holds no Network
document still comes up with DHCP, and a node that configures one interface keeps DHCP on
the others.
The firewall drops what it does not accept¶
Everything on a node runs on nftables. The NixOS firewall is configured with
flushRuleset = false, so reloading it replaces its own table and leaves the tables of other
services alone. Its input chain drops what it does not accept, whatever other tables do. On every
node it accepts chalkd's port 50000, established and related connections, the loopback interface,
ICMP echo requests, ICMPv6 other than redirects and node information queries, and DHCPv6 replies
to link-local addresses. A reverse-path check drops packets from a source address the node has no
route back to.
Ports and firewall lists each port and who needs it.
A router image has no NixOS firewall. The router layer's own table, rendered from the node's router documents, takes its place, as The router's firewall describes.
With the Kubernetes layer, Kubernetes opens more ports
A node with Kubernetes also accepts the kubelet's port, the NodePort range and VXLAN sent to its own addresses. A control plane accepts the API server's and etcd's ports. Kubernetes networking explains how.
chrony keeps the clock with authenticated servers¶
A node keeps its clock with chrony. By default it synchronises with the
Physikalisch-Technische Bundesanstalt's servers ptbtime1.ptb.de to ptbtime3.ptb.de,
authenticated with NTS. A TimeSync document names other
servers, each with NTS unless nts: false:
apiVersion: time.chalkos.dev/v1alpha1
kind: TimeSync
metadata: {name: default}
spec:
servers:
- {host: nts.example.org}
- {host: ntp.example.org, nts: false}
The built-in output chrony renders the servers into a sources file that chrony reads, so one
image serves nodes with different servers. dhcp: true adds the servers that DHCP announces,
which are not authenticated. chrony keeps its NTS cookies on VAR, so after a reboot it needs no new
key exchange with each server. A node runs no NTP server for other machines, except a router with
a TimeServer, as Serve time with NTP describes.
The clock decides which certificates a node accepts, as The clock decides whether a certificate is valid explains.
Limits¶
- The network kinds accept the keys of the image's own networkd. A key that only a newer systemd knows is refused until the image carries that systemd.
- The firewall opens only the ports chalkos's modules open. A service an image adds opens its
ports in the image's NixOS modules. On a router image, a
FirewallPolicytolocalopens them. - Time servers that DHCP announces are not authenticated.
Related pages¶
- Kubernetes networking: node addresses, dual stack, the VIP and flannel.
- Config and the inventory: how the network documents reach a node.
- Write a node's config for bonds, VLANs and static addresses.
- Ports and firewall.
- Config kinds for every field of
Network,NetDev,LinkandTimeSync.