Kubernetes networking¶
A Kubernetes cluster on chalkos runs on the node's host network, which Networking describes. On top of it, each node picks one address per IP family of the cluster, the control planes share a VIP, flannel carries the pods' traffic between nodes, and kube-proxy and CoreDNS give pods their services and names. The diagram shows a cluster of three control planes and two workers on one L2 network, with cp1 holding the VIP.
flowchart TD
client["kubectl, chalkctl"]
subgraph lan["one L2 network"]
vip(["VIP 10.0.0.10"])
cp1["cp1, VIP holder"]
cp2["cp2"]
cp3["cp3"]
w1["w1"]
w2["w2"]
end
client -->|"TCP 6443"| vip
w1 -->|"API server"| vip
w2 -->|"API server"| vip
vip --- cp1
cp1 <-->|"etcd peers, TCP 2380"| cp2
cp2 <-->|"etcd peers"| cp3
cp1 <-->|"etcd peers"| cp3
w1 <-.->|"VXLAN, UDP 8472, between all nodes"| w2
Each node picks one address per family before the kubelet starts¶
A node with Kubernetes registers one address of each IP family of the cluster. On a control plane,
its certificates name the same addresses, and etcd and the API server advertise the primary
family's. Before the kubelet starts, chalkos-kubernetes.service picks them and writes them to
/run/chalkos/kubernetes/node-ip.
A family's address is fixed or picked:
- Fixed:
nodeIP.addressesof the node'sNodedocument, at most one address per family. By default it is the first address of each family among thenetwork.addresskeys of the node'sNetworkdocuments, in the order of their names, unless subnets to pick from apply to the node. The node waits until an interface holds the address. - Picked: the node takes the first of its global unicast addresses that
nodeIP.validSubnetsof theClusterdocument selects, ornodeIP.validSubnetsof the node's ownNodedocument, which replaces the cluster's list as a whole. A leading!excludes a subnet. Addresses are compared IPv4 before IPv6, then by interface name and address, so the choice does not depend on the order the kernel lists them in. With no subnets, every global unicast address qualifies.
The diagram shows how the node decides for each family.
flowchart TD
pin{"Addresses pinned on STATE?"} -->|yes| waitpin["Wait for exactly those"]
pin -->|no| fixed{"nodeIP.addresses set?"}
fixed -->|yes| waitfixed["Wait until an interface holds it"]
fixed -->|no| subnets{"validSubnets apply?"}
subnets -->|yes| match["Pick the first address they select"]
subnets -->|no| net{"A Network document gives an address?"}
net -->|yes| waitfixed
net -->|no| any["Pick the first global unicast address"]
The node never picks an address in the pod or service ranges or on an interface of the pod network
or kube-proxy (flannel*, cni*, veth*, kube-*). It takes the endpoint's address or a VIP only
when nothing else of the family matches, because those move between control planes. With flannel
and two families, the node's IPv4 and IPv6 addresses must be on one interface.
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
endpoint: https://10.0.0.10:6443
nodeIP:
validSubnets: [10.0.0.0/24, "!10.0.0.200/29"]
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
A subnet that starts with ! is quoted, because YAML reads an unquoted ! as a tag.
DHCP leases and addresses that a routing daemon adds can arrive late, so the node looks for its
addresses every second, up to nodeIP.timeout of the Cluster document, 300 seconds by default.
A node that finds none runs neither the kubelet nor static pods. Its status stays in the state
preparation failed: <reason>, and the reason names the filter and the addresses the node has:
kubernetes worker: preparation failed: no ipv4 node address matches validSubnets 10.0.0.0/24 (the node has 192.168.1.20 on enp1s0), 5m0s after chalkos-node-addresses.target
A routing daemon can provide the node's addresses¶
A node's addresses do not have to come from networkd. A routing daemon, such as the Kubernetes
layer's BGP speaker or one an extension adds to an image, can announce an address on a loopback or
dummy interface. Such a unit orders itself before chalkos-node-addresses.target, which every
image defines, and is wanted by it. The preparation starts after the target and still waits up to
nodeIP.timeout for the addresses. frr.service, the BGP speaker's unit, is ordered this way: it
starts before the target, so FRR is running before the preparation looks for the node's
addresses.
An address on a loopback interface needs the Flannel document's spec.mtu, because flannel would
take the loopback's MTU. When it is set, a dummy interface holding a node's address needs an MTU of
at least its value, 20 more for an IPv6 address. Otherwise the node refuses to prepare, with a
message such as the node's address 10.0.0.11 is on the loopback interface lo, whose MTU flannel
would take; set the Flannel document's spec.mtu. The homelab's nodes hold their addresses on the
dummy interface node0, and its Flannel document sets spec.mtu.
A control plane's addresses are pinned once it joins etcd¶
etcd's peers know a member by the address it joined with, and the control plane's certificates
name that address. When a control plane becomes an etcd member, by bootstrap or by joining, chalkd
pins the addresses it picked on STATE. Every later boot waits for exactly those, whatever the
subnets say. A control plane's addresses can still change in its config, but a Network change
that drops a pinned address stops the preparation, so the control plane starts no etcd:
pinned address 10.0.0.11 is not present, 5m0s after chalkos-node-addresses.target; restore it, or remove the node's etcd member with chalkctl etcd remove-member cp1 and reinstall the node
chalkctl etcd leave removes the pin together with
the node's member. A node whose pinned address is gone runs no etcd member that could answer, so it
leaves with chalkctl etcd leave <node> --force, which removes its member through the other
members.
The address families of a cluster are fixed once it exists¶
ipFamilies of the Cluster document lists the families of the cluster, the primary one first:
[ipv4] by default, or both in either order. Every node has one address of each family, pods get
one of each, and services get one of each family they ask for.
Each family has its own ranges, fields of the Cluster document keyed by ipv4 and ipv6:
| Field | IPv4 default | IPv6 default |
|---|---|---|
podCIDRs |
10.244.0.0/16 |
fd00:10:244::/56 |
serviceCIDRs |
10.96.0.0/12 |
fd00:10:96::/112 |
dnsIPs |
the service range's 10th address, 10.96.0.10 |
the service range's 10th address, fd00:10:96::a |
nodeCIDRMaskSizes |
24 |
64 |
The schemas check that every range is an address range in CIDR notation and every DNS address is
an address, in both families, including a family the cluster does not use. The other checks apply
to the families of ipFamilies alone: the family of a range, host bits, overlapping ranges and
the DNS address inside the service range.
A cluster with IPv6 as its primary family and IPv4 beside it, on the default ranges:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
endpoint: https://10.0.0.10:6443
ipFamilies: [ipv6, ipv4]
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
The cluster DNS service has an address of the primary family alone, as kubeadm's does, because the kubelet gives pods one name server. It still answers A and AAAA queries.
The families and their order, the pod ranges, the service ranges and the DNS addresses are fixed
once the cluster exists, as
How Kubernetes runs on a node
explains. A worker whose Node lacks a pod range of one of the cluster's families reports
joined: <problem> in its status.
One healthy control plane holds the VIP¶
A VIP is an address of the API server that no single machine owns. One healthy control plane holds
it at a time, so the cluster endpoint stays reachable while a control plane reboots or fails.
vip.addresses of the Cluster document takes at most one address per family. The document's
endpoint is then one of them, or a name that resolves to them. The document goes into the files
every node gets:
apiVersion: kubernetes.chalkos.dev/v1alpha1
kind: Cluster
metadata: {name: default}
spec:
endpoint: https://10.0.0.10:6443
vip: {addresses: [10.0.0.10]}
images:
apiServer: registry.k8s.io/kube-apiserver:v1.37.2@sha256:3e563afb5cb5b72a10bac7012698dd09f5a42bde7d22de8d7d3c12e654df86b4
controllerManager: registry.k8s.io/kube-controller-manager:v1.37.2@sha256:28e62daf706ed815a8bd1dd27b03294ba09421b2e1674e9a83bbac38e24ceb0c
scheduler: registry.k8s.io/kube-scheduler:v1.37.2@sha256:028c847fc321b8a7f9787f7b1c0840f832d6b68439be2e5c09279ffb14203d6b
proxy: registry.k8s.io/kube-proxy:v1.37.2@sha256:b2e075697d7ab4f0878775dff9e5dccf818b67644437175210ce72fc1ec70551
etcd: registry.k8s.io/etcd:3.7.0-0@sha256:6ecefbe2510c4a30573a62a4d6dd175acf881ca67003fcd91849a16df7a724d5
coredns: registry.k8s.io/coredns/coredns:v1.14.6@sha256:900f9c109f7a33545d3c811516e8376df9019147b750f5ce3e254468769176ea
pause: registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
flannel: ghcr.io/flannel-io/flannel:v0.28.9@sha256:708a2c9c1cfbfe529d1dbb249cb7113af117f3ed26b5e3e4c8ee015e10555204
chalkd on every bootstrapped control plane takes part in an election held in etcd under
/chalkos/vip. When it starts, it removes any VIP a chalkd before it left behind. It campaigns only
while its local API server answers /readyz. The winner adds every VIP to an interface as a /32 or
/128 host address. In the l2 mode, the default, it announces each address it adds 3 times, 1
second apart, with gratuitous ARP and unsolicited neighbour advertisements, so switches and
neighbours send its traffic to the new holder at once. While it holds the VIPs, it checks every 2
seconds and adds back an address that has gone missing, which it then announces again.
The interface is the one vip.interface of the node's Node document names, or else the one
holding the node's address of that family. The interface is the node's own, so it goes into the
node's Node document, and only on a control plane: chalkctl and chalkd refuse vip.interface on
a worker, which never holds the VIP.
In the routed mode, set with vip: {addresses: [...], mode: routed}, the holder adds the VIPs the
same way but sends no gratuitous ARP and no neighbour advertisements. A routing daemon on the node,
the Kubernetes layer's BGP speaker or one an extension adds to an image, announces the addresses
of the interface instead, and withdraws them when the holder removes them. Every control plane
names that interface in
vip.interface, usually a dummy interface the routing daemon announces, and chalkctl and chalkd
refuse a routed VIP on a control plane without one. The Network document of that interface sets
keepConfiguration: "yes", KeepConfiguration=yes, so that networkd keeps addresses it did not
configure itself. chalkd adds an IPv6 VIP with a preferred lifetime of 0, which networkd counts as
dynamic, so networkd may otherwise remove the VIP when it configures the link again. The control
planes then need no shared L2 network, and clients reach the VIP through the routers that learn
the route. A failover takes the election's time plus the time the routing daemons need to
withdraw the route on the old holder, announce it from the new one and pass the change through
their peers. The homelab example works this way: FRR, the Kubernetes layer's BGP speaker, on each
control plane announces the addresses of its dummy interface node0, the node's own and the VIP
while it holds it, to the router over BGP. Announce addresses with BGP
sets it up.
The election runs on an etcd lease of 10 seconds, and every control plane checks its API server every 2 seconds. The diagram shows the states of one control plane's chalkd.
stateDiagram-v2
state "Waiting for its API server" as Waiting
state "Standby, campaigning" as Standby
state "Holder" as Holder
[*] --> Waiting: chalkd starts
Waiting --> Standby: API server answers /readyz
Standby --> Holder: wins the election
Standby --> Waiting: 3 failed checks in a row
Holder --> Waiting: 3 failed checks, or the lease lost
Holder --> [*]: chalkd stops and resigns
What happens to the VIP depends on how its holder stops:
- When the holder's API server fails three checks in a row, the holder removes the addresses and resigns, and another control plane takes over.
- When the holder stops, it resigns and revokes its lease, so the next one takes over without waiting.
- When the holder crashes or loses its network, the VIP is unheld until the lease expires, within 10 seconds. A holder that cannot renew the lease removes the addresses 2 seconds before etcd could let it expire, so two nodes never announce the same address.
- Without an etcd quorum nobody can win the election, and nobody holds the VIP.
A control plane's API server also fails its readiness when its own etcd member is unhealthy, so the
VIP moves away from a control plane whose etcd member fails.
chalkctl status shows each control plane as
vip holder or vip standby in its Kubernetes line.
flannel carries pod traffic in VXLAN¶
chalkd applies the pod network as an add-on: flannel, unless
chalkos.cni.provider is none. flannel carries
pod traffic between nodes in VXLAN on UDP port 8472, one device per family (flannel.1 and
flannel-v6.1). flanneld uses exactly the node's addresses as the kubelet registered them, and
masquerades what pods send to other nodes behind them.
flannel's devices get the MTU of the interface holding the node's address, minus 50. spec.mtu of
the Flannel document sets the network's MTU instead, and is
required when the node's address is on a loopback interface. IPv6 VXLAN needs 70 bytes rather than
50, so with IPv6 in use set it to the network's MTU minus 20:
A cluster has at most one Flannel document, and every node holds the same one. chalkctl refuses an
inventory whose nodes hold different Flannel specs. Only images whose pod network is flannel
accept the document; for another image it is an unknown kind. An image built with
chalkos.cni.provider = "none" in its modules installs no pod network and ships every CNI
reference plugin, for clusters whose Addon documents bring their own. Opening that network's ports
is then up to the image's NixOS modules.
The firewall accepts VXLAN only for the node's own addresses¶
VXLAN carries pod packets unauthenticated, so the firewall accepts it only when it is sent to one of
the node's own addresses. It accepts it on the interface that holds the address. When the address
is on a loopback or dummy interface, it accepts it on any interface but those of the pod network
and kube-proxy, where pods could send it. The node picks its addresses after the firewall started,
so a table of chalkos's own, chalkos-vxlan, which the preparation fills with the addresses it
picked, marks such packets for the firewall to accept. No VXLAN is accepted while the addresses
are being picked or when none are found.
VXLAN rule lists the table's chains and marks.
vxlanSourceSubnets of the Flannel document narrows this to sources in the given ranges, one or
more per family. They must hold every node's addresses and may not overlap the pod or service
ranges. With them, the node also drops VXLAN that pods send through it to those ranges. The node
masquerades pod traffic behind its own address, which lies in the ranges, so the other nodes would
take such VXLAN for this node's own. Without them, VXLAN is accepted from any source, so pods can
reach the other nodes' VXLAN port too. The node reads the pod ranges from its Cluster document and the source ranges from its Flannel
document. While it cannot, it drops all VXLAN it forwards.
chalkctl validate refuses a node whose fixed addresses,
or the subnets it picks its addresses from, lie outside the ranges. A node that picks its addresses
at boot checks the picked ones against the ranges again, and refuses to prepare Kubernetes when one
lies outside.
kube-proxy and CoreDNS give pods services and names¶
chalkd applies kube-proxy as a DaemonSet in nftables mode, the mode the node's firewall and flannel
use too. kube-proxy serves NodePorts only on the node's own addresses, one per family, as the
kubelet registered them. chalkd applies CoreDNS at the primary family's address of the Cluster
document's dnsIPs. The kubelet configuration of every node names the cluster's DNS address and
domain, so pods resolve through CoreDNS.
The control plane ports open only on control planes¶
Every node with Kubernetes accepts the kubelet's port 10250, the NodePort range 30000 to 32767 and
marked VXLAN. chalkos-controlplane.service opens the API server's port 6443 and etcd's ports 2379
and 2380 on a node whose Node document has the type controlplane, and closes them on a worker.
It does this at boot and whenever the Node document changes, through a table of chalkos's own,
chalkos-controlplane, that marks the ports for the firewall to accept.
Ports and firewall lists the ports and the tables.
The BGP speaker announces interfaces¶
The output bgp renders FRR's config from the node's BGPSpeaker, its BGPPeer documents and its
Node. FRR redistributes the connected routes of the speaker's export interfaces through a
route-map that matches those interfaces, so an address added to one of them, the VIP among them,
is announced at once and withdrawn when it goes. chalkd adds an IPv6 VIP with a preferred lifetime
of 0, and zebra announces such a deprecated address like any other. Each BGPPeer becomes a peer
group. Its outbound route-map passes only the routes with an empty AS path, those the node
originates, and its inbound route-map passes the default routes or nothing. A password reaches FRR
as a credential of frr.service, which fills in the placeholder of the rendered config. A change
of the documents reloads FRR through chalkos-frr-reload, which applies the difference and keeps
the sessions it leaves alone. Routers run the same FRR build and reload, which
BGP on a router describes.
Limits¶
- An
l2VIP needs the control planes on one L2 network, through which clients reach the VIP, because its holder announces it with ARP and neighbour advertisements. AroutedVIP needs no shared network, but chalkd does not announce it itself: it needs the BGP speaker, or another routing daemon, on every control plane, and its failover takes as long as that daemon and its peers need to move the route. A load balancer or a DNS name in front of the control planes works instead of either. - On control planes, etcd's ports 2379 and 2380 are open to every source, not only to the cluster's nodes. etcd accepts only clients and peers with a certificate of its CA there.
- Packets a control plane receives lose the packet mark bit
0x02000000, which the firewall's rule for the control plane's ports uses, so a pod network of the cluster's own (chalkos.cni.provider = "none") must not use it. - The address families and their order, the pod ranges, the service ranges and the DNS addresses of a cluster are fixed once it exists. A change needs a new cluster.
- Within
vxlanSourceSubnetsa source address can be forged. The kernel drops an IPv4 packet that claims the node's own address as its source, but has no such check for IPv6. - A control plane whose pinned address is gone does not start etcd or the control plane until the
address returns, or until the node leaves etcd with
chalkctl etcd leave --forceand is reinstalled.
Related pages¶
- Networking: the host network a Kubernetes node runs on.
- How Kubernetes runs on a node and The control plane.
- Run a highly available control plane and Plan a cluster.
- Configure Kubernetes
for the
Flanneldocument. - Troubleshoot Kubernetes for VXLAN and MTU problems.
- Ports and firewall.
- Config kinds for every field of
Cluster,NodeandFlannel.