Skip to content

Ports and firewall

This page is for operators who set up a network firewall in front of the nodes or read a node's nftables rules. It lists the ports a chalkos node accepts connections on, the ports its services listen on locally, the nftables tables on a node and the connections a node makes. Networking and Kubernetes networking explain the network design, and Security model what each listener trusts.

Open ports

The firewall accepts new connections on these ports alone. Every node has the first; the others depend on whether the image has Kubernetes, on the node's type (Node.spec.type: controlplane or worker), on the CNI and on the ClusterRegistry document. A router image filters with a table of its own, which Router images describes.

Port Protocol Service Nodes Connects to it
50000 TCP chalkd's node API, mutual TLS every node, the installer too chalkctl and chalklab; nodes renewing their node certificate, at a control plane through the cluster endpoint's host
6443 TCP Kubernetes API server control planes kubectl, every kubelet and kube-proxy at the cluster endpoint; pods, flannel among them, through the kubernetes service
2379 TCP etcd's client port control planes the other control planes
2380 TCP etcd's peer port control planes the other control planes
10250 TCP kubelet control planes and workers the API server
30000–32767 TCP NodePort services control planes and workers clients of NodePort services
8472 UDP flannel's VXLAN control planes and workers, with flannel the other nodes
3784 UDP BFD, FRR's bfdd: single-hop control control planes and workers whose image carries the BGP speaker the BGP neighbours of a BGPPeer with bfd
5000, or the ClusterRegistry's port TCP chalkos-registry, the cluster registry: HTTPS with the node certificate; pulls for anyone, pushes, deletes and garbage collection for a client certificate of the admin role nodes whose static address is one of the ClusterRegistry's endpoints, every control plane among them containerd on every Kubernetes node; chalkctl registry

Ports 2379 and 2380 accept only certificates of the cluster's etcd CA, but a control plane's firewall opens them to every source, not only to the other control planes' addresses.

The firewall opens the NodePort range for TCP only.

The cluster registry's port is open on a node that serves an endpoint of the ClusterRegistry, to every source. The table inet chalkos-registry, which chalkos-registry.service loads before the registry starts and deletes when it stops, marks TCP to the port with the bit 0x04000000, and the firewall accepts packets that carry it. The table clears the bit before it marks and after the firewall's chain, as the control plane's table does. A node that serves no endpoint runs no registry, loads no table and keeps the port closed.

VXLAN rule

A node accepts UDP 8472 only when it is sent to one of the addresses the node picked for Kubernetes, and only from the ranges of the Flannel document's spec.vxlanSourceSubnets when that list is set. Kubernetes networking explains why. The rule has three parts:

Part What it does
Table inet chalkos-vxlan, chain input Runs before the firewall and marks UDP 8472 sent to the node's address of each family with the bit 0x01000000: on the interface holding that address, or on any interface but the pod network's and kube-proxy's (cni*, flannel*, kube-* and veth*) when the address is on a loopback or dummy interface. With source ranges set, only packets from them get the mark, and a family without a range gets none.
Firewall's input-allow chain Accepts UDP 8472 that carries the mark.
Table inet chalkos-vxlan, chain clear Runs after the firewall and clears the mark, so no later rule sees it.

With source ranges set, a forward chain in the same table also drops UDP 8472 that pods send through the node to those ranges. On every run, the chain takes the pod range of each family in ipFamilies and the source ranges from /run/chalkos/kubernetes/cluster.json, which chalkd builds from the node's Cluster and Flannel documents. While those ranges are unknown, because the node's config holds no Cluster document or the file cannot be read, or when a family has no pod range or a source range is empty, the chain drops all UDP 8472 the node forwards.

chalkd kubernetes vxlan-rule empties the table's input chain before chalkd picks the node's addresses at boot. chalkd kubernetes prepare --vxlan-rule then fills it from /run/chalkos/kubernetes/vxlan, which it writes once it picked them. Without that file, or with one it cannot parse, the table holds no marking rule and the node refuses all VXLAN. When nft refuses the table, the node refuses all VXLAN and drops all UDP 8472 it forwards; when nft refuses that too, the table goes.

Listeners the firewall keeps closed

These services listen on a loopback address, or on every address with the firewall dropping connections from other machines. They are reachable from the node itself.

Port Protocol Service Address Nodes
53 UDP and TCP systemd-resolved's stub resolver, which the node's own name lookups use 127.0.0.53 and 127.0.0.54 every node
323 UDP chrony's command port, for chronyc; on router images with the time server it also answers clients and serverstats, which the diagnostic ntp-clients reads 127.0.0.1 and ::1 every node
2379 TCP etcd's client port, for the local API server the loopback address of the primary family control planes
2381 TCP etcd's metrics and health, plain HTTP the loopback address of the primary family control planes
8081 TCP flanneld's health every address nodes with flannel
10248 TCP kubelet's health, the kubelet's default 127.0.0.1 control planes and workers
10249 TCP kube-proxy's metrics, kube-proxy's default 127.0.0.1 control planes and workers
10256 TCP kube-proxy's health, kube-proxy's default every address control planes and workers
10257 TCP kube-controller-manager 127.0.0.1 control planes
10259 TCP kube-scheduler 127.0.0.1 control planes
179 TCP BGP, FRR's bgpd; the node opens its sessions itself every address control planes and workers whose image carries the BGP speaker

CoreDNS answers on port 53 inside the pod network, at the cluster's DNS service addresses; it opens no port on the node.

What a node does not listen on

A node runs no SSH server, so port 22 is closed, and there are no logins on the console. Status, logs and the console shows how to debug a node through kubectl debug node/<node> and the node API. chrony keeps the clock and serves no NTP clients, except on a router with a TimeServer.

Firewall tables

The NixOS firewall runs on nftables, as kube-proxy and flannel do. chalkos sets networking.nftables.flushRuleset = false, so loading or reloading the firewall replaces its own table and leaves the others in place.

Table Owner Contents
inet nixos-fw NixOS firewall The input policy: drop, except for the loopback interface, established and related connections, the open ports above, ICMP echo requests, ICMPv6 other than redirects and node information queries, DHCPv6 replies to link-local addresses, and the control plane's ports, VXLAN and the cluster registry's port carrying their marks. A reverse-path check in prerouting drops packets from a source address the node has no route back to.
inet chalkos-controlplane chalkos On a control plane, the mark of the control plane's ports and its clearing.
inet chalkos-vxlan chalkos The VXLAN mark, its clearing and the forward drop.
inet chalkos-registry chalkos On a node that serves the cluster registry, the mark of its port and its clearing.
ip kube-proxy, ip6 kube-proxy kube-proxy Service and NodePort translation, in kube-proxy's nftables mode.
flannel's tables flanneld Masquerading of pod traffic, with EnableNFTables.
the portmap plugin's table CNI portmap plugin Host ports of pods, with its nftables backend.

The input-allow chain, as NixOS renders it for a node of the default Kubernetes image, with flannel, the BGP speaker and the cluster registry:

chain input-allow {
   tcp dport { 10250, 50000, 30000-32767 } accept
   udp dport { 3784 } accept
   meta l4proto . th dport @temp-ports accept
   icmp type echo-request  accept comment "allow ping"
   icmpv6 type != { nd-redirect, 139 } accept comment "Accept all ICMPv6 messages except redirects and node information queries (type 139).  See RFC 4890, section 4.4."
   ip6 daddr fe80::/64 udp dport 546 accept comment "DHCPv6 client"
   meta mark & 0x04000000 == 0x04000000 accept
   tcp dport { 6443, 2379, 2380 } meta mark & 0x02000000 == 0x02000000 accept
   udp dport 8472 meta mark & 0x01000000 == 0x01000000 accept
}

The chain also accepts 6443, 2379 and 2380 when they carry the mark 0x02000000, which the table inet chalkos-controlplane sets on a node whose Node.spec.type is controlplane. chalkos-controlplane.service loads that table, or deletes it on a worker, at boot and whenever the node's Node document changes. The table clears the bit on every packet the node receives before it marks, and again after the firewall's chain, so a network plugin of the cluster's own must not use it.

Router images

A router image has no NixOS firewall. The router layer's table inet chalkos filters instead, and no other table of chalkos filters the router's traffic. The router's firewall describes its chains.

Port Protocol Service Accepted on
50000 TCP chalkd's node API, mutual TLS the interfaces of Router.spec.management.interfaces, or every interface while it is null; every interface while the node has no rendered ruleset, as in maintenance mode
179 TCP BGP, FRR's bgpd the interfaces of each BGPPeer of interface or interfaces, matched as sdifname, the link a packet came in on, for a peer with vrf; and from the address of each of address, with the VRF's device as iifname for a peer with vrf
3784, 3785 UDP BFD, FRR's bfdd: single-hop control and echo as port 179, for each BGPPeer with bfd
4784 UDP BFD, FRR's bfdd: multi-hop control from the address of each BGPPeer of address with bfd, as port 179
53 UDP and TCP DNS, Unbound from the ranges of each Resolver's allow to the addresses of its listen; for a resolver with vrf, on the links of its VRF alone, with the VRF's device as iifname
123 UDP NTP, chrony from the ranges of the TimeServer's allow to the addresses of its listen, in the default VRF; chrony listens on every address, and the table keeps the others closed
67 UDP DHCPv4, Kea the links of each DHCPSubnet's interfaces, through raw sockets ahead of the table, which has no line for it; see below
547 UDP DHCPv6, Kea from link-local addresses on the link of each IPv6 DHCPSubnet, with the VRF's device as iifname and the link as sdifname for a link of a VRF
67 UDP DHCPv4 relay, dnsmasq on the links of each DHCPRelay with an IPv4 server, matched as above
547 UDP DHCPv6 relay, dnsmasq from link-local addresses on the links of each DHCPRelay with an IPv6 server, matched as above

The table also accepts the loopback interface, established and related connections, ICMP and ICMPv6 errors and echo requests, ICMPv6 neighbour discovery, router solicitations and advertisements and multicast listener discovery, and DHCPv6 replies from port 547 to port 546 of a link-local address. A FirewallPolicy whose to is local accepts more. Of the listeners the firewall keeps closed, a router has those of every node. The image ignores networking.firewall options, such as allowedTCPPorts, and its build warns of each one set.

Kea, the DHCP server, reads and answers DHCPv4 on raw sockets of the links its subnets serve. Raw sockets take a link's frames before nftables sees them, and send past it, so the table needs no line for port 67 and a FirewallPolicy does not filter it. Kea's UDP socket on port 67 of each link's address takes what the input chain accepts, which is nothing by default. DHCP on a router explains why. The relays' servers answer the address and port the relay sent from, so the table lets their answers in as replies; it needs no line for them.

Table Owner Contents
inet chalkos chalkos-firewall.service The AddressSet sets, a chain per FirewallPolicy, the conntrack zones of the VRFs that have one, the reverse-path filter, the chain policy-routing of the PolicyRoute documents, which marks packets with bits of 0x00ff0000, the input and forward chains and the NAT chains, which the output router-nftables renders.

Each NAT64 runs in a network namespace of its own, joined to its VRF by the link n64-<name>, so its traffic passes the table. Lines with the comment NAT64 <name> handle it:

Chain Lines of each NAT64
forward Ahead of the policies, once the chain drops: accept IPv6 from the prefix and IPv4 from the pool, or from the link's inner address for an empty pool, out of the link, matched as sdifname for a NAT64 with vrf; accept IPv6 to the prefix and IPv4 to an explicit pool into the link.
srcnat Ahead of the NATRule lines: accept connections into the link and from the prefix and an explicit pool; for an empty pool, masquerade the inner address, TCP and UDP to the ports 61001 to 65535.
dstnat With dnat rules, ahead of them: accept connections out of the link and to the prefix and an explicit pool.
ct-prerouting, ct-output The link joins the conntrack zone of the NAT64's VRF.

Connections a node makes

A network firewall in front of the nodes needs to let these through, besides the open ports between the nodes.

Destination Port Protocol Purpose
Container registries, or the endpoints their Mirror documents name 443, or the endpoint's port TCP Images of the control plane's static pods, kube-proxy, flannel, CoreDNS and workloads. An endpoint may use plain HTTP when a Registry of its host sets spec.insecure.
The endpoints of the ClusterRegistry 5000, or its port TCP Images of the hosts the cluster registry mirrors, which containerd on every Kubernetes node looks for there first. HTTPS, verified by the OS CA.
The registry of an install's or upgrade's image, when the inventory names the image in a registry, or the endpoints of its Mirror 443, or the port of the image reference or the endpoint TCP chalkd pulling the image, on the installer during chalkctl install and on the node during chalkctl upgrade. Plain HTTP only when the Registry of the host or endpoint sets spec.insecure.
Time servers of the TimeSync document's spec.servers 4460 TCP NTS key exchange, for servers with nts, the default.
Time servers 123, or the server's port UDP NTP.
DNS resolvers of the node's network 53 UDP and TCP Names of registries and time servers.
DHCP servers, when the node's network units use DHCP 67 UDP Addresses.
The cluster endpoint 6443 TCP The kubelet and kube-proxy of every Kubernetes node.
The cluster endpoint's host 50000 TCP Renewal of a worker's node certificate at a control plane, once two thirds of its lifetime have passed; control planes sign their own.
The other control planes 2379, 2380 TCP etcd membership and replication.
The other Kubernetes nodes 10250 TCP The API server reaching kubelets, from control planes.
The other Kubernetes nodes 8472 UDP flannel's VXLAN.
The neighbours of the node's BGPPeer documents 179 TCP BGP sessions, which the node opens.
The neighbours of the node's BGPPeer documents with bfd 3784 UDP BFD's control packets.

Config kinds describes the Mirror, Registry and TimeSync documents.

A VIP needs no port of its own: chalkd adds the address to an interface of the control plane that holds it and announces it to the neighbours with ARP or IPv6 neighbour advertisements. A routed VIP travels in the BGP sessions above.

Sources: the firewall rules of modules/node/chalkd.nix, modules/node/kubernetes, modules/node/router, cmd/chalkos-render/nftables.go, pkg/kubernetes/vxlan and pkg/kubernetes/clusterregistry, and the listeners of cmd/chalkd, cmd/chalkos-registry, pkg/kubernetes/manifests/manifests.go, pkg/kubernetes/addons and modules/node/kubernetes.