Ports and firewall¶
This page is for operators who set up a network firewall in front of the nodes or read a node's nftables rules. It lists the ports a chalkos node accepts connections on, the ports its services listen on locally, the nftables tables on a node and the connections a node makes. Networking and Kubernetes networking explain the network design, and Security model what each listener trusts.
Open ports¶
The firewall accepts new connections on these ports alone. Every node has the first; the others
depend on whether the image has Kubernetes, on the node's type (Node.spec.type:
controlplane or worker), on the CNI and on the ClusterRegistry document. A router image
filters with a table of its own, which Router images describes.
| Port | Protocol | Service | Nodes | Connects to it |
|---|---|---|---|---|
| 50000 | TCP | chalkd's node API, mutual TLS | every node, the installer too | chalkctl and chalklab; nodes renewing their node certificate, at a control plane through the cluster endpoint's host |
| 6443 | TCP | Kubernetes API server | control planes | kubectl, every kubelet and kube-proxy at the cluster endpoint; pods, flannel among them, through the kubernetes service |
| 2379 | TCP | etcd's client port | control planes | the other control planes |
| 2380 | TCP | etcd's peer port | control planes | the other control planes |
| 10250 | TCP | kubelet | control planes and workers | the API server |
| 30000–32767 | TCP | NodePort services | control planes and workers | clients of NodePort services |
| 8472 | UDP | flannel's VXLAN | control planes and workers, with flannel | the other nodes |
| 3784 | UDP | BFD, FRR's bfdd: single-hop control | control planes and workers whose image carries the BGP speaker | the BGP neighbours of a BGPPeer with bfd |
5000, or the ClusterRegistry's port |
TCP | chalkos-registry, the cluster registry: HTTPS with the node certificate; pulls for anyone, pushes, deletes and garbage collection for a client certificate of the admin role | nodes whose static address is one of the ClusterRegistry's endpoints, every control plane among them |
containerd on every Kubernetes node; chalkctl registry |
Ports 2379 and 2380 accept only certificates of the cluster's etcd CA, but a control plane's firewall opens them to every source, not only to the other control planes' addresses.
The firewall opens the NodePort range for TCP only.
The cluster registry's port is open on a node that serves an endpoint of the
ClusterRegistry, to every source. The table inet chalkos-registry,
which chalkos-registry.service loads before the registry starts and deletes when it stops, marks
TCP to the port with the bit 0x04000000, and the firewall accepts packets that carry it. The
table clears the bit before it marks and after the firewall's chain, as the control plane's table
does. A node that serves no endpoint runs no registry, loads no table and keeps the port closed.
VXLAN rule¶
A node accepts UDP 8472 only when it is sent to one of the addresses the node picked for
Kubernetes, and only from the ranges of the Flannel document's
spec.vxlanSourceSubnets when that list is set.
Kubernetes networking
explains why. The rule has three parts:
| Part | What it does |
|---|---|
Table inet chalkos-vxlan, chain input |
Runs before the firewall and marks UDP 8472 sent to the node's address of each family with the bit 0x01000000: on the interface holding that address, or on any interface but the pod network's and kube-proxy's (cni*, flannel*, kube-* and veth*) when the address is on a loopback or dummy interface. With source ranges set, only packets from them get the mark, and a family without a range gets none. |
Firewall's input-allow chain |
Accepts UDP 8472 that carries the mark. |
Table inet chalkos-vxlan, chain clear |
Runs after the firewall and clears the mark, so no later rule sees it. |
With source ranges set, a forward chain in the same table also drops UDP 8472 that pods send
through the node to those ranges. On every run, the chain takes the pod range of each family in
ipFamilies and the source ranges from /run/chalkos/kubernetes/cluster.json, which chalkd
builds from the node's Cluster and Flannel documents. While those ranges are unknown, because
the node's config holds no Cluster document or the file cannot be read, or when a family has no
pod range or a source range is empty, the chain drops all UDP 8472 the node forwards.
chalkd kubernetes vxlan-rule empties the table's input chain before chalkd picks the node's
addresses at boot. chalkd kubernetes prepare --vxlan-rule then fills it from
/run/chalkos/kubernetes/vxlan, which it writes once it picked them. Without that file, or with
one it cannot parse, the table holds no marking rule and the node refuses all VXLAN. When nft
refuses the table, the node refuses all VXLAN and drops all UDP 8472 it forwards; when nft
refuses that too, the table goes.
Listeners the firewall keeps closed¶
These services listen on a loopback address, or on every address with the firewall dropping connections from other machines. They are reachable from the node itself.
| Port | Protocol | Service | Address | Nodes |
|---|---|---|---|---|
| 53 | UDP and TCP | systemd-resolved's stub resolver, which the node's own name lookups use | 127.0.0.53 and 127.0.0.54 |
every node |
| 323 | UDP | chrony's command port, for chronyc; on router images with the time server it also answers clients and serverstats, which the diagnostic ntp-clients reads |
127.0.0.1 and ::1 |
every node |
| 2379 | TCP | etcd's client port, for the local API server | the loopback address of the primary family | control planes |
| 2381 | TCP | etcd's metrics and health, plain HTTP | the loopback address of the primary family | control planes |
| 8081 | TCP | flanneld's health | every address | nodes with flannel |
| 10248 | TCP | kubelet's health, the kubelet's default | 127.0.0.1 |
control planes and workers |
| 10249 | TCP | kube-proxy's metrics, kube-proxy's default | 127.0.0.1 |
control planes and workers |
| 10256 | TCP | kube-proxy's health, kube-proxy's default | every address | control planes and workers |
| 10257 | TCP | kube-controller-manager | 127.0.0.1 |
control planes |
| 10259 | TCP | kube-scheduler | 127.0.0.1 |
control planes |
| 179 | TCP | BGP, FRR's bgpd; the node opens its sessions itself | every address | control planes and workers whose image carries the BGP speaker |
CoreDNS answers on port 53 inside the pod network, at the cluster's DNS service addresses; it opens no port on the node.
What a node does not listen on¶
A node runs no SSH server, so port 22 is closed, and there are no logins on the console.
Status, logs and the console shows how to debug a node through
kubectl debug node/<node> and the node API. chrony keeps the clock and serves no NTP clients,
except on a router with a TimeServer.
Firewall tables¶
The NixOS firewall runs on nftables, as kube-proxy and flannel do. chalkos sets
networking.nftables.flushRuleset = false, so loading or reloading the firewall replaces its own
table and leaves the others in place.
| Table | Owner | Contents |
|---|---|---|
inet nixos-fw |
NixOS firewall | The input policy: drop, except for the loopback interface, established and related connections, the open ports above, ICMP echo requests, ICMPv6 other than redirects and node information queries, DHCPv6 replies to link-local addresses, and the control plane's ports, VXLAN and the cluster registry's port carrying their marks. A reverse-path check in prerouting drops packets from a source address the node has no route back to. |
inet chalkos-controlplane |
chalkos | On a control plane, the mark of the control plane's ports and its clearing. |
inet chalkos-vxlan |
chalkos | The VXLAN mark, its clearing and the forward drop. |
inet chalkos-registry |
chalkos | On a node that serves the cluster registry, the mark of its port and its clearing. |
ip kube-proxy, ip6 kube-proxy |
kube-proxy | Service and NodePort translation, in kube-proxy's nftables mode. |
| flannel's tables | flanneld | Masquerading of pod traffic, with EnableNFTables. |
| the portmap plugin's table | CNI portmap plugin | Host ports of pods, with its nftables backend. |
The input-allow chain, as NixOS renders it for a node of the default Kubernetes image, with
flannel, the BGP speaker and the cluster registry:
chain input-allow {
tcp dport { 10250, 50000, 30000-32767 } accept
udp dport { 3784 } accept
meta l4proto . th dport @temp-ports accept
icmp type echo-request accept comment "allow ping"
icmpv6 type != { nd-redirect, 139 } accept comment "Accept all ICMPv6 messages except redirects and node information queries (type 139). See RFC 4890, section 4.4."
ip6 daddr fe80::/64 udp dport 546 accept comment "DHCPv6 client"
meta mark & 0x04000000 == 0x04000000 accept
tcp dport { 6443, 2379, 2380 } meta mark & 0x02000000 == 0x02000000 accept
udp dport 8472 meta mark & 0x01000000 == 0x01000000 accept
}
The chain also accepts 6443, 2379 and 2380 when they carry the mark 0x02000000, which the
table inet chalkos-controlplane sets on a node whose Node.spec.type is controlplane.
chalkos-controlplane.service loads that table, or deletes it on a worker, at boot and whenever
the node's Node document changes. The table clears the bit on every packet the node receives
before it marks, and again after the firewall's chain, so a network plugin of the cluster's own
must not use it.
Router images¶
A router image has no NixOS firewall. The router layer's table inet chalkos filters instead, and
no other table of chalkos filters the router's traffic.
The router's firewall describes its chains.
| Port | Protocol | Service | Accepted on |
|---|---|---|---|
| 50000 | TCP | chalkd's node API, mutual TLS | the interfaces of Router.spec.management.interfaces, or every interface while it is null; every interface while the node has no rendered ruleset, as in maintenance mode |
| 179 | TCP | BGP, FRR's bgpd | the interfaces of each BGPPeer of interface or interfaces, matched as sdifname, the link a packet came in on, for a peer with vrf; and from the address of each of address, with the VRF's device as iifname for a peer with vrf |
| 3784, 3785 | UDP | BFD, FRR's bfdd: single-hop control and echo | as port 179, for each BGPPeer with bfd |
| 4784 | UDP | BFD, FRR's bfdd: multi-hop control | from the address of each BGPPeer of address with bfd, as port 179 |
| 53 | UDP and TCP | DNS, Unbound | from the ranges of each Resolver's allow to the addresses of its listen; for a resolver with vrf, on the links of its VRF alone, with the VRF's device as iifname |
| 123 | UDP | NTP, chrony | from the ranges of the TimeServer's allow to the addresses of its listen, in the default VRF; chrony listens on every address, and the table keeps the others closed |
| 67 | UDP | DHCPv4, Kea | the links of each DHCPSubnet's interfaces, through raw sockets ahead of the table, which has no line for it; see below |
| 547 | UDP | DHCPv6, Kea | from link-local addresses on the link of each IPv6 DHCPSubnet, with the VRF's device as iifname and the link as sdifname for a link of a VRF |
| 67 | UDP | DHCPv4 relay, dnsmasq | on the links of each DHCPRelay with an IPv4 server, matched as above |
| 547 | UDP | DHCPv6 relay, dnsmasq | from link-local addresses on the links of each DHCPRelay with an IPv6 server, matched as above |
The table also accepts the loopback interface, established and related connections, ICMP and
ICMPv6 errors and echo requests, ICMPv6 neighbour discovery, router solicitations and
advertisements and multicast listener discovery, and DHCPv6 replies from port 547 to port 546 of
a link-local address. A FirewallPolicy whose to is local accepts more. Of the
listeners the firewall keeps closed, a router has those
of every node. The image ignores networking.firewall options, such as allowedTCPPorts, and its
build warns of each one set.
Kea, the DHCP server, reads and answers DHCPv4 on raw sockets of the links its subnets serve.
Raw sockets take a link's frames before nftables sees them, and send past it, so the table needs
no line for port 67 and a FirewallPolicy does not filter it. Kea's UDP socket on port 67 of
each link's address takes what the input chain accepts, which is nothing by default.
DHCP on a router
explains why.
The relays' servers answer the address and port the relay sent from, so the table lets their
answers in as replies; it needs no line for them.
| Table | Owner | Contents |
|---|---|---|
inet chalkos |
chalkos-firewall.service |
The AddressSet sets, a chain per FirewallPolicy, the conntrack zones of the VRFs that have one, the reverse-path filter, the chain policy-routing of the PolicyRoute documents, which marks packets with bits of 0x00ff0000, the input and forward chains and the NAT chains, which the output router-nftables renders. |
Each NAT64 runs in a network namespace of its own, joined to its VRF by the link n64-<name>, so
its traffic passes the table. Lines with the comment NAT64 <name> handle it:
| Chain | Lines of each NAT64 |
|---|---|
forward |
Ahead of the policies, once the chain drops: accept IPv6 from the prefix and IPv4 from the pool, or from the link's inner address for an empty pool, out of the link, matched as sdifname for a NAT64 with vrf; accept IPv6 to the prefix and IPv4 to an explicit pool into the link. |
srcnat |
Ahead of the NATRule lines: accept connections into the link and from the prefix and an explicit pool; for an empty pool, masquerade the inner address, TCP and UDP to the ports 61001 to 65535. |
dstnat |
With dnat rules, ahead of them: accept connections out of the link and to the prefix and an explicit pool. |
ct-prerouting, ct-output |
The link joins the conntrack zone of the NAT64's VRF. |
Connections a node makes¶
A network firewall in front of the nodes needs to let these through, besides the open ports between the nodes.
| Destination | Port | Protocol | Purpose |
|---|---|---|---|
Container registries, or the endpoints their Mirror documents name |
443, or the endpoint's port | TCP | Images of the control plane's static pods, kube-proxy, flannel, CoreDNS and workloads. An endpoint may use plain HTTP when a Registry of its host sets spec.insecure. |
The endpoints of the ClusterRegistry |
5000, or its port |
TCP | Images of the hosts the cluster registry mirrors, which containerd on every Kubernetes node looks for there first. HTTPS, verified by the OS CA. |
The registry of an install's or upgrade's image, when the inventory names the image in a registry, or the endpoints of its Mirror |
443, or the port of the image reference or the endpoint | TCP | chalkd pulling the image, on the installer during chalkctl install and on the node during chalkctl upgrade. Plain HTTP only when the Registry of the host or endpoint sets spec.insecure. |
Time servers of the TimeSync document's spec.servers |
4460 | TCP | NTS key exchange, for servers with nts, the default. |
| Time servers | 123, or the server's port |
UDP | NTP. |
| DNS resolvers of the node's network | 53 | UDP and TCP | Names of registries and time servers. |
| DHCP servers, when the node's network units use DHCP | 67 | UDP | Addresses. |
| The cluster endpoint | 6443 | TCP | The kubelet and kube-proxy of every Kubernetes node. |
| The cluster endpoint's host | 50000 | TCP | Renewal of a worker's node certificate at a control plane, once two thirds of its lifetime have passed; control planes sign their own. |
| The other control planes | 2379, 2380 | TCP | etcd membership and replication. |
| The other Kubernetes nodes | 10250 | TCP | The API server reaching kubelets, from control planes. |
| The other Kubernetes nodes | 8472 | UDP | flannel's VXLAN. |
The neighbours of the node's BGPPeer documents |
179 | TCP | BGP sessions, which the node opens. |
The neighbours of the node's BGPPeer documents with bfd |
3784 | UDP | BFD's control packets. |
Config kinds describes the Mirror, Registry and TimeSync documents.
A VIP needs no port of its own: chalkd adds the address to an interface of the control plane that holds it and announces it to the neighbours with ARP or IPv6 neighbour advertisements. A routed VIP travels in the BGP sessions above.
Sources: the firewall rules of modules/node/chalkd.nix, modules/node/kubernetes,
modules/node/router, cmd/chalkos-render/nftables.go, pkg/kubernetes/vxlan and
pkg/kubernetes/clusterregistry, and the listeners of cmd/chalkd, cmd/chalkos-registry,
pkg/kubernetes/manifests/manifests.go, pkg/kubernetes/addons and modules/node/kubernetes.