Setting up a Kubernetes Cluster with Raspberry PI 4

After yet another nights‑long Kubernetes cluster setup, I created this document so I don’t have to repeat all the late‑night Googling.

Consider your storage

This also cost me quite some time. Do NOT use slow storage (e.g. a USB stick), especially on the master node. This causes random crashes in etcd, kube-scheduler and apiserver causing the cluster to be unreachable.

The reason is that containers start so slowly that the readiness checks and liveness checks time out.

Enable required cgroups (very important on Pi OS)

Edit /boot/cmdline.txt (single line!). Append cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 at the end, separated by a space, or just run:

sudo sed -i 's/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1/' /boot/firmware/cmdline.txt

Reboot:

sudo reboot

Verify

grep -E 'cgroup|memory' /proc/cmdline

Turn off swap

sudo dphys-swapfile swapoff || true
sudo systemctl disable --now dphys-swapfile || true
sudo swapoff -a

Turn off zram:

# stop zram and turn off swap
sudo systemctl stop 'systemd-zram-setup@*'
sudo swapoff -a    # schaltet ALLE Swap-Quellen ab (inkl. zram)

# uninstall zram generator - this really works on bookworm
sudo apt purge systemd-zram-generator

I had issues with guides on how to disable the generator—they did not work at all.

Confirm:

free -h | grep -i swap

Kernel modules & sysctl for container networking

# Ensure required kernel modules at boot
cat <<'EOF' | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

# Netfilter & forwarding
cat <<'EOF' | sudo tee /etc/sysctl.d/99-k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

Install containerd (and configure Systemd cgroups)

sudo apt install -y containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
# Use systemd cgroups (Kubelet expects this)
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
# Use the newer pause-container, kubernetes would like
sudo sed -i 's/sandbox_image = \"registry.k8s.io\/pause:3.8\"/sandbox_image = \"registry.k8s.io\/pause:3.10\"/' /etc/containerd/config.toml
sudo systemctl enable --now containerd

Install CNI plugins (this was the missing piece when CoreDNS was stuck)

sudo apt install -y nfs-common # (optional but handy)
# On Debian/PI OS Bookworm the package is named:
sudo apt install -y containernetworking-plugins
# On some distributions it's 'cni-plugins' – but 'containernetworking-plugins' is correct here.
ls -1 /opt/cni/bin | head

Kubernetes expects CNI binaries in /opt/cni/bin (default). Don’t add –cni-bin-dir to the kubelet; newer kubelets reject that flag.

Add Kubernetes v1.33 apt repo and install kubelet/kubeadm/kubectl

sudo apt-get install -y apt-transport-https ca-certificates curl gpg
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \
https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" \
| sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

After starting the node / after join

If this is the master node, set it up first and install Flannel before this step.

This works only after the node has started or joined. /opt/cni is created after that.

In my current Version (1.33) the option --cni-bin-dir is not available anymore and causes kubectl to fail to start.

ls /opt/cni/bin/
# check, if everything is present
sudo ln -sf /opt/cni/bin/* /usr/lib/cni/

Setting up the master Node

Flannel defaults to 10.244.0.0/16. Use the same when initializing:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

Pre-flight-checks might issue a warning about not having hugetlb cgroup. You can ignore this, as it is irrelevant on Raspberry Pi.

Set up kubectl for your user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Save the join command that kubeadm prints, or regenerate anytime with:

kubeadm token create --print-join-command

Install Flannel (ARM-friendly CNI):

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Check if it’s running:

kubectl -n kube-flannel get ds,pods -o wide

Don’t forget that you might do the linking of /opt/cni/bin.

You should soon see interfaces on the node:

ip link show cni0
ip link show flannel.1

installing flannel and metal-lb (master node only)

Only needed once for the whole cluster. Just apply this one manifest:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/main/config/manifests/metallb-native.yaml

# Configure pool – Choose IPs from your LAN (must not be used by DHCP!)
cat <<'EOF' | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.200-192.168.1.220 #example, replace with actual IPs
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system
EOF

Installing an Ingress (NGINX)

Only needed once for the whole cluster.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml

Problems with kubernetes

Taints (Master node only)

If you want the master node to also run workloads (not recommended):

kubectl taint nodes $(hostname) node-role.kubernetes.io/control-plane- || true
kubectl taint nodes $(hostname) node-role.kubernetes.io/master- || true

If your cluster is properly set up, reapply the taints:

kubectl taint nodes $(hostname) node-role.kubernetes.io/control-plane:NoSchedule
kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule

Disk Pressure

see here for more information

needs to be set in /var/lib/kubelet/config.yaml on the node.

Coredns causing issues?

If the Flannel setup previously failed and was later fixed, you can restart CoreDNS with this command.

kubectl -n kube-system rollout restart deploy/coredns