Continuing on with my adventures in spinning up an HA k8s cluster at home I’m going to go a bit more into the software setup. This is by no means a guide on how to do this and honestly if you are spinning up a lightweight cluster for use at home I’d probably tell you to go find a k3s guide instead of this. I however wanted to use kubeadm to stand up a cluster, and kube-vip for getting the cluster nodes into HA.

For the uninitiated an HA (High Availability) cluster typically runs three or more controller nodes, typically keeping an odd number of controller nodes so you can maintain quorum while running the cluster. Then some arbitrary number of worker nodes that are used to run your day to day workloads. You then need some kind of load balancer to ensure traffic can be passed to any of the control nodes for API requests.

Getting the hosts ready

As stated in the previous post I have ten nodes, all Orange Pi 5 SBCs. The Ubuntu you get from them has a bunch of apt repos that point away from Canonical, and I’m a bit tinfoil hatty around this and while I could just point to the right repositories, I don’t know for sure they didn’t put some…interesting software on there. I opted to use a build from Joshua Riek called ubuntu-rockhip which supports builds for many Rockchip based SBCs.

I had originally planned on loading the OS on the NVME, but as the OS disks don’t really need the performance, and they do not need any redundancies, I can just load a new node up if it dies, I chose to put the OS on a 256GB SD card. I am curious how well they will hold up in the long run, I have had success doing this in the past but not on installs that write to the logs at this volume. As stated before though, loosing a node wouldn’t really be a big deal as I can just recreate it.

Automate the install

I picked Ansible for this task, mostly due to its ease of use being agentless, this comes with its ups and downs but it is hard to beat when it comes to writing some automation at home. I wrote my playbooks specifically around my setup, so while you can use it for inspiration, unless you have the same IP space, node count, etc. as I do you will need to tweak things.

There isn’t anything too clever here, as I wanted to keep it idempotent. The common role gets the nodes all ready, setting the timezone, hostname, installing requisite packages. I did pin kubeadm, containerd, kubelet, etc. to control upgrades. Then just setting things up for the next steps.

In the controller stage we do a couple of interesting things. We need to know the IPs of the other controllers to set up kube-vip to peer between them and generate the config for the static pods around that. I picked Cillium for my network, mostly just to tinker with it a bit more so we download the binary to install and use it to verify that the network is healthy.

Then we start kubeadm to bootstrap the cluster. We are deploying “stacked” nodes that run etcd on each controller, this is why we need an odd number of controllers as you need a majority to get quorum, and if you have an even number it is a recipe for split-brain in etcd. Once the cluster is up we need to generate the command to use to join the other controllers, and the workers. This was where I did get a bit clever. We generate a script from the output of the kubeadm token create command, the worker script is easy enough.

kubeadm token create --print-join-command > /tmp/worker-join.sh

The controller script took some figuring out as we need some extra certificates to come along for the ride.

 echo $(kubeadm token create --print-join-command) \
      --control-plane \
      --certificate-key $(kubeadm init phase upload-certs --upload-certs | grep -vw -e certificate -e Namespace) > /tmp/master-join.sh

With this we have the command the can help us find the certificates to join the controller nodes to the cluster. We then copy those scripts locally to then distribute to each node. From here we can finally install Cillium, verify the install, and then join the other control nodes to the cluster.

The workers are comparatively trivial, we just copy over the join script, then run it. Assuming no issues pop up the cluster is now ready to accept workloads! I also made a reset playbook, to get the cluster back to a clean slate when developing this.

That’s it! once the workers are in a Ready state we can start to deploy our workloads. I like to use GitOps for managing my deployments so my first task is typically installing Argo and pointing it at my repo.

Upgrades

The next step is upgrades, you can see another playbook for this, it is a bit of a rehash of some parts of the install script but we’re juggling holding and unholding some packages, and draining and uncordoning nodes. This is still a WIP but the workers part works just fine, but I still manually upgrade the controllers.