Migrating to GKE

For some time now I’ve been a fan of Kubernetes. I run a small cluster at home for tinkering, but it doesn’t really enable me to get a feel for running this in the cloud. I need to setup my own load balancer (metallb) storage is more annoying as I need to provide that as well (glusterfs). I have sorted those problems, but I still wanted to try running something in the cloud.

So several months ago I migrated to GKE for my external facing services (this blog is one of them) so I could start getting a feel for what needs to be done to run in GKE.

Why GKE

At the moment it is, in my opinion, the best managed k8s solution, with one major advantage being I do not need to pay for the master. And another being that preemptible instances have a set price that I do not need to bid for.

Some other minor reasons:

  • Faster node bringup
  • Auto node upgrades
  • Cheaper instances overall
  • Tighter integration with cloud console

Ultimately any managed solution that removes the need to manage etcd is a win in my book. So feel free to choose your own adventure, as aside from the actual cluster bringup this is pretty cloud agnostic.

Installing the cluster

Bringing up a GKE cluster is pretty straightforward and only takes about 5 minutes. We have 3 requirements.

  1. 3 Worker nodes
  2. All run on preemptible instances
  3. At least 2 vCPUs and 4GB of memory available

That last point is important, I want 4GB of memory available for containers, so that is after all of the requirements for running the worker node. I chose the n1-standard-2 instances as it gives me ample memory. I could have gone with the custom option to precisely pin down my memory, but the price difference is negligible.

I am not going to go over creating a cluster, as the docs to a great job of that. You will need to install the Google cloud SDK, and kubectl to follow the steps. You can find the docs here

Migrating the blog

I use Hugo as a static site generator, and build my blog from Markdown files. So there was a question of how do I get said static files into my cluster. I had a few ideas.

  • Create a container that has the static files
  • Create repository that is the latest static files
  • Generate the static files in the container

Create a container hosting static files

This was my first reaction, but was very quickly dismissed. While this would work just fine you would need to generate a new container every time you make a new post. That puts more manual work on making blog posts, and really “releases” should not be tied to posts.

Create a repository for static files

I actually went this route for a bit. I would simply generate and push the static files to a public repo that the container would pull from every so often. I was thinking about setting up some hooks to have the pulls be triggered, but ultimately I didn’t like having two repos for the one blog. Plus I have other blogs I want to pull over and would like a more consistent way of doing things.

Make a script to generate the static files from the repository

This is where I am now. I created a script that will do a git pull on my blog repo with the Markdown files every 5 minutes, and generate the static files.

I run this as two containers in the pod with a shared volume mount for the static files. One container is just a plain old nginx container that I have volume mounted the static files to where nginx expects them to be. The other is a container that runs my script.

Pull script

The script is pretty simple but it has one major requirement, it needs to read from a private git repo. Thankfully storing ssh keys in k8s secrets is a somewhat common pattern and is easy to do.

So on startup the script registers the ssh key, clones the repo, pulls in the submodules, then enters an infinite loop where it pulls the latest changes, generates the static files, and then sleeps for 5 minutes.

I’ve found this to be a great hands off approach, we only need to watch the one repo, and I can create branches to save drafts, then merge to master to post. I did run into some hiccups like the script being run seemingly before network was available, so I added a check to wait until it can reach github.

Getting traffic into the cluster

Next up was choosing something for ingress. I decided on Traefik mostly just because I wanted to tinker around with it. I like the fact that it is written in Go, and it is pretty simple to set up and run (when not in HA mode) so I went for it.

Installing Traefik and helm to the cluster

I decided to install this with Helm, again mostly because I wanted to tinker with Helm as well and the install with helm seemed to be pretty straightforward. So off I went and loaded up helm into the cluster. If you want to get started with Helm I’d suggest reading through the quickstart guide before installing.

Once Helm was up and running I created a set of directories in my github repo for my k8s cluster specifically helm/traefik-values, this contained the template info for helm. I initially tried to run Traefik in HA mode, and the deployment went well enough. I use letsencrypt for my certificates for external services, and Traefik supports this out of the box, but when you do this in HA mode Traefik needs to be able to share that cert via a KV store.

Deploying etcd to the cluster for Traefik (then undeploying it)

When I decided to run Traefik in HA mode I settled on etcd, it seemed to be the best supported by Traefik at the time, and I was already pretty familiar with it. I set about deploying etcd via helm using the etcd operator, and this was less than straightforward. With such a small cluster it was imperative that each instance run on a separate node, I incorrectly assumed the operator would assert this, it did not. When I lost my first instance after the install I lost quorum and etcd refused to start. After I figured this out I set about ensuring they would be launched on separate nodes. Thankfully the chart/operator let you pass in a pod affinity, I used something that looks like this:

  pod:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: etcd_cluster
              operator: In
              values: ["etcd-cluster"]
          topologyKey: kubernetes.io/hostname

Which let me get to a point where all pods were being launched on separate nodes, yay! But as I would find out soon there was another problem. When a node would go down, a new instance would be spun up, and the etcd cluster would now think it had 4 members, and 2/4 is too little to maintain quorum, so the cluster would go down, whee! Since this cluster is primarily for personal projects I decided to just ditch the HA with Traefik for now, and come back to it later. When the node hosting Traefik goes down there is about a minute that the cluster is unresponsive to http. For now I can deal with that. Hopefully Traefik 2.0 has a better solution, or I just need to figure out how to run an etcd cluster on a 100% preemptible k8s cluster.

Ingress and letsencrypt

I’ve been very happy with how well Traefik handles new ingress routes and automatically fetches a certificate for it. All I need to do is create an ingress rule for it and bam! Which looks something like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: blog
  namespace: blog
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: blog.heschlie.com
    http:
      paths:
      - path: /
        backend:
          serviceName: blog
          servicePort: http

Migrating the wiki

I keep a wiki for note taking on personal projects I want to make, or stories I want to tell for games. Most of them end their life in the wiki as ideas, but a few I’ve made at least some progress on. This was a more interesting adventure as I needed a mongodb cluster as well.

Trying out kubedb

I decided to try the kubedb operator out, it lets you install a number of databases in a cluster and manage backups and recovery. Their docs leave a lot to be desired, but I managed to at least deploy a mongodb cluster to my k8s install. I’m not taking advantage of the full power of kubedb at this point so I can’t really attest to if it is really worth it or not, but I will say that if I were to redeploy mongo I’d probably just use a helm chart.

Moving Wikijs to k8s

This one was a bit tricky, and the software doesn’t seem to like seeing the mongo nodes going up and down as it seems to frequently loose its connection to mongo. Overall this was pretty fairly easy, modify the config to work in k8s, push it into a configmap, save the DB string to a secret to be pulled in, and run the deployment. The main issue I had that threw me through a loop was figuring out the DB string. I knew that statefulsets would present a stable endpoint to talk to each instance but not having done this before it took a bit of reading to figure this out. It looks something like this when in the same namespace:

mgo-replicaset-0.mgo-replicaset-gvr
mgo-replicaset-1.mgo-replicaset-gvr
mgo-replicaset-2.mgo-replicaset-gvr

This took me longer to figure out than I care to admit. but once I had sorted it out things finally came up. Some other pain points:

  • Wikijs does not maintain an image per version, just latest, dev, and beta
  • The logs that are printed to stdout are not helpful, you need to exec into the container to look at more detailed logs
  • Wikijs seems to not cope well with mongodb instances going down and coming back up

Conclusion

It has been a fun adventure into running my own k8s cluster, I’ve since migrated another log over which I converted from wordpress to Hugo, and am getting ready to deploy a Go app I’ve been writing for some time to the cluster to get an idea of how to create and run my own app, which needs a postgres DB to back it, I’m debating on whether or not to try out kubedb for it to see if it is any better with postgres and try to take advantage of backups.

The cluster isn’t terribly expensive to run, it costs me about $60/m for the 3 nodes and a load balancer in GCP to point to my ingress. compared to the two linode boxes I was using at $20m and hosting some other services out of my server at home with only a 30mbit upload speed. I’ll likely bump to 5 nodes when I deploy my next “big” app, big being relative, my blogs are super lightweight to run being just statically generated sites.