Chee Yee Lim's Personal Website

Some Closing Thoughts on Kubernetes

· 2 min read

Sharing the services and associated configurations I used to set up the Kubernetes backend powering my personal stack. In case the design is useful to others.

All services were hosted on DigitalOcean and managed using Helm with Helmfile, with almost no manual modifications to Kubernetes values required.

Almost all services are platform-agnostic (i.e. they can be deployed in any Kubernetes setup - AWS, GCP etc.). However the load balancer, ingress and persistent volume setups are specific to DigitalOcean.

Quick introductions on the services used here and their associated purpose:

  • ingress-nginx + cert-manager : Used to manage all Kubernetes ingress with associated SSL/TLS encryption.

  • hashicorp-vault + external-secrets : Used to manage secure secret storage and exposing those secrets to Kubernetes.

  • grafana + loki + prometheus : Used for observability monitoring on operation metrics and logs.

  • metrics-server : Used to resource monitoring.

  • velero : Used for Kubernetes backups.

  • wordpress + mariadb : Used to host a personal blog with an easy-to-use content management system (CMS).

  • homepage + dash-dashboard : These are custom charts wrapping around Python Flask and Dash respectively. Used to host custom websites and dashboards.

  • airflow : Used to manage data pipelines.

  • mlflow : Used to manage machine learning models.

  • postgresql + pgbouncer : Used as the main database across many services, e.g. Airflow, Dash.

  • redis : Used for Airflow Celery executor communication.

Some learnings on Kubernetes:

  • Kubernetes is actually not hard to use.

I would actually argue that deploying apps on Kubernetes is actually easier than deploying apps on virtual machines directly.

However the prerequisites are (1) good knowledge of how Linux and Docker work and (2) good knowledge of how the apps you are trying to deploy work.

Without these prerequisites, debugging non-working apps is extremely hard in Kubernetes due to multi layer abstractions (Linux -> Docker -> Kubernetes -> Apps).

  • Maintaining infrastructure takes more time and effort than you may initially think.

This may sound obvious, but it is amplified by the ease of deployment on Kubernetes. Spinning up new services is easy, but the complexity of managing interacting services grows exponentially very quickly.

  • The cost of hosting on Kubernetes is higher than using managed services at a small scale.

This is because you will need to host utility services that support actual services that you need to run, for example services for secret management and observability monitoring.

At a larger scale, the cost of hosting these utility services diminished as a proportion of total cost due to the economy of scale. However at a smaller scale, the cost of these utility services is quite apparent. In my example, hosting these utility services accounted for 40-50% of my total cloud bill.

As mentioned in my previous post, I will pause the Kubernetes experiment for now and redesign the services to be hosted on a mix of managed and self-hosted mode.

You can find the link to the repo here : https://github.com/cheeyeelim/k8s-personal-boilerplate

Note that about 10 of the services are still pointing to the now-closed Bitnami charts/images repository, so those would require manual migration and testing.