Life pro tip: put your active kubernetes context in your prompt

Today I did an oopsie. I tried to upgrade a service in my homelab cluster (alrest) but accidentally upgraded it in the production cluster (aeacus). I was upgrading ingress-nginx to patch the security vulnerabilities released a while ago. I should have done it sooner, but things have been rather wild lately and now kernel.org runs some software I made.

Cadey

Either way, I found out that Oh my ZSH (the ZSH prompt toolkit I use) has a plugin for kube_ps1. This lets you put your active Kubernetes context in your prompt so that you're less likely to apply the wrong manifest to the wrong cluster.

To install it, I changed the plugins list in my ~/.zshrc:

-plugins=(git)
+plugins=(git kube-ps1)

And then added configuration at the end for kube_ps1:

export KUBE_PS1_NS_ENABLE=false
export KUBE_PS1_SUFFIX=") "

PROMPT='$(kube_ps1)'$PROMPT

This makes my prompt look like this:

(⎈|alrest) ➜  site git:(main) ✗

Showing that I'm using the Kubernetes cluster Alrest.

Aoi

Wouldn't it be better to modify your configuration such that you always have to pass a --context flag or something?

Cadey

Yes, but some of the tools I use don't have that support universally. Until I can ensure they all do, I'm willing to settle for tamper-evident instead of tamper-resistant.

Why upgrading ingress-nginx broke my HTTP ingress setup

Apparently when I set up the Kubernetes cluster for my website, the Anubis docs and other things like my Headscale server, I did a very creative life decision. I started out with the "baremetal" self-hosted ingress-nginx install flow and then manually edited the Service to be a LoadBalancer service instead of a NodePort service.

I had forgotten about this. So when the upgrade hit the wrong cluster, Kubernetes happily made that Service into a NodePort service, destroying the cloud's load balancer that had been doing all of my HTTP ingress.

Thankfully, Kubernetes dutifully recorded logs of that entire process, which I have reproduced here for your amusement.

Cadey

OOPS!

Numa

Pro tip if you're ever having trouble waking up, take down production. That'll wake you up in a jiffy!

Thankfully, getting this all back up was easy. All I needed to do was change the Service type back to LoadBalancer, wait a second for the cloud to converge, and then change the default DNS target from the old IP address to the new one. external-dns updated everything once I changed the IP it was told to use, and now everything should be back to normal.

Well, at least I know how to do that now!

Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.

Tags: