Trick #7 – Merge All The KUBECONFIGs!

Once you start managing more than one Kubernetes cluster, you’ll start to demand more from your $KUBECONFIG. Early on in my career spinning pods, I kept a separate .kubeconfig file for each cluster, and wrote a small shell function to set the requisite environment variables. To switch between clusters was as simple as:

→  k is buffalo-labkubernetes: buffalo-lab
  (config): /Users/jhunt/.k/buffalo-lab

I did this because the “correct” command was too much to remember:

kubectl config use-context buffalo-lab

See what I mean?

Then I met kubectx and kubens, and they quickly became invaluable.

Now I’ve got this directory full of all these disparate, discrete configuration files, and I don’t want to hand-edit them back together. Thankfully, kubectl itself can merge them for us!

Note: kubectl will only merge configuration files if you specify them via the $KUBECONFIG environment variable. You cannot use the --kubeconfig command-line flag, it won’t work.

To demonstrate this, I’m going to show you my ~/.k directory:

$ ls -l ~/.k
total 96
-rw-r--r-- 1 jhunt staff  7050 Jan  6 09:56 buffalo-lab
-rw-r--r-- 1 jhunt staff  5641 Jan  1 11:30 jhunt-vcp-prodernetes.yml
-rw-r--r-- 1 jhunt staff  5633 Jan  1 11:30 jhunt-vcp-tinynetes
-rw-rw-rw- 1 jhunt staff  5481 Jan  3 12:32 lke-eval1
-rw-r--r-- 1 jhunt staff 23633 Jan  1 11:56 merged
-rw------- 1 jhunt staff   422 Nov 17 14:25 minikube
-rw-r--r-- 1 jhunt staff  7023 Jan  1 11:30 oss
-rw-r--r-- 1 jhunt staff 25706 Jan  3 12:40 tmp

Using kubectl config get-contexts commands, we can see that each of them is different:

$ KUBECONFIG=~/.k/buffalo-lab \
kubectl config get-contexts
CURRENT   NAME              CLUSTER           AUTHINFO   NAMESPACE
*         buffalo-lab-k8s   buffalo-lab-k8s   admin      default
$ KUBECONFIG=~/.k/lke-eval1 \
kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         [email protected]   kubernetes   kubernetes-admin   k8s-cp-ubuntu

If you put two (or more) files, separated by colons, in $KUBECONFIG, then kubectl will merge them together semantically, and operate on the composite configuration. Any changes you make, like changing namespaces or setting contexts, will update the appropriate member file(s).

$ KUBECONFIG=~/.k/buffalo-lab:~/.k/lke-eval1 \
kubectl config get-contexts
CURRENT   NAME                          CLUSTER           AUTHINFO           NAMESPACE
*         buffalo-lab-k8s               buffalo-lab-k8s   admin              default
          [email protected]   kubernetes        kubernetes-admin   k8s-cp-ubuntu

To write a new, aggregate configuration that contains the results of the merge, we can use kubectl config view --raw. The --raw flag prevents kubectl from redacting user secrets, certificates, and keys. To save this combined file, we can employ some shell redirection, along with a bit of caution.

Every UNIX shell I’ve ever used truncates files that are the target of >-style output redirection, before the command producing the output gets executed. It’s the nature of UNIX pipelines. We have to be careful with the order of our operations.

This will fail spectacularly (and most assuredly destroy data in the process!):

# DON'T RUN THIS!
$ KUBECONFIG=a:b:c kubectl config view --raw > a

The a configuration file will be empty by the time kubectl gets around to trying to merge it in with the other two. Instead, use a temporary file and a rename operation:

$ KUBECONFIG=tmp:a:b:c kubectl config view --raw > tmp \
&& mv tmp a

Pro Tip: While you are merging all of these configurations together, you can also embed any referenced, external files (like certificates and private keys) by swapping out the --raw flag for --flatten. This produces “portable” kubeconfigs, which do not have any dependencies on other parts of your filesystem. They can be moved, or copied, without losing their utility.