Trick #6 – Label it to Enable it

The building block of almost all Kubernetes deployments is the pod – one or more containers sharing a network stack. Pods are where the magic happens, where we get our logs, and where we spend most of our time troubleshooting outages and malfunctions.

However, we rarely create pods ourselves. Instead, we rely on higher level constructs like ReplicaSets and Deployments to create them for us.

Unfortunately, ReplicaSets (and by extension, Deployments) are just awful at naming pods.

$ kubectl get podsNAME                           READY   STATUS    RESTARTS   AGE
admin-ui-6d87999cdf-27rdj      1/1     Running   0          43s
admin-ui-6d87999cdf-4pl64      1/1     Running   0          43s
admin-ui-6d87999cdf-8n2rq      1/1     Running   0          43s
app-server-66487d84f8-6hkcc    2/2     Running   0          42s
app-server-66487d84f8-6kwxp    2/2     Running   0          42s
app-server-66487d84f8-9fl5k    2/2     Running   0          42s
app-server-66487d84f8-dhfq7    2/2     Running   0          42s
app-server-66487d84f8-q5x6m    2/2     Running   0          42s
app-server-66487d84f8-t4t4r    2/2     Running   0          42s
app-server-66487d84f8-tlk54    2/2     Running   0          42s
db-6b649cbc68-vt9qh            1/1     Running   0          43s
db-6b649cbc68-zgz5p            1/1     Running   0          43s
redis-cache-6f54c47d4c-4rknd   1/1     Running   0          44s
redis-cache-6f54c47d4c-9mq7d   1/1     Running   0          44s
redis-cache-6f54c47d4c-crqfp   1/1     Running   0          44s
redis-cache-6f54c47d4c-xxdzq   1/1     Running   0          44s

Horrendous.

This wouldn’t matter too much if we never had to reference a pod by name. But that’s exactly what we have to give kubectl for any command involving a pod, or one of its containers. If we want to review logs, execute arbitrary commands inside the container, or even see how the pod is doing, we’re going to be constantly running kubectl get pods and copying / pasting those names around.

(Note: you can find a Kubernetes resource spec on the GitHub repository that accompanies this blog series, if you want to play along at home.)

Unless we use labels!

Labels are key+value pairs that we, as operators, get to assign to the things we deploy on Kubernetes. Here’s how we specify pod labels in a Deployment definition:

---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      labels:
        env:     prod
        role:    frontend
        app:     redis-cache
        service: redis-cache

While labels are used internally by other parts of Kubernetes (like Deployments, Services, and the like), we can also benefit from them with the kubectl get command and its -l flag. Combine that with some data extraction, and we can do some pretty amazing things:

$ kubectl get all -l role=frontend -o custom-columns=NAME:.metadata.name
NAME
app-server-66487d84f8-6hkcc
app-server-66487d84f8-6kwxp
app-server-66487d84f8-9fl5k
app-server-66487d84f8-dhfq7
app-server-66487d84f8-q5x6m
app-server-66487d84f8-t4t4r
app-server-66487d84f8-tlk54
redis-cache-6f54c47d4c-4rknd
redis-cache-6f54c47d4c-9mq7d
redis-cache-6f54c47d4c-crqfp
redis-cache-6f54c47d4c-xxdzq
app-server-66487d84f8
redis-cache-6f54c47d4c

The demo.yml definitions that we’re using define some labels that we can use to filter our queries:

• env – We set this to either prod (stuff that goes down and makes our lives unbearable) or admin (stuff that goes down and makes our lives inconvenient).
• app – This is mostly a unique-per-deployment label that we use for selectors to wire up the Deployment object through its ReplicaSet to its constituent Pods.
• service – Like app, we use this label to identify the pods that make up a named service, for wiring up Service definitions.
• role – In a front-end / back-end paradigm, which end does this piece belong to?

Armed with these semantics, we can run some neat sub-queries against our Pods:

Q: How is prod doing today?

$ kubectl get pod -l env=prod -o wide
NAME                           READY   STATUS    RESTARTS   AGE
app-server-66487d84f8-6hkcc    2/2     Running   0          8m54s
app-server-66487d84f8-6kwxp    2/2     Running   0          8m54s
app-server-66487d84f8-9fl5k    2/2     Running   0          8m54s
app-server-66487d84f8-dhfq7    2/2     Running   0          8m54s
app-server-66487d84f8-q5x6m    2/2     Running   0          8m54s
app-server-66487d84f8-t4t4r    2/2     Running   0          8m54s
app-server-66487d84f8-tlk54    2/2     Running   0          8m54s
db-6b649cbc68-vt9qh            1/1     Running   0          8m55s
db-6b649cbc68-zgz5p            1/1     Running   0          8m55s
redis-cache-6f54c47d4c-4rknd   1/1     Running   0          8m56s
redis-cache-6f54c47d4c-9mq7d   1/1     Running   0          8m56s
redis-cache-6f54c47d4c-crqfp   1/1     Running   0          8m56s
redis-cache-6f54c47d4c-xxdzq   1/1     Running   0          8m56s

Q: What images are we running on the backend?

Note: This one uses the image.fmt output format from Silly Kubectl Trick #2.

$ kubectl get pod -l role=backend -o custom-columns-file=../trick2/images.fmt
NAMESPACE   NAME                        IMAGE
trick6      admin-ui-6d87999cdf-27rdj   nginx
trick6      admin-ui-6d87999cdf-4pl64   nginx
trick6      admin-ui-6d87999cdf-8n2rq   nginx
trick6      db-6b649cbc68-vt9qh         mariadb
trick6      db-6b649cbc68-zgz5p         mariadb

Using subshell expansion in Bash and Zsh, we can combine these calls with other commands that deal with individual pods:

$ kubectl describe $(kubectl get pods ...)
$ kubectl logs $(kubectl get pods ...)

Powerful, but a little less convenient than I would like. On my machine, I put the common prefix command from the above subshells into a small script in my $PATH, called kid:

$ cat ~/bin/kid
#!/bin/sh
exec kubectl get -o jsonpath='{.items[*].metadata.name}' "[email protected]"

That shortens my describe and logs calls to just:

$ kubectl describe $(kid pod -l ...)
$ kubectl logs $(kid pod -l ...)

This also works for other resource objects, like Volumes, Services, etc.

$ kubectl describe $(kid volume -l env=prod)

You can do more than just look. For example, if the admin-ui from our sample deployment is off in the weeds, and we want to delete / recreate all of the pods:

$ kubectl delete $(kid pod -l env=admin)

Just remember: with great power, comes great responsibility.