Trick #5 – Waiting For Kubernetes

When you deploy stuff for a living, you find yourself waiting around, a lot. Wait for Terraform to spin up the AWS VPCs. Wait for the Kubernetes cluster node VMs to boot. Wait for the Kubernetes cluster to coalesce. Wait for the CNI pods, DNS pods, and kube-proxy bits to be happy. Wait, wait, wait.

kubectl feels your pain there, and has some nifty ways of helping you wait (or not!) for the things you deploy or delete to coalesce.

To wait for a single pod to do something, we can use the wait command, and pause until a condition is met on the pod:

$ kubectl wait pod/slow --for condition=readypod/slow condition met

Here’s a more rigorous shell session, with date bounding so you can better visualize the passage of time:

$ kubectl apply -f trick5/pod.yml
namespace/trick5 created
configmap/opt created
pod/slow created
$ date
Tue Feb 11 10:58:31 EST 2020
$ kubectl get pods
NAME   READY   STATUS     RESTARTS   AGE
slow   0/1     Init:0/1   0          1s
$ kubectl wait pod/slow --for condition=ready
pod/slow condition met
$ date
Tue Feb 11 10:58:39 EST 2020        # ~8s later!
$ kubectl get pods
NAME   READY   STATUS    RESTARTS   AGE
slow   1/1     Running   0          9s

(This uses the pod.yml from the Silly Kubectl Tricks GitHub repository)

You’ll probably want to wait for either ready, initialized, or containersReady. If you want, you can even define your own readiness gates, which will delay the ready condition until your custom gates resolve to true.

You can find out more, here, in the official Kubernetes documentation.

Bigger Things Than Pods

Chances are you aren’t dealing directly with individual pods, but are concerned chiefly with Deployments, StatefulSets, or DaemonSets. For deployments, you can use kubectl rollout status

kubectl rollout status deploy/slow
Waiting for deployment "slow" rollout to finish: 0 of 5 updated replicas are available...
Waiting for deployment "slow" rollout to finish: 1 of 5 updated replicas are available...
Waiting for deployment "slow" rollout to finish: 2 of 5 updated replicas are available...
Waiting for deployment "slow" rollout to finish: 3 of 5 updated replicas are available...
Waiting for deployment "slow" rollout to finish: 4 of 5 updated replicas are available...
deployment "slowdep" successfully rolled out

This command will block while the entirety of the deployment is incrementally “rolled out” and comes up to a state of readiness. As the pods in the current live replica set spin up. kubectl rollout status prints helpful progress updates, to keep you abreast of what’s happening while you wait.

This works for StatefulSets and DaemonSets, too.

You can also use kubectl wait for deployments:

$ kubectl wait deploy/slow --for condition=available
deployment.apps/slow condition met

Note that the condition for a deployment is available, not ready.

We find kubectl wait to be a useful tool for change automation, properly ordering things outside of Kubernetes, and general Zen-like meditative states. We hop you will too!