Stark & Wayne

Trick #5 - Sure, But How Did It Fail Last Time?

Logs are a great way of troubleshooting; they can provide a wealth of insight into the inner workings of a process.

But, what happens when pods crash?

If we're using a Deployment, DaemonSet, or StatefulSet, Kubernetes will unerringly recreate crashed pods (after giving them some time to breathe). To keep things neat and tidy, Kubernetes starts the new pod off with a blank log buffer.

This is handy, in that we never get stray, unrelated log messages when we issue a kubectl logs.  It is decidedly less helpful if our pod keeps crashing and being recreated. This happens a lot in the early stages of systems implementation, until we fine-tune images, commands, and environments.

How do we get at the previous log buffers? With the kubectl logs -p flag!

Here's a scenario where I need to get at previous logs. I have a service that should stay up indefinitely, but keeps restarting in the wee hours of the morning. When I check the logs, I get the successful pod's current log buffer, which looks healthy:

$ kubectl logs crashing-pod
[Mon Feb 10 16:29:19 UTC 2020] starting up...
[Mon Feb 10 16:29:34 UTC 2020] startup complete; initializing.
[Mon Feb 10 16:29:34 UTC 2020] - frobbing the data cache...
[Mon Feb 10 16:29:34 UTC 2020] - allocating grokbase fooblers...
[Mon Feb 10 16:29:34 UTC 2020] - reticulating splines...
[Mon Feb 10 16:29:34 UTC 2020] - reclaiming unused memory...
[Mon Feb 10 16:29:34 UTC 2020] - checking in on friends and family...
[Mon Feb 10 16:29:34 UTC 2020] initialization complete; entering main loop.

What I really need is the end of the log buffer from the failed / crash instance:

$ kubectl logs -p crashing-pod
[Mon Feb 10  7:18:20 UTC 2020] starting up...
[Mon Feb 10  7:18:35 UTC 2020] startup complete; initializing.
[Mon Feb 10  7:18:35 UTC 2020] - frobbing the data cache...
[Mon Feb 10  7:18:35 UTC 2020] - allocating grokbase fooblers...
[Mon Feb 10  7:18:35 UTC 2020] - reticulating splines...
[Mon Feb 10  7:18:35 UTC 2020] - reclaiming unused memory...
[Mon Feb 10  7:18:35 UTC 2020] - checking in on friends and family...
[Mon Feb 10  7:18:35 UTC 2020] initialization complete; entering main loop.
[Mon Feb 10  7:18:35 UTC 2020] ...
[Mon Feb 10  7:18:35 UTC 2020] ...
[Mon Feb 10  7:18:35 UTC 2020] ...
[Mon Feb 10  7:18:35 UTC 2020] ...
[Mon Feb 10  7:18:35 UTC 2020] UNMITIGATED DISASTER!
[Mon Feb 10  7:18:35 UTC 2020] THREAD PANIC detected; no assurances given.
[Mon Feb 10  7:18:35 UTC 2020] IT'S A BAD SCENE, MAN!
[Mon Feb 10  7:18:35 UTC 2020] <crashed>

With that information in hand, I can continue my diagnostic efforts and hopefully resolve this early morning nuisance.

Find more great articles with similar tags kubernetes k8s tricks