Quick Byte: Analyse stopped container

Problem

An application container crashed, and you do not see why. The obvious thing to do is to check the logs. But the container is already gone. So, how can I access the logs in kubernetes, then?

Logging infrastructure

Ideally, you have an infrastructure that hosts an instance of, for example, Kibana – through which you can search container logs. For this example, we will assume that this is not available. This is typical if you are running an app in your local kubernetes cluster. Or if you are testing a new deployment in your dev environment that is not wired up yet with the logging infra.

Reasons for stopped pods

A pod is not stopped for no reason. The most common cases include: timeouts while initialising the application. Usually, this is related to misconfigurations or failed upstream call for dependencies to init the application.

Retrieve the logs 💻

Before I explain to you how to retrieve logs from a gone container, Let me briefly remind you of the command that helps you to get the logs from a present pod:

kubectl logs my_pod --container my_container

Note: If your pod has only one container, you can omit the --container parameter.

Now, to retrieve the logs from the pod that preceded the current one, you will need to add --previous (or -p) to the command:

kubectl logs pod-name --previous

For more helpful options, I recommend taking a closer look at the command reference.

But where do the logs come from? 🤯

It is important to understand that the above command does not retrieve the logs from the pod resource itself. The logs are stored in logs files on the kubernetes node. As can be seen in the following picture.

When you use the kubectl command interface, it uses the Kubernetes API under the hood. Take a look at the documentation for more details.

Further reading

• Logging architecture – basic info on configuration and log processing
• Kubernetes components – how the kubernetes API is connected with the nodes