Using strace in Docker and Kubernetes: A Practical Guide for Debugging

When working with containers, we often treat them as black boxes: you run the image, check the logs, and if something fails, you rebuild. But what if the logs don’t tell you why your containerized app is crashing, hanging, or failing to connect to a service?

That’s where strace comes in.

strace is a Linux tool that intercepts and records system calls made by a process. In simple terms, it shows you what a program is doing “under the hood” — opening files, sending network packets, waiting on I/O, failing to access resources, etc. In containerized environments like Docker and Kubernetes, strace becomes invaluable for deep troubleshooting.

Why Use strace in Containers?

Containers add a layer of abstraction. Your app might run fine on your laptop but fail in production because:

  • A required library is missing in the container image.
  • File paths or permissions differ.
  • Network policies in Kubernetes block connections.
  • Syscalls are blocked by security profiles (e.g., seccomp).

Logs often won’t show these low-level issues. strace can.

For example:

  • Debugging a containerized Python app that says Connection refused. With strace, you can confirm whether it’s actually trying to connect to the expected host/port.
  • Tracking down a file-not-found error in a Go binary. strace reveals exactly which path the binary is trying to open.
  • Identifying why a container exits immediately. strace shows the failing syscall that triggers the crash.

Using strace with Docker

Installing strace

Most minimal images (Alpine, Debian-slim, Distroless) don’t ship with strace. You can:

  1. Install it temporarily inside a running container:

    docker exec -it <container_id> apt-get update && apt-get install -y strace

  2. Use a debug sidecar or custom image that includes strace.

Attaching strace to a Running Container

  1. Find the container PID:

    docker inspect -f '' <container_id>

  2. Attach strace from the host:

    sudo strace -p <PID>

    This avoids modifying the container at all.

Example: Debugging File Access

strace -p <PID> -e open,openat,read,write

You’ll see which files the process is opening, and whether it’s failing with ENOENT (file not found) or EACCES (permission denied).

Using strace in Kubernetes

Debugging inside Kubernetes is trickier because you don’t have direct access to nodes and pods may be scheduled anywhere.

Method 1: Install strace in the Pod

If your container allows:

kubectl exec -it <pod_name> -- sh
apt-get update && apt-get install -y strace
strace <command>

This is invasive but works for dev environments.

Method 2: Run strace as a Sidecar

Attach a privileged debug sidecar with strace pre-installed, then use it to enter the same process namespace as the main container.

Method 3: Attach strace from the Node

If you have node access:

  1. Get the pod’s container ID:

    crictl ps | grep <pod_name>

  2. Find the PID of the container process:

    crictl inspect <container_id> | grep pid

  3. Attach strace on the node:

    sudo strace -p <PID>

Real-World Examples

  1. Debugging “Connection Refused”

    strace -p <PID> -e connect

    Output:

    connect(3, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("10.96.0.12")}, 16) = -1 ECONNREFUSED

    → Confirms the app is trying to connect to MySQL on the right IP, but the service isn’t reachable.

  2. Investigating Crashes

    strace <binary>

    Output ends with:

    open("/lib/x86_64-linux-gnu/libssl.so.1.1", O_RDONLY) = -1 ENOENT (No such file or directory)

    → Missing library inside the container.

  3. Checking Seccomp Restrictions If a container is killed unexpectedly, strace might reveal:

    --- SIGSYS {si_signo=SIGSYS, si_code=SYS_SECCOMP, ...} ---

    → A syscall is blocked by the seccomp profile.

Best Practices

  • Don’t run strace in production unless necessary — it’s intrusive and can slow down the process. Use it as a last-resort debugging tool.
  • Prefer attaching from the host/node over installing strace in production containers.
  • Use ephemeral debug containers (kubectl debug) in Kubernetes 1.18+ for safer troubleshooting.
  • For ongoing observability, combine with tools like tcpdump, gdb, or eBPF-based tracers.

Conclusion

strace is a powerful ally when debugging tricky issues in containerized environments. In Docker and Kubernetes, where abstraction layers hide the underlying OS, strace lets you peek inside and see exactly what your app is asking the kernel to do — and why it might be failing.

The next time your container crashes with a vague error message, don’t just rebuild the image. Try strace and let the syscalls tell the story.