Debugging on Lead DevOps

coredumpctl: Finding a Binary Crash

Fri, 18 Sep 2026 00:00:00 +0000

What is coredumpctl and how it works

When a binary crashes with SEGV, the kernel can save a core dump — a snapshot of process memory at the moment of the crash. In systemd-based distributions, coredumpctl handles collecting, storing, and searching these dumps. It’s a wrapper around systemd-coredump, which stores dumps in /var/lib/systemd/coredump/ and indexes metadata through journald.

Note

Requires systemd-coredump and an active journald. In minimal containers without systemd, this tool is unavailable.

curl: HTTP Debugging in CLI

Fri, 04 Sep 2026 00:00:00 +0000

cURL is the standard tool for debugging HTTP in the terminal. It ships out of the box on Linux and macOS and is present in most Docker images. Need to quickly check an API, inspect response headers, or trace a redirect issue — one command line is enough.

Basic Debug Flags

The most common scenario: get a response and see what the server returned. The -i flag prints headers before the body, -v enables verbose mode with connection details.

lsof: which processes listen on port and hold file

Fri, 04 Sep 2026 00:00:00 +0000

Service won’t start — port 8080 is already bound. You dig into who’s holding it, and discover the same process has your config file open while you’re trying to edit it. lsof answers both questions: which processes opened which files and sockets.

Listening Ports

The classic task — find who is listening on a specific port.

lsof -i -n -P

Flag	Effect
`-i`	Show internet sockets
`-n`	Skip DNS resolution (show IP instead of hostname)
`-P`	Skip port-to-service conversion (show 80 instead of http)

Without -n -P, lsof wastes time on DNS lookups and resolves ports through /etc/services. On production hosts that’s unnecessary seconds.

strace: System Call Tracing for Diagnosing Hangs and Leaks

Thu, 03 Sep 2026 00:00:00 +0000

When a service hangs, standard tools like top, htop, and ps show the state but not the cause. If a process is in state D (uninterruptible sleep), it’s waiting on a syscall. strace attaches to a live process and outputs every system call in real time. This turns a mysterious hang into a specific syscall, its arguments, and return code.

Note

strace uses ptrace, the kernel’s debugging mechanism. On production, tracing slows a process by 2–10x. Use it sparingly, targeting a single PID.

ngrep: grep for Network Packets in Real Time

Wed, 02 Sep 2026 00:00:00 +0000

Ngrep applies grep-style pattern matching to network packets. When you need to see exactly what two services are exchanging over the wire and tcpdump drowns you in noise, ngrep isolates the payload content you care about.

Installation

Ngrep ships in the standard repositories of most distributions.

# Debian/Ubuntu
apt install ngrep

# RHEL/CentOS/Alma
yum install ngrep

# macOS
brew install ngrep

Running ngrep requires root privileges or the CAP_NET_RAW and CAP_NET_ADMIN capabilities.