Troubleshooting on Lead DevOps

journalctl: Filtering and Formatting systemd Logs

Sun, 06 Sep 2026 00:00:00 +0000

Logs disappeared. Server rebooted, and the familiar less /var/log/syslog returns nothing. On modern distros with systemd, logs are collected by journald and read with journalctl. Without knowing its filters, system debugging turns into guesswork.

Why Logs Disappear After Reboot

By default, journal stores data in /run/log/journal/ — a tmpfs that wipes on reboot. To make logs survive reboots, create the directory:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

Then restart systemd-journald:

lsof: which processes listen on port and hold file

Fri, 04 Sep 2026 00:00:00 +0000

Service won’t start — port 8080 is already bound. You dig into who’s holding it, and discover the same process has your config file open while you’re trying to edit it. lsof answers both questions: which processes opened which files and sockets.

Listening Ports

The classic task — find who is listening on a specific port.

lsof -i -n -P

Flag	Effect
`-i`	Show internet sockets
`-n`	Skip DNS resolution (show IP instead of hostname)
`-P`	Skip port-to-service conversion (show 80 instead of http)

Without -n -P, lsof wastes time on DNS lookups and resolves ports through /etc/services. On production hosts that’s unnecessary seconds.

ss: socket statistics instead of deprecated netstat

Thu, 03 Sep 2026 00:00:00 +0000

When netstat hangs on a server with tens of thousands of connections, it’s time to switch to ss. Part of the iproute2 package, ss queries the kernel directly via netlink instead of parsing /proc/net/*. The result is instant output with minimal overhead.

Why switch from netstat

netstat from net-tools relies on a deprecated approach: it reads from /proc/net/tcp, /proc/net/unix and converts numeric IDs to symbolic names. On a server with active connections, this takes seconds and spikes CPU usage.

strace: System Call Tracing for Diagnosing Hangs and Leaks

Thu, 03 Sep 2026 00:00:00 +0000

When a service hangs, standard tools like top, htop, and ps show the state but not the cause. If a process is in state D (uninterruptible sleep), it’s waiting on a syscall. strace attaches to a live process and outputs every system call in real time. This turns a mysterious hang into a specific syscall, its arguments, and return code.

Note

strace uses ptrace, the kernel’s debugging mechanism. On production, tracing slows a process by 2–10x. Use it sparingly, targeting a single PID.

SSH Escape Sequences: Reviving a Frozen Terminal

Wed, 02 Sep 2026 00:00:00 +0000

SSH session froze, Ctrl+C does nothing, Ctrl+D spits out garbage — familiar situation. Before closing the terminal and losing the session, try built-in escape sequences. They operate at the SSH client level before data reaches the remote host.

How to invoke escape sequences

The default escape character is tilde (~). The combination works only at the beginning of a line. Press Enter, then ~, then the desired symbol. For example, ~. terminates the connection.