strace: System Call Tracing for Diagnosing Hangs and Leaks

3 September 2026 In Posts 822 words 4 min

Lead DevOps

When a service hangs, standard tools like top, htop, and ps show the state but not the cause. If a process is in state D (uninterruptible sleep), it’s waiting on a syscall. strace attaches to a live process and outputs every system call in real time. This turns a mysterious hang into a specific syscall, its arguments, and return code.

Note

strace uses ptrace, the kernel’s debugging mechanism. On production, tracing slows a process by 2–10x. Use it sparingly, targeting a single PID.

Basic Flags and Syntax

Installation:

# Debian/Ubuntu
apt install strace

# RHEL/CentOS
yum install strace

# Alpine
apk add strace

Running:

# Trace a new process
strace ls -la /tmp

# Attach to a running process
strace -p 12345

# Attach including subprocesses (fork/clone)
strace -fp 12345

Core flags for diagnostics:

Flag	Purpose
`-f`	Follow child processes
`-c`	Count calls and time (summary)
`-tt`	Microsecond timestamps
`-T`	Time spent in each syscall
`-e trace=openat,read,write`	Trace only specified calls
`-e write=1,2`	Trace writes to fd 1 and 2 only
`-o output.log`	Write output to file
`-s 1024`	Truncate strings longer than N characters

Diagnosing Blocking Calls

Scenario: process is hung, ps shows state D. Need to find what it’s blocked on.

# Check process status
ps aux | grep nginx
# root     12345  0.0  0.1 ... S    pts/0    0:00 nginx: worker

# Attach for 5 seconds, output everything
strace -p 12345 -f -tt -T 2>&1 | head -50

Typical output when blocked on a file:

14:23:45.123456 read(15, "", 1024)  = 0 <2.345678>
14:23:47.469134 openat(AT_FDCWD, "/var/data/large-file.db", O_RDONLY) = 15 <0.000023>
14:23:47.469157 read(15, "", 1024)  = 0 <5.678901>

If you see read(...) <time> = 0 with large time values, the process is waiting for data. If <time> is in seconds, you’ve found the bottleneck.

Tip

Blocking on epoll_wait, poll, select is normal for an idle process. Look for read, write, openat, sendto with times exceeding 100ms.

For network sockets:

# Trace network calls only
strace -p 12345 -e trace=network,read,write -f

Hang on connect() to an unreachable host:

14:30:01.234 connect(14, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.0.100")}, 16) = -1 EINPROGRESS <3.456789>

EINPROGRESS means non-blocking socket, but long time indicates a network issue or timeout.

Finding File Descriptor Leaks

Scenario: process won’t open files, error “too many open files”. Need to find who’s holding the descriptors.

# Check limits and current usage
ps aux | grep myservice
# 12345  5123  0.0  /opt/myservice

cat /proc/12345/limits | grep "Max open files"
# Max open files            1024                 1024                 files

# Count open fd
ls /proc/12345/fd | wc -l
# 987

Attach strace with a filter on file operations:

strace -p 12345 -e trace=openat,open,close,clone -f 2>&1 | tee /tmp/strace.log

Analyze the output:

# What was opened but not closed
grep -E "openat|close" /tmp/strace.log | awk '{print $2}' | sort | uniq -c | sort -rn | head -20

Alternative — summary mode:

# Stats over 30 seconds
timeout 30 strace -p 12345 -c -f 2>&1

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- -------
 45.23    1.234567          123       1000         openat
 30.12    0.823456           45      18234       close
 20.45    0.559123         8905        63         read
------ ----------- ----------- --------- --------- -------
100.00    2.617146               19297        63 total

If close is called less than openat — you found a leak.

Warning

With high call frequency (thousands per second), strace generates enormous output. Limit time with -tt and filter by syscall via -e trace=.

Analyzing Slow Requests

Scenario: API endpoint responds in 5 seconds instead of 200ms. Need to find where time is lost.

# Start process with tracing and timings
strace -f -tt -T -o /tmp/slow.log ./myservice

# Or attach to running process
strace -p 12345 -f -tt -T 2>&1 | tee /tmp/live.log

Find calls with high execution time:

# Highlight syscalls taking more than 500ms
grep -E "\> [0-9]\.[0-9]{3}" /tmp/live.log | sort -t '>' -k2 -rn | head -20

14:45:23.123456 write(7, "HTTP/1.1 200 OK\r\n"..., 512) = 512 <0.000045>
14:45:23.890123 openat(AT_FDCWD, "/opt/app/cache.json", O_RDONLY) = 12 <0.523456>
14:45:24.413679 read(12, "{\"key\":\"value\"}"..., 4096) = 4096 <0.000089>

openat with 523ms — investigate: file on NFS, missing permissions, remote filesystem.

For SQL-like queries (PostgreSQL, MySQL), trace the socket:

# Find socket fd of process
ls -la /proc/12345/fd | grep socket
# lr-x 14 -> socket:[1234567]

# Trace specific fd
strace -p 12345 -e write=14 -f -tt -T

Slow database query looks like a series of write/read with long time between them:

14:50:01.123 write(14, "SELECT * FROM orders"..., 45) = 45 <0.000234>
14:50:05.890 read(14, "", 4096)          = 2048 <4.766890>

4.7 seconds between sending the query and receiving data — problem is on the database side or network path to it.

Summary

strace turns a hang with no visible cause into a specific syscall. Attach to PID, filter calls via -e trace=, check execution time with -T. For leaks — compare open/close counts in summary mode. For slow requests — find calls with time exceeding 100ms.

Tip

On production use -e trace=write,read,openat instead of tracing all calls — reduces overhead by 3–5x.