Systemd on Lead DevOps

ulimit and systemd LimitNOFILE — why ulimit -n inside a unit doesn't stick

Sat, 19 Sep 2026 00:00:00 +0000

What is nofile and where it lives

nofile is the maximum number of open file descriptors per process. That’s not just regular files — it covers sockets, pipes, stdin/stdout/stderr, logs shipped through journald — everything counts. When nginx or a Go app crashes with too many open files, this is the limit to blame.

Limits live at three levels:

Level	Where to check	What it controls
Kernel (system-wide)	`/proc/sys/fs/file-max`, `/proc/sys/fs/nr_open`	Absolute ceiling for the whole system
PAM / login	`/etc/security/limits.conf`, `/etc/security/limits.d/`	For sessions via `pam_limits.so`
systemd	`LimitNOFILE=` in unit, `DefaultLimitNOFILE=` in `system.conf`	For systemd-managed services

Note

/proc/sys/fs/nr_open is the upper bound you can raise nofile to for a single process. It defaults to 1073741816 (≈1B) on most distros, but in practice you rarely need more than 1048576.

coredumpctl: Finding a Binary Crash

Fri, 18 Sep 2026 00:00:00 +0000

What is coredumpctl and how it works

When a binary crashes with SEGV, the kernel can save a core dump — a snapshot of process memory at the moment of the crash. In systemd-based distributions, coredumpctl handles collecting, storing, and searching these dumps. It’s a wrapper around systemd-coredump, which stores dumps in /var/lib/systemd/coredump/ and indexes metadata through journald.

Note

Requires systemd-coredump and an active journald. In minimal containers without systemd, this tool is unavailable.

Docker logs and journald: choosing a logging driver

Fri, 18 Sep 2026 00:00:00 +0000

When a container crashes, logs are the first thing you need to see. docker logs looks simple, but under the hood different logging drivers are at work, and the choice affects how logs are stored, rotated, and accessed. Here is what you should know before trusting the default.

How docker logs works

The docker logs <container> command reads the container’s stdout/stderr stream and outputs it to the terminal. Behind this sits a logging driver — a component that determines where the data actually goes. By default it is json-file: each container gets a JSON file on the host into which every output line is written.

Chrony Instead of ntpd

Thu, 17 Sep 2026 00:00:00 +0000

Chrony has replaced ntpd as the default NTP client in most modern Linux distributions. It converges to accurate time faster, handles intermittent network connections better, and consumes fewer resources. If your machine still runs ntpd, switching takes only a few minutes.

Installation

On RHEL-based systems:

sudo dnf install chrony -y
sudo systemctl enable --now chronyd

On Debian/Ubuntu:

sudo apt install chrony -y
sudo systemctl enable --now chronyd

If ntpd was running on this machine before, stop and disable it to avoid port conflicts:

journalctl: filters and follow

Thu, 17 Sep 2026 00:00:00 +0000

Systemd’s journal is the first place to look when a service crashes or a node starts burning CPU. journalctl does far more than dump the entire log in sequence: it can filter by units, priorities, time windows, and stream in real time. Below is the working set I use daily.

Follow in real time

Behavior similar to tail -f, but aware of journald’s structured format:

logrotate for Custom Daemons

Tue, 08 Sep 2026 00:00:00 +0000

Log rotation is missing for your daemon, the log file has grown to dozens of gigabytes, the disk is full, and monitoring is screaming. systemd-journald and syslog-ng rotate on their own, but if your custom daemon writes directly to a file, rotation falls to logrotate. Here is how to configure it for a specific service.

Why write a custom logrotate config

Packages from the repository usually drop their config into /etc/logrotate.d/, but for self-built daemons or those compiled from source, there is none. Without a config the file grows without limits. logrotate runs via a systemd timer (logrotate.timer) or cron and reads all files from /etc/logrotate.d/. Creating a single file is enough to start the rotation cycle.

journalctl: Filtering and Formatting systemd Logs

Sun, 06 Sep 2026 00:00:00 +0000

Logs disappeared. Server rebooted, and the familiar less /var/log/syslog returns nothing. On modern distros with systemd, logs are collected by journald and read with journalctl. Without knowing its filters, system debugging turns into guesswork.

Why Logs Disappear After Reboot

By default, journal stores data in /run/log/journal/ — a tmpfs that wipes on reboot. To make logs survive reboots, create the directory:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

Then restart systemd-journald:

systemd-run: Run Services Without Unit Files

Sat, 05 Sep 2026 00:00:00 +0000

Sometimes you need to run a process under systemd’s control without writing a unit file — maybe you’re in a container without systemd, on someone else’s machine, or just need a quick one-off. That’s where systemd-run comes in.

Why systemd-run

The tool creates a transient unit — a unit that exists only in systemd’s memory, with no file on disk. This is useful when you need:

Creating a Custom Systemd Service

Wed, 02 Sep 2026 00:00:00 +0000

Your application needs to start on boot, restart on crash, and log output. Shell scripts in /etc/rc.local give you none of that. Systemd solves all three with a single declarative file.

Why Write a Custom Unit File

Supervisord and init scripts are overkill for most cases. Systemd provides a unified interface for service management: socket-based activation, dependency tracking, resource limits, and built-in logging via journald. You get all of it without additional tooling.

systemd-timer: scheduling instead of cron

Tue, 01 Sep 2026 00:00:00 +0000

cron works, but its logs are flat text files with no structure, and service dependencies require workarounds like embedding Requires= logic inside shell scripts. systemd-timer fixes this: unified management interface, logs in journald, dependencies through the familiar After= and WantedBy= directives — all in one stack.

Structure: .service and .timer

A timer is a separate unit that triggers a .service. The separation is intentional: the service can be invoked manually or on a schedule.