# ulimit and systemd LimitNOFILE — why ulimit -n inside a unit doesn't stick

LLMS index: [llms.txt](/en/llms.txt)

---

## What is nofile and where it lives

`nofile` is the maximum number of open file descriptors per process. That's not just regular files — it covers sockets, pipes, stdin/stdout/stderr, logs shipped through journald — everything counts. When nginx or a Go app crashes with `too many open files`, this is the limit to blame.

Limits live at three levels:

| Level | Where to check | What it controls |
|---|---|---|
| Kernel (system-wide) | `/proc/sys/fs/file-max`, `/proc/sys/fs/nr_open` | Absolute ceiling for the whole system |
| PAM / login | `/etc/security/limits.conf`, `/etc/security/limits.d/` | For sessions via `pam_limits.so` |
| systemd | `LimitNOFILE=` in unit, `DefaultLimitNOFILE=` in `system.conf` | For systemd-managed services |

> [!NOTE]
> `/proc/sys/fs/nr_open` is the upper bound you can raise `nofile` to for a single process. It defaults to `1073741816` (≈1B) on most distros, but in practice you rarely need more than `1048576`.

## LimitNOFILE in a systemd unit

In a unit file the directive looks like this:

```ini
[Service]
LimitNOFILE=65536
```

You can set both soft and hard limits at once, separated by a space:

```ini
LimitNOFILE=65536:1048576
```

The first value is the soft limit, the second is the hard limit. If you specify only one, it becomes the soft limit and the hard limit is taken from the system maximum.

> [!TIP]
> The global default for all units is `DefaultLimitNOFILE=` in `/etc/systemd/system.conf` (and `user.conf`). Modern distros often default to `1048576`, but older ones may have `4096` or `1024` — that's exactly what catches you off guard.

## Why ulimit -n in ExecStart doesn't work

The typical mistake is trying to set the limit directly in the launch command:

```ini
[Service]
ExecStart=/bin/sh -c 'ulimit -n 65536 && exec /usr/bin/myapp'
```

`ulimit -n` inside `ExecStart` **does not take effect** on the process systemd tracks as `MainPID`. systemd sets limits **before** `ExecStart` runs, and the shell wrapper already operates in a context where the limit is fixed. Worse, `ulimit -n` may fail with `Operation not permitted` if the requested value exceeds the hard limit systemd assigned.

If you need to change the limit, use the `[Service]` directive — not a shell wrapper.

> [!WARNING]
> `ulimit -n` in a shell script called from `ExecStartPre` also does not propagate the limit to the main process. Limits are a property of the process, not the shell session.

## How to check applied limits

Three ways, from simple to authoritative:

```bash
# 1. From inside the process
cat /proc/self/limits | grep "Max open files"

# 2. For a specific PID
cat /proc/<pid>/limits | grep "Max open files"

# 3. What systemd sees for the unit
systemctl show myservice.service -p LimitNOFILE
```

`systemctl show` reports exactly what systemd applied at `fork()` — that's the authoritative source. `/proc/<pid>/limits` is what the kernel sees for the process. If they disagree, the problem is in an intermediate layer (PAM, container runtime, sudo).

For debugging `ExecStart`, you can add:

```ini
[Service]
ExecStartPre=/bin/sh -c 'cat /proc/self/limits | grep "Max open files"'
```

This shows the limits **before** the main process starts — what systemd set for the service.

## PAM limits vs systemd limits

On most modern distros (RHEL 8+, Ubuntu 20.04+, Debian 11+) systemd **does not invoke** `pam_limits.so` for system services. Limits from `/etc/security/limits.conf` are **not applied** to unit files. They only work for login sessions (SSH, local login, `su`).

| Mechanism | Applies to | Configured in |
|---|---|---|
| `LimitNOFILE=` in unit | Specific systemd service | `/etc/systemd/system/*.service` |
| `DefaultLimitNOFILE=` | All systemd services | `/etc/systemd/system.conf` |
| `pam_limits.so` / `limits.conf` | User login sessions | `/etc/security/limits.conf` |
| `ulimit` in shell | Current shell and children | Interactive session |

If a service isn't started through systemd (e.g., via supervisor, docker `--ulimit`, or directly), `LimitNOFILE` in the unit file doesn't apply at all. In Docker that's `--ulimit nofile=65536:1048576`, in supervisor the `stdout_maxbytes` and similar options aren't related to `nofile` directly — you configure it at the OS or container level.

> [!TIP]
> After changing `LimitNOFILE` in a unit file, always run `systemctl daemon-reload` and `systemctl restart <unit>`. `systemctl reload` doesn't restart the process and doesn't reapply limits.