Rich Gibbs

systemd Kept Guessing Which PID to Track

homelab · systemd · honeypot · linux · operations

A lab note on why a honeypot's first systemd unit looked healthy while its restart safety net was silently disconnected, why Type=forking's PID guess goes stale, what Type=simple changed, and the three-way PID match adopted as the health standard.

The honeypot had no persistence at first. A reboot or a crash left it dead, with no auto-restart to bring it back. Getting it under systemd was the obvious answer. What I did not expect was that the first unit I wrote would look correct, report healthy, and quietly have no restart safety net at all.

The first unit was a reasonable design

I wrote the unit file with Type=forking. The reasoning was sound: the honeypot's own start command is a wrapper script that forks into the background, and Type=forking is the standard systemd pattern for exactly that shape of program. It ran under a dedicated unprivileged service account with Restart=on-failure, and I baked an authbind environment variable in — the start wrapper read that variable internally and added the authbind prefix for me.

Once real problems showed up I added a hard memory ceiling of 512 MB, after an actual out-of-memory incident. The reasoning there: better that systemd kills and cleanly restarts at the cap than that the kernel's OOM killer picks a victim and risks taking the whole VPS down with it. I also added control-group kill mode and a disabled start rate limit to fight the orphan-process bug below. Both helped. Neither fully fixed it.

The failure mode, in two shapes

Early on it looked like this: after a failed start — a bad config, a port collision — an old daemon process would survive holding ports 22 and 23, while the PID systemd was tracking was a different, already-dead process. The socket table and systemctl status disagreed about which PID actually held the port.

The later shape is the one worth remembering. The daemon was alive and genuinely serving real attacker traffic, while systemctl status reported inactive (dead), because the PID systemd had tracked from the fork had been replaced and an untracked PID had taken over the listening ports. Restart=on-failure would never have fired if that process ever died. Persistence was silently non-functional the whole time traffic was flowing normally.

A service that looks healthy while its actual safety net is disconnected is worse than one that is visibly broken. Nothing flags the first kind.

Root cause: Type=forking requires systemd to guess which child PID to track after the wrapper forks, and that guess goes stale under real conditions — a crash mid-restart, an orphan from a failed start. The guess is the bug.

Removing the guess

The fix was to stop forking and let systemd own the process directly with Type=simple. Four things changed with it:

  • Type=forking to Type=simple: systemd tracks the exact process it launches. No fork, no PID guessing, no staleness.
  • ExecStart invokes the daemon directly instead of the wrapper, with the no-daemon flag required so the process stays in the foreground where systemd can own it.
  • The authbind prefix moved into the command itself. The old environment variable only ever worked because the wrapper read it — bypass the wrapper and nothing reads it anymore. My first attempt at the redesign missed that and failed with CannotListenError: Permission denied on ports 22 and 23, because an unprivileged account cannot bind privileged ports without authbind. Calling authbind explicitly fixed it.
  • The explicit stop command and PID file came out. Not needed under Type=simple; systemd sends SIGTERM straight to the PID it is tracking.

The service section below is trimmed to the directives that mattered — argument list shortened, account name and path replaced with placeholders. Not the literal file, and only the service section is shown.

[Service]
Type=simple
User=<service-account>
Group=<service-account>
WorkingDirectory=<service-account-home>
Environment=PATH=<service-account-home>/bin:/usr/bin:/bin
ExecStart=/usr/bin/authbind --deep <service-account-home>/bin/twistd --nodaemon
Restart=on-failure
RestartSec=10
MemoryMax=512M
KillMode=control-group

The check that actually counts

After any restart, three numbers have to match: the PID in the process list for the daemon, the PID in the socket table bound to the actual ports, and the Main PID in systemctl status. That three-way match is the only proof of health I accept now. A single status read is not enough — it had already been shown wrong in both directions, saying dead while the process was alive and alive while it was not.

Stress test: two systemctl restart runs back to back, the exact sequence that used to reliably orphan processes, produced exactly one clean process each time. Then a full unattended overnight run held one consistent PID across all three checks, with both ports live, capturing 1038 new connections and 76 file downloads with zero manual intervention.

Those two counters belong to that one overnight run. They are a record of what that sensor saw, not a general claim.

Same pattern, second service

The email honeypot had been running manually in a foreground terminal and was confirmed dead on wake-up the next morning, as expected. It got the same Type=simple pattern immediately, with no forking-era design to fight through. No authbind needed: its listen port is unprivileged and bound to loopback only, never internet-facing — the SSH honeypot connects to it locally. Differences from the first unit: a 256 MB ceiling instead of 512, a ten-second restart delay, and an explicit interpreter path, working directory, bound address and port pair, and a production sensor label argument. It worked cleanly on the first try.

And then I broke it myself. Right after starting the systemd service I ran a manual launch out of old habit, which briefly made two processes fight over the same port. I killed the manual one and confirmed via the process list that only the systemd-owned PID remained. The lesson is not the port conflict. It is that the old manual-launch reflex is the wrong instinct once systemd owns the service, and it has to be deliberately unlearned.

What I'm keeping

  • "Running" and "serving traffic" are two different claims. Type=forking let them diverge silently; Type=simple removes the guess that made it possible.
  • The dangerous failure is the one that looks fine. Visible breakage gets fixed. A disconnected safety net just waits.
  • Prove health with the three-way PID match, not one status read.
  • A stress test that reproduces the original bug is the only real proof the fix worked.
  • When you bypass something that was doing work for you — the wrapper that read that environment variable — the behaviour it provided has to be re-added explicitly.