Dionaea froze again. Last time, the call was to accept the risk and keep watching rather than rebuild anything; this time the freeze landed 24.5 hours after that restart, and it broke more than just the honeypot.
The heartbeat caught it fast — under 20 minutes from the freeze to the first alert. Then nothing happened for about a day, because the alert fired on every single run after that and nobody could tell one from the next. 73 near-identical alerts went out in roughly 24 hours, because the suppression check compared the full alert text, and the text always carried a different number (CPU percentage, hours since the last database write). Two messages that say the same thing with different numbers in them never compare as identical, so suppression never engaged. Detection without a response acted on is only half a system — I'd caught the failure in 20 minutes and then let a flood of "it's still broken" messages turn into noise I stopped reading.
Worse: the second freeze took apart the theory written up after the first one. That theory blamed a flood of SIP traffic for piling up per-call timer threads until the event loop starved, with a burst of 711 cleanup log lines as the evidence — read as roughly 11.8 hours of backlogged timer fires landing in one second. None of that held up against a second data point.
- No flood preceded the second freeze. Call volume in the day before was ordinary, nowhere near the volume from the single source that preceded the first incident. If Dionaea froze under ordinary load, a flood isn't a required trigger.
- The container's thread/process count was flat and identical across both freezes. If timers were piling up, that count would climb. It didn't move — and that number had already been sitting in the first incident's own notes, never checked against the theory at the time.
- The cleanup-line counts were 711 and 712, from two freezes with different runtimes. A count that measures elapsed time can't land on nearly the same number from two different durations. The number that actually explains it: every evidence snapshot from both incidents came out to exactly 49,152 bytes — 12 × 4,096. That's consistent with the error log reaching disk in fixed 4 KiB blocks rather than being written in real time. It's the file's own strong reading of its own evidence, not an externally confirmed mechanism, but the byte math is hard to argue with.
Flood theory gone, thread-accumulation theory gone, log-burst-as-timeline reading gone. What's left is honest and thin: both freezes share the same signature — CPU pinned near 100% on one core, worker threads sitting idle, the database stopped writing, Docker never OOM-killing the container, and a plain restart fixing it instantly. The trigger, the loop the main thread is actually stuck in, and whether SIP is involved at all beyond logging last, are all still open. Root cause: unknown.
The one new data point is weak on purpose, not by accident. Runtime-to-failure was 17.9 hours the first time and 24.5 hours the second — 6.6 hours apart, further apart than I'd expect from anything tripping at a fixed runtime. Not ruled out, not supported either, from two points. Small bug along the way worth remembering: the first attempt to even compute that gap printed nothing, because the pattern for pulling the process start time didn't allow for the space Docker puts after the colon in its output. When a check silently returns nothing, check the check before trusting the silence.
The watchdog rewrite. Four changes, aimed straight at the two failures above:
- Suppression keys on a stable problem identifier instead of the alert text, so a changing number inside the message can't defeat deduplication. One message when the set of active problems changes, one on recovery.
- Auto-restart requires a combined signature — high CPU on two consecutive checks and a stale database — because either signal alone has an innocent explanation. High CPU alone could be real traffic. A quiet database alone could be a quiet day.
- Evidence capture happens before the restart: process start time, logs, database, and a thread listing, so the next freeze adds a data point instead of erasing one.
- A six-hour minimum gap between automatic restarts, so a failure recurring faster than that has to surface to a human instead of looping on its own forever.
The restart path got tested with a forced hook rather than waiting on a real freeze, and it worked — evidence captured, a restart message sent, worst case from freeze to recovery around 40 minutes. The honest catch: the test itself counted as a restart, which consumed the six-hour lockout window. A real freeze inside that window would have alerted but not self-healed. Worth remembering before trusting the next "it auto-recovered" message at face value — check whether a test burned the window first.
Total Dionaea downtime since it went up: about 94 hours from the first incident, about 24 hours from the second, roughly 118 hours and counting. Any before/after comparison spanning those gaps has holes in it.
Left open for next time: capture a stack trace from the stuck thread before the next restart, if that's possible against the running container; watch whether future runtime-to-failure values keep landing "within about a day" or kill that pattern outright; leave SIP alone until there's more than two data points pointing at it; a newer Dionaea version is sitting tested but not deployed; and there's a carried-forward to-do to add log rotation to the alert log and to rotate an access token before publishing any notes that reference it.
The lesson that outlasts this specific bug: a clean, fast explanation that seems to fit all the data in hand is exactly the moment to go looking for the number that breaks it. The flat process count was sitting in the first incident's own notes the whole time and never got held up against the theory being written. It would have killed the thread-accumulation idea on the spot, without needing a second freeze to do it.