Dionaea froze a third time, and for the first time the watchdog actually did its job: high CPU on two consecutive checks plus a quiet database for 30-plus minutes, the combined signature fired, evidence got preserved, the process restarted, and it reported — all with nobody awake for it. Outage this time: about 37 minutes. The first two incidents took 94 hours and 24 hours respectively to come back, both by hand. That gap is the whole point of building a watchdog in the first place, and it's the first time this one has proven it rather than just existed untested.
Uptime before the freeze doesn't tell the same happy story. Incident one ran 17.9 hours before freezing, incident two 24.5 hours, incident three only 6.4 — shortest yet, and under perfectly ordinary load (81 connections spread across nine protocols, no single source dominating, top talkers in the low teens). Compare that to incident one, where a single busy scanner alone threw nearly 6,000 SIP connections at the box in twenty hours and it kept running anyway. So load isn't the trigger, and neither is a time-on-the-clock threshold — the trend is downward, which reads more like something degrading across restarts than a fixed timer.
The other theory that didn't survive this freeze: session 37's claim that a burst of SIP cleanup-warning log lines was part of the failure signature. This freeze produced none of that burst. Worth recording honestly, I almost got the correction wrong too — my first read was that the two previous bursts had actually been the same one, counted twice. They weren't. Both earlier incidents produced genuinely separate ~712-line bursts of near-identical size, and the only duplicate was between two snapshots of the same incident-two log file, taken half an hour apart. So the accurate position is narrower than either the original claim or my own too-fast correction: the burst shows up with some freezes, not all of them, and the repeated size across two independent events is still an open question. Lesson under all of this: when you retract a correction, go back and re-check the original claim too, not just the thing you got wrong the second time.
Everything up to this point had been inferred from outside the process, which is exactly why each theory fit for a while and then fell apart. So py-spy and gdb are now wired into the watchdog, both dumps written to disk before the restart call so they capture the frozen process instead of a fresh one. First run was on a healthy process, and it still taught me something: py-spy showed the main thread "active" with no Python frames, which I read in the moment as "spinning in C." gdb showed the real story — that thread is dionaea's C event loop and never runs Python at all, so py-spy has nothing to show there whether it's spinning or idle. The gdb baseline parked at epoll_wait, which is just the event loop asleep waiting on the network — correct for an idle box, and now the reference line the next freeze's trace gets compared against. The tool didn't lie to me; I misread what "no frames" meant.
Separate thread entirely: Mailoney, the SMTP honeypot, had an empty credentials table for three weeks and I finally ran down why. It advertises AUTH LOGIN but only ever implemented AUTH PLAIN, so every bot — and they overwhelmingly use LOGIN — died at the first step. Even if that path had worked, it lowercases every received line before storing it, and SMTP credentials are base64, which is case-sensitive, so the one auth path that did function would have stored corrupted data anyway. On top of that, a failed attempt was getting answered with the SMTP success code instead of a failure code — bots read the number, not the text, so a login that should have looked rejected looked accepted to nothing in particular. And the EHLO banner advertised pipelining support that the handler never implemented, so a pipelining client's commands just got dropped on the floor. Four bugs, each independently enough to explain a silent sensor.
Fixed all four: AUTH LOGIN now actually prompts and reads back credentials, the raw (correctly-cased) line gets logged instead of the lowercased one, the rejection code is an actual rejection code, and the pipelining advertisement is gone to match what's really supported. Verified with a test credential pair end to end rather than assumed. I also added a staged response on top of the fix — reject the first five attempts from an IP so the bot cycles through its wordlist, then accept the sixth so it proceeds far enough to hand over a full message body too, not just credentials. Mailoney never actually delivers anything, so nothing leaves the box either way. Small implementation note worth remembering later: the per-IP counter is just a dict in memory, so it resets on every service restart — a returning bot gets five free rejections again after any restart. Acceptable at this traffic level, not worth fixing yet.
While I was in there I also finally understood what the four emails already sitting in the capture folder actually were. Each one is just the word "QUIT" — because the bot's real goal wasn't delivering mail, it was an open-relay test: send the envelope, get mailoney's "go ahead, send the body" reply, and disconnect the moment that reply confirms the server would have relayed for anyone. The envelope addresses are free webmail, probably drop-boxes the tester controls rather than real targets, and one sender address showed up from two different source IPs three days apart — more likely one toolkit or operator reused than two unrelated probes.
Open items carried forward: the next freeze's gdb trace is the priority now that there's a healthy baseline to compare it to. The restart guard is 6 hours but the last uptime before freezing was only 6.4 — close enough that a freeze could now land inside the lockout window, where the alert still fires but recovery goes manual; tightening the guard window would help that case at the cost of less protection against a restart loop. Mailoney's disk usage needs watching now that it stores a session log per attempt and full bodies for accepted ones — this box has already had one disk-full incident, so that's not a theoretical worry. There's a dropper count from the sensor pull that's known to be inflated and needs recomputing properly. One captured sample is WannaCry-sized but shows no kill-switch string under a plain strings search, still unresolved. Captured mail bodies appear to be missing their trailing newline, which wants confirming with a cleaner test case. And two things carried over from before: log rotation still isn't set up for the alerting log, and there's still an access token that needs rotating before any notes containing it go anywhere public.
The short version of what this session actually taught: a watchdog only proves itself the first time it saves something real, not the day it's deployed. Two data points are a coincidence wearing a signature's clothes, not a signature. And a sensor sitting at zero isn't evidence that nothing's happening — it's a question about whether the sensor itself is broken, until proven otherwise.