12 September 2026 started with an abuse report. The hosting provider forwarded a Spamhaus notice saying the box's address was on the Botnet Controller List for hosting a live command-and-control server for the win.pure_rat family, with 24 hours to reply or risk the address being locked.
The temptation was obvious. I run a honeypot. Honeypots look compromised to outside observers — that is the entire point of them. The easy move was to reply "that's just the honeypot being a honeypot" and get on with the day. I want to remember why I didn't, because the distinction is the useful part: a report alleging a live, currently-controlling C2 is a categorically stronger claim than "this address scanned us" or "this address is listed for spam." So I worked it as a possible real compromise until I could prove otherwise.
The audit I ran before writing a word of the reply
Ground truth first, explanation second. Listening sockets. A targeted check for the one port that session 27's captured shellcode had tried to bind. Container inventory. Top processes by CPU and by memory. Established outbound connections. User and root crontabs plus the cron drop-in directory. Recent logins.
Every check came back clean, and the one that mattered most came back empty — the port from that capture was completely absent from the listening-socket table. Nothing real had ever been bound there. Container inventory showed only the honeypot container I already knew about. Every other listening port traced back to a service I could account for. No cron persistence beyond the distro defaults. The process list was the processes I expected. No evidence of an actual running C2 anywhere on the host.
That last sentence is the one I want to keep accurate when I come back here. Not "the box was clean" — "every check I ran came back clean and I found no evidence of a live C2." They are different claims, and the second is the one the evidence actually supports.
Two real problems that had nothing to do with the report
The established-connections check showed two simultaneous sessions on the real admin port. One was mine. The other was an address I didn't recognise, still mid-authentication. In the middle of an open abuse investigation that went straight to the top of the list, ahead of the original report, until it was cleared.
It resolved as ordinary internet background noise — a mass automated SSH scanner working through common usernames, each attempt failing and disconnecting inside about two seconds. The shape of opportunistic scanning rather than anything targeted, and not connected to the Spamhaus report in any way.
But it exposed something real. The SSH config permitted root login with no password-authentication override, so password login for root was genuinely allowed on the real admin port. I didn't have to infer that from the config file either — the log showed the scanner's attempts drawing a real password prompt instead of an immediate rejection. That became the second workstream of the day, entirely independent of the first.
Then, trying to restrict that port, something more fundamental turned up. The firewall front-end wasn't installed. No firewall had ever been active on this box, at any point in this project. Every port opened for every honeypot service had been reachable on its own service-level binding alone, with nothing in front of it. In hindsight that explains how casually the admin port got found and hammered: it looked like any other open port to a scanner, because nothing distinguished it as restricted.
The package removal that would have broken much later
Installing the firewall front-end removed the existing iptables rule-persistence package. The two are conflicting rule-persistence managers, so installing one takes out the other. I caught it only by reading what the package manager said it was removing, rather than skimming the output for errors.
It mattered immediately. The mail honeypot's SMTP redirect — a NAT rule sending the real SMTP port to a loopback listener — had been a hand-added rule for this whole project, and the thing keeping it alive across reboots was exactly the package that had just been removed. Nothing would have broken that day. It would have broken at the next reboot, whenever that happened, silently dropping mail-honeypot traffic with no error message anywhere.
So: inspect the live rule, capture its exact syntax before it could be lost, then fold it into the firewall's own pre-rules file in a new NAT block placed above the existing filter block, because NAT has to be evaluated before the filter rules. One system owning all firewall and NAT state going forward instead of two overlapping ones.
The lockout, and what actually caused it
With the firewall rules staged — default-deny incoming, explicit allows for every honeypot port, plus the restricted admin rule — and the NAT fix written, I applied the SSH hardening. Root login switched to key-only, password authentication appended as disabled.
Total lockout within minutes. Connection reset or timed out on every attempt, including from the terminal that had been open and working the entire session.
Diagnosis went through the provider's web console, which doesn't depend on SSH or the network at all: the SSH unit's journal, and uptime. The cause was an unattended, unrelated reboot. The unattended-upgrades service had acted on the pending kernel upgrade that the earlier package install had already flagged in its own output. The journal showed a clean boot marker at the exact time of the lockout, and uptime confirmed only about six minutes of runtime. The hardening edits did not cause this. The box was simply down for the ninety seconds or so a reboot takes, and every connection attempt in that window failed exactly as any attempt against a rebooting host would.
That is the reminder worth carrying out of this day. The instinct when a lockout follows a security edit is to assume the edit broke something, and acting on that instinct would have had me unpicking changes that were correct. The boot marker and the six-minute uptime were the actual proof.
Two side effects of the reboot, both worth knowing about: the firewall, never yet enabled, came back inactive, and the config edit — written but not yet applied with a daemon restart when the reboot hit — came back as whatever the fresh boot's defaults were.
The worse problem underneath the coincidence
Recovered through the console, re-applied the hardening — this time with a config syntax check before restarting the daemon, and this time it actually took effect — and a fresh connection was still rejected for want of a public key.
No SSH key had ever existed for root. Every login throughout this entire project, every session in every prior write-up, had been password-based.
Disabling password authentication therefore hardened nothing against an attacker. It removed the only access method that had ever existed. The sole surviving path back in was one console session that happened to still be open from before the change. That is not a safety net; that is luck.
The order I should have used, written down here so I actually use it next time: generate the key, install it, confirm a working key-based login from a fresh session, and only then disable password auth. Add and confirm, then remove. Never the reverse.
Recovery itself was short. Key pair generated on the Windows machine, public half pasted into root's authorized_keys through the still-open session, permissions tightened on both the .ssh directory and the file itself — specifically because the daemon silently refuses overly-permissive files there, and that failure looks identical to a missing key, which would have been a second and much harder hole to climb out of. A fresh third window confirmed key-based login worked. Only then did I close the original session.
The redo, with the undo armed first
Because the day already contained one real lockout, I redid the firewall step with an automatic, guaranteed undo armed before the risky change, instead of relying on a currently-open session as the only fallback. Installed and enabled the at scheduler, queued a job to disable the firewall five minutes later, and checked the queue to confirm the pending job really existed before going ahead. If enabling the firewall locked anything out again, the job would turn it back off on its own — no console trip, no dependence on any session staying open.
It came up clean. Every honeypot port open to the internet as intended, the admin port restricted to one known address alone, and the mail-honeypot NAT rule correctly reloaded from the new NAT block, confirmed present and back in the chain by re-listing the NAT table. A fresh window confirmed key-based login worked immediately. Then I cancelled the rollback job on purpose, so it couldn't fire later and undo a working configuration.
I didn't stop at "the firewall looks right on paper" either: service-manager status for the systemd-managed honeypot and web services, container status and port mapping, and a listening-socket check for the mail honeypot's loopback port. Every service had survived both the reboot and the reconfiguration intact.
The honest read on the two firewall attempts is that the rules were correct both times. The difference in outcome wasn't more carefully written rules. It was the five-minute timer.
What went to the abuse desk, and what's still open
The conclusion I submitted was hedged, and it should stay hedged. The listing was very likely triggered by the honeypot's shellcode-emulation module. Session 27's captured loader contained shellcode which, when emulated — the module's documented, intended behaviour of tracing what a payload's Windows API calls would do — executed a bind-and-listen sequence on the port in question. External automated scanning, plausibly the reporter's own detection infrastructure, most likely observed that sandboxed emulation and classified it as genuine live C2 behaviour. The statement explained what the emulation module does and why, the correlation to the September 9 loader capture and that port, the full list of checks performed, the conclusion that no live C2 exists, and a note that the module's network exposure is under review to prevent recurrence.
Submitted is all it is. Whether it lands isn't something I know sitting here, and four items are still open:
- Whether to restrict the emulation module's network reach via container network policy, or accept the small recurrence risk as the cost of the capture data. I told the abuse desk it was under review, and I genuinely haven't decided.
- A deliberate, planned reboot to confirm the mail-honeypot NAT rule survives, rather than trusting proof from one unplanned reboot.
- Whether any other project machine — the hypervisor, the VMs — has the same missing-key gap.
- Confirming the box is on the expected kernel with no further reboot pending, after the upgrade that fired the unattended one.
That third item is the one that nags. The missing key wasn't found by an audit, a checklist or a scan. It was found because an unrelated reboot happened to fire in the same few minutes I disabled the only way in.