The last analysis pass turned up four defects in the sensor stack. Today was about fixing them. Three closed clean. The fourth I diagnosed all the way down to the mechanism and then chose to stop, which is its own kind of lesson.
The NAT redirect was blinding mailoney
Every one of 517 SMTP sessions across 24 days had logged the loopback address as the source. Not "mostly" — every single one. The credentials table was empty too.
The service itself was innocent. It already defaulted to listening on all interfaces and it logs the real peer address correctly. The deployment was wrong: it was bound to loopback and fed through an iptables NAT redirect on port 25, so the socket only ever saw a loopback peer. The packet rewrite happened before the application ever got a chance to see who actually connected, and that data isn't recoverable after the fact — it was never captured in the first place.
Reading that redirect rule all the way through paid off twice. It also carried a rate limit — twenty connections per minute, burst of ten — silently dropping anything over that threshold. So it wasn't just an attribution bug, it was quietly discarding traffic too.
The fix was to bind the service to the real port directly instead of routing around it. Port 25 is privileged, so that means granting the capability to bind it rather than running the whole thing as root — the ambient-capability approach, with the bounding set capped to just that one capability as a defence-in-depth extra. Then the redirect had to come out of both places: the live iptables table and ufw's persistent config. Missing the persistent copy would have quietly restored the redirect on the next reboot. And the firewall port itself had never been opened, because traffic had only ever arrived through the redirect, bypassing the filter chain entirely — so that needed opening too.
Verification showed a real external source address landing in the session log where the loopback address used to be. Fixed. Also: removing the rate limit along with the redirect means genuinely more SMTP traffic from here on, both the attribution that was missing and the volume that was previously being silently dropped. Worth remembering that jump isn't attackers suddenly noticing the box — it's the sensor finally seeing what was already arriving.
Lesson for later me: a NAT redirect destroys source attribution by construction. If the whole point of a sensor is recording who connected, it has to bind the real port directly.
The download cap that wasn't
A single incomplete transfer — a partial file download that never finished — reached about 690 MB before it failed, and it was part of what fed the disk exhaustion from an earlier incident. The setting that would have prevented it was already sitting in the config file. Commented out.
Default behavior when that line is commented out is no limit at all. Not a conservative default — none. Uncommenting it and setting it to 10 MB was the whole fix.
The honest tradeoff: a 10 MB cap truncates anything bigger, and somewhere out there a legitimate large payload could get cut off. But the largest legitimate captures in this corpus are a few hundred KB. For what's actually showing up here, that's the right trade, not a free one.
Lesson: a commented-out setting is not a safe default. It's just unlimited with extra steps.
The HTTP fingerprint mismatch — mostly fixed
The web-facing honeypot was answering requests with a header claiming one well-known server while serving a body that server would never produce — a directory listing in a style that doesn't match. That's a trivially detectable tell to anyone actually looking.
Three things were stacked on top of each other: the identity header had no version string, the nicer template set that would have rendered realistic pages was disabled, and the content root was empty, which meant every request to the site root fell through to an autoindex instead.
I went for the template fix first since the templates were already there and genuinely well-made — matched the real server's spacing and date formatting closely. Turning the flag on didn't error, didn't warn, just silently did nothing, because the Python templating library it depends on isn't in the container image, and neither is the tool that would install it. That's worse than a crash: a config flag that parses fine and changes nothing. Reverted it and left a comment explaining why, so I don't burn another hour rediscovering the same dead end.
What I shipped instead: a versioned identity header consistent with another fingerprint header already configured elsewhere, and real placeholder content at the root instead of the autoindex, targeting the path that gets the large majority of the captured HTTP traffic. Verification showed the new header and the new content coming back live.
Caveat, stated plainly so I don't forget it: the error-page markup is better but still not an authentic match to the real server's actual error-page structure. Closing that gap needs the same missing dependency. The tell is reduced, not gone.
Lesson: test that a config change is actually reachable, not just that it parses and restarts clean. A silent no-op is worse than an error, because an error tells you something's wrong.
The credential pair that wouldn't die
There's a specific credential pair scanners use purely to fingerprint honeypots — a real system rejects it, a permissive one accepts it and gives itself away. This box was accepting it, 519 times over 24 days.
Blocking just that one pair felt like the right scope: the sensor's permissiveness elsewhere is load-bearing, it's why real tooling and real human exploration get captured at all, so a narrow deny rule shouldn't cost anything. Added the deny rule ahead of the catch-all in the auth rules file, in the right order. Then actually ran the auth module's own rule-matching logic against the file directly, outside the live service, to confirm the logic was sound before trusting it. It was — the rule denied exactly the pair it should have and let everything else through.
And live testing still showed it succeeding.
The login logs showed no password had ever been offered for that pair, and verbose SSH client output explained why: the service accepts the SSH "none" authentication method — the method a client offers first just to ask what's supported — and grants a session on it before any password or key is ever evaluated. The deny rule, and the entire rules file behind it, is never consulted for a client that completes "none" and stops there.
That's the bigger finding, bigger than the one credential pair: any policy in that rules file is only enforced against clients that actually negotiate password authentication. It doesn't touch key-based or keyboard-interactive attempts, and it doesn't touch anything that completes "none" first. That's a plausible explanation for a ratio I'd been puzzled by before — 35,195 login "successes" against only 474 failures. A real server doesn't authenticate a client that offered nothing at all, so this is a realism problem in its own right, separate from any one credential.
I stopped here on purpose. The rule is correct and still useful against clients that do send a password. Chasing the auth-dispatch layer itself would have been disproportionate next to the other three fixes, which closed real data loss and a real detectability gap. Documenting the mechanism felt like the better use of the time than continuing to patch around it today.
Lesson: testing a rule in isolation proves the rule, not the system it runs inside. And an auth log field that should never be empty — no credential offered at all — is a diagnostic signal, not noise. ssh -v turned that one blank field into an exact answer in a single command.
Still open, for when I'm back here
- Whether Cowrie's "none"-auth acceptance is fixable without rebuilding the auth dispatch path — worth understanding properly, since it undermines any credential policy in that file.
- The error-page fidelity gap on the HTTP side, blocked on the same missing dependency as the templates.
- Confirming the mail honeypot is actually recording credentials now that real connections are landing — the source-IP fix doesn't guarantee the credential-capture path downstream is working. The credentials table being empty for 24 days was the original symptom; I want to see it populate before calling this done.
- Whether any of today's changes survive a reboot. The firewall and NAT state especially — there's a precedent for an unattended reboot landing mid-change and locking things out. Don't assume persistence just because it's correct right now.
Taking stock
Three real fixes today, each traceable to one specific, findable cause: a redirect hiding the thing a sensor exists to record, a comment character standing between a setting and its own default, and a dependency that silently amputates a feature instead of erroring. The fourth is still open, but it's open with a name and a mechanism attached instead of being a mystery — "the credential rule doesn't work" turned into "the credential rule is correct and the auth method that matters never reaches it." That's worth more going forward than a clean four-for-four would have been if one of them were still unexplained.