Rich Gibbs

Trust the Process List, Not the Success Message

homelab · honeypot · cowrie · ssh · log-analysis

Diary notes from an overnight session building a Cowrie SSH honeypot on a small cloud VPS — the move off a free tier after signup friction, a socket-activation port change that restarting the SSH service does not make, an install process that has drifted from most findable guides, a decoy file that needs two separate registrations, and three deliberately separated categories of overnight traffic.

Session 12, 2026-08-23, overnight. This is the start of the honeypot track — the first thing built specifically for the SOC-analyst portfolio rather than for infrastructure cost or convenience. By the end of the night there was a live Cowrie SSH honeypot catching real internet attacker traffic, and the setup was not finished yet.

The box

Tried a cloud free tier first and signup went sideways: card verification failed, then a request-rate lockout. The note is explicit that this is a commonly reported pain point with that provider rather than something specific to this account. Worth knowing before the next free-tier signup.

Pivoted to Hetzner Cloud, CX23 plan, roughly $6/month, Ubuntu 26 LTS.

Three decisions from that night I would make the same way again:

  • A dedicated, passphrase-protected SSH key, never reused from any other homelab key. This box is deliberately internet-exposed bait; its key should never be able to reach anything else.
  • Deliberately generic server naming, not honeypot-flavoured. The console name is visible only to me and to the provider — an attacker only ever sees what the honeypot presents them, completely independent of the real console name. So a honeypot-flavoured name fools nobody, and it risks drawing the provider's own fraud review onto a brand-new account.
  • Admin SSH moved off the default port and verified working before the honeypot was allowed to answer on the default one. Same "test the new door before locking the old one" discipline as an earlier migration.

That last one cost real time to catch, so here is the shape of it: on this Ubuntu release sshd's port is managed through systemd socket activation, and systemctl restart sshd does not move the listening port. It takes systemctl daemon-reload and then systemctl restart ssh.socket. On a socket-activated service, "restart the service" and "change the port it listens on" are two different instructions.

Cowrie, and the version drift

Cowrie 3.0 restructured its install process, and most guides findable by search still describe the old one. Writing the current process down so the next version bump does not cost the same time:

  • Old guides: an in-tree bin/ control script for start/stop/status, and copying the .cfg.dist config template by hand.
  • Current: bin/ is gone. An editable pip install -e . inside a virtual environment registers a cowrie command directly, and cowrie init generates the config from the bundled template.

Then the real gotcha. After a stop and a restart, a process check showed a stale orphaned process still holding the port. The success message from cowrie stop was not proof that the process had actually died; it needed a manual kill. That is the one I keep coming back to — the exit message and the process list are two different claims, and only one of them is an observation.

Expected, and not a bug: handing the default SSH port from real sshd to Cowrie genuinely changes the host key behind that port, because the honeypot generates its own fake key. Any client shows a host-identification-changed warning on reconnect. That warning is safe to clear only because the cause is known and deliberate:

# example address from the documentation range — not the real box
ssh-keygen -R 192.0.2.10

The decoy file

Planted a fabricated file inside the honeypot's fake filesystem, named to look like a private customer export and populated with invented names, addresses at example.com and numbers from the reserved fictional range — the point being that an attacker who logs in finds something that looks worth taking.

The mechanic worth keeping: a fake file needs two separate registrations — a metadata entry so it appears in a directory listing, and loaded content so reading it returns something. Register only one half and you get a file that either shows up empty or reads as nothing. The fake filesystem's own shell does the create for the entry, then loads the content from a local copy of the file. I checked it end to end over an external SSH connection: the listing showed the file at a realistic size, and reading it returned the invented data.

What showed up the same night

Three categories, and keeping them apart is the whole point.

  1. An SSH key persistence attempt from two different source addresses. Both ran an identical command sequence within seconds of connecting: wipe the existing SSH directory, recreate it, append the attacker's own public key to the authorized-keys file, tighten permissions. Identical scripted sequences from separate addresses seconds apart is the tell of one automated campaign hitting from multiple compromised hosts, not two people. It is standard persistence as well — the planted key keeps working after a later password change. The key comment is a recognizable signature that turns up in many public honeypot write-ups, so this is a known, widely circulated botnet doing opportunistic work rather than anything aimed at this box. Nothing came of it: the planted key exists only inside the fake filesystem, so a reconnect with it finds nothing.
  2. A relay pivot attempt. Someone logged in with a guessed username and password pair, then immediately tried to open a connection out through the SSH session to the mail-submission port on a third-party address — the classic move to make spam appear to originate from the compromised box instead of from the attacker. Cowrie logs that forwarded-connection request in its default mode and never forwards it, so this one is evidence with no real risk attached.
  3. Background scanner noise, explicitly separated from the two above. Most connections show no login attempt at all: a TCP connect, sometimes enough handshake to record a fingerprint, then a disconnect within milliseconds. That is internet-wide cataloguing confirming the port is open, not a deliberate attack. I want it logged, and I do not want future-me counting it as an attacker.

Lessons

  • Verify a stopped process by checking the process list, not the success message.
  • A changed host-key warning is expected after deliberately swapping what answers a port — safe to dismiss only when the reason for it is mine and known.
  • A fake file needs both a directory entry and loaded content. Registering only one half is a common, easy-to-miss mistake.
  • One attacker action can produce several log entries under different event types. Several data points might be one story, not several.
  • Not every connection is an attack. Telling automated background scanning apart from deliberate behaviour is an analytical skill, not noise to filter out.

The thread running through the whole night is the part I would actually want back in six months: every mistake was trusting something that reported, instead of looking at the thing itself. The command that said the process stopped. The install guides describing a version that no longer exists. The log line that looks like an attacker because something connected. Same habit each time — after changing something, go and look at the thing, and stay clear about which of the evidence in front of me is a claim and which is an observation.