Session 18, 2026-08-26. Two threads that only look like one story because I was watching both when they crossed: a threat actor pushing a complete multi-architecture kit at the honeypot, and the honeypot service itself being killed over and over for a reason the earlier persistence work had put in place.
The kit arrived over an SFTP subsystem, not a shell
A remote host connected and was authenticated straight away as the highest-privilege account. That acceptance is the honeypot's user database doing its job — the login was permissive on purpose. Not a break-in.
The delivery method is the part to keep. An SFTP subsystem, not a shell one-liner. Somebody who already carries a complete kit does not need to type commands against a shell.
The kit: five architecture-specific ELF binaries — ARM 32-bit, ARM 64-bit, i386, RISC-V, x86_64 — all static and stripped. Around them, the two scripts that make a kit usable on a host it has never seen.
- A cleanup script that stops and disables rival miners and backdoors already present, strips other backdoor patterns out of cron and shell startup files, and wipes the usual temp locations except its own directory. Turf war: clear the CPU of competitors so this payload owns it uncontested.
- A setup script that detects CPU architecture, looks for a writable and executable directory while explicitly skipping
noexecmounts, copies the architecture-matching binary to a randomised hidden filename, and launches it.
There was also an authorized_keys file holding a public key whose comment matched previously published threat-intel reporting on the same botnet family. Everything was hashed, cataloged and left unexecuted.
The correlation is the part I want to be able to come back to. The SSH client fingerprint and a Go-based client signature matched an actor I had seen earlier — the one that embedded a key and pulled a loader from a separate command-and-control host. Same campaign, now at its payload-delivery stage, not somebody new. Worth remembering that this is an inference from matching fingerprints whose values I never reproduced, not a demonstrated match.
VirusTotal came back empty on the hashes. I read that as expected rather than a failure of the process: this family rebuilds its ELF payloads often, so binary hashes drift while the artifacts around them stay stable. The identity lives in the stable things — filenames, the key comment, the client fingerprint, the C2 host — not in the binary hash on its own.
"Something keeps taking the server down"
Investigating the apparent downtime, I ruled out the obvious suspects in order: not a disk-space crash (downloads and logs were both well under capacity), not a duplicate-process conflict, not the second honeypot service, which was idle and light.
The answer was in the kernel logs. systemd's own per-service memory cgroup cap was killing the honeypot process every time it grew to almost exactly the MemoryMax=512M ceiling set back during the persistence work, the killed process sitting at roughly 522,988 kB of anonymous RSS. Restart=on-failure then brought it straight back up, it grew again, it was killed again. From outside, the honeypot looked intermittently down; the host itself stayed healthy the whole time.
The growth traced to traffic rather than an inherent leak: four sequential addresses in a single network block, all sharing one SSH client fingerprint — one operator on adjacent hosts — ran roughly 840 single-command sessions of uname -a over about 9.5 hours, starting right at midnight. Every session means the honeypot builds a fresh simulated shell environment, and at that volume, with per-session cleanup less than perfect, memory climbed until it reached the cap.
One misreading worth writing down. RSS pinned almost exactly at the cap looked, at first glance, like an active crash loop. Uptime rather than memory disproved it: the process had been stable for 6+ hours, plateaued at the ceiling instead of still climbing. Memory alone cannot separate "stable at a hard cap" from "about to die again."
Raising the ceiling, and the trap in doing it
I raised the limit with a systemd drop-in override instead of editing the unit file directly, so a future package update or reinstall cannot silently wipe the change.
The trap is in how the override gets written. systemctl edit opens an editor showing the current unit file's settings as commented-out reference lines. Those lines are not the configuration being written. An override left empty, or left fully commented out, changes nothing at all — silently, and nothing tells you. The override only does anything once it carries real, uncommented directives under [Service]: MemoryMax=1536M and MemoryHigh=1200M. Then daemon-reload, a restart, and a systemctl show of the memory properties to confirm what actually landed.
It came back applied and holding steady well under the new ceiling, with no more kill-restart cycling.
Considered and deliberately not done: adding 1 GB of swap as extra margin, since the host runs with zero swap. The new cap already resolved the actual symptom, so that stays a note for future hardening if memory pressure returns — not a change I made.
Reminders for next time
- A honeypot going down can be its safety mechanism working correctly. Find the cause before blaming the safeguard.
- A hash miss on a public scanning service is an identification problem, not an unknown-threat verdict.
- Commented lines in
systemctl editare reference, not configuration. A no-op override is silent. - Uptime, not just current memory, separates "stable at a cap" from "actively crash-looping." Check both before diagnosing.
- Addresses sharing one client fingerprint are one operator, not several. The same correlation technique from the log-analysis work, here used to explain a real outage rather than just sketch a profile.
Sessions 12 through 18 now cover the build, the hardening, the training, and this incident — that closes out the honeypot-arc catch-up.