I sat down to migrate the honeypot to T-Pot and stood up again without migrating anything. That decision is worth writing down, because the reasoning behind it will still hold the next time the idea looks attractive.
The migration was scoped properly, against a table of options and requirements rather than a mood. T-Pot Standard wanted 8-16 GB of RAM and 128 GB of disk, which this box does not have. T-Pot Sensor fit, but the role it plays is already claimed by another tool in the plan. T-Pot Mini fit and lost the coverage that motivated the switch in the first place. Then the two upsize options: the cheaper cost-optimised instance was not available in the box's region at all, and the larger one costs something like four times what this box does per month.
So: stay hand-built, and deepen the shell that already exists. Building it by hand is where the transferable skill lives; a turnkey stack hands you a dashboard and hides the internals. The same argument parked high-interaction mode — real containers sitting behind the shell — because that swaps an attacker trapped in an emulator for an attacker inside a real container that could, in principle, be escaped from. Do that last, deliberately, not first.
Let the failure log pick the targets
Guessing which commands are worth emulating wastes a session, and it is unnecessary. The honeypot logs every command it could not handle, so that log is a priority list ranked by what attackers actually type. One pass over the rotated JSON logs — keyed on the stable event identifier rather than the human-readable log wording, which changes between releases — gives the frequency list.
The list is also where the session nearly went wrong. The top of it was five commands: system, enable, shell, linuxshell, enablelinuxshell. Around 3,300 hits between them. Every instinct says build them.
They need nothing. That group is Mirai/Gafgyt-family handshake probes — before dropping a payload, the botnet works out what kind of device it landed on. enable and system are IOS-style commands and are not valid on real Linux either. The only question that mattered was whether the rejection looks like real bash's, and the rejection string in the source is identical to bash's character for character; that was the whole check, and it passed.
The lesson goes on the wall: the loudest signal is not the fault. A high count in that log can mean the emulation is already correct and simply being probed hard. Verify what "correct" looks like before building a fix for something that already works.
After filtering, the genuine gaps were four commands: ip, ll, la, nano. Most of the rest of the session came out of those four.
The trap inside the obvious fix
ip first, because on modern Ubuntu the ip tooling is standard while ifconfig is often not installed at all — so a shell that has one and not the other is a fingerprinting tell, not a cosmetic gap. Cowrie ships ifconfig and no ip.
The obvious implementation is a new command file that prints plausible addresses. The trap sits underneath it. ifconfig generates its hardware address and IPv6 values once per process, at module import time. A separately written ip that invented its own hardware address would then report a different one from ifconfig, for the same interface in the same session — a much louder tell than not shipping ip at all. So the new command imports the existing values instead of generating its own, and the two commands cannot disagree.
ll and la were the same shape of gap. Ubuntu's stock shell skeleton enables the ll alias, so real users have it. Cowrie registers alias as a no-op and has no alias-resolution layer anywhere, and building a general alias system is a large lift for one convenience. The shortcut: register ll and la as commands that inject their own flags and then hand off to the real ls, which inherits all the existing formatting for free.
ls itself had a quieter bug. The layout applied one global width to every name, so a short name was padded out to the length of the longest name in the directory and the output grew enormous inconsistent gaps. GNU ls lays out down-then-across with each column sized to its own longest entry, so the columnization was replaced with that — including the detail that is easy to miss, which is no padding on the last column, because trailing whitespace at the end of a line is a tell in itself.
Restart, and nothing works
After a restart the new commands still failed, while ifconfig kept working. That combination told me the restart had succeeded and the service was healthy. Not a crash. A discovery problem.
Cowrie does not auto-discover command files. It iterates an explicit module list, and a new file that is not on that list is never imported. Worse, the loader's import error is swallowed into the log, so a module that fails to load does so silently — no crash, nothing on the console.
The technique I would keep from this whole session is embarrassing for how cheap it was: reproduce the loader by hand with the swallowed exception printed, and make the silence loud.
Two files, both telling the truth
Then a search said my new module was on a particular line of that list, and the interpreter said it was not in the list at all. Both were right, about different files.
Printing the package's resolved path showed the interpreter reading the live installation under the service account's home. The earlier edit had run at a prompt where, as root, ~ expanded somewhere else entirely — root's home, which held a second, complete copy of the Cowrie source. The edit landed there. The service reads the other tree.
~ is user-relative, and su does not always change your working directory. Check who you are and where you are, not the hostname in the prompt. This is a failure class I have met before: a stale second copy of an application sitting alongside the live one, with a command quietly landing on the wrong instance.
I am not deleting a root-owned application tree on the strength of "it is probably unused." Two checks turned the guess into evidence. The stale files' birth and modify times fell in the exact minute the misdirected edit ran, with nothing older and nothing else ever touching them, and a search of the systemd unit directory plus a crontab listing showed nothing referencing the tree. Then it went.
The real fix went into the correct tree, anchored on the existing indented list entry so the insertion stayed alphabetical and unambiguous, followed by a confirming search — because an in-place edit reports neither zero matches nor two.
Four aborts well spent
nano needed Ctrl+O and Ctrl+X, and Cowrie routes control keys through a fixed dispatch table rather than by naming convention. Both byte values were absent, so a handler defined on the command alone would never be reached; the keystrokes just get absorbed into the input buffer as garbage.
That meant editing shared infrastructure that every command inherits, so the safety requirement was that the new handlers be no-ops everywhere by default, with nano the only override. Three classes need the methods, because the top of the command stack can be any of them — including the idle shell. Miss that one and every session sitting at a prompt raises an attribute error, which is a far worse bug than the feature being missing.
For multi-line Python with real indentation, sed is the wrong tool. The pattern that worked is a patch script that verifies every anchor in every file before writing any of them: two passes, all or nothing, rejecting both "not found" and "ambiguous". The earlier version of that script verified and wrote per file, so an abort on the second file left the first already modified.
It aborted four times before it ran clean. Every failure was one blank line between two methods — a line I had already looked at and read straight past. Dumping the region with non-printing characters visible (cat -A) is how an exact-match anchor gets built; reading the output by eye is not sufficient. The abort is the feature: four failed runs cost minutes and left the source untouched every time, where one unverified in-place edit could have quietly broken shared shell infrastructure on a live honeypot.
The bugs that only testing found
Both of the worst finds came from using the thing rather than reading it.
Logging in twice as the same non-existent user produced two identically-named home directories, which real Linux cannot do. The file-creation routine has a de-duplication check; the directory-creation routine does not, and the session code calls it on every login for a temporary user. The fix mirrors the existing check and sits at the filesystem layer rather than in the login path, so every future caller inherits it instead of just the one that exposed it.
The one I actually want to remember is two characters long. Every auto-created home directory showed a garbled mode instead of the expected drwxr-xr-x. The mode had been written as decimal 755 rather than octal 0o755. Decimal 755 is octal 1363, which sets the sticky bit and scrambles the permission triads into exactly the garbage I was looking at — meaning every temporary user's home directory had been wrong. Permission literals are always octal, and both forms are valid integers, so nothing errors; the mode is simply wrong. Verified with a fresh username, because an existing user keeps the directory it already created.
Smaller: cd - had never worked. The original printed "OLDPWD not set" unconditionally, and no old-directory tracking existed anywhere. Real bash keeps the previous directory in the environment, and Cowrie already has a per-session environment dictionary that the command base binds by reference, so state written there survives across commands. The parts that matter are the ordering ones: capture the old directory before reassigning the current one, set it only after validation so a failed change does not clobber it, make the error conditional so a first cd - behaves like real bash, and keep the current directory in sync so the prompt and $PWD agree.
The editor had two of its own. An empty file saved and reopened had quietly gained a phantom blank line, and characters mid-typed at the moment the save key was pressed bled into the next shell command — both of which only showed up when the editor was driven the careless way an attacker actually drives it.
What counted as verified
A syntax check is a gate, not proof. py_compile parses; it does not execute, so a name error from an undefined constant sails right through it. Every change here was verified by function: a compile pass across the edited files, a service restart with an active-state check, a regression list from a fresh session (listing and tab-completion, interrupt, clear-screen, and ifconfig all unchanged), and feature checks written as assertions — the new address command's hardware address has to match the interface command's in the same session, a saved empty file has to be genuinely empty with no lone blank line, a file written with content has to read back correctly without bleeding into the next listing, and a cd - toggle sequence has to go there and back. The duplicate-home check is a fresh login as a new username ending in exactly one home directory with the right mode.
Left open on purpose
tee writes files that cannot be read back: it updates the size and never the content, so a file that ls -l shows as non-empty returns nothing from cat. That is a pre-existing bug found incidentally while building the editor, logged as an open item rather than fixed here, and the same content-update method applies to it.
The new nano is line-oriented — no cursor movement, no arrow keys, no live status bar — which is enough for the paste-and-save path it exists to cover. The new ip handles only addr and link; route, neighbour, and statistics subcommands fall through to the unknown-object error. Auto-created home directories are empty where a real Ubuntu home would carry dotfiles. And the directory de-duplication gap and the decimal-mode bug are genuine upstream defects rather than local quirks, so both are worth a pull request.
The shape to carry forward: before building a fix for something, check whether what is already there is right — and before trusting or deleting a second copy of anything, check who you are, where you are, and prove it is dead first.