Rich Gibbs / Notes from the work

Field notebook.

Practical, opinionated notes on Linux server security, AWS hygiene, and indie-founder operations from Rich Gibbs.

DECISIONS. DETOURS. THINGS WORTH KEEPING.Follow via RSS

Earlier field notes

10 preceding entries · search this page

  1. 01

    Count Every Event Type First, and Copy the Samples Before They Rotate

    Roughly a week of captures across three co-located honeypot sensors, and the two mistakes in the analysis are worth more than the malware. A complete event-type histogram surfaced an upload category and a relay-abuse category that download-based greps never see, and the captured binaries had already been rotated off disk before the write-up.

  2. 02

    Six Cowrie Fixes, One Phantom, and an Edit in the Wrong Tree

    One session spent deepening a hand-built Cowrie honeypot shell: six defects fixed, a command-failure list whose top entries needed no fix at all, a silent module-loader trap, and an edit that landed in the wrong copy of the source tree.

  3. 03

    The Second Cowrie OOM Had a Different Cause, and the Cap Was Never the Ceiling

    Session 19, 2026-08-28 into 2026-08-29. The honeypot started getting killed again two days after the last cap change — same symptom, different mechanism — and the live memory limits were coming from a drop-in override the unit file never mentioned. Swap went in this time; the ceiling stayed where it was.

  4. 04

    Redtail's Payload Kit, and the Memory Cap That Kept Killing Cowrie

    Session 18, 2026-08-26: a five-binary multi-architecture kit pushed in over SFTP, and a real outage caused by the systemd memory cap I set myself — plus the trap in systemctl edit that changes nothing and tells you nothing.

  5. 05

    Filters Are Triage, Not Verdicts

    Notes from one long honeypot session: two passes at the traffic filter, one upload I could finally call confirmed benign, and two sessions I nearly filed as sophisticated attackers.

  6. 06

    Session 16: a pipe carries stdout, not stderr

    Practice notes from session 16: the permission-denied lines that survived a pipe into grep were stderr walking around the pipe, not a bug in grep — plus the recursive content search I left unconfirmed.

  7. 07

    The Month-Old Hardware Doc, and What I Left Out of It

    A documentation-only pass that pulled the lab node's hardware doc a month forward: RAM settled by asking the machine, and two unsettled decisions deliberately left out.

  8. 08

    The Most Active Session Was Mine

    Notes from two days of log-analysis practice against a honeypot's real attacker traffic: why the top of my own most-active ranking was my test session, and the verify-the-number habit that came out of it.

  9. 09

    systemd Kept Guessing Which PID to Track

    Notes from moving a honeypot off Type=forking after a subtle failure where the service served real traffic while systemd reported it dead, and the three-way PID check that replaced a single status read.

  10. 10

    Trust the Process List, Not the Success Message

    Notes from one overnight honeypot build: a provider pivot, a port a service restart did not move, a stopped process that was still running, a decoy file that needs two halves, and three different kinds of nightly traffic.