Overnight session, 2026-08-21. The thing I want out of these notes in six months isn't the build order — it's the reminder that a service saying it's "running" proves nothing, and that the only honest evidence a cutover happened is a real action round-tripping end to end.
One agent left on the old box
The agent system was the last major one still living on the AWS EC2 box. Rather than repurpose an existing VM, it got its own: built from scratch on an existing Proxmox host, RHEL 10.2, registered under a free Red Hat Developer subscription. Real RHEL rather than a rebuild distribution, for the boring reason that it's what a lot of enterprise environments actually run, which matters for certification and resume relevance.
The move wasn't just the code. The whole gateway went across — state, credentials, dependencies.
The accounts are part of the config
Three Linux service accounts had to exist on the new VM with IDs matching the old host exactly, because the agent's config contained hardcoded home-directory paths. Get them wrong and every path reference breaks silently: no loud error, just a thing that quietly doesn't resolve. One account's group ID deliberately equals another account's user ID — not a typo — which forced a specific creation order.
Matching them exactly was cheaper than debugging permission ghosts later. That trade is worth remembering: doing it right once beat hunting for the reason a path silently didn't resolve.
The pull lost, the push won
The plan was to pull from the old host. It failed: that host's SSH config was old and tangled from months of ad-hoc changes, and pulls kept dying on silent auth failures. I reversed the direction — pushed from the old host into the clean new VM — and it worked immediately.
The rule I'm keeping: when one direction of an SSH transfer is a fight, flip the direction instead of debugging the messy end. The clean machine is the wrong place to be troubleshooting from.
Exclude patterns match exact names
Small one, worth writing down because it cost a dry run: .cache/ does not match audio_cache/. The pattern has to be the actual directory name. The fix was a wildcard, *cache*/. Until that was caught, the first dry run came out roughly twice the size it should have been, because the exclusion was silently missing a differently-named cache directory on its way into the transfer.
Rebuild the runtime, don't copy it
Python and Node dependencies were rebuilt fresh on the new VM rather than copied over, to avoid architecture and library mismatches. The Python path was a virtual environment plus an editable install of the project:
python3 -m venv venv
source venv/bin/activate
pip install -e .
Node failed on the first try:
npm install
# FAILS: node-pty native build error — "not found: make"
Minimal RHEL has no compiler installed by default. Fix:
sudo dnf install -y gcc gcc-c++ make
npm install
node-pty is the module the agent uses for shell and PTY features — a real dependency, not something to skip.
Also deliberate: I avoided the agent's self-update command and a forced dependency audit-fix during the migration. The goal was matching the old host's exact known-good versions, not the newest or the cleanest. Auto-upgrading mid-migration risks breaking something that already worked, for no benefit. "Newer" and "audit-clean" are jobs for a later day.
The constraint that fixed the order
First attempt at starting the agent on the new VM, with the old host still running, was rejected by the messaging platform with a polling conflict — the old host still held the bot credential. Failing there was the point: it confirmed the real constraint, that only one instance can poll a given bot credential at a time, and that constraint is what determined the cutover order.
The sequence:
- Stop the gateway on the old side.
- Run one final delta sync — safe now, because nothing on the old side is still writing to the database.
- Start the gateway on the new VM in the foreground; the messaging platform connected cleanly this time.
- Send the bot a real message and get a real reply from the new VM. That round-trip is the proof of cutover, not the service status.
- Install it as a
systemctl --userservice with lingering enabled, so it starts on boot without a login session. - Reboot the VM and confirm it came back on its own and resumed responding with zero manual intervention.
Step 6 is the one that's easy to skip. A user service with lingering that survives a reboot is permanent; one that works in the current terminal session isn't.
Related caution baked into the same step: the live SQLite databases — state, kanban, verification evidence — were only trustworthy once copied after the old gateway had stopped writing to them. Copy a live database while the other side is still writing and you risk a partial or inconsistent file.
Still open, deliberately
- The trading bot's final home is unconfirmed. It's currently paused, with none of its 24 scheduled jobs enabled. That's not decided yet, and that's on purpose — the money involved is minimal and there's no rush to get it wrong.
- The old AWS EC2 host was kept fully intact as a fallback. Nothing decommissioned.
- Several web apps and APIs were still being migrated when the session closed. Each one gets verified by actually loading it, not by trusting a status report.
Carry forward
Verify by function, not by status. Flip a transfer direction that's fighting you. A reboot is the real durability test for anything installed as a user service with lingering. Match known-good versions during a migration instead of auto-upgrading. And a minimal RHEL install has no compiler — native npm or pip modules need the toolchain first.