Rich Gibbs

Recreating an Old Account Name Beat Rewriting 20,062 Paths

homelab · proxmox · debian · docker · linux · migration · systemd

Notes from standing up the first guest on the Proxmox host and moving a self-hosted multi-agent system off AWS onto it. The load-bearing lesson was path hardcoding: 20,062 files carried the cloud account's absolute home path, so the fix was to recreate that account name rather than rewrite the files. The post also records how the guest was sized (4 vCPU with the host CPU type, 8 GB RAM, 32 GB disk), why RAM must not be oversubscribed while vCPUs can be time-shared, and why a partition-order check comes before any growpart.

The first guest on the Proxmox host is up, Docker went in from the official repo method, and the self-hosted agent system that used to run in the cloud now runs from the house. The install isn't the part I want to remember in six months. The part worth remembering is the twenty thousand files that had a username baked into them — a username that didn't exist on the new machine.

The symptom showed up at the very end

Everything looked migrated. Then the app's own doctor command failed with a permission error trying to create a home directory: the cloud account's home directory, on a machine that had never had that account. The cloud box ran the service as one user; the new guest's admin account is a different one. The whole migrated tree — config, agent memory, dream logs, a handful of sqlite databases — had absolute paths hardcoded, because nothing about it had ever needed to be portable.

A recursive grep for the old home path across the migrated directory counted 20,062 files with matches.

Two ways out:

  • Option A: recreate the account name the cloud box used, move the tree under that account's home, chown it recursively, and reinstall the app as that account so its per-user npm prefix matches the old layout. Every baked-in path then resolves untouched.
  • Option B: mass find-and-replace across 20,062 files.

I took A. B looks cleaner in a diff and is riskier in a way that only shows up months later: miss one path — one sqlite file, one memory entry — and the breakage is subtle and arrives far from the change that caused it. A costs one extra account to know about and one move. Framing it as "recreate the environment the app expects" rather than "fix the files" made the trade obvious.

Permission denied can be proof it's working

After the recursive chown, reading the tree as the other account failed with permission denied on a directory-size check. That was correct: the sensitive directories are drwx------, owner-only. The earlier read had worked because it ran as root. Worth writing down, because the instinct is to "fix" that error — and the fix would be to loosen permissions that should stay tight.

Environment differences leak out through the service definition

The gateway was a user unit in the cloud and is a system unit on the guest, so commands need sudo systemctl, not systemctl --user. Same binary, different control path. The error message said so almost literally. System units are also the more robust choice for an always-on server, since they start at boot regardless of who logs in.

That answer also explained an EADDRINUSE on the gateway port: the migrated unit had already started at boot. The migration had succeeded and my manual start was redundant.

RAM is reserved, CPU is time-shared

The agent system, asked to spec its own machine, asked for 6 vCPU / 24 GB RAM / 100 GB disk. It had been living on a small cloud instance using roughly 2 GB in practice. Taking that ask literally would have reserved most of the host's memory for idle use and starved both the hypervisor and the second VM I have planned.

The principle I keep relearning: RAM assigned to a VM is physically reserved and cannot be handed to anything else, so over-assigning risks the host's OOM killer. vCPUs are time-shared, so over-assigning costs contention, not failure. Don't commit more than roughly 75–80% of host RAM across all guests.

I settled on 4 vCPU with the CPU type set to host (the generic default compatibility mode is slower; host passes the real CPU features through), 8 GB of RAM — about four times the observed working set — and a 32 GB disk, which is 6.4 GB used.

The ask wasn't wrong, and that's the part worth keeping. It was right about CPU: deploying sub-agents is genuinely bursty, and that was the actual bottleneck. An agent can see its own workload and can't see the other tenants. Take the diagnosis seriously; do the sizing yourself.

Don't growpart until you know what sits after root

The disk was meant to go 32 → 64 GB. New space is always appended at the end of a disk, and a partition can only expand into free space directly after it. Here root is the first partition and swap is the second, so swap sits between root and the new space — a blind growpart would fail or do damage.

The correct order (deliberately not run yet): snapshot the VM first, grow the virtual disk on the host, swapoff, remove the swap partition, grow and resize the root filesystem, then recreate swap as a 4 GB swapfile inside root. A swapfile lives in the root filesystem, so it never blocks future growth again — that's what removes the ordering constraint permanently. Not urgent at 6.4 GB used of 32 GB, which is exactly why it's the kind of job that gets botched by someone in a hurry.

Smaller things I'd otherwise forget

  • The official-repo Docker install failed once with "no installation candidate" because the long pasted repo-definition one-liner had silently never landed — apt update listed only the distro repos. Re-running it, checking the sources file, then verifying the repo appears in apt update output fixed it. After adding any repo, confirm it shows up in the update output.
  • A fresh minimal Debian install ships without sudo, rsync, or curl. That's by design, not breakage. Install what you hit; before sudo exists at all, su - is the way in.
  • The hypervisor's browser console is for the OS install and for recovery when the network is down — it doesn't resize, can't scroll, and mangles pasted text. Daily work is SSH.
  • The guest's address moved after a reboot and SSH stopped working until I read the new one off the console. Servers want a DHCP reservation bound to the NIC, or a static address.
  • The data was pulled from the cloud box, not pushed: the guest sits behind NAT on a client-isolated building WiFi uplink (the apartment's wired jack is dead — confirmed NO-CARRIER on two cables), so it has no inbound path and must initiate. The copy used archive mode so permissions and ownership survived the trip; those owner-only directories are load-bearing.
  • The cloud instance is stopped and disabled but not terminated, kept as a fallback for a few days. Separately, one integration was already broken with a missing key while it still ran in the cloud — that failure predates the migration, and it's written down here so future me doesn't blame the move for it.

What I still owe myself

The DHCP reservation isn't done. And 4 vCPU is a considered guess until I watch top during a real sub-agent deploy before deciding whether 6 is justified. The sizing was judgment; the measurement is still unpaid.