The nightly dump copies the whole VM disk into one compressed file, and across three consecutive nights that file grew: 4.8 GB on Jul 22, 6.2 GB on Jul 23, 7.3 GB on Jul 24. Two things about that stopped being ignorable at the same time. Capacity: the offsite bucket's free tier is 10 GB, so at 7.3 GB a night there is no room for even two of these without paying. Time: 4.8 GB took 34 minutes over this connection, and 7.3 GB takes well over an hour, every night, on a link that keeps dropping. So the session had one question — what is actually eating the disk, and what of it does not need to be there.
One move, repeated
Everything in that session was one technique applied over and over: start at the top of the filesystem, find the biggest thing, descend into it, repeat. The tool is du, disk usage, which adds up the size of everything under a path.
du -sh <path> gives one total for the whole path instead of a line per file (-s for summary) in human-readable units (-h), so a directory reads as 2.5G rather than a long byte count. To compare siblings and see which is biggest, du -xsh <dir>/* | sort -h — that * is every item directly inside the directory, and sort -h orders the output by human-readable size, smallest first, so the biggest result lands at the bottom where I will actually notice it.
Two flags carried most of the weight.
-x keeps du on one filesystem. In an earlier session a separate 16 GB disk was mounted inside the application's own backup directory, and without -x the tool walks into that mount and counts its contents as though they lived on the root disk, which inflates the total and hides what is really there. That is exactly why the same backups folder read 2.5G one way and 370M another: pointed straight at the mount it measured the mounted disk, and measured with -x from above it measured only what sits on the root disk. Both numbers were correct; they were answering different questions. Worth pinning down now, because these two are easy to merge later: that 2.5G is not the same figure as the 2.5 GB stale copy further down this note. They only happen to share a magnitude.
The second flag is the glob. Anything starting with a dot is hidden here — .cache, .local, and the application's own dot-directory — and a plain * matches none of them, so a glob over a home directory walks straight past the directory holding the gigabytes. .[!.]* is the fix: starts with a dot, second character is not a dot, which catches dot-entries while excluding . and .., the two that would otherwise send du walking the whole system. The simpler spelling that avoids the cryptic glob entirely is du -xh --max-depth=1 <dir> | sort -h — totals for the immediate children only, hidden or not, one level down. That is the form that finally surfaced a 2.5 GB dot-directory, after the glob version had come back with nothing at all.
What the descent turned up
The hunt went whole disk, then the home tree, then each account, then each account's directories. The first real signal was a mismatch: the home tree measured 12 GB while the folders I could see only added up to about 6. Data does not vanish and space does not appear from nowhere, so a difference that size is real and it is somewhere. That mismatch is what kept me drilling instead of writing the number off as rounding.
I want the next part written down properly, because it is the thing most likely to get garbled in memory. The descent turned up a complete stale copy of the agent application, 2.5 GB, sitting in the original account's home. That copy is a separate finding from the 12-against-6 mismatch. It is not an explanation of that gap, and the note never balances the gap at all — I never wrote down what filled that difference, and I should not later pretend that I did. Two findings, two numbers, kept apart.
The stale copy itself was a migration leftover. During the migration everything was first copied into that original home, then a copy was moved to the second account's home because the application expected that user, and the original in the first home was never deleted. Worse, the duplicate gateway service that was disabled in an earlier session had been running against this copy — so it was not only dead space, it was also the source of that duplicate-gateway problem.
The other leftovers had the same shape. A ~831 MB pre-repair copy of a browser sat beside its live 416 MB install, taken before a repair on Jul 18 — a browser is reinstallable, and a saved copy of one never needed backing up. The duplicate gateway's runtime installs, ~665 MB, were left across two directories in the same home. And ~370 MB of old one-off repair snapshots sat under the application's backup directory, safety copies from past operations, left in place for the application to confirm.
The pattern is the part worth keeping: almost none of the bloat was precious data. It was duplicates and reinstallable tooling left behind by a migration and a run of repairs, while the genuinely irreplaceable material — agents, memory, workspace, state, secrets — is about 5.5 GB. Everything above that figure was cruft nobody had ever budgeted a cleanup pass for.
Proving the copy was dead before deleting it
Before removing 2.5 GB that had been the entire agent state a week earlier, I wanted written evidence rather than a feeling. That is what stat is for. The timestamp that answers a staleness question is Modify, because it changes only when the contents change. Access updates on a mere read, even a listing, so it is nearly useless for this. Change is the last metadata change, such as permissions, and Birth is creation.
The comparison: the stale copy's Modify time read 2026-07-17, a week earlier, and before the duplicate gateway was disabled. The live copy's Modify time read 2026-07-24, that same day. Nothing had written to the old copy since the week before, and the live one was current. That comparison is what turned deleting it into a check instead of a guess.
The cleanup, and the habit that goes with it
Snapshot first. The 2.5 GB about to be removed had been the whole agent state a week earlier, so the snapshot was not ceremony: it was taken through the VM's snapshot interface and labelled for the cleanup. Then four recursive deletes — the stale application copy, its two runtime directories, and the pre-repair browser copy.
rm -rf is the most dangerous command in common use, and it is worth writing down why: it deletes immediately and permanently, with no trash can and no undo. The only safety in the command itself is reading the path three times before pressing Enter. The snapshot is the real safety net.
Then df -h / showed used space dropping from 15G to about 11G. Keep both of those as approximations, the way the note records them — the four deletions add up only roughly to that drop, there is no exact after-figure, and I am not going to compute a tidier number than the one I have. After that I confirmed the application was still answering on its messaging integration and that the gateway was healthy, and then deleted the cleanup snapshot so it would not turn into the next thing eating space itself.
The distinction underneath all of this is worth keeping straight, because it explains most of the confusion that comes later: df asks the filesystem how full it is — fast, and authoritative for that question — while du adds up the files in a tree, which is slower and tells you where the space went. When the two disagree, something interesting is happening: deleted files still held open by a process, or files hidden underneath a mount, as in an earlier session.
The offsite side was a different failure
The offsite copy had its own problem, independent of the disk cleanup, and it was not a fault in the transfer tool. The nightly backup log showed DNS timeouts at 03:56 and 04:23, and the job had fired at 03:30 while the network interface was hung from the earlier session's driver bug, so the upload never reached the destination at all. The last two nights failed for that reason and no other. It is the kind of failure that reads as a backup problem until you look at the log.
The bucket itself held three snapshots totalling about 15.4 GB, over the 10 GB free tier, and nothing had aged out because the prune rule only removes objects older than 8 days. That makes it a small monthly charge unless it is cut to one snapshot. The failed uploads also mean what is actually offsite is the 20th, 21st and 22nd, not the most recent nights.
The decision is still open, and deliberately so: keep one offsite snapshot, which is free, or keep three, which costs a few cents a month, about $0.05. Either is fine. It just needs to be chosen rather than left to drift.
What is still open
The unchecked items are the honest part of the note. Tonight's dump should land under 5 GB after the cleanup, and that is a prediction to confirm, not a result I have. The ~370 MB of old repair snapshots stay where they are until the application is asked, because it created them and it is the thing that knows what is still referenced.
Carried over and unresolved: access hardening in three steps — key-based authentication, then refusing interactive password login for the root account, then a banning tool; whether to encrypt the backups before upload, given that the dump carries live credential material; a leftover timestamped repair directory under the superuser's home; an unidentified credential dated May 2026 whose origin is still unknown; a checksum setting in the transfer tool's config that would quiet the retry noise; a leftover distribution image still attached to the VM's virtual optical drive; and VM memory, which was at 78% after four days of uptime. The network-interface hang fix from the earlier session still needs verifying over the coming days — no new hang messages in the kernel log, and the offload settings confirmed to have stuck after any reboot — with the note that the failed uploads should resume on their own once that fix is confirmed.
One longer-term idea came out of this session and I do not want to lose it: most of this VM is reinstallable tooling rather than precious state. If the dumps keep growing, back up only the ~5.5 GB that actually matters instead of the whole VM. Smaller and faster, at the cost of a more manual restore, because the OS and the tools would have to be rebuilt by hand.
The part worth remembering
Both problems in this session were leftovers rather than failures of anything that was running: an old duplicate of the application that nobody had deleted, quietly the source of a second and apparently unrelated problem, and an offsite copy sized for a nightly dump that had outgrown its free tier.
Three habits came out of it. A stale copy is not neutral — it costs space, and it can still be the thing some old process is running against. The way to prove one is safe to remove is to compare Modify times against the live copy first, take a snapshot, then delete with the path read three times, and delete that snapshot afterwards so it does not become the next thing eating space. And when a total does not match the sum of the visible parts, the mismatch is the lead, not an error to round off: measure at each level and let the numbers point down, with -x so a mount inside the tree cannot lie to you and --max-depth=1 so a dot-directory cannot hide. If the descent does not explain the whole difference, write down what is left over. This note does not, which is exactly why one of its findings is easy to misread six months from now.
The small one I do not want to lose either: retention is the kind of cheap decision that drifts for months when it is not written down. It is written down here. Pick one before the next prune.