Roughly 4 GB of junk had just come off the VM, and the nightly backup did not care. It stayed at 7.5 GB compressed. The VM's own files measured 11 GB with du. Those numbers do not sit together: 11 GB of real files, reasonably compressible, should land well under 7.5 GB, and after deleting 4 GB the backup should have gone down rather than sat still.
While chasing it I got a scare that turned out to be nothing, and the nothing is the useful part. The dump directory reported "23 GB" and looked like the VM had ballooned again. It had not: that was three retained nightly backups of about 7.5 GB each, which is keep-last-3 retention working as intended. Twenty-three divided by three kept copies lands back on the size of one dump. A big number is not automatically one problem. Divide before panicking.
It de-indexes, it does not erase
Deleting a file does not scrub its bytes. The filesystem just marks those blocks available in its index, the way removing a book's catalogue entry leaves the book on the shelf. The data stays until something else writes over it. Normal use never notices, because the space does show as free and new files land in it. A raw disk-image backup notices everything.
vzdump does not copy files. It reads the whole 32 GB virtual drive block by block, compressing as it goes, so it reads and compresses the leftovers of every file ever deleted inside that VM: old caches, stale directories, an outdated browser copy. The filesystem forgot them. The disk image did not.
Compression is what makes this visible. Genuinely empty blocks, all zeroes, are one long run of identical bytes and compress to almost nothing. Leftover deleted data is real, varied data, and it compresses like it. A disk full of deleted-file ghosts produces a fat backup; the same free space properly zeroed produces a thin one.
Which is also why three tools gave three different answers. df counts ghost blocks as free, because to the filesystem they are free. du sums real files and never sees ghost data, because deleted files are not files any more. vzdump reads the raw disk image and does see it. So du said 11 GB, df agreed there was free space, and the backup was 7.5 GB because it read the physical disk including the ghosts.
TRIM is the step that reaches the storage
TRIM tells the storage layer that blocks are genuinely free and it can forget their contents. sudo fstrim -av walks mounted filesystems that support it and issues TRIM for free blocks, reporting how much was released per filesystem. It reported 13.2 GiB trimmed on the backup disk and 19.3 GiB on the root disk. That root figure is deleted-file ghost data released at the storage layer, and the next dump should read those blocks as empty and compress them away. That expectation was the payoff still to verify, not a result.
One precondition matters more than the command. TRIM only reaches the underlying storage if the virtual disk was created with discard enabled. Both disks here had discard=on, and I am going on the config read back when the backup disk was added rather than a fresh read that day. Without it, fstrim would run and accomplish nothing.
The automation is a timer: sudo systemctl enable --now fstrim.timer, weekly by default, enabled at boot and started immediately. I still ran it by hand that day, because a lot of deleting had just happened and waiting a week for the timer made no sense.
The bucket had five copies and a rule that could not fix that
Separate problem, same session. The object-storage bucket held five dumps, roughly 31 GB against a 10 GB free tier, with a new object of about 8 GB landing every night. The old prune was age-based, deleting anything older than 8 days, which only works if objects age out faster than they arrive. They do not: nothing reaches 8 days old before several have piled up, so the bucket grew nightly and never corrected itself. Age-based pruning is the wrong tool when the goal is to keep exactly one object.
The replacement uploads the newest local dump, then deletes everything in the bucket except it, but only inside a conditional. The guard lists the bucket's filenames, pipes them through a quiet grep, and anchors the pattern so an entire line has to match the uploaded name exactly. No confirmed upload, no delete.
The guard exists because of specific nights. Two uploads failed with DNS timeouts, the network wedged. Without the check, the script would have deleted everything except today's file, and today's file had never uploaded, so it would have wiped the last good copy and left the bucket empty. With the check, a failed night changes nothing and the previous copy survives. The general question is worth keeping: what does this automated delete do on the night the previous step failed?
I ran it by hand to clear the five stragglers, and the listing afterwards showed one file. That is one cleanup and one listing, not a track record, and a full nightly cycle under the new script is still the thing to watch.
The sizing guess that did not survive measurement
The migration plan carried an assumption that the incoming agent needed far more RAM than the one it replaces. Measurement killed it. The agent process itself sits at about 647 MB resident, and the whole cloud box, meaning the agent plus both web apps plus an accessibility app plus a trading bot plus the MCP servers, comes to 2.5 GB total. They are comparable. The plan is a Rocky Linux 10 VM with 8 GB RAM and 4 vCPU, 6 if bursts demand it, which fits alongside the other VM on a 16 GB host without a RAM upgrade.
Decisions out of the same pass: the trading bot stays in the cloud on its own cheap VPS, because it runs unattended with real money and this home network hangs daily, which is the wrong place for it. Everything else comes home, where downtime is acceptable because there are no customers. Web-facing pieces go behind an outbound tunnel from home instead of port forwarding, which also replaces the fragile reverse-SSH tunnels. Order: trading bot to the VPS, build the VM, move the agent, move the accessibility app, bring up the tunnel, cut over DNS, verify, decommission.
Two questions stayed open, and they are the reason not to start yet: whether a VPS already exists or standing one up is part of the work, and how well the trading bot's own code is understood, since it has real money on it and moves slowest. That box also has a cleanup waiting, a real payload of 3-4 GB sitting under about 22 GB of cache and duplicates.
Still open
Kept short, because the list is texture and not the point: SSH hardening in three steps, keys then restricting root login to keys then fail2ban; the undecided question of whether to encrypt backups before upload, given the dump holds secrets material, credential material, identity material and several providers' API keys; an unidentified token noticed in older notes and never chased down; a leftover repair directory on the root filesystem; a leftover install image still mounted in a VM's virtual CD drive; a config flag to stop checksum retry noise; a memory watch after the VM sat at 78% following four days of uptime; and a look at dmesg for new hardware-hang messages, since the two nights that lost uploads traced back to the hang and uploads were present again once the offload fix took effect.
If any of this stops making sense in six months, three sentences carry it. Freeing space in the filesystem is not freeing space on the disk; TRIM is what closes that gap, and it only works if the virtual disk was created with discard enabled. And any automation that deletes has to be tested against the night its previous step failed: upload-then-prune is only safe when the prune waits for the upload to be confirmed. Also, measure before sizing a move. The guess was wrong by a lot.