Notes from 2026-07-20, the session where backups went from planned to actually leaving the building: rclone on the hypervisor talking to one S3-compatible bucket, a Proxmox backup job, and a small script to push the newest snapshot offsite. The config is forgettable. What belongs on paper is the debugging pattern that cracked it, the arithmetic that decided how many copies exist and where they live, and the two items still unticked — one of which is the only one that turns any of this into a backup.
Three 403s, three different causes
Worth writing out in full, because each 403 had a different cause and the progression is the useful part rather than the fix.
The first one was mine. Listing all buckets on the account is an account-level operation, and the token is deliberately scoped to a single bucket, so the provider refused it — correctly. A bad first test command, not a broken remote. The right test is an object listing inside the bucket. The other thing that reads like failure and is not: an object listing that returns empty output with no error is a success when the bucket is empty.
The second one is the one that taught me something. Reads worked and writes did not. That asymmetry is diagnostic by itself, because the endpoint, the account, the bucket name and the credentials all had to be right or the read would have failed too — which leaves only the permission level in question. My first theory was a read-only token, and the provider dashboard showing Object Read & Write on the correct bucket killed that theory.
The third was the actual cause. Before writing, rclone checks that the bucket exists, a HeadBucket call and possibly a CreateBucket after it. Those are bucket-level operations, and a token scoped to Object Read & Write on one bucket cannot perform them, so the denial surfaces as a 403 on the copy. The fix is one flag on the copy command, --s3-no-check-bucket, made permanent with no_check_bucket = true in that remote's config section.
The rule the notes land on: compare the failure against what still works. Staring at the broken call told me less than noticing that the read on the same credentials had succeeded. The harder half of that lesson is the discipline — do not widen a credential to make an error go away. The token was correct and the command was wrong, and the fix belonged in a flag.
One config detail that will bite again if forgotten: the endpoint has to be the account-level one with nothing appended. The URL shown on the bucket's own settings page includes the bucket name on the end, and using that form breaks path resolution.
The 501 that resolves itself
The first copy attempt failed with a 501 NotImplemented and a retry on the same operation succeeded. Older rclone — the version Debian packages, v1.60.1-DEV — sends an S3 feature the provider does not implement, usually traced to checksum handling. Harmless, noisy in the logs, and the candidate fix recorded in the notes is adding disable_checksum = true to the config. Recorded as a consider, not applied.
The dropdown that defaults to yearly
The backup job was created in the Proxmox UI under Datacenter → Backup → Add. The settings that actually matter, and the one that is easy to get wrong:
- Schedule, set to 02:00. The dropdown defaults to "yearly", which is the trap — it must be changed explicitly. The field also accepts systemd calendar expressions.
- Mode: Snapshot, so the VM keeps running through the backup.
- ZSTD compression, include the selected VM, keep last 3.
- Test it with "Run now" rather than waiting for the schedule to come round.
Recorded result: one compressed image of 4.8 GB built from a 32 GB disk, done in about a minute, with a log and a tiny notes file timestamped alongside it. That listing is the only captured output in these notes, so there is nothing else to quote from the run.
The honest caveat on snapshot mode: keeping the VM running means a database mid-write can land in a slightly inconsistent moment. Snapshot is the right trade here, and if a write-heavy database ever moves into that VM, this is the line to come back to. Suspend pauses the VM briefly for a more consistent image, and stop is the most consistent and the most disruptive.
Three local, one offsite, and one loose end
The retention shape was decided by arithmetic rather than preference. The free tier is 10 GB and one snapshot is 4.8 GB, so exactly one fits comfortably offsite. Two would be 9.6 GB with no margin, and during an upload both copies are held at once, which breaks the tier for the whole transfer. So: 3 local for fast restores and recent history, 1 offsite for the disaster case — fire, theft, drive failure. Beyond the free tier the recorded price is about $0.015 per GB per month, which makes 3 offsite roughly a nickel a month if deeper history is ever wanted.
The push script is four small decisions stacked: take the newest .vma.zst by modification time, exit non-zero when nothing matched rather than running the copy against an empty string, write to a log file at INFO level instead of a terminal nobody is sitting at, and prune remote objects older than 8 days. The logging line is the one that is easy to drop and the one that matters most — the job runs unattended, so a file is the only record of what happened.
Scheduling is a daily 03:30 entry: the backup job runs at 02:00 and takes about a minute, so 03:30 leaves plenty of margin. The first full upload of 4.8 GB over a WiFi-as-WAN link took roughly 38 minutes.
Two notes from the same section that cost nothing to remember. Storage class Standard rather than Infrequent Access: IA is cheaper per gigabyte but adds a retrieval fee and a 30-day minimum storage duration, so a nightly rotating snapshot would pay for 30 days even when deleted after two. Rotating snapshots are the wrong shape for it entirely; IA is for write-once, keep-for-years material. And for an unattended job, use an account-level token rather than a user token — account tokens stay valid independently of the user, while user tokens go inactive when account membership changes.
The loose end I want flagged rather than solved: the constraint section argues that only one snapshot fits the tier, while the script uploads the newest snapshot every run and prunes only objects older than 8 days. On those numbers several could sit in the bucket at once, which does not reconcile with keeping 1 offsite. The notes do not address it. Either the prune window or the retention plan is wrong, and I would rather work that out now than during a restore. Related: the same file's checklist still lists adding the cron entry as unticked, so the 03:30 line above is the intended schedule, not evidence that the push is running.
Where rclone has to live
The backup tool runs on the hypervisor, not inside the guest — the host is what has access to the VM's disk image, so that is where the files land and therefore where rclone has to be installed and configured. Configure it as any other user and the scheduled job running at system level cannot find the credentials. That is the kind of "why here and not there" detail that is obvious for a week and then gone.
Still untested
The verification rule in these notes is a round trip, not an upload: write a test object, copy it up, list it, read it back down from the far side, delete it. The session applied that rule to a tiny test file. It has not been applied to the real image. An upload that returns success proves nothing, and a backup you cannot read back is not a backup. The notes say this plainly — the restore test is marked not yet done.
There is no excuse available for leaving it there, because Proxmox can restore to a different VM ID and boot it without touching the running VM. The restore can be verified while production keeps running.
The decision with a cost attached to delay
Open decision: encrypt before upload. The snapshot image contains the entire VM, including its credential stores, identity material, and API keys for seven providers. The provider encrypts at rest, so this is not about their carelessness — it is about who holds the keys. Layering a crypt remote on top would mean the provider stores ciphertext it cannot read. The trade is blunt: lose the passphrase and the backups are permanently unrecoverable, and no support ticket fixes that. Because setting it up later means re-uploading everything, the cheap moment to decide is before the next cycle rather than in a month.
The mistake that was not technical
A secret access key was pasted verbatim while sharing terminal output. Rotation happened immediately, and the order is the part worth keeping: delete the old token first so there is no window where the leaked key is still live, then create the replacement and update rclone. The habit to build: redact anything after secret_access_key, password, token, or api key before sharing output anywhere.
Still open
Confirm the first offsite upload actually finished inside that roughly 38-minute estimate; settle whether the nightly push is really scheduled; test the restore; decide the encryption question. Hardening has not started: key-based login for the privileged account, then disallowing password root login, then fail2ban. Deferred: grow the disk from 32 GB with a snapshot first, swap off, and replace the swap partition with a 4 GB swapfile; re-price the 32 GB RAM kit against the DRAM rise recorded in these notes, around 172% — which is a figure from my own notes recorded 2026-07-20, not market data I checked; and a future migration that starts by inventorying its source host before sizing hardware to measured need. Housekeeping: clear out a leftover dated directory from an earlier messaging-bridge repair, identify an extra token with broad read-only scope that nobody remembers creating and decide whether it is needed, and try the checksum flag to quiet the 501 noise.
The session's honest summary: the backup runs, the offsite copy is scripted, and the one step that turns a file into a backup — reading it back — has not happened yet. That is the first item next time.