The Backups That Weren't: A Stale Fingerprint and Two Months of Silence

Every backup horror story starts the same way: “I thought it was working.” Ours started with a one-line question to the AI agent driving our homelab — “can you confirm backups are running for PBS?” — and ended several hours later with the uncomfortable discovery that our virtual machines hadn’t been backed up off the local disk in two months, and that the very alerting system meant to warn us had been silently failing the entire time.

This post is the story of that afternoon: what was broken, why we never found out, and how we fixed it so it can’t happen the same way again.

”Are backups running?”

We have a textbook 3-2-1 setup on paper:

Local — Proxmox vzdump of every VM and LXC to a second disk on the hypervisor.
PBS — a second vzdump job to a dedicated Proxmox Backup Server (deduplicated, incremental, easy point-in-time restore).
Offsite — the PBS datastore mirrored to Google Drive with rclone.

So when we asked the agent to confirm backups, we expected a quick green checkmark. Instead, checking the actual contents of the PBS datastore told a different story. The datastore held exactly 2.3 GB and a single backup group — our Docker volumes. No VM images. No LXC archives. For a fleet of a dozen guests, that’s almost nothing.

The schedule looked fine. The job was enabled. The local backups were healthy (629 GB of recent archives on the second disk). But nothing from the nightly PBS job had landed in months.

Following the thread

The Proxmox task log made the cause embarrassingly clear. Every single night, the PBS backup job failed with the same error:

could not activate storage 'pbs': pbs: error fetching datastores -
fingerprint '15:CF:E2:…' not verified, abort!

Proxmox VE pins the TLS certificate fingerprint of the Backup Server it talks to — a good security default, so a man-in-the-middle can’t impersonate your backup target. But that pin is only as good as its freshness. We compared the two values:

The fingerprint PVE expected: 63:7B:D2:… (pinned when the storage was first added)
The fingerprint PBS actually presented: 15:CF:E2:…

They didn’t match. The certificate’s own metadata told us why: PBS had been rebuilt on a date about two months prior, and a fresh PBS install mints a brand-new self-signed certificate. The new server had a new identity; PVE was still holding the old one and, correctly but catastrophically, refused to talk to it. Every VM backup had been aborting before it even started.

The local backups kept working (they don’t touch PBS), which is exactly why nothing looked obviously on fire. The redundant copy was the one quietly failing — the worst kind, because you only discover it when you need it.

The deeper problem: nobody was being told

Here’s the part that turned a backup bug into a real lesson. We had an alert for this. Our monitoring stack ships a PVEBackupFailed rule that fires when a backup job reports failure. It had been evaluating correctly and firing for two months. So why no email?

Because the alert delivery was broken too. Our Alertmanager was configured to send via Gmail SMTP, and the logs were a wall of:

535 5.7.8 Username and Password not accepted … BadCredentials

The app password had been revoked at some point. Every alert — not just the backup one — had been failing to send. The smoke detector was screaming into a disconnected phone line.

And it got one layer deeper still. When we looked at which hosts were even being monitored, three were dark:

PBS itself had no node_exporter installed — so the metric that powers the “offsite sync failed” alert physically couldn’t reach the monitoring server. That alert was structurally incapable of firing, regardless of delivery.
The control node had lost its exporter during an earlier VM migration.
The reverse proxy sits in an isolated DMZ where the firewall only permits port 443, so it couldn’t be scraped on the usual metrics port at all.

The scrape target list, it turned out, hadn’t been regenerated since the rebuild — it was missing hosts added since, and still listed hosts that no longer applied. A quiet cascade: broken delivery, missing exporters, a stale target list. Any one of them alone would have blinded us to the backup failure. We had all three.

Fixing it — and making it stay fixed

A fix you have to remember to repeat isn’t a fix. So for each problem we did two things: repair the live system, and codify the repair so a future rebuild can’t reintroduce it.

The fingerprint. Re-pinning it on PVE was a one-liner and VM backups immediately started flowing again — the datastore went from 2.3 GB to 95 GB overnight as the whole fleet finally landed. But the real fix was teaching our automation to reconcile the fingerprint on every deploy: read the Backup Server’s live certificate, and update the hypervisor’s stored value to match. Now a PBS rebuild self-heals instead of silently severing backups for two months.

The offsite copy. While we were in there, we found the Google Drive sync had its own issues — it was choking on a filesystem artifact and overlapping with garbage collection, and it was scheduled weekly. That last one bothered us: our local and PBS copies have a ~1-day recovery point, but the offsite disaster copy was up to a week stale. The whole point of offsite is the house-burns-down scenario; a week-old copy is a weak position. Since the sync is incremental, we moved it to nightly. Re-enabling the timer immediately kicked off the first full upload of the now-complete 95 GB datastore.

The alerting. Gmail had burned us, and email is a single fragile channel anyway. We switched the primary alert channel to Telegram — push notifications straight to a phone, no SMTP, no app-password rot. We installed the missing exporters so every reachable host now reports a heartbeat, taught the reverse proxy to expose its metrics through the one port the DMZ firewall allows, and — importantly — wired monitoring into the master deploy playbook so a freshly provisioned host can’t exist without a heartbeat. As one of us put it: a host in the inventory with no heartbeat isn’t a host, it’s a blind spot.

We also found, almost by accident, that our master site.yml playbook used an invalid Ansible keyword and would have errored on the first line if anyone ever ran it. Two months of nobody running the full deploy had hidden that too. Fixed.

What we actually learned

A green schedule is not a backup. “The job is enabled” and “the job is succeeding” are different claims. Verify the artifacts — open the datastore, count the snapshots, check their dates. We now verify by looking at what landed, not what was scheduled.
Test your alerting like you test your backups. A broken alert channel is worse than no alerting, because it manufactures false confidence. If we’d ever sent ourselves a test alert, we’d have caught the dead Gmail credentials — and through them, the backups.
An absent metric is a silent failure. up == 0 only fires for a host that’s being scraped. A host that was never added, or whose exporter was never installed, produces no series and trips no alert. Coverage has to be guaranteed at provisioning time, not hoped for.
Pinned trust needs a refresh plan. Certificate pinning is the right call for a backup target — but a pin with no reconciliation is a time bomb that goes off the next time the other end is rebuilt. Automate the re-pin.
Codify the fix, not just the repair. Every fix here became a few lines of Ansible so the next rebuild reconciles automatically. The incident is only truly closed when the mechanism that caused it can’t recur.

The most valuable thing in this whole episode cost nothing: a healthy dose of suspicion. “Confirm backups are running” turned out to be the single most useful sentence we’d said to the homelab in months. If it’s been a while since you looked — not at the schedule, at the snapshots — go look. We’ll wait.