Phase 3: Backups with Proxmox Backup Server and Google Drive

The Problem

We had been running a homelab with 14 VMs and LXCs for years and had no real backup strategy. Proxmox’s built-in snapshot mechanism existed, but snapshots live on the same host — when the host dies, they go with it. This is not a backup. It is a comfort blanket.

Phase 3 of the homelab IaC project was dedicated to fixing this. We needed something that could answer a simple question: if pve died tonight, how long until we are back up?

Choosing the Right Tools

Why Not Restic?

Restic is excellent for file-level backups. But we run Proxmox, and the right tool for backing up a Proxmox environment is Proxmox Backup Server. PBS speaks native Proxmox backup protocol — it understands the difference between a dirty and clean VM state, supports incremental chunk-based deduplication, and integrates directly into the PVE web UI for scheduling and restore.

With restic, backing up a running VM means backing up a disk image as a file. PBS is aware of VM state and coordinates with the QEMU agent for consistent snapshots. The dedup ratio on our fleet (lots of similar Debian/Ubuntu base images) was around 40% — 14 VMs backed up in roughly the space of 8.

Why Not Just PVE Snapshots?

PVE snapshots are fast and storage-efficient, but they live on the same physical disk pool as the VMs. They also cannot be moved offsite without significant effort. PBS decouples the backup storage from the live storage and provides a clean API for remote restore.

B2 vs Google Drive

Our original plan was Backblaze B2 for offsite storage — it is cheap, fast, and has a well-supported rclone backend. We reconsidered when we looked at the actual data volume. Our PBS datastore would top out around 150–200GB compressed and deduplicated. We are already paying for a 2TB Google Drive subscription through Google One. The incremental cost of using Drive for offsite backup was zero dollars. B2 would have been a few dollars a month, which is not the point — the point is that we were introducing a new dependency and billing relationship when an equivalent one already existed.

Simpler wins.

What We Built

PVE (nightly) → PBS local (300GB HDD) → Google Drive (weekly rclone sync)

PBS LXC (VMID 115)

We provisioned a Debian 12 LXC at 10.0.0.25 (pbs) via Terraform and added it to the backup group in Ansible inventory. The LXC has two storage allocations:

Root disk (8GB) on local-lvm — OS and PBS software
Data disk (300GB) on local-disk1 — the backup datastore, mounted at /mnt/datastore/main

Using a separate physical disk (local-disk1 is a spinning HDD, while local-lvm sits on the SSD pool) gives us the second copy in the 3-2-1 rule on a different failure domain. An SSD failure does not take the backups.

The Ansible Role

The pbs role handles the full installation:

Disable the enterprise repo before any apt update. This is critical — the proxmox-backup-server package activates the enterprise repo in /etc/apt/sources.list.d/pbs-enterprise.list as a post-install step. On the next run without a subscription, apt update returns a 401 and the play fails. We disable it first.
Add the no-subscription repo and install proxmox-backup-server.
Create the datastore at /mnt/datastore/main using proxmox-backup-manager:

- name: Create PBS datastore
  ansible.builtin.command:
    cmd: >
      proxmox-backup-manager datastore create {{ pbs_datastore_name }}
      {{ pbs_datastore_path }}
      --keep-last {{ pbs_retention_keep_last }}
      --keep-daily {{ pbs_retention_keep_daily }}
      --keep-weekly {{ pbs_retention_keep_weekly }}
      --keep-monthly {{ pbs_retention_keep_monthly }}
  when:
    - pbs_datastore_list.rc == 0
    - pbs_datastore_name not in (pbs_datastore_list.stdout | default(''))

Our retention defaults: keep last 3, daily for 7 days, weekly for 4 weeks, monthly for 3 months.

Create the admin@pbs user and grant DatastoreAdmin access. This user is how PVE authenticates to PBS. The root@pam account on PBS exists but is not appropriate for regular backup operations — it has no password by default and is meant for emergency console access only.
UFW rules restricting the PBS web UI (port 8007) to 10.0.0.0/24. PBS has no business being accessible from the internet.

Retention Policy

# roles/pbs/defaults/main.yml
pbs_retention_keep_last: 3
pbs_retention_keep_daily: 7
pbs_retention_keep_weekly: 4
pbs_retention_keep_monthly: 3

This gives us granular rollback for recent incidents (keep-daily covers the past week), and longer-horizon recovery for problems that go undetected for weeks or months (keep-weekly, keep-monthly). The 300GB datastore is more than sufficient — PBS dedup means the incremental cost of a daily backup is typically a few hundred MB once the base is established.

PVE Backup Job

After deploying PBS, we configured PVE to back up all 14 VMs and LXCs nightly. This is done through the PVE web UI under Datacenter → Backup — there is no Terraform or Ansible for this yet (it is on the Phase 4 list). The job runs at 02:00 UTC, targets all guests, and uses the main datastore on pbs.

rclone + Google Drive

For offsite sync we use rclone with a Google Drive OAuth2 app. The rclone config lives at <rclone-config-path> and is deployed via Ansible from a Jinja2 template:

[gdrive-homelab]
type = drive
client_id = {{ gdrive_client_id }}
client_secret = {{ gdrive_client_secret }}
scope = drive
token = {{ gdrive_token | to_json if gdrive_token is mapping else gdrive_token }}

The credentials flow from Ansible Vault: vault_gdrive_client_id, vault_gdrive_client_secret, vault_gdrive_token are encrypted in vault.yml and exposed through group_vars/backup/vars.yml.

The sync runs weekly on Sunday at 03:00, driven by a systemd timer:

[Timer]
OnCalendar=Sun *-*-* 03:00:00
Persistent=true
RandomizedDelaySec=300

Persistent=true means if the system was powered down at 03:00 Sunday, the sync triggers automatically when it next boots. The RandomizedDelaySec spreads any potential fleet-wide timer collisions by up to five minutes.

The service unit caps resource usage to prevent the backup sync from interfering with live workloads:

CPUQuota=50%
IOWeight=100

After each run, the exit code is written to a node_exporter textfile collector path so Grafana can alert on sync failures:

ExecStartPost=/bin/sh -c 'echo "rclone_gdrive_sync_exit_code $?" > \
  /var/lib/node_exporter/textfile_collector/rclone_gdrive_sync.prom'

Gotchas

1. The Google Drive OAuth Token Format

rclone authorize outputs the token as a Python dict, not valid JSON:

{'access_token': '...', 'token_type': 'Bearer', ...}

Ansible Vault stores it as a string. Jinja2’s default serialization passes it through unchanged — which means rclone gets a Python dict literal where it expects JSON, and authentication fails with an unhelpful parse error.

The fix is the conditional to_json filter in the template:

token = {{ gdrive_token | to_json if gdrive_token is mapping else gdrive_token }}

If Ansible has already parsed the token as a YAML mapping (which happens if the value is valid JSON stored in vault), to_json serializes it back to proper JSON. If it comes in as a raw string (Python dict format), we pass it through and rely on the user to have converted it manually before vaulting. The right workflow is: run rclone authorize, copy the output, replace the outer single quotes with braces to make it valid JSON, then vault the result.

2. The Enterprise Repo Activation Trap

The proxmox-backup-server package ships a post-install hook that activates the enterprise repository at /etc/apt/sources.list.d/pbs-enterprise.list. Without a subscription, the next apt update returns a 401 from enterprise.proxmox.com and the entire apt operation fails.

The mitigation is task ordering. Our role disables the enterprise repo as its first task — before the package is even installed:

- name: Disable PBS enterprise repository (requires subscription)
  ansible.builtin.replace:
    path: /etc/apt/sources.list.d/pbs-enterprise.list
    regexp: '^(deb .*enterprise.*)'
    replace: '# \1'
  failed_when: false  # file may not exist on first run

failed_when: false handles the first run where the file does not yet exist. On subsequent runs, the file is there (written by the package post-install), and we disable it before any apt update can touch it.

3. The LXC CA Bundle

The fresh Debian 12 LXC did not include the Let’s Encrypt R12 intermediate CA in its certificate bundle. When our role tried to download the Proxmox GPG key over HTTPS:

SSL: CERTIFICATE_VERIFY_FAILED

We used validate_certs: false for the GPG key download, which is acceptable because the GPG key itself is what provides trust for the packages — once the key is in trusted.gpg.d/, apt verifies every package against it regardless of how the key was retrieved. The security model here is correct.

The permanent fix is to run update-ca-certificates or install the ISRG Root X1 certificate, but for a fresh LXC that only needs to do this once, the pragmatic approach is fine.

4. `keep-last: 0` Is Invalid

PBS requires --keep-last to be at minimum 1. Our initial draft had 0 as a possible value for “don’t use this retention dimension.” PBS rejects it with a validation error. We set the minimum to 3.

5. `admin@pbs` Doesn’t Exist By Default

PBS ships with only root@pam. The admin@pbs user (in the PBS PAM realm, not the Linux PAM realm) must be explicitly created and granted ACL access on /datastore/main before PVE can connect. The ACL must be on the datastore path — granting it at / does not work as expected for backup operations.

The Restore Test

A backup system that has never been tested is not a backup system.

We restored wastebin (CT 104, approximately 3GB compressed) to VMID 999 using the PBS web UI. Steps: select the backup, click Restore, choose target VMID and storage, start.

Time from initiation to container ready: approximately 60 seconds.

We started the container, confirmed the wastebin service was active on port 6544, and retrieved a paste we had created before the backup ran. Clean.

One operational note: the restore used the original network config, meaning it came up with 10.0.0.6 — the same IP as the live wastebin. In a real disaster recovery scenario, you want to either shut down the live container first or change the restored container’s network config before starting it. IP conflicts on the LAN are disruptive.

3-2-1 Achieved

Copy	Location	Medium	Failure Domain
Live	Proxmox hosts	SSD pool	Single host
2nd	PBS datastore	HDD (`local-disk1`)	Different disk, same host
3rd	Google Drive	Cloud	Offsite

The HDD (local-disk1) is a separate physical drive from the SSD pool running the VMs. An SSD failure does not touch the backups. A host-level failure (power supply, motherboard, fire) is covered by Google Drive.

We are not protecting against the case where the entire physical location is destroyed, because for a homelab that threat model implies consequences that extend well beyond “how long until the services are back up.” The 3-2-1 rule is satisfied.

Lessons Learned

1. Order tasks around failure modes, not installation order. The enterprise repo gotcha is a perfect example. The “natural” ordering is install, then configure. But because the package activates a repo on install, the right ordering is configure-the-thing-that-will-break, then install.

2. rclone authorize outputs Python, not JSON. If you are vaulting an OAuth token for rclone, convert it manually before storing. Or use the Jinja2 conditional filter and document why it exists.

3. Test the restore before you need it. We found one real gap — the IP conflict scenario — during a calm planned test. That is the right time to find it.

4. PBS dedup ratio is real. Our 14-guest fleet runs mostly Debian and Ubuntu base images with similar package sets. Initial backup was about 80GB. With daily incrementals running for a week the datastore grew by about 15GB total. 300GB will last years at this rate.

What’s Next

The PVE backup job configuration is still manual (web UI). Phase 4 will bring that under Terraform or Ansible so the full backup pipeline — not just PBS itself — is declarative and reproducible. We also want to add a Grafana dashboard panel pulling from the node_exporter textfile to show last sync time and exit code for the Google Drive offsite job.