Backups and Recovery

There are three backup jobs. All three write to the same Hetzner Storage Box (account u348035, 20 TB):

What	How	Schedule	Kept
Every VM on hetzner-main	Proxmox vzdump, over a CIFS mount	21:00 daily	7 (keep-weekly=7)
Secops	Borg, over SSH	22:00 daily	everything (no pruning yet)
Vaultwarden	Borg, over SSH	22:30 daily	everything (no pruning yet)

Amergin and Puca aren't backed up. Amergin's configuration is in the infrastructure repo, and its app code and models are on Gitea, so the services can be rebuilt from those. Pyannote's diarisation models are the exception: they're downloaded from Hugging Face. Puca copies its recordings to the lab NAS every hour.

Server configuration (packages, nginx, firewalls, certificates) isn't backed up either, because Ansible can recreate it. The backups cover what the repo can't recreate: VM disks, databases and app data.

Admin-only

Setting up or changing a backup job, and every restore below, is admin-only.

VM backups (vzdump)

Set up by make setup-proxmox-host-backup, the backup tag of the Proxmox Host playbook:

• The Storage Box share is mounted over CIFS at /mnt/backup. The credentials are in /etc/cifs-creds/pve-backup, and the mount is in /etc/fstab.
• A Proxmox storage called pve-backup points at /mnt/backup/pve-backup.
• A backup job covers every VM on the host, including scealai. It runs at 21:00 in snapshot mode (no downtime), keeps 7 copies, and Proxmox prunes older ones itself.
• Files are written to /mnt/backup/pve-backup/dump/vzdump-qemu-<vmid>-<timestamp>.vma.zst.

The playbook only creates the job if the host has no backup job at all. To change the schedule or retention of an existing job, edit it in Proxmox.

Why the backup setup is in Ansible

The mount, the storage and the job were originally set up by hand. When the host was reinstalled on 2026-08-26, nothing recreated them, and no backups ran for about three weeks. The gap was found on 2026-09-15. They are now in the playbook so a reinstall can't silently drop them again. After any rebuild, check that new files are appearing in the dump/ directory.

Restoring files from a VM backup

vzdump has no per-file restore, and restoring a whole VM next to the live one would clash with its IP address. ansible/proxmox-host/restore-from-vzdump-backup.sh pulls individual files out instead, without booting anything. Run it on the Proxmox host as root:

./restore-from-vzdump-backup.sh <vmid> <dest-dir> <path-in-vm> [<path-in-vm> ...]
# e.g.
./restore-from-vzdump-backup.sh 119 /root/restore opt/apps/analytics-plausible

Paths are relative to the VM's root filesystem, with no leading /. The script:

1. takes the newest backup of that VM (you can't pick an older one);
2. checks that /root has about three times the compressed backup size free;
3. decompresses and extracts the disk image;
4. attaches it with qemu-nbd, checks the filesystem, and mounts it read-only;
5. copies each path into <dest-dir>;
6. unmounts and cleans up, even if something fails.

It needs zstd, vma, qemu-nbd and e2fsck on the host.

To restore a whole VM (for example scealai after a rebuild), use qmrestore <backup-file> <vmid> --storage local, then qm start <vmid>.

Secops and Vaultwarden (Borg)

Secops and Vaultwarden aren't VMs on hetzner-main, so vzdump can't reach them. Each runs its own encrypted Borg backup to the Storage Box:

	Secops	Vaultwarden
Make target	setup-secops-backup	setup-vaultwarden-backup
Borg repository	./borg-secops	./borg-vaultwarden
Schedule	22:00	22:30
Backs up	/opt (all stacks, including compose and .env files) and /var/lib/docker/volumes (Wazuh indexer, Loki, Prometheus, Grafana, Uptime Kuma)	/opt/vaultwarden-stack (database, attachments, keys, .env)
Before backing up	pg_dump of Gitea's database to /opt/gitea-stack/postgres-dump.sql.gz, and mongodump of Rocket.Chat to /opt/rocketchat-stack/mongo-dump.archive.gz	sqlite3 .backup of the database to db.sqlite3.backup
Passphrase (Vaultwarden)	Borg Passphrase - secops	Borg Passphrase - vaultwarden

Each playbook:

• installs Borg;
• creates an SSH key on the host itself (/root/.borg/id_ed25519), which never leaves the host;
• writes the passphrase to /etc/borg/passphrase;
• initialises a repokey-blake2 repository;
• installs borg-backup.service with a daily borg-backup.timer.

The repository URL is ssh://u348035@u348035.your-storagebox.de:23/./borg-<host>.

One-time manual step

The playbook can't register the host's SSH key with the Storage Box. On the first run it stops and prints the public key. Then:

1. In Hetzner Robot, go to Storage Box → SSH keys and add the plain public key. The panel rejects keys with a command= prefix.
2. Download .ssh/authorized_keys over SFTP (sftp -P 23 u348035@u348035.your-storagebox.de). Add a restriction in front of that key, and upload the file again:

   command="borg serve --append-only --restrict-to-path ./borg-secops",restrict ssh-ed25519 AAAA... borg-secops

3. Run the same make target again to finish the setup.

What actually protects the backups

--restrict-to-path works: each host's key can only reach its own repository. --append-only doesn't work on the Storage Box. A test on 2026-09-15 showed that a restricted key can still delete archives and reclaim the space. A compromised host could therefore wipe its own backups over SSH.

The real protection is the Storage Box's own Snapshots feature (Robot → Storage Box → Snapshots). It is enabled for the account, and snapshots can only be changed from the Robot panel, not over SSH.

Pruning

Nothing prunes the Borg repositories yet, so they grow forever. Prune by hand from time to time (monthly is fine):

export BORG_PASSPHRASE="..."
REPO=ssh://u348035@u348035.your-storagebox.de:23/./borg-secops
borg prune --keep-daily 14 --keep-weekly 8 --keep-monthly 12 -v --list "$REPO"
borg compact "$REPO"

Repeat for borg-vaultwarden.

Restoring

export BORG_PASSPHRASE="..."
REPO=ssh://u348035@u348035.your-storagebox.de:23/./borg-secops
borg list "$REPO"                                  # find an archive, e.g. secops-2026-09-15_22-00-00
borg extract "$REPO::secops-2026-09-15_22-00-00"   # everything, into the current directory
borg mount "$REPO::secops-2026-09-15_22-00-00" /mnt/restore   # or browse it

Then:

• Gitea: gunzip -c postgres-dump.sql.gz | docker exec -i gitea-db psql -U gitea gitea
• Rocket.Chat: docker exec -i rocketchat-mongodb mongorestore --archive --gzip < mongo-dump.archive.gz
• Vaultwarden: stop the container, copy db.sqlite3.backup over db.sqlite3, then start it again.

Known gaps

• No restore from either Borg repository has been tested, not even a borg extract --dry-run. Don't assume a restore will work until someone has tried one.
• Nothing alerts when a backup fails. Check the timers (systemctl list-timers borg-backup.timer) and the Storage Box contents from time to time.
• The Borg repository keys haven't been exported (borg key export). With repokey encryption, each key exists only inside its own repository. If that part of the repository is damaged, the backups can't be decrypted, even with the passphrase. The passphrases themselves are in Vaultwarden.
• /etc on Secops and Vaultwarden isn't backed up. That's by design, since Ansible recreates it.