Backups and snapshots that actually restore

Know what each layer is for

Different failures need different copies. Match the tool to the failure before you write a single cron line.

If any row has no answer for your setup, that is the gap to close first.

Layer 1: snapshots before risky changes

In the FastNode portal, open the server and take a snapshot before a distribution upgrade, a major config rewrite or a schema migration. It captures the full disk, so a rollback returns the server to that exact moment, including anything written after it. That last part matters: roll back after an hour of live traffic and that hour is gone.

Good habits:

• Name snapshots with the reason and date, e.g. pre-noble-upgrade-2026-10-27.
• Delete them once the change has bedded in. Old snapshots are clutter, not insurance.
• For databases, stop writes for the few seconds it takes, or accept crash-consistent state and let recovery replay the WAL or redo log.

Layer 2: scheduled backups

The portal also runs backups on a schedule you set per server. Use them as the "yesterday" layer: nightly is a sensible default, with retention long enough to notice a slow problem, such as a log rotation bug that has been eating files for a week. Check in the portal what retention applies to your plan before you rely on it.

Layer 3: database dumps

Disk-level copies of a busy database are fine for disaster recovery, but a logical dump is portable, version-tolerant and lets you restore one table. Dump to local disk first, then let the off-site job ship it.

# Postgres: custom format, compressed, restorable per table
sudo -u postgres pg_dump -Fc -d appdb -f /var/backups/db/appdb-$(date +%F).dump

# MySQL / MariaDB: consistent InnoDB dump without locking tables
mysqldump --single-transaction --routines --triggers appdb \
  | zstd -q -o /var/backups/db/appdb-$(date +%F).sql.zst

# Keep 7 local copies
find /var/backups/db -type f -mtime +7 -delete

Pitfall: --single-transaction only gives a consistent view of InnoDB tables. Any MyISAM tables are still dumped while being written to. Convert them or accept the risk.

Layer 4: off-site with restic

restic deduplicates, encrypts client-side and talks to S3-compatible storage, SFTP and more. Install it from your distribution (apt install restic or dnf install restic from EPEL).

# /root/.restic.env  (chmod 600)
export RESTIC_REPOSITORY="s3:https://s3.example-storage.com/my-bucket/web01"
export RESTIC_PASSWORD_FILE=/root/.restic.pass
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

# one-off: create the repository
. /root/.restic.env && restic init

The backup run itself:

#!/bin/sh
# /usr/local/sbin/offsite-backup
set -eu
. /root/.restic.env
restic backup /etc /home /srv /var/backups/db \
  --exclude-caches --exclude '/srv/*/node_modules'
restic forget --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --prune
restic check --read-data-subset=2%

Store the repository password somewhere that is not this server: a password manager, printed in a safe, both. Lose it and the backups are noise. Give the storage credentials write-only or append-only permissions if your bucket provider supports it, so an attacker with root on the VPS cannot delete history.

Prefer borg? Same shape: borg init --encryption=repokey-blake2, borg create, borg prune. borg needs SSH access to a host running borg; restic only needs a bucket.

Schedule it

Run the dump first, then the off-site job. A systemd timer gives you logs and missed-run catch-up for free; the timers guide has the full unit files. Short version:

# /etc/systemd/system/offsite-backup.timer
[Timer]
OnCalendar=*-*-* 03:20:00
RandomizedDelaySec=15m
Persistent=true
[Install]
WantedBy=timers.target

Run a restore drill

Order a spare Spark for an afternoon, or use a local VM, and restore onto it from nothing but the off-site copy and your password manager.

restic snapshots --host web01
restic restore latest --target /restore --include /srv/app
sudo -u postgres createdb appdb_restore
sudo -u postgres pg_restore -d appdb_restore /restore/var/backups/db/appdb-2026-10-26.dump
psql -d appdb_restore -c 'select count(*) from orders;'

Write down how long each step took. That number is your real recovery time, and it is usually longer than anyone guessed. Things drills tend to uncover:

• Secrets in /etc were backed up but the app reads them from an environment file that was excluded.
• The dump restores but extensions such as pgcrypto or postgis are missing on the target.
• File ownership comes back as numeric UIDs that do not exist on a fresh image.
• Nobody knows where the restic password is.

Monitor the backups, not just the server

A silent failure is the common one. Have the job ping a dead-man's-switch URL on success, or alert when the newest snapshot is older than 26 hours:

restic snapshots --latest 1 --json | jq -r '.[0].time'

Feed that into whatever you use from the monitoring basics guide. Backups are only finished when a missed one wakes someone up.