Incremental Backup with rsync and cron
Back up one Linux server to another with rsync and cron: dedicated, restricted SSH key, hard-link snapshots, rotation and a restore test.
On this page
This guide explains how to back up a server every night to a second server with rsync and cron. Each backup looks like a full, dated copy, but unchanged files are not stored again: they are linked to the previous backup with hard links.
The backed-up server uses a dedicated SSH key, limited to writing into a single folder. Even if this server is compromised, the key cannot open a shell on the backup server, nor read back or delete older backups through rsync.
Prerequisites
- Two servers running Debian 12 or 13, or Ubuntu 22.04 or 24.04, with
sudoaccess - The source server, the one to back up, called
web01in this guide - The backup server, with enough disk space, ideally in another location
- A file system that supports extended attributes on the backup server, such as ext4 or XFS
Replace BACKUP_SERVER_IP and SOURCE_SERVER_IP with the addresses of your servers.
Port to open
Only the backup server receives a connection: port 22/tcp, or your SSH port, must be reachable from the source server's address. With UFW, on the backup server:
sudo ufw allow from SOURCE_SERVER_IP to any port 22 proto tcpKeep the rule that lets you administer this server yourself. Our guide Secure your server details the SSH configuration.
Prepare the backup server
-
Install the packages.
python3is required by therrsyncscript, which ships with rsync:Terminalsudo apt update sudo apt install -y rsync python3 cron -
Create a dedicated account, with no usable password, and the destination folder:
Terminalsudo useradd --create-home --home-dir /srv/rsyncbackup --shell /bin/sh rsyncbackup sudo install -d -o rsyncbackup -g rsyncbackup -m 700 /srv/rsyncbackup/.ssh /srv/rsyncbackup/web01
Prepare the source server
-
Install the packages:
Terminalsudo apt update sudo apt install -y rsync cron openssh-client -
Create a key reserved for the backup. It has no passphrase, because cron must be able to use it on its own: this is why it will be heavily restricted in the next step.
Terminalsudo install -d -m 700 /root/.ssh sudo ssh-keygen -t ed25519 -N '' -C 'backup-rsync@web01' -f /root/.ssh/backup_rsync sudo cat /root/.ssh/backup_rsync.pub
Restrict the key on the backup server
-
Open the authorized keys file:
Terminalsudo nano /srv/rsyncbackup/.ssh/authorized_keys -
Add a single line, with your public key at the end:
Textfrom="SOURCE_SERVER_IP",restrict,command="/usr/bin/rrsync -wo -no-del /srv/rsyncbackup/web01" ssh-ed25519 AAAA...your_public_key backup-rsync@web01Element Purpose from=Accepts this key from the source server only restrictDisables the terminal and SSH forwarding command=Forces rrsyncin place of any other command-woAllows write-only access, with no reading back -no-delRejects rsync's deletion options -
Apply the permissions:
Terminalsudo chown rsyncbackup:rsyncbackup /srv/rsyncbackup/.ssh/authorized_keys sudo chmod 600 /srv/rsyncbackup/.ssh/authorized_keys -
Display the server fingerprint to compare it in the next step:
Terminalsudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
Check the connection
On the source server, record the backup server's key, then compare its fingerprint with the previous one:
ssh-keyscan -t ed25519 BACKUP_SERVER_IP | sudo tee -a /root/.ssh/known_hosts
sudo ssh-keygen -lf /root/.ssh/known_hostsIf the fingerprints differ, delete the added line and find the cause before continuing. Then test the key:
sudo ssh -i /root/.ssh/backup_rsync -o IdentitiesOnly=yes rsyncbackup@BACKUP_SERVER_IPThe expected response is an rrsync error containing Not invoked via sshd. It proves that the key is accepted and that it does not give a shell.
Create the backup script
On the source server:
-
Create the exclusion list:
Terminalsudo nano /etc/backup-rsync.excludeText/root/.ssh/backup_rsync* /root/.ssh/restore_rsync* /root/.cache/ /home/*/.cache/ /var/www/*/cache/ *.tmp -
Create the script, filling in the backup server's address and the folders to copy:
Terminalsudo nano /usr/local/sbin/backup-rsync.shTerminal#!/bin/bash set -uo pipefail BACKUP_HOST="BACKUP_SERVER_IP" BACKUP_USER="rsyncbackup" SSH_PORT=22 SSH_KEY="/root/.ssh/backup_rsync" SOURCES=(/etc /home /root /var/www /var/backups) EXCLUDES="/etc/backup-rsync.exclude" STATE_DIR="/var/lib/backup-rsync" SNAPSHOT="$(date +%Y-%m-%d_%H%M%S)" exec 9>/run/lock/backup-rsync.lock if ! flock -n 9; then echo "$(date -Is) backup already running" exit 0 fi mkdir -p "$STATE_DIR" EXISTING=() for path in "${SOURCES[@]}"; do [ -e "$path" ] && EXISTING+=("$path") done SSH_CMD="ssh -i $SSH_KEY -p $SSH_PORT -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=20" OPTIONS=(-aHAX --numeric-ids --relative --timeout=600 --stats --exclude-from="$EXCLUDES" -M--fake-super -e "$SSH_CMD") if [ -s "$STATE_DIR/last" ]; then OPTIONS+=("--link-dest=/$(cat "$STATE_DIR/last")") fi echo "$(date -Is) starting $SNAPSHOT" rsync "${OPTIONS[@]}" "${EXISTING[@]}" "$BACKUP_USER@$BACKUP_HOST:$SNAPSHOT/" status=$? # Code 24 means some files vanished during the copy. if [ "$status" -ne 0 ] && [ "$status" -ne 24 ]; then echo "$(date -Is) failed with rsync code $status" exit "$status" fi MARKER="$(mktemp)" rsync -e "$SSH_CMD" "$MARKER" "$BACKUP_USER@$BACKUP_HOST:$SNAPSHOT/.backup-complete" status=$? rm -f "$MARKER" if [ "$status" -ne 0 ]; then echo "$(date -Is) could not write completion marker, rsync code $status" exit "$status" fi echo "$SNAPSHOT" > "$STATE_DIR/last" echo "$(date -Is) finished $SNAPSHOT" -
Make it executable and run the first backup. It copies everything and can take a while:
Terminalsudo chmod 700 /usr/local/sbin/backup-rsync.sh sudo /usr/local/sbin/backup-rsync.sh -
Wait a minute, then run it again to create a second snapshot:
Terminalsudo /usr/local/sbin/backup-rsync.sh
| Option | Purpose |
|---|---|
-a |
Preserves the directory tree, timestamps, permissions and symbolic links |
-H, -A, -X |
Preserve hard links, ACLs and extended attributes |
--numeric-ids |
Preserves numeric user and group IDs |
--relative |
Reproduces full paths in the snapshot |
--link-dest |
Links unchanged files to the previous snapshot |
-M--fake-super |
Stores owners and permissions in extended attributes, without being root at the destination |
Check the snapshots
On the backup server:
sudo ls -la /srv/rsyncbackup/web01
sudo sh -c 'du -sh /srv/rsyncbackup/web01/*'
sudo sh -c 'stat -c "%h %n" /srv/rsyncbackup/web01/*/etc/hostname'du counts each shared file only once: the second snapshot should therefore look much smaller than the first. stat must show a link count of at least 2 for an unchanged file. Each completed snapshot contains a .backup-complete file.
Schedule with cron
On the source server:
-
Create the scheduled task:
Terminalsudo nano /etc/cron.d/backup-rsyncTextSHELL=/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 15 3 * * * root /usr/local/sbin/backup-rsync.sh >> /var/log/backup-rsync.log 2>&1This file must be owned by
rootand end with a newline. -
Limit the size of the log:
Terminalsudo nano /etc/logrotate.d/backup-rsyncText/var/log/backup-rsync.log { weekly rotate 8 compress missingok notifempty } -
Check that cron is active, then check the log the next day:
Terminalsudo systemctl is-active cron sudo tail -n 30 /var/log/backup-rsync.log sudo cat /var/lib/backup-rsync/last
Delete old snapshots
Rotation runs on the backup server. The source server therefore has no way to erase the history.
-
Create the script:
Terminalsudo nano /usr/local/bin/backup-rsync-pruneTerminal#!/bin/bash set -euo pipefail ROOT="/srv/rsyncbackup/$1" KEEP="${2:-14}" cd "$ROOT" mapfile -t COMPLETE < <(find . -mindepth 2 -maxdepth 2 -type f -name .backup-complete -printf '%h\n' | sed 's|^\./||' | sort) COUNT="${#COMPLETE[@]}" [ "$COUNT" -gt "$KEEP" ] || exit 0 OLDEST_KEPT="${COMPLETE[COUNT-KEEP]}" for dir in 20*/; do snapshot="${dir%/}" if [[ "$snapshot" < "$OLDEST_KEPT" ]]; then rm -rf -- "${ROOT:?}/$snapshot" fi doneIt keeps the last 14 completed snapshots and removes everything older, including interrupted snapshots. Deleting a snapshot does not remove files still used by the following ones.
-
Schedule it:
Terminalsudo chmod 755 /usr/local/bin/backup-rsync-prune echo '45 5 * * * rsyncbackup /usr/local/bin/backup-rsync-prune web01 14' | sudo tee /etc/cron.d/backup-rsync-prune -
To be alerted if backups stop, monitor this check, which fails if no snapshot has completed in the last 26 hours:
Terminalsudo find /srv/rsyncbackup/web01 -mindepth 2 -maxdepth 2 -name .backup-complete -mmin -1560 | grep -q . && echo "OK" || echo "NO RECENT BACKUP"
Test the restore
Do this test right away, then regularly. Since the backup key cannot read the data, create a second key, read-only, reserved for restores.
-
On the source server, create this key:
Terminalsudo ssh-keygen -t ed25519 -N '' -C 'restore-rsync@web01' -f /root/.ssh/restore_rsync sudo cat /root/.ssh/restore_rsync.pub -
On the backup server, add this line to
/srv/rsyncbackup/.ssh/authorized_keys:Textfrom="SOURCE_SERVER_IP",restrict,command="/usr/bin/rrsync -ro /srv/rsyncbackup/web01" ssh-ed25519 AAAA...restore_key restore-rsync@web01 -
On the source server, list the snapshots:
TerminalBACKUP_HOST="BACKUP_SERVER_IP" SSH_RESTORE="ssh -i /root/.ssh/restore_rsync -p 22 -o BatchMode=yes -o IdentitiesOnly=yes" sudo rsync --list-only -e "$SSH_RESTORE" "rsyncbackup@$BACKUP_HOST:/" -
Restore
/etcinto a temporary folder, from the latest completed snapshot:TerminalSNAPSHOT="$(sudo cat /var/lib/backup-rsync/last)" sudo mkdir -p /var/tmp/rsync-restore sudo rsync -aHAX --numeric-ids -M--fake-super -e "$SSH_RESTORE" "rsyncbackup@$BACKUP_HOST:$SNAPSHOT/etc/" /var/tmp/rsync-restore/etc/ -
Compare the content, owner and permissions:
Terminalsudo diff -r /etc /var/tmp/rsync-restore/etc | head -n 20 sudo stat -c '%U:%G %a %n' /etc/shadow /var/tmp/rsync-restore/etc/shadowOnly files modified since the backup should differ, and the two
statlines must show the same owner and permissions. -
Delete the test copy, which contains sensitive data:
Terminalsudo rm -rf /var/tmp/rsync-restore -
If you have nothing to restore, remove the
restore-rsynckey line on the backup server. You can add it again when needed.
Restore after an incident
Pick from the list a snapshot containing .backup-complete and put its name in SNAPSHOT. First simulate the operation with --dry-run and check the list, because it can overwrite files:
sudo rsync -aHAX --numeric-ids -M--fake-super --dry-run --itemize-changes -e "$SSH_RESTORE" "rsyncbackup@$BACKUP_HOST:$SNAPSHOT/var/www/" /var/www/If the result looks right, run the command again without --dry-run. If the source server is lost, prepare the new server, generate a restore key, add it on the backup server with the new address in from=, then restore your folders one by one.
Update
rsync is updated with the system, on both servers:
sudo apt update
sudo apt upgradeAfter an update, check the log of the next backup. If you replace the backup server, its host key changes: update /root/.ssh/known_hosts on the source server after verifying the new fingerprint.
Troubleshooting
Permission denied (publickey)
On the backup server, check the public key, the address in from= and the permissions. The SSH log often shows the cause:
sudo ls -ld /srv/rsyncbackup /srv/rsyncbackup/.ssh
sudo ls -l /srv/rsyncbackup/.ssh/authorized_keys
sudo journalctl -u ssh -n 30 --no-pagerThe home folder must not be writable by the group or other users.
Host key verification failed.
The backup server's key is missing from /root/.ssh/known_hosts or has changed. Redo the verification step, without disabling the check.
python3: No such file or directory
rrsync cannot start. Install python3 on the backup server.
do not use .. in --link-dest
rrsync rejects paths containing ... The --link-dest path must start with /, as in the script: it is then relative to the allowed folder.
reading from write-only server is not allowed
You are trying to read with the backup key. Use the restore_rsync restore key.
Another instance of rrsync is already accessing this directory.
A backup is already running, or an old connection has not ended. Check with pgrep -a rsync on both servers.
Code 23: some files/attrs were not transferred
Some files could not be copied. The log lists them, just before this message. The script treats this snapshot as failed: fix the cause and run it again.
Operation not supported (95) on attributes
The backup server's file system does not support the extended attributes used by --fake-super. Put the backups on an ext4 or XFS volume.
Code 255 or 12
The SSH connection failed or was cut. Run the connection test again and check the network, the firewall and the disk space with our guide Check disk space.
All files are copied again on every run
Check /var/lib/backup-rsync/last: it must contain the name of an existing snapshot. Also check that this snapshot contains the expected files and that the destination file system supports hard links.