Centralized Rsync Backups
Making a post based on old notes on setting up backups, in the hopes of ensuring current validity, updating as needed and perhaps even monitoring. The inspiration for and original source of much of the information listed here can be found at MadHacking Backup System.
Install and Configure Backup Software On NAS
- Update firmware on NAS to latest version and make sure all is working well
- Install rsync and rsnapshot (Version 1.3.1-4+deb8u1 as of Nov 2019)
sudo apt-get install rsnapshot
- Configuration
sudo mkdir /etc/rsnapshot
sudo cp /etc/rsnapshot.conf.default /etc/rsnapshot/rsnapshot.base
- Edit rsnapshot.base
sudo vim /etc/rsnapshot/rsnapshot.base
- This is how it looks as of December 2016
config_version 1.2
ssh_args -p 22 -i /home/rsync-backup/.ssh/id_rsa
no_create_root 0
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_du /usr/bin/du
cmd_rsnapshot_diff /usr/bin/rsnapshot-diff
rsync_long_args --delete --numeric-ids --relative --delete-excluded --acls --xattrs --sparse --delete-after --hard-links --rsync-path=/usr/bin/rsync
# Verbose level, 1 through 5.
# 1 Quiet Print fatal errors only
# 2 Default Print errors and warnings only
# 3 Verbose Show equivalent shell commands being executed
# 4 Extra Verbose Show extra verbose information
# 5 Debug mode Everything
#
verbose 2
# Same as "verbose" above, but controls the amount of data sent to the
# logfile, if one is being used. The default is 3.
#
loglevel 3
# If you enable this, data will be written to the file you specify. The
# amount of data written is controlled by the "loglevel" parameter.
#
logfile /var/log/rsnapshot.log
# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile /var/run/rsnapshot.pid
Install and Configure Backup Software On Client
- Client needs rsync and sshd installed and working
- Gentoo boxes are ready to go (Ubuntu, Raspbian and MacOS probably are too but should be validated)
- Windows, well that is a story for another day…
# emerge -pv1 openssh rsync
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild N ] net-misc/openssh-5.8_p1-r1 USE="hpn ldap pam tcpd -X -X509 -kerberos -libedit -pkcs11 -skey -static"
[ebuild N ] net-misc/rsync-3.0.8 USE="acl iconv ipv6 -static xattr"
Backing up NAS
I am running the backups centrally from my NAS appliance. As a result it is fairly important that the NAS backup and other configuration files are backed up. The backup of the NAS is a little different from backups of other targets in that it will essentially be doing a local backup instead of the remote backup process that will be used everywhere else. The first step is to create a common configuration file for all the backup rotations for the localhost (NAS) backup target. This file will simply include the base rsnapshot configuration file and set the snapshot_root for this backup target. This file will then be sourced in all configurations for this target.
- Note: where ever
<HOSTNAME>or<DOMAINNAME>are used below, replace with the actual host name and domain name of the device the backup config file is related. For example if target host name were space and domain name were example.com<HOSTNAME>.<DOMAINNAME>would bespace.example.com.
sudo vi /etc/rsnapshot/<HOSTNAME>.common
# Include the base configuration file.
include_conf /etc/rsnapshot/rsnapshot.base
# Set the snapshot_root for this host
snapshot_root /data/Backup/<HOSTNAME>
Create a second file in which we shall configure an additional set of rotations and backup points. As you can see we are making a backup of the /etc/, /frontview/, /home/ directories. We are storing 7 daily, 8 weekly, and 12 monthly backups. A call is made to a script which will be run after the backups are performed which will merge the 7th daily backup with the weekly backup performed by this configuration file (merge-snapshots).
sudo vi /etc/rsnapshot/<HOSTNAME>.1
# Include the common configuration for this host
include_conf /etc/rsnapshot/localhost.common
# Configure the retention period for daily backups
retain daily 7
retain weekly 8
retain monthly 12
# Configure the backup source locations
backup /etc/ ./
backup /frontview/ ./
backup /home/ ./
backup /usr/local/sbin ./
# Merge the daily backup from one week ago with the weekly backup from the same time
backup_script /usr/local/sbin/merge-snapshots localhost daily.7 weekly.1 .merge/
By default, rsnapshot package does not provide a script to merge snapshots in the manner needed so one must be created. This is a simple task consisting of checking for the existence of the source and destination directories for the snapshots in question and, if they both exist, copying the contents of one to the other using hard linked files in the same way as the rsnapshot application.
sudo vi /usr/local/sbin/merge-snapshots
#! /bin/bash
# If the source and destination directories exist then perform the merge
[ -d /data/Backup/$1/$2 ] && [ -d /data/Backup/$1/$3/ ] && \
/bin/cp -al /data/Backup/$1/$2/* /data/Backup/$1/$3/
exit 0
- Make it executable
chmod +x /usr/local/sbin/merge-snapshots
The final step is to create another script which will be used to execute rsnapshot with the correct parameters to perform the backups we have defined in the correct sequence. The example script below can be run daily and will execute all rsnapshot configurations which are required for that particular day performing a daily backup every day, a weekly backup on every Sunday and a monthly backup on the first Sunday of every month.
sudo vi /usr/local/sbin/do-snapshots
#! /bin/bash
# Set default configuration values
SNAPSHOT_WEEKLY_DAY=6
# Get the current day of the week and day of the month
DAY_OF_THE_WEEK=$(date +%u)
DAY_OF_THE_MONTH=$(date +%d)
DO_MONTHLY=false
DO_WEEKLY=false
if [ $DAY_OF_THE_WEEK -eq $SNAPSHOT_WEEKLY_DAY ]; then
DO_WEEKLY=true
fi
if $DO_WEEKLY && [ $DAY_OF_THE_MONTH -le 7 ]; then
DO_MONTHLY=true
fi
if $DO_MONTHLY ; then
echo "Performing monthly snapshots:"
awk '/^retain.+monthly/ { print FILENAME }' /etc/rsnapshot/* | \
xargs -r -n 1 -I{} bash -c "echo ' {}' && rsnapshot -c {} monthly"
fi
if $DO_WEEKLY ; then
echo "Performing weekly snapshots:"
awk '/^retain.+weekly/ { print FILENAME }' /etc/rsnapshot/* | \
xargs -r -n 1 -I{} bash -c "echo ' {}' && rsnapshot -c {} weekly"
fi
echo "Performing daily snapshots:"
awk '/^retain.+daily/ { print FILENAME }' /etc/rsnapshot/* | \
xargs -r -n 1 -I{} bash -c "echo ' {}' && rsnapshot -c {} daily"
- Make it executable too
chmod +x /usr/local/sbin/do-snapshots
- Test by executing do-snapshots script. If all is well, there should now be a snapshot of the directories specified in the daily backup plan locate in /data/Backups/
/daily.0. If executed on a Sunday, there should also be a weekly snapshot created in /data/Backup/ /weekly.0.
do-snapshots
- If do-snapshots appears to be working as desired, we can automate the process so it does not get forgotten, using cron.
crontab -e
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
0 3 * * * /usr/local/sbin/do-snapshots
Creating and Collecting Snapshots from Remote Targets
- I will be using rsync via ssh, and will be connecting via ssh without a password to facilitate automation. To do this we will be creating an ssh key pair for backup use on the NAS appliance. The user rsync-backup already ll be used for remote backups exists on my NAS and I will use its public key. There are plenty of examples on how to create public an private ssh keys without a password available on the Internet, so I will not detail that process in this post.
su rsync-backup
sudo ssh-keygen -t rsa -b 4096 -N ""
- I use rsync-backup’s id_rsa.pub key instead for the remainder of this document.
- On each remote target I intend to backup, I create an rsync-backup user, with nothing but a home directory and a standard UID.
useradd --create-home --no-user-group --gid nogroup --uid 900 rsync-backup
-
The next step is to copy the NAS rsync-backup ssh public key to the client we are going to backup. Also need to change ownership of authorized_keys file to match that of the target backup account UID and GID.
-
From the NAS:
sudo scp /home/rsync-backup/.ssh/id_rsa.pub rsync-backup@<HOSTNAME>.<DOMAINNAME>:/home/rsync-backup/.ssh/authorized_keys
sudo ssh rsync-backup@<HOSTNAME>.<DOMAINNAME> chown rsync-backup.nogroup /home/rsync-backup/.ssh/authorized_keys
sudo ssh rsync-backup@<HOSTNAME>.<DOMAINNAME> chmod 700 /home/rsync-backup/.ssh/authorized_keys
- Test from NAS that ssh to backup client works as expected connecting without requiring entry of a password.
su - rsync-backup -c 'ssh <HOSTNAME>.<DOMAINNAME>'
- Once working, the next step is to control where password less connections are allowed from and what they allow. This work is done on the client.
sudo vi /home/rsync-backup/.ssh/authorized_keys - Add from= and command= directives before key, where
is replaced with actual TCP/IP address of the device running the rsnapshot command.
from="<IPADDRESS>",command="/home/rsync-backup/validate-backup-cmd.sh" ssh-rsa AAAAB3NzaC1y...
- Create validate-backup-cmd.sh on client in home directory of rsync-backup.
vi ~/validate-backup-cmd.sh
#! /bin/bash
# $SSH_ORIGINAL_COMMAND
case "$SSH_ORIGINAL_COMMAND" in
*\&*|*\|*|*\;*|*\>*|*\<*|*\!*)
exit 1
;;
/usr/bin/rsync\ --server\ --sender*)
logger "Rsync central backup to NAS started."
sudo $SSH_ORIGINAL_COMMAND
logger "Rsync central backup to NAS ended."
;;
*)
exit 1
;;
esac
- Empower rsync-backup to use rsync as if root on client.
visudo
# User privilege specification
root ALL=(ALL) ALL
rsync-backup ALL=(root) NOPASSWD: /usr/bin/rsync
- At this point NAS should be able to connect to client and execute rsync as root. All that remains is to configure rsnapshot on the NAS to set what to back up from client and how long to retain those backups.
sudo vi /etc/rsnapshot/rsnapshot.base
config_version 1.2
ssh_args -p 22 -i /home/rsync-backup/.ssh/id_rsa
no_create_root 0
cmd_cp /bin/cp
cmd_rm /bin/rm
cmd_rsync /usr/bin/rsync
cmd_ssh /usr/bin/ssh
cmd_du /usr/bin/du
cmd_rsnapshot_diff /usr/bin/rsnapshot-diff
rsync_long_args --delete --numeric-ids --relative --delete-excluded --acls --xattrs --sparse --delete-after --hard-links --rsync-path=/usr/bin/rsync
# Verbose level, 1 through 5.
# 1 Quiet Print fatal errors only
# 2 Default Print errors and warnings only
# 3 Verbose Show equivalent shell commands being executed
# 4 Extra Verbose Show extra verbose information
# 5 Debug mode Everything
#
verbose 2
# Same as "verbose" above, but controls the amount of data sent to the
# logfile, if one is being used. The default is 3.
#
loglevel 3
# If you enable this, data will be written to the file you specify. The
# amount of data written is controlled by the "loglevel" parameter.
#
logfile /var/log/rsnapshot.log
# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile /var/run/rsnapshot.pid
sudo vi /etc/rsnapshot/client.common
# Include the base configuration file.
include_conf /etc/rsnapshot/rsnapshot.base
# Set the snapshot_root for this host
snapshot_root /data/Backup/<HOSTNAME>
sudo vi /etc/rsnapshot/<HOSTNAME>.1
# Include the common configuration for this host
include_conf /etc/rsnapshot/<HOSTNAME>.common
# Configure the retention period for daily backups
retain daily 7
retain weekly 8
retain monthly 12
# Configure the backup source locations
backup rsync-backup@<HOSTNAME>.<DOMAINNAME>:/ ./
# Merge the daily backup from one week ago with the weekly backup from the same time
backup_script /usr/local/sbin/merge-snapshots <HOSTNAME> daily.7 weekly.1 .merge/
# Excludes
exclude /usr/portage/distfiles
exclude /dev
exclude /proc
exclude /sys
exclude /var/cache
exclude /var/tmp
exclude /var/log
exclude /mnt/music
Troubleshooting Tips
- Review /var/log/rsnapshot.log on NAS
- Error code 255 generally points to an SSH problem
- Test whether root can connect to remote host presenting rsync-backup SSH RSA id.
- Sometimes old entries need to be removed .ssh/known_hosts
- Run
/usr/loca/sbin/do-snapshotson NAS as root, accepting host keys for new remotes, before first automatic backup.
- Make sure
rsyncis installed on remote hosts. - Make sure rsync-backup can use rsync as root in *sudoeors” file.
- Make sure path to rsync is correct in sudoers file, and the validate-rsync-cmd.sh script file.
- Make sure /etc/ssh/sshe_config file has the following entries:
- Error code 255 generally points to an SSH problem
ClientAliveInterval 30
ClientAliveCountMax 6
Chrls is currently a technology transformation leader at one of the oldest American investment banking services holding companies, headquartered in New York City. Previous roles at current employer include project manager, front office support, retail and enterprise global Windows server support. Currently residing in Tennessee on the Cumberland Plateau and enjoying life, family and open source.
Mastodon Diaspora Funkwhale Pixelfed