To make backups we use a combination of sheepdog, borg, sshfs, rsync. sheepdog is the monitor and status is tracked using a redis queue. borg does incremental backups as a local user. Next we drop the backup to a remote machine using a special user on the remote with very limited access - only one directory can be mounted with sshfs and there is no ssh login. Finally the data gets rsync'd across.
This system proves pretty resilient over time. Only on the synology server I can't get it to work because of some CRON permission issue.
It is advised to use a backup password and not store that on the remote.
Disable the password, just to be sure. Next add the following to /etc/ssh/sshd.conf:
-# override default of no subsystems -Subsystem sftp /usr/lib/openssh/sftp-server +Subsystem sftp internal-sftp -f AUTH -l VERBOSE + +Match User bacchus + # ChrootDirectory /export/backup/%u + ChrootDirectory /export/backup/%u + X11Forwarding no + AllowTcpForwarding no + ForceCommand internal-sftp + PasswordAuthentication no + PubkeyAuthentication yes
systemctl reload ssh
You may need to add to allowusers for ssh access. If you use allowusers (recommended) you can even specify the sending host with
where remote can be an IP address.
Warning: if you introduce this `AllowUsers` command all users should be listed or people may get locked out of the machine.
Next create a special key on the backup machine's ibackup user (just hit enter):
su ibackup ssh-keygen -t ecdsa -f $HOME/.ssh/id_ecdsa_backup
and copy the public key into the remote /home/bacchus/.ssh/authorized_keys
Now test it from the backup server with
ssh -v bacchus@dropserver
it should give a Permission denied (publickey).
On the drop server you can track messages by
tail -40 /var/log/auth.log
ssh -v -i ~/.ssh/id_ecdsa_backup bacchus@dropserver
should give a Broken pipe(!). In auth.log you may see something like
fatal: bad ownership or modes for chroot directory component "/export/backup/"
This is a tricky bit. This directory should be owned by root and have permissions. The inside user directory has different permissions:
ls -ld /export/backup/ drwxr-xr-x 3 root root 4096 Oct 21 02:08 /export/backup/ drwxr-xr-x 3 root root 4096 Oct 21 02:07 /export/backup/bacchus drwx------ 3 bacchus bacchus 4096 Oct 21 02:26 /export/backup/bacchus/drop
So, as root
cd /export mkdir -p backup/bacchus/drop chown bacchus.bacchus backup/bacchus/drop/ chmod 0700 backup/bacchus/drop/
If auth.log says error: /dev/pts/11: No such file or directory on ssh, or received disconnect (...) disconnected by user we are good to go!
Next try sshfs
su ibackup mkdir -p ~/mnt/dropserver sshfs -o IdentityFile=~/.ssh/id_ecdsa_backup bacchus@dropserver:/ ~/mnt/dropserver/ df -h|grep mnt touch ~/mnt/dropserver/drop/HELLO
And the remote directory should be ready for dropping files!
To unmount the dir
fusermount -u ~/mnt/dropserver
IMPORTANT: it is important to try ssh and read /var/log/auth.log to deal with permission issues. sshfs and the underlying sftp protocol are fussy.
A backup script with sheepdog may look like
sheepdog_run.rb -h rabbit --always -v --tag "drop-mount-dropserver" -c "sshfs -o IdentityFile=~/.ssh/id_ecdsa_backup bacchus@dropserver:/ ~/mnt/this" sheepdog_run.rb -h rabbit --always -v --tag "drop-rsync-dropserver" -c "rsync -vrltDP this/* borg/* ~/mnt/this/drop/this/ --delete" sheepdog_run.rb -h rabbit --always -v --tag "drop-unmount-dropserver" -c "fusermount -u ~/mnt/this"
It may be useful to add the following options to sshfs:
sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,IdentityFile=~/.ssh/id_ecdsa_backup ...
The recent scripts can be found at