Despite our precautions it is advised to use a backup password and *not* store that on the remote.

• sheepdog monitor running on redis queue
• current web monitor for GN sheepdog on rabbit

adduser bacchus

Disable the password, just to be sure. Next add the following to /etc/ssh/sshd.conf:

-# override default of no subsystems
-Subsystem      sftp    /usr/lib/openssh/sftp-server
+Subsystem      sftp    internal-sftp -f AUTH -l VERBOSE
+
+Match User bacchus
+    # ChrootDirectory /export/backup/%u
+    ChrootDirectory /export/backup/%u
+    X11Forwarding no
+    AllowTcpForwarding no
+    ForceCommand internal-sftp
+    PasswordAuthentication no
+    PubkeyAuthentication yes

And run

systemctl reload ssh

You may need to add to allowusers for ssh access. If you use allowusers (recommended) you can even specify the sending host with

AllowUsers bacchus@remote

where remote can be an IP address.

Warning: if you introduce this `AllowUsers` command all users should be listed or people may get locked out of the machine.

Next create a special password-less key on the backup machine's ibackup user (just hit enter):

su ibackup
ssh-keygen -t ecdsa -f $HOME/.ssh/id_ecdsa_backup

and copy the public key into the remote /home/bacchus/.ssh/authorized_keys.

Now test it from the backup server with

ssh -v bacchus@dropserver

it should give a Permission denied (publickey).

On the drop server you can track messages by

tail -40 /var/log/auth.log

or on recent linux with systemd

journalctl -r

Next

ssh -v -i ~/.ssh/id_ecdsa_backup bacchus@dropserver

should give a Broken pipe(!) or -- more recently -- it says `This service allows sftp connections only`. When running sshd with a verbose switch you may see something like

fatal: bad ownership or modes for chroot directory component "/export/backup/"

This is a tricky bit. This directory should be owned by root and have permissions. The inside user directory has different permissions:

ls -ld /export/backup/
drwxr-xr-x 3 root root 4096 Oct 21 02:08 /export/backup/
drwxr-xr-x 3 root root 4096 Oct 21 02:07 /export/backup/bacchus
drwx------ 3 bacchus bacchus 4096 Oct 21 02:26 /export/backup/bacchus/drop

So, as root

cd /export
mkdir -p backup/bacchus/drop
chown bacchus:bacchus backup/bacchus/drop/
chmod 0700 backup/bacchus/drop/

Another error may be:

fusermount3: mount failed: Operation not permitted

This means you need to set the suid on the fusermount3 command. Bit nasty in Guix.

apt-get install fuse(3) sshfs
chmod 4755 /usr/bin/fusermount

If auth.log says error: /dev/pts/11: No such file or directory on ssh, or received disconnect (...) disconnected by user we are good to go!

Note: at this stage it may pay to track the system log with

journalctl -u ssh -f

and, in a pinch, increase DEBUG output in sshd_config with

LogLevel DEBUG3

In my case I had to disable PAM!

Next try sshfs

su ibackup
mkdir -p ~/mnt/dropserver
sshfs -o IdentityFile=~/.ssh/id_ecdsa_backup bacchus@dropserver:/ ~/mnt/dropserver/
df -h|grep mnt
touch ~/mnt/dropserver/drop/HELLO

And the remote directory should be ready for dropping files!

rsync  -vrltDP  /export2/backup/borg/borg-tux01 ~/mnt/balg01/drop/ --delete

To unmount the dir

fusermount -u ~/mnt/dropserver

IMPORTANT: it is important to try ssh and read /var/log/auth.log to deal with permission issues. sshfs and the underlying sftp protocol are fussy.

A backup script with sheepdog may look like

sheepdog_run.rb -h rabbit --always -v --tag "drop-mount-dropserver" -c "sshfs -o IdentityFile=~/.ssh/id_ecdsa_backup bacchus@dropserver:/ ~/mnt/this"

sheepdog_run.rb -h rabbit --always -v --tag "drop-rsync-dropserver" -c "rsync -vrltDP this/* borg/* ~/mnt/this/drop/this/ --delete"

sheepdog_run.rb -h rabbit --always -v --tag "drop-unmount-dropserver" -c "fusermount -u ~/mnt/this"

It may be useful to add the following options to sshfs:

sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,IdentityFile=~/.ssh/id_ecdsa_backup ...

The recent scripts can be found at

• https://github.com/genenetwork/gn-deploy-servers/blob/master/scripts/tux01/backup_drop.sh