We use borg backups. Load a recent borg, for example from my profile
source /home/wrk/opt/borg/etc/profile
The backup passphrase needs to be set.
source /home/ibackup/.borg-pass
Now run against a repo on 'tux02'
borg list /export3/backup/tux01/borg/borg-tux01/ borg-backup-mariadb-20221004-03:13-Tue Mon, 2022-10-03 22:13:11
The last backup on 'tux02' is from October 2022 - after I did a reinstall. That is not great.
According to sheepdog the drops are happening to 'space' and 'epysode', but 'tux02' is missing:
The first step is to find where the current mariadb is running. This will be a symlink in /var/lib/mysql -> some dir.
mysql -> /export2/mysql/borg-backup-mariadb-20211024-03:09-Sun
We typically run the database on an nvme partition. Check if there is enough space(!). It may be you need to remove the old database after making a backup (if it is not in borg).
Next, find the backup. Typically it is symlinked from /export/backup. On
tux02:/export5/backup/bacchus/drop/tux01
We need a place to unpack the DB. On tux02 export5 is large, so let's try that and pick the last backup
cd /export5/mysql tux02:/export5/mysql# borg list /export5/backup/bacchus/drop/tux01/borg-tux01 borg extract --progress /export5/backup/bacchus/drop/tux01/borg-tux01::borg-backup-mariadb-20230207-02:31-Tue
As it happened, this was an incomplete backup! Check original size with
borg info --last 3 /export5/backup/bacchus/drop/tux01/borg-tux01
so I went with
borg extract --progress /export5/backup/bacchus/drop/tux01/borg-tux01::borg-backup-mariadb-20230205-03:38-Sun
Mariadb's own ibdata1 is still at 100Gb. Too large, and we'll muck it out at some point. The total mariadb database at this point is 430Gb.
To move the new DB in place we first have to stop mariadb, move the old database out of the way, move the new one in, and restart the DB after making sure permissions and symlink are right.
Just to be sure we copy the old DB somewhere.
systemctl stop mariadb cd /export2/mysql cp -vau /export2/mysql/borg-backup-mariadb-20211024-03:09-Sun . root@tux02:/export2/mysql# cp -vau /export5/mysql/home/backup/tux01_mariadb_new/latest/* borg-backup-mariadb-20230205-03:38-Sun cd /var/lib rm mysql ln -s /export2/mysql/borg-backup-mariadb-20230205-03\:38-Sun mysql systemctl start mariadb systemctl status mariadb
Oops. Can't create test file /var/lib/mysql. We need to set permissions.
root@tux02:/var/lib# chown mysql.mysql mysql
And we are running.
Next stop update genotype files.
GN has a file repo for genotype files, lmdb and such, next to MariaDB. Typically
root@tux02:/home/gn2/production/genotype_files# du -sh . 38G .
the backup in drop lists, for example
root@tux02:/export5/backup/bacchus/drop/tux01# borg list borg-genenetwork/ borg-GN2-home-20230207-04:00-Tue Mon, 2023-02-06 22:00:02 borg-GN1-home-20230207-04:40-Tue Mon, 2023-02-06 22:40:01 borg-GN3-home-20230207-04:41-Tue Mon, 2023-02-06 22:41:27 borg-GN2s-home-20230207-04:41-Tue Mon, 2023-02-06 22:41:45 borg-ZACH-home-20230207-04:42-Tue Mon, 2023-02-06 22:42:01 borg-datafiles-home-20230207-04:42-Tue Mon, 2023-02-06 22:42:32
At this poing ZACH-home is the main dir, so this fetches all files
borg extract --verbose borg-genenetwork::borg-ZACH-home-20230207-04:42-Tue home/zas1024/gn2-zach/genotype_files
and we copy it to /home/gn2/production.
We also have a separate data files 'warehouse', but that is not linked up with the webserver right now.