Edit this page | Blame

Update production checklist

Last migration round is the move to tux03 (Sept2025)!

Tasks

[X] Install underlying Debian
[X] Get guix going
[X] Check database settings
[X] Check gemma working
[X] Check global search
[X] Check authentication
[X] Check sending E-mails
[X] Check SPARQL
[X] Make sure info.genenetwork.org and 'space' can reach the DB
[ ] Backups

The following are at the system level

[X] Firewalling and other security measures (sshd)
[X] Check tmpdirs (cleanup?)
[X] Make sure journalctl persistent (check for reboots)
[X] Update certificates in CRON (no longer if not part of Guix)
[X] Run trim in CRON
[ ] Monitors (sheepdog)

Install underlying Debian

For our production systems we use Debian as a base install. Once installed:

[X] set up git in /etc and limit permissions to root user
[X] add ttyS0 support for grub and kernel - so out-of-band works
[X] start ssh server and configure not to use with passwords
[X] start nginx and check external networking
[X] mount old root
[X] Clean up /etc/profile (remove global profile.d loading)
[X] set up E-mail routing

It may help to mount the old root if you have it. Now it is on

mount /dev/sdd2 /mnt/old-root/

Get Guix going

[X] Mount bind /gnu on a large partition
[X] Move /gnu/store to larger partition
[X] Install Guix daemon
[X] Update Guix daemon and setup in systemd (if necessary)
[X] Make available in /usr/local/guix-profiles

Next move the /gnu store to a large partion and hard mount it in /etc/fstab with

/export2/gnu /gnu none defaults,bind 0 0

We can bootstrap with the Debian guix package (though I prefer the guix-install.sh script these days, mostly because it is more modern).

https://guix.gnu.org/manual/en/html_node/Binary-Installation.html

Run guix pull

guix pull --url=https://codeberg.org/guix/guix  -p ~/opt/guix-pull

Use that also to install guix in /usr/local/guix-profiles

guix package -i guix -p /usr/local/guix-profiles/guix

and update the daemon in systemd accordingly. After that I tend to remove /usr/bin/guix

The Debian installer configures guix. I tend to remove the profiles from /etc/profile so people have a minimal profile.

Check database

[X] Install mariadb
[X] Recover database
[X] Test permissions
[X] Mariadb update my.cnf

Basically recover the database from a backup is the best start and set permissions. We usually take the default mariadb unless production is already on a newer version - so we move to guix deployment.

On tux02 mariadb-10.5.8 is running. On Debian it is now 10.11.11-0+deb12u1, so we should be good. On Guix is 10.10 at this point.

apt-get install mariadb-server

Next unpack the database files and set permissions to the mysql user. And (don't forget) update the /etc/mysql config files.

Restart mysql until you see:

mysql -u webqtlout -p -e "show databases"
+---------------------------+
| Database                  |
+---------------------------+
| 20081110_uthsc_dbdownload |
| db_GeneOntology           |
| db_webqtl                 |
| db_webqtl_s               |
| go                        |
| information_schema        |
| kegg                      |
| mysql                     |
| performance_schema        |
| sys                       |
+---------------------------+

topics/systems/mariadb/mariadb.gmi

Recover database

We use borg for backups. First restore the backup on the PCIe. Also a test for overheating!

Check sending E-mails

The swaks package is quite useful to test for a valid receive host:

swaks --to testing-my-server@gmail.com --server smtp.network
=== Trying smtp.network:25...
=== Connected to smtp.network.
<-  220 mailrouter8.network ESMTP NO UCE
 -> EHLO tux04.network
<-  250-mailrouter8.network
<-  250-PIPELINING
<-  250-SIZE 26214400
<-  250-VRFY
<-  250-ETRN
<-  250-STARTTLS
<-  250-ENHANCEDSTATUSCODES
<-  250-8BITMIME
<-  250-DSN
<-  250 SMTPUTF8
 -> MAIL FROM:<root@tux04.network>
<-  250 2.1.0 Ok
 -> RCPT TO:<pjotr2020@thebird.nl>
<-  250 2.1.5 Ok
 -> DATA
<-  354 End data with <CR><LF>.<CR><LF>
 -> Date: Thu, 06 Mar 2025 08:34:24 +0000
 -> To: pjotr2020@thebird.nl
 -> From: root@tux04.network
 -> Subject: test Thu, 06 Mar 2025 08:34:24 +0000
 -> Message-Id: <20250306083424.624509@tux04.network>
 -> X-Mailer: swaks v20201014.0 jetmore.org/john/code/swaks/
 ->
 -> This is a test mailing
 ->
 ->
 -> .
<-  250 2.0.0 Ok: queued as 4157929DD
 -> QUIT
<-  221 2.0.0 Bye                                                                                                                             === Connection closed with remote host

An exim configuration can be

dc_eximconfig_configtype='smarthost'
dc_other_hostnames='genenetwork.org'
dc_local_interfaces='127.0.0.1 ; ::1'
dc_readhost=''
dc_relay_domains=''
dc_minimaldns='false'
dc_relay_nets=''
dc_smarthost='smtp.network'
CFILEMODE='644'
dc_use_split_config='false'
dc_hide_mailname='false'
dc_mailname_in_oh='true'
dc_localdelivery='maildir_home'

And this should work:

swaks --to myemailaddress --from john@network --server localhost

Backups

[ ] Create an ibackup user.
[ ] Install borg (usually guix version)
[ ] Create a borg passphrase