- Database Migrations
issue opened on Oct 19 2022 by Frederick Muriuki Muriithi
Database Migrations
## Tags
* assigned:
* type: feature
* priority: high
* keywords: database migrations
* status: pending
## Description
There might need to be some form of database migration...
- Tux02 Production
issue opened on Oct 07 2021 by Pjotr Prins, last updated on Dec 14 2022 by Pjotr Prins; 17 of 22 tasks done
...borg backups
* [ ] create check list for manual testing
* [ ] look at performance
## Info
We have a protocol for updating GN2 on Tux02.
### Restore database from backup
Databases no longer get copied.
- Errors, defects and missing data in the database
issue opened on Oct 27 2022 by Arun Isaac, last updated on Oct 30 2022 by Arun Isaac
...database, which we try to track in this issue. These are best fixed directly in the database rather than by working around them in code.
## LRS values listed as 0.000
Some LRS values in the database...
- MGAMMA Convert
issue opened on Apr 17 2024 by Pjotr Prins, last updated on May 18 2024 by Artyom Bologov; 7 of 11 tasks done
...that information
```
{type: "GRM", version:0.01, float: true, symmetric: true}
```
* [ ] Support genodb database format:
=> ../../topics/database/genotype-database.gmi See the genotype-database topic
- Database: `ProbeSetSE` Schema Bug
issue opened on Mar 22 2024 by Frederick Muriuki Muriithi, last updated on Mar 22 2024 by Frederick Muriuki Muriithi
Database: `ProbeSetSE` Schema Bug
## Tags
* type: bug
* priority: critical
* status: open
* keywords: database, mariadb, schema
* assigned:
## Description
The schemas are defined as follows...
- Use genodb in genenetwork
issue opened on Jul 19 2022 by Arun Isaac, last updated on Mar 31 2023 by Arun Isaac
This will serve as an example based on which the team can port the rest of genenetwork to genodb.
=> https://issues.genenetwork.org/topics/genotype-database Design and use of the genodb database
- Tux04/Tux05 disk issues
issue opened on May 03 2024 by Pjotr Prins, last updated 7 days ago by Pjotr Prins; 10 of 10 tasks done
...-AdpAllInfo -aAll
```
# Database
During a backup the DB shows this error:
```
2025-03-02 06:28:33 Database page corruption detected at page 1079428, retrying...\n[01] 2025-03-02 06:29:33 Database page...
- Databases Getting Out of Wack
issue opened on Mar 03 2022 by jgart, last updated on Mar 16 2023 by Pjotr Prins
...database, gemma, reaper
## Let's use Gemma instead of Reaper
Zachary:
> If we're using GEMMA, we'll need to recalculate all other trait Max LRS scores using
> GEMMA as well (so I think we should just...
- GN1 Time machines
issue opened on Oct 07 2021 by Pjotr Prins, last updated on Jul 01 2022 by Arun Isaac
...databases, source code and etc files to set up the containers. Start with the most recent one and see if you can get that to run on Penguin2. After that we'll do the others. The database are named...
- ProbeSetData
issue opened on Dec 31 2021 by Pjotr Prins, last updated on Mar 13 2023 by Pjotr Prins; 15 of 17 tasks done
...copy the database to a new partition:
```
root@tux01:/export4/local/home/mariadb/database/db_webqtl# rsync -vaP /var/lib/mysql/db_webqtl/* . --delete --bwlimit=20M
```
Note I throttle the speed...
- Clean Up
issue opened on Jan 02 2022 by Pjotr Prins, last updated on Jul 01 2022 by Arun Isaac
...database administration
* keywords: database, mariadb
## Description
Find all larger tables
```
SELECT TABLE_SCHEMA,TABLE_NAME,DATA_LENGTH FROM information_schema.TABLES WHERE DATA_LENGTH>10000...
- Support searching using SNP names
issue opened on Feb 15 2023 by Arun Isaac
...SNP name in an external database and resolve it to coordinates before searching. This is a needless extra step and can be automated.
Implementing this will require us to have a database (perhaps dbSNP)...
- Replace Redis with SQL
issue opened 14 hours ago by Frederick Muriuki Muriithi
...SQL databases (we'll probably use SQLite3 anyway), which are more standardised, and easier to migrate data from and to. It has the added advantage that we can open multiple connections to the database...
- Add mouse data-set
issue opened on Jun 30 2022 by BonfaceKilz, last updated on Apr 18 2023 by Munyoki Kilyungi; 0 of 4 tasks done
.../dump-genenetwork-database/blob/master/csv-dump.scm
Remaining tasks:
* [ ] Share latest changes.
* [ ] Test the script in a copy of the production database.
* [ ] Make this more generic
* [ ] Integrate...
- Hanging database
issue opened on Dec 21 2021 by Pjotr Prins, last updated on Mar 11 2023 by Pjotr Prins
...'hanging'
In the last 12 hours GN2 monitoring shows the website is responding intermittendly. A quick check shows the database is blocking. Rather than simply restarting the database - which is known...
- Some correlations running very slowly
issue opened on Mar 06 2023 by Zachary Sloan, last updated on Mar 07 2023 by Pjotr Prins
...database
## Description
Some correlations (it specifically seems to be ones done against ProbeSet databases) are running extremely slowly.
After looking into this, the cause seemed to be a specific...
- GN2 Time Machines
issue opened on Aug 19 2022 by Pjotr Prins, last updated on Sep 05 2022 by Alexander_Kabui; 4 of 10 tasks done
...'show databases'
```
### Mariadb database from backup
We have daily incremental backups on P2, Tux02 and Epysode. First restore the files with
```
. ~/.borg-pass
cd /export2/tux01-restore
borg extract...
- ProbeSE
issue opened on Dec 30 2021 by Pjotr Prins, last updated on Jul 01 2022 by Arun Isaac
...database, mariadb, innodb, ProbeSE
## Description
Zach pointed out that ProbeSE is used on GN1 with
=> http://gn1.genenetwork.org/webqtl/main.py?FormID=showProbeInfo&database=HC_M2_0606_P&ProbeSetID...
- LMM precomputed scores
issue opened on Mar 16 2023 by Pjotr Prins, last updated on May 07 2024 by Pjotr Prins; 0 of 1 tasks done
Interestingly, this ties in with our xapian search and fast querying of value ranges.
# Tags
* assigned: pjotrp
* priority: high
* type: bug, enhancement
* status: ongoing
* keywords: database, gemma...
- Fetch trait names for phenotypes
issue opened on Mar 22 2022 by Frederick Muriuki Muriithi
...database.
### Describe the solution you'd like
For example, downloading the data as CSV via the web interface includes the following header information:
```
Record ID,10620
Symbol,WMZTrgtQuadTime...
- Add Cis-Trans plot
issue opened on Apr 07 2023 by Pjotr Prins
See also
=> ../../topics/systems/mariadb/precompute-mapping-input-data.gmi
# Tags
* assigned: zsloan, pjotrp
* priority: medium
* type: enhancement
* status: unclear
* keywords: database, gemma
- Capture state of phenotypes in a HASH
issue opened on Mar 23 2022 by Frederick Muriuki Muriithi
.../551 From GitHub
To represent the state of the database we need to start using HASH values or UUIDs. For phenotypes we should create these for phenotype columns within a dataset - i.e. the column one...
- Simplify `dataset.py` in GeneNetwork2
issue opened on Sep 13 2022 by Frederick Muriuki Muriithi
.../wqflask/base/data_set.py#L740-L832
and split it into various chunks, that
* compute the `self.sample_list`
* retrieve `sample_ids` values from the database using the `self.sample_list` values computed...
- Production on tux04
issue opened 3 weeks ago by Pjotr Prins, last updated 13 days ago by Pjotr Prins; 10 of 12 tasks done
...databases (ugh).
Fred copied all data to an enterprise level storage, and we rolled back to some older DBs, so hopefully we'll be OK for now.
# Reinstating backups
In the next step we need to restore...
- Cannot Connect to MariaDB
issue opened on Nov 05 2024 by Frederick Muriuki Muriithi, last updated on Nov 08 2024 by Frederick Muriuki Muriithi
...scm?id=46a1c4c8d01198799e6ac3b99998dca40d2c7094#n47 [^1] Lines 47 to 49 of production.scm
=> https://guix.gnu.org/manual/en/html_node/Database-Services.html#index-mysql_002dconfiguration [^2] Guix's...
- Cool Interfaces We Should Emulate
issue opened on Apr 13 2022 by BonfaceKilz, last updated on Apr 14 2022 by BonfaceKilz
...help.
#### Thu 14 Apr 2022 16:08:54 EAT
[Zach] Was this [shared above] figure generated with some tool, or manually?
[Dave] All plots/visualizations/tools in the Mouse Phenome Database are dynamic...
- ProbeData
issue opened on Dec 30 2021 by Pjotr Prins, last updated on Mar 14 2023 by Arun Isaac
...database, mariadb, innodb, ProbeData
## Description
Probe level data is used to examine the correlation structure among the
N probes that have the same nominal target. Sometimes several probes
are...
- MariaDB: Move to InnoDB Engine
issue opened on Dec 28 2021 by Pjotr Prins, last updated on Apr 24 2024 by Pjotr Prins; 0 of 8 tasks done
...database, mariadb, innodb
## Report
With the SQL database we need to move from myisam to innodb format,
mostly to stop the problem of full table locks. Also I expect the
occasional crashes we see...
- Genewiki conversion
issue opened on Aug 26 2022 by Pjotr Prins, last updated on Sep 05 2022 by Alexander_Kabui; 0 of 3 tasks done
...to (1) migrate the existing genewiki data in the database to named markdown documents in that repository and (2) create a rendered page that is found through
=> https://genenetwork.org/doc/genes/BRCA2...
- Fallbacks and backups
issue opened on Aug 31 2021 by Pjotr Prins, last updated on Jun 28 2024 by Pjotr Prins; 3 of 25 tasks done
...too. Incremental copies work with rsync - so that is fast. To restore the full MariaDB database from a local borg repo takes a few minutes:
```
wrk@epysode:/export/restore_tux01$ time borg extract -v...
- Handling Tissue in Uploader
issue opened on Mar 28 2024 by Frederick Muriuki Muriithi, last updated on Mar 28 2024 by Frederick Muriuki Muriithi; 2 of 2 tasks done
...is via the `ProbeFreeze`
table that refers to the `InbredSet` table that then refers to the `Species`
table. Even with that, on the **Tux02** database, we have 48 tissues that are
not connected to any...
- slow text search query
issue opened on Mar 12 2023 by Pjotr Prins
- Export Uploaded Data to LMDB and RDF Stores
issue opened on Nov 03 2023 by Frederick Muriuki Muriithi, last updated on Nov 14 2023 by Frederick Muriuki Muriithi; 0 of 6 tasks done
.../pull/130 2: Munyoki's Pull request
=> https://github.com/BonfaceKilz/gn-dataset-dump 3: Dataset -> LMDB export repository
=> https://github.com/genenetwork/dump-genenetwork-database 4: Metadata -> RDF...
- Fetch trait data using genofiles
issue opened on Jul 11 2023 by Alexander_Kabui, last updated on Jul 13 2023 by Alexander_Kabui; 2 of 3 tasks done
...database does not have all genotype files when fetching sample data use genotypes to fetch trat data given a dataset and the trait
Having fetched the sample names of a given group from the genofiles...
- ProbeSetXRef
issue opened on Dec 29 2021 by Pjotr Prins, last updated on Jul 01 2022 by Arun Isaac
...database, mariadb, innodb
* type: enhancement, documentation
* assigned: pjotrp
* status: unclear
* priority: medium
## Table ProbeSetXRef
Juggling indexes and transforming to InnoDB led to a massive...