- Make xapian index rebuild conditional on database checksums
✓ issue opened on Jun 01 2023 by Arun Isaac, last updated on Dec 19 2023 by Arun Isaac; 3 of 3 tasks done
...conditional on database checksums
* assigned: arun
Currently, we unconditionally rebuild the xapian index once every day regardless of whether the database has actually changed over the last day. Not...
- Tests for the genodb genotype database
✓ issue opened on Aug 30 2022 by Arun Isaac, last updated on Sep 05 2022 by Alexander_Kabui
...database
Write tests for the genodb genotype database Common Lisp code.
=> https://git.genenetwork.org/GeneNetwork/cl-gn cl-gn repository which contains the genodb code
Perhaps, imitate similar tests...
- Implementing Efficient Database Caching for Query Responses in GN-LLM system
✓ issue opened on Jan 17 2024 by Alexander_Kabui, last updated on Sep 09 2024 by Alexander_Kabui; 3 of 3 tasks done
...task aims to enhance the performance and responsiveness of our GN-LLM (Large Language Model) system by incorporating a robust database caching mechanism. The focus will be on utilizing a database...
- Check Genotypes in the Database for R/qtl2 Uploads
✓ issue opened on Jun 26 2024 by Frederick Muriuki Muriithi, last updated on Jul 01 2024 by Frederick Muriuki Muriithi
...already in its database.
This issue tracks the implementation for the check of the genotypes against both the genotypes provided in the bundle, and those already in the database.
### Updates
Fixed...
- Move Uploader to tux02
✓ issue opened on Mar 12 2024 by Frederick Muriuki Muriithi, last updated on Jul 03 2024 by Frederick Muriuki Muriithi; 3 of 3 tasks done
.../var/lib/mysql3307
socket = /var/run/mysqld/mysqld3307.sock
︙
```
### SQLite
- [ ] Provide separate path for the SQLite database file
- [ ] Run migrations on SQLite database...
- Full text search
✓ issue opened on Jun 30 2022 by Arun Isaac, last updated on Feb 13 2023 by Arun Isaac; 1 of 1 tasks done
...-database repo using guile-xapian.
=> https://xapian.org/ Xapian search engine library
=> https://git.genenetwork.org/arunisaac/dump-genenetwork-database dump-genenetwork-database repository...
- Upload GeneWiki RDF metadata to CD
✓ issue opened on Apr 17 2023 by Munyoki Kilyungi, last updated on Apr 18 2023 by Munyoki Kilyungi
...-database$ curl -I localhost:9082/sparql
curl: (7) Failed to connect to localhost port 9082: Connection refused
```
### Resolution
There was a database format mismatch due to a virrtuoso upgrade. Now...
- QC: Fix Integration Tests
✓ issue opened on Jan 09 2024 by Frederick Muriuki Muriithi, last updated on Feb 27 2024 by Frederick Muriuki Muriithi
...a new database for the test session, enabling the tests to run unhindered, but also without tainting the production redis databases.
### Update 2024-02-27
The system was updated to use prefixed keys...
- Materialised Views for Correlations
✓ issue opened on Oct 19 2022 by Frederick Muriuki Muriithi, last updated on Dec 19 2022 by Frederick Muriuki Muriithi
...CI/CD) database to get similar results.
The problem here, is that the migration might be moot, if the data is then moved out of the database, as is being planned.
### Queries to Materialise
Possible...
- Read Samples/Cases/Individuals From Database
✓ issue opened on Jan 20 2024 by Frederick Muriuki Muriithi, last updated on Feb 27 2024 by Frederick Muriuki Muriithi
...database.
This bug is even "encoded" in
=> https://gitlab.com/fredmanglis/gnqc_py/-/blob/6200a60eb6f04a5d50bfe0ad366674dc49a08119/README.org#L26 the original specifications.
> - check strain headers...
- Fix Broken UTF-8 characters in our Database
✓ issue opened on Nov 29 2022 by Munyoki Kilyungi, last updated on Aug 29 2023 by Munyoki Kilyungi
Fix Broken UTF-8 characters in our Database
## Tags
* assigned: bonfacem, arthur
* type: database
* priority: high
## Description
We have jumbled up text in our database and this has been the case...
- Improve Menu Generation and Move it to GN3
✓ issue opened on May 16 2022 by Frederick Muriuki Muriithi, last updated on Jun 23 2022 by Frederick Muriuki Muriithi; 2 of 3 tasks done
...database within loops.
This ruins the performance of the system significantly.
The queries should be reworked, and the code should be moved to GN3 since it does database access.
### TODOs
* [ ] Rework...
- Virtuoso: Shutdown Clears Data
✓ issue opened 4 weeks ago by Frederick Muriuki Muriithi, last updated 4 weeks ago by Frederick Muriuki Muriithi
...Set up virtuoso database configurations
=> https://git.genenetwork.org/gn-machines/commit/?id=46a1c4c8d01198799e6ac3b99998dca40d2c7094 Explicitly name virtuoso database files.
- Clean up Authorisation
✓ issue opened on Nov 19 2021 by BonfaceKilz, last updated on Dec 04 2023 by Frederick Muriuki Muriithi; 4 of 4 tasks done
...extra value
* [x] Fetch complete list of samples from database and genotype file
instead of only fetching that list from the database. Look at trait
page for reference.
* [X] Extend idea of csv...
- Phenotype Correlation Error
✓ issue opened on Sep 28 2022 by Zachary Sloan, last updated on Nov 28 2023 by Frederick Muriuki Muriithi
.../production/gene/wqflask/base/trait.py", line 599, in retrieve_trait_info
raise KeyError(repr(trait.name)
KeyError: "'1422223_at' information is not found in the database."
```
so far, triangulated...
- Quality Control of Data in Uploaded R/qtl2 Bundles
✓ issue opened on Feb 02 2024 by Frederick Muriuki Muriithi, last updated on Sep 16 2024 by Frederick Muriuki Muriithi; 10 of 15 tasks done
...database, prior to attempting to parse the file and load data into the database
* [ ] If listed samples/cases do not exist in database, verify they are all listed in the "geno" file(s)
### [ ] phenose...
- Login issues with gn-auth
✓ issue opened on Mar 01 2024 by Pjotr Prins, last updated on Jul 03 2024 by Frederick Muriuki Muriithi
...52 cursor.execute("INSERT INTO users VALUES (?, ?, ?)",
2024-03-02 01:53:52 sqlite3.OperationalError: attempt to write a readonly database
```
Looks like the container cannot write to the database.
- Data Uploads: Zero Representation
✓ issue opened on Feb 08 2022 by BonfaceKilz, last updated on Nov 11 2022 by Munyoki Kilyungi
...gn2 the value still remains as "x" and isn't updated in the database.
Also, ATM you cannot edit a value to "x", which is similar to removing
a field in the database.
#### Wed 25 May 2022 22:01:44 EAT...
- Add HTML Page for ProbeSet Page
✓ issue opened on Dec 14 2023 by Munyoki Kilyungi, last updated on Jan 09 2024 by Munyoki Kilyungi
...were url-encoded to form valid urls. See: 4a62e1781692, 56d09222742c, and 03df1227c419 from:
=> https://git.genenetwork.org/gn-transform-databases/ gn-transform-databases
Other Relevant PR's/Commits...
- Minor Phenotype Page UI updates
✓ issue opened on Nov 30 2023 by Munyoki Kilyungi, last updated on Dec 01 2023 by Munyoki Kilyungi; 6 of 6 tasks done
...-logP"(LOD-Score), peak-location, effect-size
Resolved in: 77f9036298e8, abe23c624c66, a850acb21152, 7b2a0e1be7d8 in:
=> https://git.genenetwork.org/gn-transform-databases/tree/ gn-transform-databases...
- Dump GeneWiki data
✓ issue opened on Mar 30 2023 by Munyoki Kilyungi, last updated on Apr 17 2023 by Munyoki Kilyungi
...database maintained by NCBI. In GeneNetwork, this is stored in GeneRIF_BASIC. [Authorised] Users of GN can add their own entries so that they are associated with a given symbol. In SQL, these edits...
- Quality Control Project
✓ issue opened on Nov 19 2021 by Arthur Centeno, last updated on Jul 20 2022 by Frederick Muriuki Muriithi
...database. GN1 does some of that. This is
where Arun comes in - we need to have a common handler for data that
is in the database and data that is in escrow. My idea is that this
will all be text files...
- Wrong CSV in ITP_10001 longevity dataset
✓ issue opened on Apr 11 2022 by BonfaceKilz, last updated on Apr 12 2022 by BonfaceKilz
...database, some characters are inserted with control sequences that need to be stripped out. Here's a current snip of how that looks like:
```
JL00005,896.000000,x,x,896,4/22/04,,4OHPBN_J,Oct,,0^M,M,JL...
- Editing Metadata [Improvements to Make]
✓ issue opened on Mar 31 2022 by BonfaceKilz, last updated on Apr 18 2023 by Munyoki Kilyungi; 7 of 11 tasks done
...Published Database*
* [X] Hmm, this header is not very good-- *Edit Trait for Published Database.* a. We mean the word "edit" as a verb, but status of this word is ambiguous. b. Published Database...
- Capture Data on the BXDs in RDF
✓ issue opened on Mar 23 2022 by Frederick Muriuki Muriithi, last updated on Oct 12 2022 by Munyoki Kilyungi
...Annotate traits page with metadata from RDF
Work on dumping RDF has already been done in:
=> https://github.com/genenetwork/dump-genenetwork-database dump-genenetwork-database
Also, vector/matrix data...
- tux01 running out of RAM
✓ issue opened on Sep 03 2022 by Pjotr Prins, last updated on May 03 2024 by Pjotr Prins; 1 of 6 tasks done
...(2.523 sec). So it is fine now. It might be that on reboot the table got fixed, but we'll check the tables anyway. First take a look at the state of the engine itself as described in
=> ../database-...
- MySQLdb._exceptions.OperationalError: (1040, 'Too many connections')
✓ issue opened on Aug 29 2022 by Munyoki Kilyungi, last updated on Sep 05 2022 by Alexander_Kabui; 2 of 2 tasks done
MySQLdb._exceptions.OperationalError: (1040, 'Too many connections')
## Tags
* assigned: bonfacem, fredm, aruni
* type: bug
* keywords: mysql, database
## Tasks
* [x] Figure out root cause
* [x] Send...
- Rework Fetching Settings
✓ issue opened on Sep 25 2022 by Munyoki Kilyungi, last updated on Oct 11 2023 by Munyoki Kilyungi
..."wqflask/base/data_set/datasetgroup.py". ATM, now we have a more generic way of fetching settings from wqflask.database, appropriately called "get_setting". Perhaps, when this task is being worked...
- Upload Strains
✓ issue opened on Dec 06 2023 by Frederick Muriuki Muriithi, last updated on Feb 27 2024 by Frederick Muriuki Muriithi; 5 of 5 tasks done
...for the strains (think InbredSet)
* [x] UI to select the CSV file with the strains data
* [x] UI to select the way to interprete the CSV file
* [x] Code to insert the new strains, into the database
- Global search does not close connections properly (and is slow)
✓ issue opened on Mar 23 2022 by Frederick Muriuki Muriithi, last updated on Oct 14 2022 by Arun Isaac
...generating 6Mb of log file info.
In fact, every row in the table has a SQL query that does not close the connection properly.
## Resolution
The new xapian search does not use the SQL database at...
- Remove everything elastic search
✓ issue opened on Oct 22 2021 by Pjotr Prins, last updated on Apr 01 2022 by BonfaceKilz
Remove everything elastic search
We are no longer using that database
Seems related to
=> ../issues/remove-elastic-search...
- Troubleshoot CD Menu Failure
✓ issue opened on Apr 21 2023 by Munyoki Kilyungi, last updated on Apr 26 2023 by Munyoki Kilyungi
...in CD fails. This is because the database in CD is out of sync with the one in production. In particular:
```
2023-04-21 11:54:41 MySQLdb._exceptions.OperationalError: (1054, "Unknown column 'Family'...
- Delete Rejected Diffs from Database
✓ issue opened on Jul 25 2023 by Frederick Muriuki Muriithi, last updated on Jul 25 2023 by Frederick Muriuki Muriithi
...Database
## Tags
* type: feature request
* status: closed
* assigned: fredm
* keywords: editing, metadata audit
* priority: high
## Description
The rejected diffs will be maintained, but will simply...
- Do Bulk Query for Correlation Results' Display
✓ issue opened on Oct 21 2022 by Frederick Muriuki Muriithi, last updated on Oct 24 2022 by Frederick Muriuki Muriithi
.../wqflask/wqflask/correlation/show_corr_results.py#L112-L220 This loop
in lines 118 to 120 (call to `create_trait(...)) queries the database at least once every iteration, which leads to performance...
- DOL group mapping issues
✓ issue opened on Apr 07 2022 by Zachary Sloan, last updated on Apr 20 2022 by Zachary Sloan
...addressed yet with displaying mitochondrial markers in the GN2 figure. GEMMA is outputting the results, so I suspect this is because mitochondria isn't included in the databased list of chromosomes...
- Rewrite qc and qc-uploads in Python3
✓ issue opened on Apr 11 2022 by Frederick Muriuki Muriithi, last updated on Nov 28 2023 by Frederick Muriuki Muriithi
...to the database, to link it properly.
### Answers to Questions
#### Question 01
The first field will be treated as text, and will not undergo any verification
#### Question 02
The line-endings will...
- Edit OAuth2 Clients
✓ issue opened on Apr 24 2024 by Frederick Muriuki Muriithi, last updated on May 02 2024 by Frederick Muriuki Muriithi
...recent updates to use JWT in place of simple "AuthorizationCode" tokens, we needed to update the database to ensure the OAuth2 clients had the appropriate grant types set up.
It turns out, at least...
- Upload probeset metadata
✓ issue opened on Jun 19 2023 by Munyoki Kilyungi, last updated on Jun 26 2023 by Munyoki Kilyungi
...-database$ ls -lah data/
total 3.2G
drwxr-xr-x 2 bonfacem bonfacem 4.0K Jun 16 08:04 .
drwxr-xr-x 9 bonfacem bonfacem 4.0K Jun 15 07:04 ..
-rw-r--r-- 1 bonfacem bonfacem 1.2G Jun 16 07:48 dump-...
- Fix load-rdf.scm script
✓ issue opened on Apr 05 2023 by Munyoki Kilyungi, last updated on Apr 05 2023 by Munyoki Kilyungi
...-pipe _ _ _ . _)
load-rdf.scm:117:24: In procedure call-with-pipe:
Invocation of program failed ("isql")
```
See the following for the fix:
=> https://github.com/genenetwork/dump-genenetwork-database/...
- Inspect Discrepancies Between Xapian and SQL Search.
✓ issue opened on Jul 10 2024 by Munyoki Kilyungi, last updated on Jul 29 2024 by Munyoki Kilyungi
...org/gn-transform-databases/commit/?id=d95501bd2bd41ef8cf3584118382e83cbbbe0c87 [gn-transform-databases] Add missing RIF symbols.
Now these 2 queries return the same exact results:
=> https://cd...
- Correlations fail for at least some ProbeSet datasets (as the target dataset)
✓ issue opened on Oct 17 2022 by Zachary Sloan, last updated on Nov 22 2022 by Frederick Muriuki Muriithi
...the database, leading to issues with the final path.
In this case, the dataset name used to generate the file is (** Note the forward slash **):
```
EPFL/ETHZ BXD Liver Proteome CD-HFD (Nov19)
```
The...
- Temp traits don't seem to be handled by the authorization system
✓ issue opened on Jul 28 2023 by Zachary Sloan, last updated on Feb 27 2024 by Frederick Muriuki Muriithi
...publicly readable. This is necessary since the "Temp" traits are not attached to any resources. It is also because unlike all the other traits, "Temp" traits are not saved in the database, rather...
- Slow Correlations and UI crashes
✓ issue opened on Oct 18 2021 by BonfaceKilz, last updated on Jul 08 2022 by Alexander_Kabui
According to Rob, GN1 does not rely on a cache. Instead it is
computing from a materialized view of the database that is
intentionally designed for a fast web service.
# Notes
### Tue, 12 April 2022...
- Genenetwork3 Effective UID
✓ issue opened on Jun 05 2023 by Frederick Muriuki Muriithi, last updated on Jun 09 2023 by Frederick Muriuki Muriithi
...KeyError: 'getpwuid(): uid not found: 1000'
2023-06-05 03:46:38
2023-06-05 03:46:48 [2023-06-05 03:46:48,918] ERROR in errors: unable to open database file
2023-06-05 03:46:48 unable to open database...
- Autogenerate documentation: trees, and labels
✓ issue opened on Jun 23 2023 by Munyoki Kilyungi, last updated on Oct 11 2023 by Munyoki Kilyungi
...-database/pull/11
Given an s-expression say:
```
(define-dump dump-species
(tables (Species))
(schema-triples
(gn:name rdfs:range rdfs:Literal)
(gn:displayName rdfs:range rdfs:Literal)
(gn...
- Migrate User Accounts from Redis to new Auth DB
✓ issue opened on Dec 22 2022 by Frederick Muriuki Muriithi, last updated on May 22 2023 by Frederick Muriuki Muriithi
...to register anew and their access details reconfirmed.
--------------------
Currently, on GN2, user details are stored in Redis. We need to migrate these to the new auth database (SQLite3) in order...
- Error when fetching SNPs in a search page
✓ issue opened on Sep 28 2022 by Munyoki Kilyungi, last updated on Apr 18 2023 by Munyoki Kilyungi
...local database, you get an error because the "RatSnpPattern" table does not exist:
```
ERROR:wqflask:http://localhost:5004/snp_browser?first_run=true&species=mouse&gene_name=BG976607&limit_strains...
- Add to Collection Error
✓ issue opened on Oct 09 2023 by Frederick Muriuki Muriithi, last updated on Nov 28 2023 by Frederick Muriuki Muriithi
..."Calculate Correlations" accordion
* Set Method = "sample r"
* Set Database = "GTEXv8 Human Kidney-Cortex RNA-Seq (Feb20) TPM log2"
* Set "Limit to" = 500
* Set Samples = GTEx_v8
* Set Type = Pearson...
- GEMMA output differs from R/qtl2
✓ issue opened on Aug 01 2024 by Pjotr Prins, last updated on Sep 20 2024 by Pjotr Prins
...database, gemma, reaper, rqtl2
# Description
When running trait BXD_21526 results differ significantly.
=> https://genenetwork.org/show_trait?trait_id=21526&dataset=BXDPublish
=> https://genenetwork...
- Set-up Virtuoso+Xapian on Production
✓ issue opened on Nov 22 2023 by Munyoki Kilyungi, last updated 4 weeks ago by Frederick Muriuki Muriithi; 1 of 1 tasks done
...-transform-databases/tree/generate-ttl-files.scm Run "generate-ttl-files" to generate the TTL files:
```
time guix shell guile-dbi -m manifest.scm -- \
./generate-ttl-files.scm --settings conn-dev.scm...
- Designing an issue tracker on gemini
✓ issue opened on Jul 25 2021 by Pjotr Prins, last updated on Feb 02 2022 by Pjotr Prins
.../gn-gemtext-threads/blob/main/issues/database-not-responding.gmi
## Process
We leverage git to pull out dates and people contributions (pjotrp wrote ...) for display in a document/web page. This page...
- R/qtl JSONDecodeError
✓ issue opened on Dec 23 2022 by Pjotr Prins, last updated on Dec 24 2022 by Pjotr Prins; 3 of 3 tasks done
.../lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/home/gn2/gn3_production/genenetwork3/gn3/db_utils.py", line 55, in xapian_database
db = xapian.Database(path)...
- Annotate traits page with metadata from RDF
✓ issue opened on Sep 30 2022 by Munyoki Kilyungi, last updated on Dec 15 2022 by Munyoki Kilyungi; 16 of 16 tasks done
...112
* [X] Work out what type of datasets have accession id's
* [X] Refactor the dataset fetch fn in GN3 to use the Maybe Monad
* [X] Write tests for the above
* [X] Test on test database upstream - if...
- Automated Testing
✓ issue opened on Feb 10 2022 by Frederick Muriuki Muriithi, last updated on Oct 12 2022 by Munyoki Kilyungi; 0 of 10 tasks done
...(among others):
* Each API endpoint responds within a specified amount of time
* Select computation-heavy functions respond within a specified amount of time for given data
* Database-querying...