Edit this page | Blame

Implementing parallel correlation with rust


In an attempt to speed the current gn2 correlation we are doing an reimplentation in rust to support parralel computation


  • assigned:alem,Fred
  • type: upgrade
  • priority: medium
  • status: closed, completed
  • keywords: correlation,rust,parralel


  • [X] implementation of basic pearson and spearman correlation in rust
  • [X] add unittests and benchmarks
  • [X] loading datasets;format
  • [X] package the library as a guix package
  • [x] gn2 integration
  • [x] parsing input datasets
  • [x] benchmark for all datasets rust v gn2 version
  • [ ] Figure out how to tell cargo to use the declared dependencies, rather than dowloading the dependencies
  • [*] db queries migration to gn3 or use partial correlation queries
  • [x] add parralel computation
  • [x] code optimization and minor fixes
  • [x] lmbd parser
  • [x] implement biweight mid correlation for rust
  • [] fix lmdb-rust packaging
  • [] production deployment


You can call the lib as an external process from any language or directly call it in a rust cargo

code can be found here:

added a documentation for correlation rust

PR for integrating to genenetwork:

8 ,July 2022

worked on input data parsing and enhancements

PR for parsing datasets

17,Aug 2022

integration of correlation rust ongoing with @fred working also including optimizing of slow queries,code refatoring

5 Sept 2022

20 Sept 2022

integrated codebase to store db in text files for probeset types

11 july 2023

  • implemented Parallel computation implementation with rust completed
  • implemented both lmdb and csv parser in rust
(made with skribilo)