Edit this page | Blame

CI/CD for genetwork projects

Continuous intergration (CI) and continuous deployment (CD) are critical parts of making sure software development does not mess up our deployment(s).

We need various levels of tests to be run, from unit tests to the more complicated ones like integration, performance, regression, etc tests, and of course, they cannot all be run for each and every commit, and will thus need to be staggered across the entire deployment cycle to help with quick iteration of the code.

Tags

  • assigned: bonfacem, fredm, efraimf, aruni
  • keywords: deployment, CI, CD, testing
  • status: in progress
  • priority: high
  • type: enhancement
  • status: closed

Tasks

As part of the CI/CD effort, it is necessary that there is

#### Ideas

GeneNetwork is interested in doing two things on every commit (or periodically, say, once an hour/day):

  • CI: run unit tests on git push
  • CD: rebuild and redeploy a container running GN3

Arun has figured out the CI part. It runs a suitably configured laminar CI service in a Guix container created with `guix system container'.

We have the quick running tests, e.g unit tests, run on each commit to branch "main". Once those are successful, the CI/CD system we choose should automatically pick the latest commit that passed the quick running tests for for further testing and deployment. Once the next battery of tests is passed, the CI/CD system will create a build/artifact to be deployed to staging and have the next battery of tests runs against it. If that passes, then that artifact could be deployed to production with details of the commit and deployment dependencies.

Adding a web-hook

Github hooks

IIRC actions run artifacts inside github's infrastracture. We use webhooks: e.g.

Update the hook at

A web hook basically calls an endpoint on a git push event. The webhook for genenetwork3 has recently not been called (ever? it says: This hook has never been triggered. ). The webhook for genenetwork2, however, has been called.

To trigger CI manually, run this with the project name:

curl https://ci.genenetwork.org/hooks/example-gn3

I just tested and it appeared this triggered a redeploy of gn2:

curl -XGET "https://ci.genenetwork.org/hooks/genenetwork2

For gemtext we have a github hook that adds a forge-project and looks like

(define gn-gemtext-threads-project
  (forge-project
   (name "gn-gemtext-threads")
   (repository "https://github.com/genenetwork/gn-gemtext-threads/")
   (ci-jobs (list (forge-laminar-job
                   (name "gn-gemtext-threads")
                   (run (with-packages (list nss-certs openssl)
                          (with-imported-modules '((guix build utils))
                            #~(begin
                                (use-modules (guix build utils))

                                (setenv "LC_ALL" "en_US.UTF-8")
                                (invoke #$(file-append tissue "/bin/tissue")
                                        "pull" "issues.genenetwork.org"))))))))
   (ci-jobs-trigger 'webhook)))

The normal trigger is automatic, you push code in any of the two repos (three? I'll verify), GN2 and GN3 and the laminar runs the jobs and updates the code in the container restarts services, as appropriate.

If you want to trigger the CI manually, there are webhooks available for that that can be triggered manually with something like:

curl -XGET "https://ci.genenetwork.org/hooks/genenetwork2"

for GN2. Change the part after /hooks/ for each of the different repos as follows:

GN2: /genenetwork2
GN3: /genenetwork3
gn-auth: /gn-auth (I need to verify this)
gn-uploader: Does not exist right now

Guix forge can be found at

git.genenetwork.org hooks

TBD

#### Possible Steps

Below are some possible steps (and tasks) to undertake for automated deployment

##### STEP 01: Build package

  • Triggered by a commit to "main" branch (for now)
  • Trigger build of the package
  • Run unit tests as part of the build: This has been done with the laminar scripts under `scripts/laminar` in genenetwork3. Maybe just change the command to ensure only specific tests are run, especially when we add in non-functional tests and the like
  • If the build fails (tests fail, other failures): abort and send notifications to development team
  • If build succeeds, go to STEP 02

##### STEP 02: Deploy to Staging

  • Triggered by a successful build
  • Run in intervals of maybe one hour or so...
  • Build the container/VM for deployment: here's the first time `guix system container ...` is run
  • Deploy the container/VM to staging: the details are fuzzy here
  • Run configuration tests here
  • Run performance tests
  • Run integration tests
  • Run UI tests
  • Run ... tests
  • On failure, abort and send out notification to development team
  • On success go to STEP 03

##### STEP 03: Deploy to Release Candidate

  • Triggered by a successful deploy to Staging
  • Run in intervals of maybe 6 hours
  • Pick latest successful commit to pass staging tests
  • Build the container/VM for deployment: run `guix system container ...` or reuse container from staging
  • Update configurations for production
  • Run configuration tests
  • Run acceptance tests
  • On failure, abort and send out notification to development team
  • On success go to STEP 04

##### STEP 03: Deploy to Production

  • Triggered by a successful Release Candidate
  • Tag the commit as a release. Maybe include the commit hash and date e.g gn3-v0.0.12-794db6e2-20220113
  • Build the container/VM for deployment: run `guix system container ...` or reuse container from staging; tag container as a release container
  • Deploy container to production
  • Generate documentation for tagged commit
  • Generate guix declaration for re-generating the release
  • Archive container image, documentation and guix declaration for possible rollback

#### See also

This contains a check-list of things that need to be done:

(made with skribilo)