Edit this page | Blame

Fallback issues

This page tracks outstanding problems with

Note that some issues may be related to production (i.e. the same). Note also I created my own development VM in just an hour, so I don't have to mess with the fallback while others are using it.



502 timeout errors (fixed)

These should no longer happen. It required relaxing timeouts on nginx and gunicorn.

Rqtl2/HK/Pairscan not working (fixed)

URL error on result of

      GeneNetwork 2.11-rc2  https://fallback.genenetwork.org/run_mapping (10:01AM UTC Mar 22, 2024)
Traceback (most recent call last):
  File "/gnu/store/a39cgbdawj9vp24nsz9q25sf9g5vda7c-profile/lib/python3.10/site-packages/requests/models.py", line 434, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/gnu/store/a39cgbdawj9vp24nsz9q25sf9g5vda7c-profile/lib/python3.10/site-packages/urllib3/util/url.py", line 397, in parse_url
    return six.raise_from(LocationParseError(source_url), None)
  File "<string>", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: http://localhost:8893api/rqtl/compute

The error points out a missing slash in the local URL: http://localhost:8893api/rqtl/compute

Grepping machines we get:

gn3-port 8893
("GN3_LOCAL_URL" ,(string-append "http://localhost:" (number->string gn3-port)))

On production we set a trailing slash

GN3_LOCAL_URL = "http://localhost:8081/"

and in the code we mostly see another addition of the slash - except for this particular link. Arguably the slash is not part of the URL prefix (it is part of the path), so I think setting GN3_LOCAL_URL without the trailing slash is preferred. See also

R/qtl JSONDecodeError (fixed)

After adding the slash to GN3 URLs (see above) led to the next error:

simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This looks similar to the earlier documented

suggesting a missing path for R. In the container log GN3 actually gives the error

2024-03-24 08:02:24     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
2024-03-24 08:02:24   File "/gnu/store/yi76sybwqql4ky60yahv91z57srb2fr0-profile/lib/python3.10/site-packages/gn3/api/rqtl.py", line 25, in compute
2024-03-24 08:02:24     raise FileNotFoundError
2024-03-24 08:02:24 FileNotFoundError

Now two things are bad here becuase there is no context other than the error lines! For FileNotFound pass in the filename as suggested here:

Patched in

That should help clarify the missing path or file.

  File "/export/source/fallback-debug/genenetwork3/gn3/fs_helpers.py", line 21, in assert_paths_exist
    raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/gn2/83f0831af5911ab6dc62cbbd37d13741.csv'

So, GN2 is miscommunicating the path to GN3.

This also led to documenting how one should test GN2 and GN3 in a running VM:

and fixing the shared TMPDIR. The next step however fails with

sh: line 1: Rscript: command not found
2024-04-01 07:33:13   File "/gnu/store/yi76sybwqql4ky60yahv91z57srb2fr0-profile/lib/python3.10/site-packages/gn3/computations/rqtl.py", line 6
5, in process_rqtl_mapping
2024-04-01 07:33:13     with open(
2024-04-01 07:33:13 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/output/d41d8cd98f00b204e9800998ecf8427ePr1axOhhSF5DMEMowkGu6AvGj6hf+TA2Ra7FIrlrT4Pw-output.csv'

So, for some reason Rscript is not in the path for GN3. So I added that to the propagated inputs. Now we still have the output error. There is no information the Rscript has actually been invoked and successfuly (or unsuccessfully) wrote the file. To fix this we mount the GN3 source directory in the container again. See

That way I was able to troubleshoot the paths and added them to the containers.

QTLReaper showing font problem (fixed)

Try HK on

  File "/gnu/store/hvv0r5nzhbbsnd9s68cmx5q0sznjhnrp-profile/lib/python3.10/site-packages/PIL/ImageFont.py", line 956, in freetype
    return FreeTypeFont(font, size, index, encoding, layout_engine)
  File "/gnu/store/hvv0r5nzhbbsnd9s68cmx5q0sznjhnrp-profile/lib/python3.10/site-packages/PIL/ImageFont.py", line 247, in __init__
    self.font = core.getfont(
OSError: cannot open resource

This error in on gn2. An image file appears in /tmp/gn2/generated/.

The fonts exist in the GN2 directory, e.g. /gnu/store/mymfiz2jgkfc196az1c0vcc2digc84jy-genenetwork2-stable-stable-3.11-1.42b37bb/lib/python3.10/site-packages/gn2/wqflask/static/fonts/courbd.ttf, but are not found. In the code the path is set to

COUR_FILE = "./gn2/wqflask/static/fonts/courbd.ttf"

so most likely GN2 is not running from that actually source dir.

Pair scan runs, but does not show image (fixed)

Pair scan shows a table on the fallback, but not the image. On CD and production no pair scan results!

Takes about a minute for 26 mice. The csv files are generated in /tmp/gn3, but no image. The relevant code is

   <div class="qtlcharts" id="chart_container">
        <div id="pairscan_chart"></div>

The problem is with Karl's D3 panels, we get an error 'TypeError: d3.scaleLinear is not a function'. That 1.7.1 release is a bit old, so let's try updating that first. After updating to 1.8.4 we have the same error. D3 is at 3.5.17 - that is 2016(!).

Now the latest versions of D3 do *not* provide the actual JS through github. The page

has links to

We can't use those because the SHA may change. So we have to host a local copy. That is fine.

413 Request Entity Too Large (fixed)

On mapping from

This resolved itself - probably a permission issue.

Files generated in /tmp (later)

Not unique file names appearing in TMPDIR:

-rw-r--r-- 1 gunicorn-genenetwork2 gunicorn-genenetwork2 337 Apr  2 07:03 175.png
-rw-r--r-- 1 gunicorn-genenetwork2 gunicorn-genenetwork2 287 Apr  2 07:03 75.png
-rw-r--r-- 1 gunicorn-genenetwork2 gunicorn-genenetwork2 303 Apr  2 07:03 25.png
-rw-r--r-- 1 gunicorn-genenetwork2 gunicorn-genenetwork2 348 Apr  2 07:03 125.png
-rw-r--r-- 1 gunicorn-genenetwork2 gunicorn-genenetwork2 835 Apr  2 07:03 -logP.png

These are created by the HK run. Harmless for now, but if two people run HK at the same time we'll get a conflict.

Unique file names, but lingering on file system - note they are root because I created them outside a container

-rw-r--r-- 1 root root   10M Apr  2 07:03 4OJCzmGV.cross
-rw-r--r-- 1 root root   10M Apr  2 07:03 6cExgwkd.cross

Purescript browser buttons not showing (fixed)

The following button does not display:

                       <button id="scrollLeft"  type="button" >
                          <i class="fas fa-arrow-left"></i>

it is part of font-awesome and apparently not loaded properly. Actually the stylesheet is loaded twice(!) in mapping_results.html AND we use two different class names (fas and fa). Changing the class name fixed it.

BNW gives a jquery error

This is probably a jquery update is causing

Uncaught TypeError: data1.join is not a function
    <anonymous> https://bnw.genenetwork.org/sourcecodes/layout_cyto.php?My_key=jFv:102
    jQuery 4
(made with skribilo)