Edit this page | Blame

Binding notebooks

Binder and Cocalc are tools for deploying Jupyter notebooks, R/shiny notebooks and Pluto notebooks. These binding tools do a lot more and do not leverage GNU Guix lightweight container infrastructure. In this project we want to create a binding-lite.

Tags

  • keywords: binder, jupyter, notebooks
  • status: unclear
  • priority: medium
  • assigned: jgart, pjotrp

Members

  • jgart
  • pjotr

Kanban

  • jgart

Implementation

Key is to leverage what is in Guix and what we already have in GeneNetwork as much as possible.

We should create this separately from genenetwork because all we want to do is run notebooks in containers. We should be able to use and external authentication system, fetch notebooks from git elsewhere and push them on git elsewhere.

binding-lite will be packaged in GNU Guix. GeneNetwork will essentially be a 'rebranding' of binding-lite.

Later we'll add an API to create notebooks on the fly. That is a separate topic.

Background

The current way of using guix-jupyter is to create a guix environment that contains the guix-jupyter kernel and jupyter.

The user can then run the `jupyter notebook` command which prints a url to the terminal for the user to copy/paste into the browser.

Once the url is opened in the browser, a user has to select the guix kernel and choose the notebook from the file manager menu.

Initial Design Ideas: guix-notebook-launcher

The first goal is to fire up a jupyter notebook from a local git repository.

Afterwards, this would be implemented with a remote git repository.

We need a command line interface/tool and guile library to automate the process of opening a jupyter notebook from a local git repository.

Some of the tasks that `guix-notebook-launcher` will need to automate is preselecting the guix kernel and opening the browser to the notebook view of the given url.

Here is an example of how the command line interface would be used:

$ guix-notebook-launcher file:///path/to/<git-repository>

To access the notebook, open this file in a browser:
        file:///path/to/<git-repository>
    or copy and paste one of these URLs:
        http://localhost:8890/?token=fc21c17b9ec31eddbea28c06ce086cf71b5605a09610439b
    or  http://127.0.0.1:8890/?token=fc21c17b9ec31eddbea28c06ce086cf71b5605a09610439b

Managing Jupyter Notebook Dependencies

We'll need a manifest.scm file that will specify the dependencies needed by the guix-jupyter notebook.

Below is an example of the contents of a guix-jupyter notebook repository that can be given to `guix-notebook-launcher`:

$ tree <git-repository>
.
├── LICENSE
├── README.md
├── index.ipynb
└── manifest.scm

An example of the contents of manifest.scm:

(specifications->manifest
  '("python-numpy" 
    "python-pandas"
    "python-matplotlib"))

Additional Features and Ideas

One possibility for easing the creation of a manifest.scm in the notebook repository is to parse the requirements.txt/setup.py, in the case of a python notebook, and generate the manifest.scm file for the notebook user.

We would have to write similar manifest generators for Julia and R project dependencies in order to support this feature.

I will ask on the guix mailing lists for feedback on this idea.

Once we have this library we can start building binder-lite on top of it.

Use Cases

The use cases for `guix-notebook-launcher` are driven by the following design decisions:

  • Automated guix container building guile library to be used by binder-lite.
  • Manual guix container building and running the container from the command line,

`guix-notebook-launcher`, by users on their workstations.

Initial Ideas for binder-lite Front End and Backend

  • Single-page application for logging in to a guix-jupyter notebook view.
  • Leverage oauth for reusing user registration credentials from other platform websites.
  • Git repository url input form for sending to guix-notebook-launcher on the backend.
  • Push changes made in notebook session to remote origin repository on third-party git forge.

Next Development Steps

Current Notebook Launcher Issues

Current WIP Code

Script / Command Line Interface / Web UI Mock / GNU Artanis

  • Test loading a preconfigured guix-jupyter kernel with `MultiKernelManager.default_kernel_nameUnicode`.
  • Fire up a preselected guix kernel from a guile or bash script with $BROWSER.
  • Understand what the data is that the UI will be interacting with.
  • Start mocking a web UI while continuing to develop guix-notebook-launcher and the data that it will provide to the front-end.
  • Finish reading, the source code of binderhub. It is only 10,212 lines of python as of Wed 28 Jul 2021 12:20:10 AM EDT.
  • Finish reading the source code of repo2docker. It is only 12,356 lines of python as of Wed 28 Jul 2021 12:21:22 AM EDT.
  • Meet with Efraim on Thursday to discuss guix containers and my progress on this project.
  • Create a mock similar to notebook.gesis.org with GNU artanis, `sxml->xml` from `(sxml simple)`, or just plain html and vanilla javascript.
  • Research production readyness of GNU Artanis and whether we'll need it. Contact Nala Ginrut.
  • Launch guix-jupyter session from binder-example git repository similar to binder-example by GeorgianaElena.

Reference Links

Current Questions I Have to Answer

  • Is OAuth still needed?

Yes

  • What type does a manifest return?

A manifest! Which of course is just another record :)

scheme@(guix-user)> ,use (gnu packages) (guix profiles)
scheme@(guix-user)> (manifest? (specifications->manifest '("python-ipython")))
$1 = #t
  • Pushing? Pulling? Authentification? Via the github API?
  • Will we build binder-lite completely from scratch or are there useful pieces from binderhub that we can salvage and while stripping out the docker integration?
  • How will we parse the new dependencies that a user introduces during a notebook session?
  • Will guix-jupyter magics be available to the notebook user? How will they interact with guix-notebook-launcher data?
  • Will dependencies be written back to requirements.scm to update the state at the end of a notebook session? Is this already implemented in Binderhub?

Reference Links

Misc Binder Lite Ideas

Should binder-lite be a distribution of GNU Guix System?

GeneNetwork can provide custom ISO and QEMU images that contain binder-lite ready to be deployed to bare metal or a VM.

This is inspired by the-littlest-jupyterhub distribution.

Reference Links

Basic Working Demo

TODOs

  • Add tests
  • Add cloned repository to a uuid generated path so that app doesn't fail on reclone due to previously existing git repo
  • Updgrade Flask to 2.0 in GNU Guix.
  • Add favicon.
  • Add css styling and html meta headers.
  • Add Guix logo loading css animation similar to https://github.com/jupyterhub/binderhub/blob/master/binderhub/static/loading.css
  • Add xtermjs component while waiting for container to build.
  • Cull container after a period of inactivity.

(made with skribilo)