Create isolated Jupyter ipython kernels with pyenv and virtualenv

Share On Facebook
Share On Twitter
Share On Google Plus
Share On Linkedin
Share On Reddit
Contact us

Everyone loves isolation. Makes our life easier and our systems much more robust. Isolating Jupyter notebooks makes no exception. Maybe you want to try some cutting edge scientific library, or more simply your latest project dependencies are not compatible with your current system setup.

Whatever is your situation, follow me in this simple tutorial on how to create an isolated python notebook kernel.

Introduction

If you are a day-to-day Jupyter user you probably know what kernels are. If you are not, a kernel is simply a language virtual machine running behind the scenes and connected to the Jupyter browser interface. Each time you create a new notebook you have to select the kernel from the top right drop down menu of the interface as shown here:

Screen Shot 2015-12-10 at 19.42.29

The goal of this blog post is to add a new kernel on a python environment that is different from the one that I have installed already. This is especially useful if you are trying out some project and you want the dependencies of the new project to be installed in a separate python local setup. We can do that in 3 "easy" steps 🙂

Step 1: Install pyenv, virtualenv, and pyenv-virtualenv

First we need a way to create different python environments. Each environment will have its own version and isolated package set. On paper it is far from easy, but thanks to Pyenv 1 and virtualenv2 the job turns out to be pretty simple.

To install it on MacOSx all you need to do is:

and add to your .bashrc the following:

Installing virtualenv is also fairly easy:

Finally, install pyenv-virtualenv 3:

and add to your .bashrc the following:

Step 2: Create an isolated python environment

Let's assume now that I want to test my latest project in a Jupyter notebook running a Python kernel with Python 2.7.3. Since I have several projects running on this Python version I also want to have a dedicated Python 2.7.3 environment for my latest project.

First let's see all the python versions available to pyenv:

and let's install the one we wants:

Now it is time to create a dedicated environment for our project. Suppose I am working on a bleeding age implementation of k_means clustering algorithm (...). This is how I create my working environment:

and I now see the following:

Let's switch our python environment to the one we created for our new project

and let's install the basic scientific packages we need:

these packages will be local to our k_means python installation and will not affect our system python (for example).
To access this environment from Jupyter you need the python kernel too, so let's install it:

Finally, let's deactivate our environment.

Step 3: Create your isolated Jupyter python kernel

Now we have to connect our Jupyter to the isolated python enviroment we created in the previous two steps. I am assuming you have already jupyter installed in your system Python. If not, go ahead and install it here.

If you are in the right environment with jupyter installed you should see the following:

In order to install a new Jupyter kernel you need to check where Jupyter is reading its configuration files. To do that simply run the following:

Jupyter search for the kernels in the data directories in the order they are displayed. First, we need to find out where pyenv is storing our k_means environment, and we do it by executing the following:

Now we are ready to create our kernel. First let's create the folder:

and let's add the following kernel.json file:

If you now run jupyter notebook you will have the new kernel available!

Screen Shot 2015-12-10 at 19.25.10

Conclusions

In this blog post I presented how to create an isolated python kernel in your jupyter installation. This is not the only way to do it, so please share in the comments if you have better ways to achieve the same result!

If you enjoyed this blog post you can also follow me on twitter.

References


Also published on Medium.

  1. Pyenv homepage
  2. Virtualenv homepage
  3. Pyenv-virtualenv homepage

6 thoughts on “Create isolated Jupyter ipython kernels with pyenv and virtualenv

  1. This is great, but one can go further! By installing a pyenv hook, one can get pyenv to automatically set the environment variables used by Jupyter to determine the locations of its configuration and data directories.

    In my shell startup scripts, I define the following two “template” environment variables:


    export _JUPYTER_CONFIG_DIR='${PYENV_ROOT}/versions/$(pyenv version-name)/etc/jupyter’

    export _JUPYTER_DATA_DIR='${PYENV_ROOT}/versions/$(pyenv version-name)/share/jupyter’

    Next, I create the hook script as ~/.pyenv/pyenv.d/exec/jupyter-paths.bash. This script contains only two lines:


    export JUPYTER_CONFIG_DIR=
    eval "echo ${_JUPYTER_CONFIG_DIR}"
    export JUPYTER_DATA_DIR=eval "echo ${_JUPYTER_DATA_DIR}"

    Now you can use Jupyter’s standard toolset for installing kernels, etc., and everything ought just to work.

  2. If I try to use matplotlib I get the following error messages.

    import matplotlib.pyplot as plt

    Do you have the same problem?

    RuntimeError Traceback (most recent call last)
    in ()
    1 #Laden der Module
    ----> 2 import matplotlib.pyplot as plt
    3 import os,sys
    4
    5 #Background Color Figure

    /Users/chrischi/.pyenv/versions/geoEnv/lib/python2.7/site-packages/matplotlib/pyplot.py in ()
    112
    113 from matplotlib.backends import pylab_setup
    --> 114 _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
    115
    116 _IP_REGISTERED = None

    /Users/chrischi/.pyenv/versions/geoEnv/lib/python2.7/site-packages/matplotlib/backends/init.pyc in pylab_setup()
    30 # imports. 0 means only perform absolute imports.
    31 backend_mod = import(backend_name,
    ---> 32 globals(),locals(),[backend_name],0)
    33
    34 # Things we pull in from all backends

    /Users/chrischi/.pyenv/versions/geoEnv/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.py in ()
    22
    23 import matplotlib
    ---> 24 from matplotlib.backends import _macosx
    25
    26

    RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are Working with Matplotlib in a virtual enviroment see 'Working with Matplotlib in Virtual environments' in the Matplotlib FAQ

    1. I have had this problem before, in the end I gave up and didn't try to use a virtual env...not ideal I know.

Leave a Reply

Your email address will not be published. Required fields are marked *

Answer the question * Time limit is exhausted. Please reload CAPTCHA.