Menu
Alfredo Motta
  • Home
  • Newsletter
  • Articles
  • About me
  • Events
  • Books
  • Research
  • Learn to code in London
Alfredo Motta

Create isolated Jupyter ipython kernels with pyenv and virtualenv

Posted on December 10, 2015May 18, 2022

Everyone loves isolation. Makes our life easier and our systems much more robust. Isolating Jupyter notebooks makes no exception. Maybe you want to try some cutting edge scientific library, or more simply your latest project dependencies are not compatible with your current system setup.

Whatever is your situation, follow me in this simple tutorial on how to create an isolated python notebook kernel.

Introduction

If you are a day-to-day Jupyter user you probably know what kernels are. If you are not, a kernel is simply a language virtual machine running behind the scenes and connected to the Jupyter browser interface. Each time you create a new notebook you have to select the kernel from the top right drop down menu of the interface as shown here:

Screen Shot 2015-12-10 at 19.42.29

The goal of this blog post is to add a new kernel on a python environment that is different from the one that I have installed already. This is especially useful if you are trying out some project and you want the dependencies of the new project to be installed in a separate python local setup. We can do that in 3 “easy” steps 🙂

Step 1: Install pyenv, virtualenv, and pyenv-virtualenv

First we need a way to create different python environments. Each environment will have its own version and isolated package set. On paper it is far from easy, but thanks to Pyenv 1 and virtualenv2 the job turns out to be pretty simple.

To install it on MacOSx all you need to do is:

1
2
$ brew update
$ brew install pyenv

and add to your .bashrc the following:

1
eval "$(pyenv init -)"

Installing virtualenv is also fairly easy:

1
2
$: pip install virtualenv
$: pip install virtualenvwrapper

Finally, install pyenv-virtualenv 3:

1
2
$: brew install pyenv
$: brew install pyenv-virtualenv

and add to your .bashrc the following:

1
eval "$(pyenv virtualenv-init -)"

Step 2: Create an isolated python environment

Let’s assume now that I want to test my latest project in a Jupyter notebook running a Python kernel with Python 2.7.3. Since I have several projects running on this Python version I also want to have a dedicated Python 2.7.3 environment for my latest project.

First let’s see all the python versions available to pyenv:

1
$ pyenv install --list

and let’s install the one we wants:

1
$ pyenv install 2.7.3

Now it is time to create a dedicated environment for our project. Suppose I am working on a bleeding age implementation of k_means clustering algorithm (…). This is how I create my working environment:

1
$: pyenv virtualenv 2.7.3 k_means

and I now see the following:

1
2
3
4
5
:~$ pyenv versions
system
* 2.7.3 (set by /Users/motta/.pyenv/version)
3.5.0
k_means

Let’s switch our python environment to the one we created for our new project

1
2
3
4
5
6
:~$ pyenv virtualenvs
k_means (created from /Users/motta/.pyenv/versions/2.7.11rc1)
:~$ pyenv activate k_means
:~$ pyenv virtualenvs
* k_means (created from /Users/motta/.pyenv/versions/2.7.11rc1)
:~$

and let’s install the basic scientific packages we need:

1
2
:~$ pip install numpy
:~$ pip install pandas

these packages will be local to our k_means python installation and will not affect our system python (for example).
To access this environment from Jupyter you need the python kernel too, so let’s install it:

1
:~$ pip install ipykernel

Finally, let’s deactivate our environment.

1
:~$ pyenv deactivate

Step 3: Create your isolated Jupyter python kernel

Now we have to connect our Jupyter to the isolated python enviroment we created in the previous two steps. I am assuming you have already jupyter installed in your system Python. If not, go ahead and install it here.

If you are in the right environment with jupyter installed you should see the following:

1
2
3
:~$ pip list | grep jupyter
jupyter-client (4.1.1)
jupyter-core (4.0.6)

In order to install a new Jupyter kernel you need to check where Jupyter is reading its configuration files. To do that simply run the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
:~$ jupyter --paths
config:
    /Users/motta/.jupyter
    /Users/motta/.pyenv/versions/2.7.3/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /Users/motta/Library/Jupyter
    /Users/motta/.pyenv/versions/2.7.3/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /Users/motta/Library/Jupyter/runtime

Jupyter search for the kernels in the data directories in the order they are displayed. First, we need to find out where pyenv is storing our k_means environment, and we do it by executing the following:

1
2
3
4
:~$ pyenv activate k_means
:~$ pyenv which python
/Users/motta/.pyenv/versions/k_means/bin/python
:~$ pyenv deactivate

Now we are ready to create our kernel. First let’s create the folder:

1
:~$ mkdir /Users/motta/Library/Jupyter/kernels/k_means

and let’s add the following kernel.json file:

JavaScript
1
2
3
4
5
6
{
"argv": [ "/Users/motta/.pyenv/versions/k_means/bin/python", "-m", "ipykernel",
          "-f", "{connection_file}"],
"display_name": "k_means",
"language": "python"
}

If you now run jupyter notebook you will have the new kernel available!

Screen Shot 2015-12-10 at 19.25.10

Conclusions

In this blog post I presented how to create an isolated python kernel in your jupyter installation. This is not the only way to do it, so please share in the comments if you have better ways to achieve the same result!

If you enjoyed this blog post you can also follow me on twitter.

References

  1. Pyenv homepage
  2. Virtualenv homepage
  3. Pyenv-virtualenv homepage

9 thoughts on “Create isolated Jupyter ipython kernels with pyenv and virtualenv”

  1. Johann Visagie says:
    January 19, 2016 at 12:15 pm

    This is great, but one can go further! By installing a pyenv hook, one can get pyenv to automatically set the environment variables used by Jupyter to determine the locations of its configuration and data directories.

    In my shell startup scripts, I define the following two “template” environment variables:


    export _JUPYTER_CONFIG_DIR='${PYENV_ROOT}/versions/$(pyenv version-name)/etc/jupyter’

    export _JUPYTER_DATA_DIR='${PYENV_ROOT}/versions/$(pyenv version-name)/share/jupyter’

    Next, I create the hook script as ~/.pyenv/pyenv.d/exec/jupyter-paths.bash. This script contains only two lines:


    export JUPYTER_CONFIG_DIR=
    eval "echo ${_JUPYTER_CONFIG_DIR}"
    export JUPYTER_DATA_DIR=eval "echo ${_JUPYTER_DATA_DIR}"

    Now you can use Jupyter’s standard toolset for installing kernels, etc., and everything ought just to work.

    Reply
    1. mottalrd says:
      January 20, 2016 at 10:24 am

      Hi Johann, thank you for your comment! In your solution you need to have jupyter installed in each pyenv you are willing to use?

      Reply
  2. Pingback: Create isolated Jupyter ipython kernels with pyenv and virtualenv – Alfredo Motta
  3. Anna says:
    May 29, 2016 at 8:24 pm

    If I try to use matplotlib I get the following error messages.

    import matplotlib.pyplot as plt

    Do you have the same problem?

    RuntimeError Traceback (most recent call last)
    in ()
    1 #Laden der Module
    —-> 2 import matplotlib.pyplot as plt
    3 import os,sys
    4
    5 #Background Color Figure

    /Users/chrischi/.pyenv/versions/geoEnv/lib/python2.7/site-packages/matplotlib/pyplot.py in ()
    112
    113 from matplotlib.backends import pylab_setup
    –> 114 _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
    115
    116 _IP_REGISTERED = None

    /Users/chrischi/.pyenv/versions/geoEnv/lib/python2.7/site-packages/matplotlib/backends/init.pyc in pylab_setup()
    30 # imports. 0 means only perform absolute imports.
    31 backend_mod = import(backend_name,
    —> 32 globals(),locals(),[backend_name],0)
    33
    34 # Things we pull in from all backends

    /Users/chrischi/.pyenv/versions/geoEnv/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.py in ()
    22
    23 import matplotlib
    —> 24 from matplotlib.backends import _macosx
    25
    26

    RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are Working with Matplotlib in a virtual enviroment see ‘Working with Matplotlib in Virtual environments’ in the Matplotlib FAQ

    Reply
    1. mottalrd says:
      June 1, 2016 at 6:36 am

      Hello Anna, no sorry I did not have this problem. Are you using some of the code I posted here?

      Reply
    2. anonymous coward says:
      August 11, 2016 at 6:15 pm

      I have had this problem before, in the end I gave up and didn’t try to use a virtual env…not ideal I know.

      Reply
  4. karthick says:
    September 6, 2017 at 4:07 am

    hi,

    i have installed hydrogen in atom. i wanted to use the python kernel installed by anaconda. the kernel and other files are installed under /opt/anaconda/bin.

    how to let hydrogen know of the kernel and the jupyter notebook run from the anacondoa env.

    Reply
  5. Felix Franz says:
    January 28, 2018 at 1:02 pm

    Hi,

    I’ve had the very same problem. The solution I found was to install a pyenv plugin called “pyenv-register”.

    You can get it here: https://github.com/doloopwhile/pyenv-register

    Then you can register the anaconda python with pyenv:
    pyenv register /opt/anaconda/bin/python3.6

    And set it as your default system python:
    pyenv global system-3.6.3

    Finally register the already installed kernel for hydrogen to find it:
    python -m ipykernel install –user

    Reply
  6. Pingback: Virtual environment – Howl's Moving Castle

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Answer the question * Time limit is exhausted. Please reload CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Tweets by mottalrd

Recent Comments

  • mottalrd on Not So Random Software #21 – Technical debt
  • mottalrd on Not So Random Software #24 – Climate Change Tech
  • mottalrd on A/B Testing, from scratch
  • mottalrd on A/B Testing, from scratch
  • Karim on A/B Testing, from scratch
©2023 Alfredo Motta | Powered by SuperbThemes & WordPress