Python Virtual Environments

Overview

Many of us work on multiple Python projects at the same time. Multiple projects may depend on different versions of the same library. This is a problem. Even if you work with a single project and you deploy it to production, you may run into trouble, because the system's Python on your production server might change due to OS upgrade or security patch, and your application might fail as a result. In general, you want full control over the Python environment of your projects. Enter virtual environments...

The basic idea of a virtual environment is to have a Python interpreter and its site-packages separate from the system one. Also, you can have many of them. That solves both problems. You can assign a separate virtual environment with its own dependencies for each project and forget about conflicts with other projects and the system's Python.

In this tutorial, you'll learn the concepts behind virtual environments and how to create and use them, and you'll discover various alternatives for special situations.

Virtualenv

The virtualenv package supports this concept. You can install virtualenv using pip install virtualenv.

Once virtualenv is installed, you can start creating virtual environments. Let's create two environments called "venv_1" and "venv_2".

~ > virtualenv ~/venv_1

Using real prefix '/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7'

New python executable in /Users/gigi/venv_1/bin/python2.7

Also creating executable in /Users/gigi/venv_1/bin/python

Installing setuptools, pip, wheel...done.



~ > virtualenv ~/venv_2

Using real prefix '/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7'

New python executable in /Users/gigi/venv_2/bin/python2.7

Also creating executable in /Users/gigi/venv_2/bin/python

Installing setuptools, pip, wheel...done.

Let's see what happened.

~ > ls -la ~/venv_1

total 16

drwxr-xr-x    7 gigi  staff   238 Mar 29 23:12 .

drwxr-xr-x+ 254 gigi  staff  8636 Mar 29 23:12 ..

lrwxr-xr-x    1 gigi  staff    79 Mar 29 23:12 .Python -> /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/Python

drwxr-xr-x   16 gigi  staff   544 Mar 29 23:12 bin

drwxr-xr-x    3 gigi  staff   102 Mar 29 23:12 include

drwxr-xr-x    3 gigi  staff   102 Mar 29 23:12 lib

-rw-r--r--    1 gigi  staff    60 Mar 29 23:12 pip-selfcheck.json

Inside the "bin" sub-directory, you'll find a few executables and symlinks. Those include the Python interpreter itself, pip, easy_install, and most importantly a few activate scripts.

~ > ls -la ~/venv_1/bin

total 136

drwxr-xr-x  16 gigi  staff    544 Mar 29 23:12 .

drwxr-xr-x   7 gigi  staff    238 Mar 29 23:12 ..

-rw-r--r--   1 gigi  staff   2077 Mar 29 23:12 activate

-rw-r--r--   1 gigi  staff   1019 Mar 29 23:12 activate.csh

-rw-r--r--   1 gigi  staff   2217 Mar 29 23:12 activate.fish

-rw-r--r--   1 gigi  staff   1137 Mar 29 23:12 activate_this.py

-rwxr-xr-x   1 gigi  staff    249 Mar 29 23:12 easy_install

-rwxr-xr-x   1 gigi  staff    249 Mar 29 23:12 easy_install-2.7

-rwxr-xr-x   1 gigi  staff    221 Mar 29 23:12 pip

-rwxr-xr-x   1 gigi  staff    221 Mar 29 23:12 pip2

-rwxr-xr-x   1 gigi  staff    221 Mar 29 23:12 pip2.7

lrwxr-xr-x   1 gigi  staff      9 Mar 29 23:12 python -> python2.7

-rwxr-xr-x   1 gigi  staff   2336 Mar 29 23:12 python-config

lrwxr-xr-x   1 gigi  staff      9 Mar 29 23:12 python2 -> python2.7

-rwxr-xr-x   1 gigi  staff  12744 Mar 29 23:12 python2.7

-rwxr-xr-x   1 gigi  staff    228 Mar 29 23:12 wheel

The activate script is the key. In order to activate a specific virtual environment, you source the activate script, as in: source ~/venv_1/bin_activate. The effect is setting a bunch of environment variables and changing the prompt to the name of the activated virtual environment. It also adds a deactivate() shell function that will reset everything. Once a virtual environment is activated, typing python will launch its Python with its dependencies.

~ > source ~/venv_1/bin/activate

(venv_1) ~ > which python

/Users/gigi/venv_1/bin/python

(venv_1) ~ >

Let's deactivate:

(venv_1) ~ > deactivate

~ > which python

/usr/local/bin/python

If you have multiple Python interpreters installed on your systems, you can specify which one to use for your virtual environment using the -p option. Here is a Python 3 virtual environment:

~ > virtualenv ~/venv_3 -p /usr/local/bin/python3

Running virtualenv with interpreter /usr/local/bin/python3

Using base prefix '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5'

New python executable in /Users/gigi/venv_3/bin/python3.5

Also creating executable in /Users/gigi/venv_3/bin/python

Installing setuptools, pip...done.



~ > source ~/venv_3/bin/activate

(venv_3)~ > python

Python 3.5.1 (default, Jan  9 2016, 19:28:52)

[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>>

Virtualenv works even on pypy.

~ > virtualenv ~/venv_pypy -p `which pypy`

Running virtualenv with interpreter /usr/local/bin/pypy

New pypy executable in /Users/gigi/venv_pypy/bin/pypy

Installing setuptools, pip...done.

~ > source ~/venv_pypy/bin/activate

(venv_pypy)~ > python

Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Feb 11 2016, 20:39:39)

[PyPy 4.0.1 with GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>>>

You can switch directly from one environment to the other without deactivating first:

(venv_3)~ > source ~/venv_2/bin/activate

(venv_2) ~ > which python

/Users/gigi/venv_2/bin/python

OK. Let's see how to use two different versions of the same package in two different virtual environments. This is as simple as activating each environment and installing the desired version. The environments are totally independent, so the fact that there is a different version in another environment is a non-issue.

Let's install the sh package version 1.0.0 to "venv_1".

(venv_1) ~ > pip install sh==1.0

Collecting sh==1.0.0

  Downloading sh-1.0.tar.gz

Building wheels for collected packages: sh

  Running setup.py bdist_wheel for sh ... done

  Stored in directory: /Users/gigi/Library/Caches/pip/wheels/f9/fb/a1/383f6dc2834b319a788a006d3ab7cc014ee852485f00b9e8c3

Successfully built sh

Installing collected packages: sh

Successfully installed sh-1.0



(venv_1) ~ > pip freeze | grep sh

sh==1.0

Let's switch to "venv_2" and install version 1.11.

(venv_1) ~ > source ~/venv_2/bin/activate

(venv_2) ~ > pip install sh==1.11

Collecting sh==1.11

  Downloading sh-1.11.tar.gz

Building wheels for collected packages: sh

  Running setup.py bdist_wheel for sh ... done

  Stored in directory: /Users/gigi/Library/Caches/pip/wheels/ba/4f/a5/ec77d662c3bf38f564b5ab16f1f3dbb9575922826fe810961c

Successfully built sh

Installing collected packages: sh

Successfully installed sh-1.11

(venv_2) ~ > pip freeze | grep sh

sh==1.11

Now, let's switch back to "venv_1" and verify that its version of the sh package is still 1.0.

(venv_2) ~ > source ~/venv_1/bin/activate

(venv_1) ~ > pip freeze | grep sh

sh==1.0

(venv_1) ~ >

Virtualenvwrapper

All that activating, deactivating and switching can get old after a while. If you manage a lot of virtual environments, it can get out of control. That's where virtualenvwrapper comes in. Virtualenvwrapper lets you list, create, delete and copy virtual environments. It also lets you switch environments easily. 

Here are all the commands:

~ > virtualenvwrapper



virtualenvwrapper is a set of extensions to Ian Bicking's virtualenv

tool.  The extensions include wrappers for creating and deleting

virtual environments and otherwise managing your development workflow,

making it easier to work on more than one project at a time without

introducing conflicts in their dependencies.



For more information please refer to the documentation:



    https://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html



Commands available:



  add2virtualenv: add directory to the import path



  allvirtualenv: run a command in all virtualenvs



  cdproject: change directory to the active project



  cdsitepackages: change to the site-packages directory



  cdvirtualenv: change to the $VIRTUAL_ENV directory



  cpvirtualenv: duplicate the named virtualenv to make a new one



  lssitepackages: list contents of the site-packages directory



  lsvirtualenv: list virtualenvs



  mkproject: create a new project directory and its associated virtualenv



  mktmpenv: create a temporary virtualenv



  mkvirtualenv: Create a new virtualenv in $WORKON_HOME



  rmvirtualenv: Remove a virtualenv



  setvirtualenvproject: associate a project directory with a virtualenv



  showvirtualenv: show details of a single virtualenv



  toggleglobalsitepackages: turn access to global site-packages on/off



  virtualenvwrapper: show this help message



  wipeenv: remove all packages installed in the current virtualenv



  workon: list or change working virtualenvs

I use pretty much two commands: mkvirtualenv and workon. All the virtual environments are created under ~/.virtualenvironments.

Here is how to create a new virtual environment:

~ > mkvirtualenv venv

New python executable in venv/bin/python2.7

Also creating executable in venv/bin/python

Installing setuptools, pip...done.



(venv)~ >

This is similar to virtualenv, but you don't specify a directory, just a name. Your new environment is here:

(venv)~ > tree -L 2 ~/.virtualenvs/venv/

/Users/gigi/.virtualenvs/venv/

├── bin

│   ├── activate

│   ├── activate.csh

│   ├── activate.fish

│   ├── activate_this.py

│   ├── easy_install

│   ├── easy_install-2.7

│   ├── get_env_details

│   ├── pip

│   ├── pip2

│   ├── pip2.7

│   ├── postactivate

│   ├── postdeactivate

│   ├── preactivate

│   ├── predeactivate

│   ├── python -> python2.7

│   ├── python2 -> python2.7

│   └── python2.7

├── include

│   └── python2.7 -> /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/include/python2.7

└── lib

    └── python2.7

To switch between environments, you use the workon command, which without arguments just lists all the virtual environments. I have quite a few:

(venv)~ > workon

acme_server

conman

curr-gen

nupic

over-achiever

pandas

prime_hunter

pypy

quote-service

venv

work



(venv)~ > workon conman

(conman) ~ >

Virtualenv-Burrito

Virtualenv-Burrito is a project to install both virtualenv and virtualenvwrapper in one command.

Python 3 Venv

The venv module was added to Python 3.3 and provides built-in virtual environment creation and management just like virtualenv. The command to create virtual environments is pyenv. Otherwise it is pretty similar to virtualenv.

Conda

Virtual environments are very useful for isolating dependencies between different projects. But that doesn't extend to native libraries. Many C extensions depend on particular versions of native libraries, and those can't be virtual environment specific. 

Conda can address this problem. It is a package management system that handles all dependencies, not just Python dependencies. It can even be used for packaging non-Python software.

Alternatives to Virtual Environments

Do you have to use virtual environments? Not really. If for some reason you're not fond of the magic of virtual environments, there are other options.

Manually Vendorize

The functionality of a virtual environment is pretty simple. If you need total control, you can just do it yourself and copy exactly the subset of tools and packages you want into a target directory structure, set up some environment variables, and you're good to go.

VM or Dockerized System Python

If you bake your applications into a docker container or a cloud image then there will be just one project, and you may not need a virtual environment in the middle. You can just build on top of the system Python, being sure it will not be changed.

Tox

If all you care about is testing your code under different interpreters and environments then Tox can do all the heavy lifting for you. It will still use virtual environments under the covers, but you don't have to deal with them.

Best Practices

There are some packaging and virtual environment best practices that have emerged over time for robust Python systems.

Pin Versions in Your Requirements Files

Pinning means specifying precisely the versions of your dependencies. If a new version comes out and you install your application on a new server, you'll still use the version you tested against and that works, and not the latest and greatest. There is a downside here—you'll have to explicitly upgrade versions if you want to keep up with the progress made by your dependencies—but it is usually worth it.

Never Use the System Python

Relying on the system version is bad practice because there are other tools that rely on it, and if you start upgrading system packages, you may break them. You may also be affected by security updates that modify system packages, or in general if you want to upgrade your OS you may find that the system Python is now different.

Use a Private Package Index When Baking Images

PyPI may be down. If you need to bake a new image and can't access PyPI, you're in trouble. Devpi is a good option to avoid frustration.

Conclusion

There are many options for managing multiple Python projects on the same machine without conflicts. Figure out which option is best for you and use it. It is fast and easy to create virtual environments. Don't hesitate to take advantage of this useful tool. If you have special requirements, there are many solutions too.