Python - Understanding and demistifying virtualenv

Introduction

With Python, when installing a product, for example Alerta server, many dependencies can also be installed.

python@vpsfrsqlpac2$ pip3 search alerta
alerta (7.4.0)              - Alerta unified command-line tool and SDK
alerta-server (7.4.1)       - Alerta server WSGI application
...

python@vpsfrsqlpac2$ pip3 install alerta-server
Installing collected packages: pymongo, certifi, urllib3, chardet, idna, requests,
MarkupSafe, Jinja2, Werkzeug, itsdangerous, click, Flask, blinker, sentry-sdk, Flask-Compress,
pyyaml, six, python-dateutil, pycparser, cffi, bcrypt, cryptography, pyparsing, Flask-Cors,
PyJWT, pytz, alerta-server
  Running setup.py install for MarkupSafe ... done
  Running setup.py install for blinker ... done
  Running setup.py install for Flask-Compress ... done
  Running setup.py install for pyyaml ... done
  Running setup.py install for pycparser ... done
Successfully installed Flask-1.1.1 Flask-Compress-1.4.0 Flask-Cors-3.0.8 Jinja2-2.10.3
MarkupSafe-1.1.1 PyJWT-1.7.1 Werkzeug-0.16.0 alerta-server-7.4.1 bcrypt-3.1.7 blinker-1.4
certifi-2019.11.28 cffi-1.13.2 chardet-3.0.4 click-7.0 cryptography-2.8 idna-2.8
itsdangerous-1.1.0 pycparser-2.19 pymongo-3.10.1 pyparsing-2.4.6 python-dateutil-2.8.1
pytz-2019.3 pyyaml-5.3 requests-2.22.0 sentry-sdk-0.14.0 six-1.13.0 urllib3-1.25.7

It becomes tedious to manage polluting the Python global distribution : dependencies, package versions conflicts, binaries… Further more some packages may be needed only for one user. By default all packages and dependencies are downloaded and installed in $PYTHON_HOME/lib/python<version>/site-packages.

The Python package virtualenv solves this kind of issue. A user can manage and run its own isolated environment without any package installation in the global Python distribution. The virtual environment is indenpendent of the Python source distribution, however an option allows the use of the packages installed in the source distribution.

In the schema below :

The Python virtual environment py-influxdb is completely isolated with 2 packages installed.
In the Python virtual environment py-alerta for the user alerta, alerta packages are installed, but this environment can also use the packages installed in the system distribution (PyMySQL, psycopg2, pymongo).

So, the packages used by the most users can be installed in the system distribution, and packages needed to only one user/product in a virtual environment using virtualenv.

Let’s see how to create and use virtual environments, and how packages and versions are managed in virtual and system distributions.

Context

In this article, Python 3.8 has been compiled and installed in the custom directory /opt/python/python-3.8. Our philosophy : no installation in the system directories (/usr…).

The user python (group: wapp) is the owner of the system distribution /opt/python/python-3.8.

Python 3.8 environment is setup by sourcing the file $HOME/.python-3.8 below.

$HOME/.python-3.8

#!/bin/bash
export PYHOME=/opt/python/python-3.8
export PATH=$PYHOME/bin:$PATH
export LD_LIBRARY_PATH=$PYHOME/lib:$LD_LIBRARY_PATH
export PYTHONPATH=/opt/python/packages

export PGLIB=/opt/postgres/pgsql-11/lib
export LD_LIBRARY_PATH=$PGLIB:$LD_LIBRARY_PATH

python@vpsfrsqlpac2$ . $HOME/.python-3.8
python@vpsfrsqlpac2$ which python3
python@vpsfrsqlpac2$ which pip3

/opt/python/python-3.8/bin/python3
/opt/python/python-3.8/bin/pip3

The custom environment variable $PYHOME specifies the Python 3.8 root installation directory (/opt/python/python-3.8).

virtualenv installation (optional)

2 methods to create virtual environments :

virtualenv (PyPI package).
python3 -m venv (native mode).

virtualenv is verbose when creating a virtual environment compared to the native mode.

The native method allows several virtual environments creations at the same time.

To install the package virtualenv in the global distribution with pip3 :

python@vpsfrsqlpac2$ pip3 install virtualenv

Collecting virtualenv
  Downloading virtualenv-16.7.9-py2.py3-none-any.whl (3.4 MB)
     |..............................| 3.4 MB 4.3 MB/s
Installing collected packages: virtualenv
Successfully installed virtualenv-16.7.9

virtualenv is then available in $PYHOME/bin.

Creating and using virtual environments

Create a repository where all virtual environments will be installed. It will be a simple directory created outside python distributions :

python@vpsfrsqlpac2$ cd /opt/python
python@vpsfrsqlpac2$ mkdir virtualenvs

The environment variable $PYVENV identifies this directory.

python@vpsfrsqlpac2$ export PYVENV=/opt/python/virtualenvs

Let’s create the virtual environment py-alerta for the user alerta

Using virtualenv :

python@vpsfrsqlpac2$ virtualenv $PYVENV/py-alerta

Using base prefix '/opt/python/python-3.8'
New python executable in /opt/python/virtualenvs/py-alerta/bin/python3.8
Also creating executable in /opt/python/virtualenvs/py-alerta/bin/python
Installing setuptools, pip, wheel...
done.

Native mode :
```
python@vpsfrsqlpac2$ python3 -m venv --copies $PYVENV/py-alerta
```
The --copies option is used to copy the Python binaries and libraries into the virtual environment : by avoiding symbolic links, future python version upgrades will be easier (topic not covered in this paper).

Python interpreter (python3…) is copied in the directory $PYVENV/py-alerta/bin. The directory $PYVENV/py-alerta/lib/python3.8 is also prepared from $PYHOME/lib/python3.8 for packages installation then the pip and setuptools packages are copied there. The command can be run multiple times, it does not alter already existing packages unless the option --clear is used, only binaries and source scripts are overwritten (python3, pip3, activate…)

Just run the script $PYVENV/py-alerta/bin/activate to use the virtual environment :

python@vpsfrsqlpac2$ source $PYVENV/py-alerta/bin/activate
          
(py-alerta) python@vpsfrsqlpac2:~$

The prompt is modified with the name of the virtual environment (prefix). The environment variable $PATH is modified accordingly :

(py-alerta) python@vpsfrsqlpac2:~$ echo $PATH
/opt/python/virtualenvs/py-alerta/bin:/opt/python/python-3.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Default python3 and pip3 executables are the virtual environment ones.

(py-alerta) python@vpsfrsqlpac2:~$ which python3
(py-alerta) python@vpsfrsqlpac2:~$ which pip3
        
/opt/python/virtualenvs/py-alerta/bin/python3
/opt/python/virtualenvs/py-alerta/bin/pip3

Using Python, the path is modified accordingly too for packages installation paths :

(py-alerta) python@vpsfrsqlpac2:~$ python3

import sys
sys.path
['', '/opt/python/packages', '/opt/python/virtualenvs/py-alerta/lib/python38.zip',
'/opt/python/virtualenvs/py-alerta/lib/python3.8', '/opt/python/virtualenvs/py-alerta/lib/python3.8/lib-dynload',
'/opt/python/python-3.8/lib/python3.8', '/opt/python/virtualenvs/py-alerta/lib/python3.8/site-packages']

In the virtual environment, packages will be installed in the directory /opt/python/virtualenvs/py-alerta/lib/python3.8/site-packages and binaries in the directory /opt/python/virtualenvs/py-alerta/bin.

Example : installing the package chardet in the virtual environment py-alerta.

(py-alerta) python@vpsfrsqlpac2:~$ pip3 install chardet
Collecting chardet
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Installing collected packages: chardet
Successfully installed chardet-3.0.4
(py-alerta) python@vpsfrsqlpac2:~$ pip3 show chardet
Name: chardet
Version: 3.0.4
Summary: Universal encoding detector for Python 2 and 3
Home-page: https://github.com/chardet/chardet
Author: Daniel Blanchard
Author-email: dan.blanchard@gmail.com
License: LGPL
Location: /opt/python/virtualenvs/py-alerta/lib/python3.8/site-packages
Requires:
Required-by:

Virtual environments do not alter any existing definition in the environment variable $PYTHONPATH.

To quit the virtual environment, run the function deactivate, function created when running the script activate.

(py-alerta) python@vpsfrsqlpac2:~$ deactivate
        
python@vpsfrsqlpac2:~$

The environment is then back to the global distribution Python 3.8.

python@vpsfrsqlpac2:~$ which python3
        
/opt/python/python-3.8/bin/python3

Virtual environments and system packages

What about a package already installed in the system distribution ? For example, PyMySQL is installed to be available by default :

python@vpsfrsqlpac2:~$ pip3 list
        
Package    Version
---------- -------
PyMySQL    0.9.3

Obviously, this package won’t be available in the virtual environment. Use the option --system-site-packages when creating the virtual environment to be able to use the packages installed in the global source distribution :

Using virtualenv :

python@vpsfrsqlpac2:~$ virtualenv --system-site-packages $PYVENV/py-alerta

Native mode :

python@vpsfrsqlpac2:~$ python3 -m venv --copies --system-site-packages $PYVENV/py-alerta

python@vpsfrsqlpac2:~$ source $PYVENV/py-alerta/bin/activate

(py-alerta) python@vpsfrsqlpac2:~$ pip3 list

Package    Version
---------- -------
PyMySQL    0.9.3

The directory $PYHOME/lib/python<version>/site-packages is added in the path with the option --system-site-packages :

(py-alerta) python@vpsfrsqlpac2:~$ python3

import sys
sys.path
['', '/opt/python/packages', '/opt/python/virtualenvs/py-alerta/lib/python38.zip', '/opt/python/virtualenvs/py-alerta/lib/python3.8',
'/opt/python/virtualenvs/py-alerta/lib/python3.8/lib-dynload', '/opt/python/python-3.8/lib/python3.8',
'/opt/python/virtualenvs/py-alerta/lib/python3.8/site-packages', '/opt/python/.local/lib/python3.8/site-packages',
'/opt/python/python-3.8/lib/python3.8/site-packages']

No worries about security, the virtual environment knows when it is not the owner of a package, trying to remove a system package from the virtual environment fails :

(py-alerta) python@vpsfrsqlpac2:~$ pip3 uninstall PyMySQL

Found existing installation: PyMySQL 0.9.3
Not uninstalling pymysql at /opt/python/python-3.8.1/lib/python3.8/site-packages, outside environment /opt/python/virtualenvs/py-alerta
Can't uninstall 'PyMySQL'. No files were found to uninstall.

About upgrades, it is slightly different. In the example below, the package chardet 3.0.0 is installed in the system repository :

(py-alerta) python@vpsfrsqlpac2:~$ pip3 list

Package    Version
---------- -------
chardet    3.0.0

The upgrade fails trying to remove the system package, but the upgrade installs the new version in the virtual environment :

(py-alerta) python@vpsfrsqlpac2:~$ pip3 install --upgrade chardet

Collecting chardet
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Installing collected packages: chardet
  Attempting uninstall: chardet
    Found existing installation: chardet 3.0.0
    Not uninstalling chardet at /opt/python/python-3.8/lib/python3.8/site-packages, outside environment /opt/python/virtualenvs/py-alerta
    Can't uninstall 'chardet'. No files were found to uninstall.
Successfully installed chardet-3.0.4

(py-alerta) python@vpsfrsqlpac2:~$ pip3 list

Package    Version
---------- -------
chardet    3.0.4

The virtual environment re-uses the system one when the package is removed from the virtual environment :

(py-alerta) python@vpsfrsqlpac2:~$ pip3 uninstall chardet

Found existing installation: chardet 3.0.4
Uninstalling chardet-3.0.4:
  Would remove:
    /opt/python/virtualenvs/py-alerta/bin/chardetect
    /opt/python/virtualenvs/py-alerta/lib/python3.8/site-packages/chardet-3.0.4.dist-info/*
    /opt/python/virtualenvs/py-alerta/lib/python3.8/site-packages/chardet/*
Proceed (y/n)? y
  Successfully uninstalled chardet-3.0.4

(py-alerta) python@vpsfrsqlpac2:~$ pip3 list

Package    Version
---------- -------
chardet    3.0.0

A specific version can be installed in the virtual environment and it takes precedence over the system version, the version can even be lower than the system version :

(py-alerta) python@vpsfrsqlpac2:~$ pip3 install chardet==3.0.2

Collecting chardet==3.0.2
  Downloading chardet-3.0.2-py2.py3-none-any.whl (133 kB)
     |...........................| 133 kB 5.0 MB/s
Installing collected packages: chardet
  Attempting uninstall: chardet
    Found existing installation: chardet 3.0.0
    Not uninstalling chardet at /opt/python/python-3.8/lib/python3.8/site-packages, outside environment /opt/python/virtualenvs/py-alerta
    Can't uninstall 'chardet'. No files were found to uninstall.
Successfully installed chardet-3.0.2

(py-alerta) python@vpsfrsqlpac2:~$ pip3 list

Package    Version
---------- -------
chardet    3.0.2

When a system package version does not meet requirements, the appropriate version is installed in the virtual environment : for example the package alerta-server needs chardet <3.1.0,>=3.0.2, but system version is 3.0.0.

(py-alerta) python@vpsfrsqlpac2:~$ pip3 install alerta-server
...
Collecting chardet<3.1.0,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
...
  Attempting uninstall: chardet
    Found existing installation: chardet 3.0.0
    Not uninstalling chardet at /opt/python/python-3.8/lib/python3.8/site-packages, outside environment /opt/python/virtualenvs/py-alerta
    Can't uninstall 'chardet'. No files were found to uninstall.
Successfully installed Flask-1.1.1 Flask-Compress-1.4.0 Flask-Cors-3.0.8 Jinja2-2.10.3 MarkupSafe-1.1.1 PyJWT-1.7.1 Werkzeug-0.16.0
alerta-server-7.4.1 bcrypt-3.1.7 blinker-1.4 certifi-2019.11.28 cffi-1.13.2 chardet-3.0.4 click-7.0 cryptography-2.8 idna-2.8
itsdangerous-1.1.0 pycparser-2.19 pymongo-3.10.1 pyparsing-2.4.6 python-dateutil-2.8.1 pytz-2019.3 pyyaml-5.3 requests-2.22.0
sentry-sdk-0.14.1 six-1.14.0 urllib3-1.25.8
(py-alerta) python@vpsfrsqlpac2:~$ pip3 list
Package    Version
---------- -------
chardet    3.0.4

Libraries compilation

Some packages need libraries compilation (*.so): psycopg2 (Python package for PostgreSQL).

No particular issue when compiling libraries in a virtual environment.

(py-alerta) python@vpsfrsqlpac2:$ export PATH=/opt/postgres/pgsql-11/bin:$PATH
        
(py-alerta) python@vpsfrsqlpac2:$ pip3 install psycopg2

Collecting psycopg2
  Using cached psycopg2-2.8.4.tar.gz (377 kB)
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py) ... done
  ...
Successfully built psycopg2
Installing collected packages: psycopg2
Successfully installed psycopg2-2.8.4

The library _psycopg.cpython-38-x86_64-linux-gnu.so is compiled in the directory $PYVENV/py-alerta/lib/python3.8/site-packages/psycopg2.

(py-alerta) python@vpsfrsqlpac2:$ python3

import psycopg2

Just need to be sure the environment variable $LD_LIBRARY_PATH contains the path to the library libpq.so.5 if not installed in the system directories (/usr). No difference compared to the installation procedure in a classic Python distribution.

export LD_LIBRARY_PATH=/opt/postgres/pgsql-11/lib:$LD_LIBRARY_PATH

Virtual Python environment for Alerta

Now let’s create a complete virtual environment for Alerta, the necessary packages are very numerous and we don’t want to install all of them in the global Python distribution. The option --system-site-packages is used, Alerta requires the package psycopg2.

python@vpsfrsqlpac2$ virtualenv --system-site-packages $PYVENV/py-alerta
        
python@vpsfrsqlpac2$ source $PYVENV/py-alerta/bin/activate
        
(py-alerta) python@vpsfrsqlpac2:~/virtualenvs$ pip3 install alerta-server
        
(py-alerta) python@vpsfrsqlpac2:~/virtualenvs$ pip3 list

Package         Version
--------------- ----------
alerta-server   7.4.1
bcrypt          3.1.7
blinker         1.4
certifi         2019.11.28 ...

The package "Alerta unified command-line tool and SDK" is also installed :

(py-alerta) python@vpsfrsqlpac2:~/virtualenvs$ pip3 install alerta
        
(py-alerta) python@vpsfrsqlpac2:~/virtualenvs$ pip3 list

Package         Version
--------------- ----------
alerta          7.4.0
alerta-server   7.4.1 ...

The executables alertad (Alerta daemon server) and alerta (Alerta command line) are installed in $PYVENV/py-alerta/bin.

The script $HOME/.profile of the user alerta is updated to activate the virtual environment py-alerta when executed :

$HOME/.profile (/opt/alerta/.profile)

if [ -f "/opt/python/.python-3.8" ] ; then
        . /opt/python/.python-3.8

        if [ -f $PYVENV/py-${USER}/bin/activate ] ; then
                source $PYVENV/py-$USER/bin/activate
        fi
fi

The command pip3 list shows the right informations about packages :

alerta@vpsfrsqlpac2$ . $HOME/.profile

(py-alerta) alerta@vpsfrsqlpac2$ pip3 list

Package         Version
--------------- ----------
alerta          7.4.0
alerta-server   7.4.1 ...

Everything is ready :

(py-alerta) alerta@vpsfrsqlpac2:~$ alertad run --port 20003 --host vpsfrsqlpac2
        
2020-01-24 16:50:15,311 werkzeug[1944]:  * Running on http://vpsfrsqlpac2:20003/ (Press CTRL+C to quit)

Conclusion

Most common packages (PyMySQL, psycopg2…) are installed in the system Python distribution.
Python virtual environments are used when "exotic" packages are needed for only very few users, this avoid polluting the system distribution with versions conflicts and discrepancies for other packages and applications.
Package versions issues can be solved using Python virtual environments.
Python virtual environments are very useful for testing dependencies installations and development purposes.