The open blogging platform. Say no to algorithms and paywalls.

Unit Testing in Python — tox and nox

Develop Fast Test Thoroughly

Image derived by Martin Thoma from Andrea Caprotti (nox project)Image derived by Martin Thoma from Andrea Caprotti (nox project)

When I started developing Python packages, there was one mistake I did quite often: I forgot to add all dependencies. Additionally, I only tested on my machine for a single Python version.

After reading this article, you will know how to locally and automatically test multiple Python versions in isolated environments. This is a preparation for Continuous Integration tools like Travis. I assume you already know the basics of unit testing in Python and how to package your code.


pyenv is a tool which lets you easily install and switch Python environments on your system. Take a look at the official installation instructions; it's not a Python package but hooks directly into your shell.

Once it is installed, you can get a list of all available Python environments. In July 2020, there were 427 different versions! I've shortened them here to show you the ones I think are interesting:

$ pyenv install --list
Available versions:

Install a Python version:

$ pyenv install 3.8.4
Downloading Python-3.8.4.tar.xz...
-> [](
Installing Python-3.8.4...
Installed Python-3.8.4 to /home/moose/.pyenv/versions/3.8.4

And use it:

$ pyenv local 3.8.4

$ python --version
Python 3.8.4

$ pip --version
pip 20.1.1 from /home/moose/.pyenv/versions/3.8.4/lib/python3.8/site-packages/pip (python 3.8)

Virtual environment basics

A virtual environment encapsulates the installed packages. Different virtual environments still share the same operating system, the same installed c libraries and executables. The only difference is which packages are available.

You can create a new virtual environment called venv-tutorial like this:

python -m venv venv-tutorial

It creates a folder which contains all installed packages and few other things. To use it, you need to activate it:

source venv-tutorial/bin/activate

This will add the prefix (venv-tutorial) in front of your prompt. It will make sure that later calls to python and pip use this environment. If you want to stop it again, type deactivate in the shell. If you delete this folder, the virtual environment is gone.

Testing multiple Python versions without tox

When you claim that your project supports Python 2.7 and 3.5 to 3.8, then you better test those versions. In order to make sure that you install the packages properly, you should create a virtual environment. You might end up with creating a shell script which creates those virtual environments, starts the tests and deletes the virtual environments again.

How to use tox

tox uses a tox.ini file which is in the package root directory. So your project structure might look like this:

your-awesome-project/            # The git repository
├──                     # Dependencies / Package Meta data
├── your_awesome_package/        # Code of the package
│   ├──
│   └──
├── tests/                       # Unit tests
│   ├──
│   └──
└── tox.ini                      # Why you're reading this article

The tox.ini file to run the tests in an isolated Python environment for Python 3.6, Python 3.7 and Python 3.8 looks like this:

envlist = py36,py37,py38

deps =
    -r requirements-dev.txt
commands =
    pip install -e .[all]
    pytest .

Note that you need to have the different Python versions already installed. You can do this with pyenv and make them available with the following command:

$ pyenv local 3.8.4 3.7.8 3.6.11

Run tox within the root folder — the same folder that contains your tox.ini file.

The next thing you might want to do is to break a couple of things out of the pytest run. For example, it's not necessary to run the linter flake8 and blackin every single environment. Instead, you can define a linter environment which is run once:

envlist = linter,py36,py37,py38

deps =
    -r requirements-dev.txt
commands =
    pip install -e .[all]
    pytest .

deps =
commands =
    black --check .

And finally, you want to run the different environments in parallel for speed:

tox -p

If one of them fails, you get this type of output:

Run tox in parallel, showing a linter issue. This output is way cleaner than if you had run flake8 and black via pytest. It might be a tiny bit faster, but that difference is not relevant. Screenshot taken by Martin ThomaRun tox in parallel, showing a linter issue. This output is way cleaner than if you had run flake8 and black via pytest. It might be a tiny bit faster, but that difference is not relevant. Screenshot taken by Martin Thoma

If you want to run just the linter, tox -e linter is your friend 🙂

You can also create a matrix of different combinations of Python environments and dependencies you install. Thea Flowers showed this at PyCon 2019 (video), but I never had the need to go down that rabbit hole.

Now… what is nox?

nox is a spin-off of tox. Instead of using a tox.ini configuration file, it uses a Python file. It's pretty similar to tox, but more flexible as it uses Python code:

# Third party
import nox

@nox.session(python=["3.6", "3.7", "3.8"])
def test(session):
    session.install("-r", "requirements-dev.txt")"pytest")

def lint(session):
    session.install("-r", "requirements-lint.txt")"flake8", ".")"black", ".", "--check")

You can run a single session of nox with nox -s lint:

Screenshot taken by Martin ThomaScreenshot taken by Martin Thoma

Overall, not a huge difference. The output of nox is way nicer and for people who get started, I think the Python code written with nox is a bit simpler to read and understand than the tox.ini configuration file.

What's next?

In this series, we already had:

Let me know if you're interested in other topics around testing with Python.

Continue Learning