11 minute read

pytest-fixtures-notes-examples
Image Source: Pexels

Introduction

I’ve been utilizing pytest for quite some time now, and I find concepts such as patching, fixtures, and parametrization quite intuitive. Nonetheless, at one point, I started thinking about the architectural aspects of the entire testing module and how they influence the actual test code. Improving performance can be achieved by scoping fixtures, particularly when dealing with fixtures that involve network access. However, this approach raises questions about multiple tests accessing the same fixture instance and the teardown process. Not to mention complexities that always arise when introducing test parallelization, for example using pytest-xdist.

The answers to most of these questions are strongly tied to a specific use case, and having a deep understanding of all the ways we can utilize fixtures can lead to better architectural decisions.

This post is a collection of notes and examples that I’ve gathered during my research, mainly serving as a reference. The documentation on fixtures is very extensive and covers all the details, so be sure to check it out.

If you possess some familiarity with pytest, I hope that this post will inspire you to delve further into the specifics and make architectural choices that can enhance your project and simplify your work.

Features

Autouse

As we are already know, we can request a fixture by specifying it in a test’s signature. For example:

import pytest


@pytest.fixture
def empty_list():
    return []


def test_empty_list(empty_list):
    assert len(empty_list) == 0

The signature of test_empty_list(empty_list) requests fixture empty_list. Of course, we can request as many fixtures as we want by specifying them next to each other in the signature.

The autouse fixtures are a convenient way to make all tests automatically request them without specifying them in the signature explicitly. This eliminates unnecessary calls and reduces the clutter in the signature.

To create an autouse fixture, just pass the autouse=True to the fixture’s decorator:

import pytest


@pytest.fixture
def value():
    return []


@pytest.fixture(autouse=True)
def empty_list(value):
    return value.append(1)


def test_empty_list(value):
    assert len(value) == 1

This is useful when we need to apply a side-effect before and/or after each test unconditionally.

Scope

The scope of a fixture defines the level on which it’s being invoked and destroyed. Fixtures are created when first requested by a test, and are destroyed based on their scope, which can take the following values:

  • function: the default scope, the fixture is destroyed at the end of the test.
  • class: the fixture is destroyed during teardown of the last test in the class.
  • module: the fixture is destroyed during teardown of the last test in the module.
  • package: the fixture is destroyed during teardown of the last test in the package.
  • session: the fixture is destroyed at the end of the test session.

This is beneficial for generating resources that typically involve substantial time and effort in their creation, such as HTTP servers, Docker containers, and so on. In that case we can specify the scope to be session and the same instance will persist for all tests. Multiple tests will receive the same fixture instance, which saves time and increases the performance.

Fixtures have to adhere to the scope hierarchy. This means that a fixture of higher scope cannot depend on a fixture of lower scope.

For example, let’s say that we create a value on a session level, and then use a fixture that modifies it:

import pytest


@pytest.fixture(scope="session")
def value():
    return []


@pytest.fixture(scope="function")
def add(value):
    return value.append(1)


def test_list(value, add):
    assert len(value) == 1

This is fine, since add is scoped on a function level, while value is of higher scope. If you switch the scopes, you’ll get ScopeMismatch stating:

ScopeMismatch: You tried to access the 'function' scoped fixture 'value' 
with a 'session' scoped request object, involved factories

Yield

The return from a fixture can be swapped with yield, which still allows us to run some code and pass an object to requesting fixture or test.

However, the difference is that any code after yield will get executed. It’s important to note that the order in which the yield fixtures are executed is linear. In setup it’s going top to bottom and in reverse order during teardown. Next example illustrates this:

import pytest


@pytest.fixture()
def a():
    print("starting a")
    yield
    print("exiting a")


@pytest.fixture()
def b(a):
    print("starting b")
    yield
    print("exiting b")


def test(b):
    print("executing test...")

Executing this with appropriate flags for output gives the following:

starting a
starting b
executing test...
exiting b
exiting a

Setup/Teardown

Having in mind the autouse, scope, and yield, we can construct a fixture that will run before or after each or all tests depending on the scope. This can be useful in many ways. For example, creating a table, setting up an http server before all tests but also cleaning the state before and after each test. What’s important is that by using the yield we can define the teardown process in each fixture.

The following code illustrates the process:

import pytest


@pytest.fixture(autouse=True, scope="module")
def a():
    print("Setup before all tests")
    yield
    print("Teardown after all tests")


@pytest.fixture(autouse=True)
def b():
    print("Before each test")
    yield
    print("After each test")


def test_1():
    print("Executing test 1...")


def test_2():
    print("Executing test 2...")

Executing this gives:

Setup before all tests
Before each test
Executing test 1...
After each test
Before each test
Executing test 2...
After each test
Teardown after all tests

Having the right order of fixtures and teardowns doesn’t guarantee a safe teardown. The documentation goes into details on this topic, so be sure to check Safe teardowns. In summary, it’s advisable not to define extensive setup and teardown procedures that include numerous state changes, as a failure in one of these steps can disrupt the execution of others. This can lead to complications, as when a setup step fails, none of the teardown code will be executed. Instead, we should strive to create fixtures in a limited way, creating only one state-changing action. Doing so, we are lowering the chances of leaving the resources hanging in the case when something fails.

Inspecting fixtures

Of course, in real life, we could have a lot of fixtures with different scopes and complicated orders, therefore a simple print won’t work. Getting the grasp of the fixture setup can be done using the --setup-only flag, like this:

pytest --setup-only pytest_test.py::test_1

This will give us an idea of what’s going on. Using the previous example, we get:

pytest_test.py 
    SETUP    M a
        SETUP    F b
        pytest_test.py::test_1 (fixtures used: a, b)
        TEARDOWN F b
    TEARDOWN M a

Besides the SETUP and TEARDOWN, we can see that M and F give us the information about the scope, module and function respectively.

If we just want the list of fixtures used for a particular test, we can use the --fixtures-per-test flag.

The --fixtures flag will output all available custom and built-in fixtures defined in our testing module.

Fixture based on arguments

We can create a fixture whose returned value depends on a parameter by returning a function that generates the data, rather than returning the data directly.

import pytest


@pytest.fixture
def generate_list():
    def _generate_list(*args):
        return list(args)

    return _generate_list


def test_customer_records(generate_list):
    list_1 = generate_list(1, 2, 3)
    list_2 = generate_list("a", "b", "c")
    print(list_1)
    print(list_2)

The output:

[1, 2, 3]
['a', 'b', 'c']

The fixture is instantiated when it’s first requested by the test. However, in this case, it returns a function which allows us to have a dynamic behavior based on input arguments.

Combining this approach with the use of yield, we have the capability to define a dynamic fixture that generates return values based on arguments while also retaining them for future teardown.

import pytest


@pytest.fixture
def generate_list():
    generated_lists = []

    def _generate_list(*args):
        result = list(args)
        generated_lists.append(result)
        return result

    yield _generate_list

    print(generated_lists)


def test_customer_records(generate_list):
    list_1 = generate_list(1, 2, 3)
    list_2 = generate_list("a", "b", "c")
    print(list_1)
    print(list_2)

which gives:

[1, 2, 3]
['a', 'b', 'c']
[[1, 2, 3], ['a', 'b', 'c']]

Design

When we are designing a testing module, we should think about the definition and availability of fixtures. Questions like:

  • What should be the scope of this fixture?
  • Do I need to teardown something?
  • When do I need to set it up and tear it down?
  • Should it be dynamic based on the arguments?

should be answered before we start building up the architecture of fixtures.

The file structure holds significance, and a particular file named conftest.py fulfills a unique purpose. Fixtures defined within the conftest.py can be accessed by any test within the same package without requiring explicit imports.

Moreover, we can have a nested directories/packages containing tests and each directory can have a separate conftest.py with its own fixtures. Note that child conftest.py also extends parent. This allows us to build a clear and concise fixture structure.

Let’s imagine that we have the following file structure:

tests/
    conftest.py -> fixture: top
    service_a/
        conftest.py -> fixture: fixture_service_a
        test_service_a_1.py -> fixture: fixture_a_1
        test_service_a_2.py -> fixture: fixture_a_2
    service_b/
        conftest.py -> fixture: fixture_service_b
        test_service_b_1.py -> fixture: fixture_b_1
        test_service_b_2.py -> fixture: fixture_b_2

The following diagram could be used to illustrate this structure:

pytest-structure-diagram
Depiction of fixtures' file structure

Note that we are talking about fixture availability and file structure, this has nothing to do with order which is defined using scope and hierarchy.

Let’s look at some examples, it’ll provide greater clarity.

Examples

Docker

If we are building a service that interacts with an SFTP server and/or a database, for example. We can run a docker container in the background and simulate the interaction. This allows us to have the proper testing structure of defining the state, action, and asserting the result.

Running a Docker container and configuring its state for each individual test can consume a significant amount of time. Thus, it is crucial to effectively manage a fixture with the appropriate scope.

We want to dynamically get an unused port that will be used for interacting with the service inside docker. This could be achieved with the following fixture:

import time
import socket
import pytest
import docker


@pytest.fixture(scope="session")
def unused_port():
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
        sock.bind(("localhost", 0))
        return sock.getsockname()[1]

This fixture is then used to run a docker container in another fixture:

@pytest.fixture(scope="session")
def run_docker(unused_port):
    client = docker.from_env()
    container = client.containers.run(
        image="<image>",
        auto_remove=True,
        name="<container-name>",
        ports={"<port>": unused_port},
        detach=True,
        remove=True,
        command="<command>",
    )

    time.sleep(5)

    yield

    container.stop()

Here, we are using time.sleep(5) to wait for a docker container to star. However, in some real-life scenario, you would define some kind of check to confirm that the instance is up and running, for example by making backoff API calls as health checks.

The key point to emphasize is that we are executing a Docker container instance within the session scope using yield, and subsequently, we ensure that the container is stopped.

SFTP

Now, let’s observe this in practice by employing an SFTP server within a Docker container. There’s an easy to use SFTP server with OpenSSH as a pre-built docker image: emberstack/sftp. To interact with it, let’s use paramiko.

We can run the SFTP server with the following fixture:

@pytest.fixture(scope="session", autouse=True)
def run_docker(unused_port):
    client = docker.from_env()
    container = client.containers.run(
        image="emberstack/sftp",
        auto_remove=True,
        name="sftp",
        ports={"22": unused_port},
        detach=True,
        remove=True,
    )

    time.sleep(5)

    yield unused_port

    container.stop()

Please be aware that the use of time.sleep(5) should be enhanced and replaced with a backoff mechanism for SFTP connection initialization. I will just leave it as is for now, as it holds no significance within the scope of this discussion.

Note that we are using autouse=True, which means that the run_docker fixture will create a docker container for the whole testing session and we don’t have to request it in each test.

Having an SFTP server doesn’t mean much if we don’t connect to it. The connection can also be established by utilizing a yield fixture within a with context manager, providing a more straightforward method for handling the connection’s closure.

@pytest.fixture(scope="session")
def sftp_client(unused_port):
    with paramiko.Transport(("localhost", unused_port)) as transport:
        transport.connect(username=USERNAME, password=PASSWORD)
        with paramiko.SFTPClient.from_transport(transport) as client:
            yield client

Finally, we pass the client fixture to a test:

def test_mkdir(sftp_client):
    assert len(sftp_client.listdir()) == 0
    sftp_client.mkdir("example")
    assert sftp_client.listdir() == ["example"]

The sftp_client.mkdir() can be designated as an independent fixture that generates directories according to the provided input parameter and maintains a record of them for cleanup. This can be demonstrated as follows:

@pytest.fixture()
def mkdir(sftp_client):
    paths = []

    def _mkdir(name):
        paths.append(name)
        sftp_client.mkdir(name)

    yield _mkdir

    for path in paths:
        sftp_client.rmdir(path)

Please take note that the fixture’s scope is limited to the function level, which implies that the state will be reset between different tests. To illustrate this, we can create two analogous tests:

def test_mkdir_1(sftp_client, mkdir):
    assert len(sftp_client.listdir()) == 0
    mkdir("a")
    mkdir("b")
    assert sftp_client.listdir() == ["b", "a"]


def test_mkdir_2(sftp_client, mkdir):
    assert len(sftp_client.listdir()) == 0
    mkdir("c")
    mkdir("d")
    assert sftp_client.listdir() == ["d", "c"]

which runs without any errors. The same should be done for other SFTP actions.

Inspecting the order and hierarchy of fixtures using --setup-only, we get:

SETUP    S unused_port
SETUP    S run_docker (fixtures used: unused_port)
SETUP    S sftp_client (fixtures used: unused_port)
        SETUP    F mkdir (fixtures used: sftp_client)
        pytest_test.py::test_mkdir_1 (fixtures used: mkdir, run_docker, sftp_client, unused_port)
        TEARDOWN F mkdir
        SETUP    F mkdir (fixtures used: sftp_client)
        pytest_test.py::test_mkdir_2 (fixtures used: mkdir, run_docker, sftp_client, unused_port)
        TEARDOWN F mkdir
TEARDOWN S sftp_client
TEARDOWN S run_docker
TEARDOWN S unused_port

With any luck, this provides greater clarity, and I hope it’ll inspire you to embark on your own journey of exploration and enhance the design of your testing module.

Resources

Updated:

Leave a comment