============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/runner/work/py-techguides/py-techguides/codebase
configfile: pyproject.toml
plugins: anyio-4.8.0
collecting ... collected 2 items
tests/test_increment.py::test_inc PASSED [ 50%]
tests/test_increment.py::test_inc_fail FAILED [100%]
=================================== FAILURES ===================================
________________________________ test_inc_fail _________________________________
def test_inc_fail():
> assert inc(3) == 5
E assert 4 == 5
E + where 4 = inc(3)
tests/test_increment.py:8: AssertionError
=========================== short test summary info ============================
FAILED tests/test_increment.py::test_inc_fail - assert 4 == 5
+ where 4 = inc(3)
========================= 1 failed, 1 passed in 0.06s ==========================pytest
Introduction
pytest emerges as a powerful, mature, and full-featured testing framework conducive to writing effective and readable tests. Its versatility from simple unit tests to complex functional testing scenarios has made it a choice tool for enterprise solutions.
Setting up pytest
You can install pytest using the following package managers:
pip install pytestconda install -c conda-forge pytestpixi add pytestRunning Tests
Multiple tests can be run via pytest by calling pytest <directory_path> (in which case all test files that pytest can find are run), or on a specific file via pytest <test_file_relative_path.py>.
Tests can also be run using the -k flag, which allows for running tests which contain names that match the given string expression (case-insensitive), which can include Python operators that use filenames, class names and function names as variables. The example below will run TestMyClass.test_something but not TestMyClass.test_method_simple.
pytest -k 'MyClass and not method'Features
pytest brings a set of features that facilitate effective testing practices. In the following sections, we’ll delve into some of them.
Assertions: At the core of pytest’s simplicity lies the
assertstatement, which serves as the fundamental mechanism for validating test conditions.Modular Fixtures: Fixtures in pytest provide a powerful way to manage test resources, enabling setup and teardown operations that are reusable and parametrizable.
Auto-Discovery: pytest automatically detects and executes test modules and functions, eliminating the need for manual test configuration.
Compatibility: pytest seamlessly integrates with other testing frameworks like
unittest, doctest andnose2, allowing you to run a variety of test suites without modification.Parametrization: pytest allows you to run the same test function on different inputs, promoting concise and maintainable test suites.
Assertions
The basic testing statement in Python is assert, which is also the primary assertion mechanism in pytest. This statement is used within test functions to verify that certain conditions hold true. pytest enhances the default Python assert by providing detailed introspection, which offers more informative error messages when an assertion fails.
In the example below the first of the two tests test_inc() passes, while the second test_inc_fail() fails, which is expected based on the arithmetics of the inc() function:
# tests/test_increment.py
def inc(x):
return x + 1
def test_inc():
assert inc(3) == 4
def test_inc_fail():
assert inc(3) == 5pytest encourages the use of plain assert statements for simple checks as a means to introspect the assert statement to provide detailed error messages, although it’s important to avoid “abusing” assert. Assertions should in fact be used uniquely for testing and debugging purposes, and not for handling errors in production code. In other words, assertions should check for conditions that should never occur if the code is working correctly. We’ll see a number of examples in accordance with this as well as a couple of no-gos:
# tests/test_area.py
import math
def calculate_circle_area(radius):
return math.pi * radius ** 2
def test_circle_area():
assert math.isclose(calculate_circle_area(2), 12.56, rel_tol=1e-2)============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/runner/work/py-techguides/py-techguides/codebase
configfile: pyproject.toml
plugins: anyio-4.8.0
collecting ... collected 1 item
tests/test_area.py::test_circle_area PASSED [100%]
============================== 1 passed in 0.01s ===============================In this test, we use math.isclose() to compare the calculated area with the expected value, accounting for potential floating-point inaccuracies. For more complex conditions though, it is recommended to use helper functions or custom assertions to improve readability and maintainability. This is especially true for data validation and/or critical checks.
import statistics
def is_valid_dataset(data):
return len(data) > 0 and statistics.mean(data) > 0 and statistics.stdev(data) < 10
def test_complex_condition():
dataset = [1, 2, 3, 4, 5]
assert is_valid_dataset(dataset), f"Dataset {dataset} is not valid"A worse way to write the same test is:
def test_complex_condition():
dataset = [1, 2, 3, 4, 5]
assert len(dataset) > 0 and statistics.mean(dataset) > 0 and statistics.stdev(dataset) < 10, f"Dataset {dataset} is not valid"Assertions should not be used for code that has side effects and they should only be used for checking conditions:
# Bad practice
def test_side_effect_bad():
global_var = 0
assert (global_var := global_var + 1) == 1 # No-go
# Good practice
def test_side_effect_good():
global_var = 0
global_var += 1
assert global_var == 1It’s not considered good practice to increment a variable’s value in an assertion, especially if a global one like global_var as in the example above.
Assertions can also be used to check for exceptions:
import pytest
def divide(a, b):
if b == 0:
raise ValueError("Cannot divide by zero")
return a / b
def test_divide_by_zero():
with pytest.raises(ValueError, match="Cannot divide by zero"):
divide(10, 0)Whereby the test passes if divide(10, 0) raises a ValueError of which the error message contains the string "Cannot divide by zero".
Here we employ pytest.raises() to check for the specific error type and message. pytest.raises() is used to assert that a specific exception is raised within its context. It checks that both the error type and the message match the expected values. This type of nuance around the types of errors can be appropriate in certain contexts like the one above, e.g. functions like divide(), whereby certain scenarios like dividing by 0 should be treated as errors.
It’s discouraged to use assertions for data validation and benchmarking in production code. Let’s see the better and worse way to validate data:
# Bad practice
def process_user_input(value):
assert isinstance(value, int), "Input must be an integer"
return value * 2
# Good practice
def process_user_input(value):
if not isinstance(value, int):
raise ValueError("Input must be an integer")
return value * 2
# Test
def test_process_user_input():
assert process_user_input(5) == 10
with pytest.raises(ValueError, match="Input must be an integer"):
process_user_input("not an int")In the example above, the first assertion in test_process_user_input() ensures that the function works correctly for valid input values. It verifies that the function doubles the input value as expected.
The second assertion using pytest.raises() ensures that the function handles invalid input correctly by raising an appropriate exception, which is relevant for enforcing input constraints and providing feedback when such constraints are violated.
Fixtures
A fixture is a feature used to provide a fixed baseline upon which tests can reliably and repeatedly execute. Fixtures are used to set up initial conditions that are common to multiple tests. They help in providing a consistent and controlled testing environment.
Features of Fixtures in pytest
- Setup and Teardown: Fixtures in pytest are used for setting up preconditions (setup) and clean-up actions (teardown) for test functions. This is particularly useful when you have multiple test cases that need to use the same test data or state.
- Scope: pytest fixtures can have different scopes such as function, class, module, or session, determining how often the fixture is invoked:
- function: the default scope, the fixture is created anew for each test function.
- class: The fixture is created once per class of tests.
- module: The fixture is created once per module, i.e., a Python file with tests.
- session: The fixture is created once per session (all test runs).
- Dependency Injection: Fixtures are a way to provide test functions with the objects they need without requiring them to know how to create these objects. This is a form of dependency injection, where the test function simply declares its need for a particular fixture, and pytest takes care of providing it.
- Use with Decorators: To declare a fixture, you use the @pytest.fixture decorator on a function. This function returns the data or state needed for the test. When a test function needs a fixture, it mentions the fixture name as a parameter, and pytest automatically calls the fixture function and passes its return value to the test.
- Reusable and Modular: Since fixtures can be defined in separate modules and plugins, they promote reusability and modularity. You can write common fixtures in a separate file and use them across multiple test modules.
- Parametrization: Fixtures can also be parametrized, meaning you can run the same test function multiple times with different sets of data coming from a fixture.
Let’s now see some examples. In each of the following, we’ll create a fixture with a different scope. We also assume a clear() method to clean up the fixture after each test (aka “teardown” step). Then we test each fixture in a test function and finally we test that multiple fixtures can be used in a single test function.
import pytest
# Fixture with function scope (default)
@pytest.fixture
def sample_data():
data = {"name": "Alice", "age": 30}
yield data
data.clear()
# Fixture with module scope
@pytest.fixture(scope="module")
def module_data():
data = {"module": "test_module"}
yield data
data.clear()
# Fixture with session scope
@pytest.fixture(scope="session")
def session_data():
data = {"session": "test_session"}
yield data
data.clear()
def test_sample_data(sample_data):
assert sample_data["name"] == "Alice"
assert sample_data["age"] == 30
def test_module_data(module_data):
assert module_data["module"] == "test_module"
def test_session_data(session_data):
assert session_data["session"] == "test_session"
def test_multiple_fixtures(sample_data, module_data, session_data):
assert sample_data["name"] == "Alice"
assert module_data["module"] == "test_module"
assert session_data["session"] == "test_session"Parametrization
As introduced as part of pytest’s features, parametrization allows you to run the same test multiple times with different input values, reducing code duplication and making it easier to test multiple scenarios. Fixtures can be parametrized via the params argument of the decorator. The parameters can be accessed via the special request object:
@pytest.fixture(params=[1, 2, 3])
def parametrized_data(request):
return request.param
def test_parametrized_data(parametrized_data):
assert parametrized_data in [1, 2, 3]Another way of parametrizing tests is using the @pytest.mark.parametrize decorator, which allows the same test to run multiple times with different sizes of the input list of numbers.
import pytest
@pytest.mark.parametrize("test_input, expected_output", [
(5, 10),
(7, 14),
(9, 18),
])
def test_double(test_input, expected_output):
result = test_input * 2
assert result == expected_outputIn this example, the test_double function will run three times:
- With test_input as 5 and expected_output as 10.
- With test_input as 7 and expected_output as 14.
- With test_input as 9 and expected_output as 18.
Differences between pytest.mark.parametrize and pytest.fixture
Both @pytest.mark.parametrize and @pytest.fixture with the params argument facilitate running tests with multiple input values. However, they serve different purposes and are best suited for different scenarios.
The @pytest.mark.parametrize decorator is used to define multiple sets of arguments and fixtures for a single test function. This approach is straightforward and is ideal when you want to run the same test logic with different inputs and expected outcomes.
In the latter example, the test_double() function is executed three times with different values for input and expected. Parametrizing fixtures using the params argument allows you to provide multiple values to a fixture. Each parameter value will result in separate invocations of tests that use the fixture. This is particularly useful when multiple tests require the same setup but with different configurations or data:
import pytest
@pytest.fixture(params=[2, 3, 4])
def input(request):
return request.param
@pytest.fixture
def expected(input):
return input * 2
def test_double(input, expected):
assert input * 2 == expectedHere, the input fixture is parametrized with values [2, 3, 4]. The test_double() function will run three times, each time with a different input and its corresponding expected values.
@pytest.mark.parametrize applies directly to the test function, making it explicit which parameters are being used for that specific test. It’s best for scenarios where different tests require different input sets. Here the parameters are specific to a single test and not reused elsewhere. It’s thus suited for simple, flat parameter sets without dependencies between them.
Parametrized Fixtures (fixtures decorated via @pytest.fixture(params=)) on the other hand, are reusable across multiple test functions. This means avoiding repetition when several tests need to use the same set of parameters. They allow for more complex setups where fixtures can depend on each other, enabling more intricate test configurations.
In some cases, it’s beneficial to combine both @pytest.mark.parametrize and parametrized fixtures together to achieve a more granular control over tests. In the following example, test_size_multiplier() runs for each combination of size and multiplier, resulting in a total of nine test cases, i.e. three times each size.
import pytest
@pytest.fixture(params=["small", "medium", "large"])
def size(request):
return request.param
@pytest.mark.parametrize("multiplier", [1, 2, 3])
def test_size_multiplier(size, multiplier):
sizes = {"small": 10, "medium": 20, "large": 30}
expected = sizes[size] * multiplier
assert sizes[size] * multiplier == expectedIntegration with PyCharm and Other IDEs
Development environments such as PyCharm, Visual Studio Code, IntelliJ IDEA, and Eclipse with PyDev support pytest, providing features like a dedicated test runner, code completion for test subjects and pytest fixtures, code navigation, and detailed failing assert reports, among others.
Test Code Organization
Test File Locations
Test files should live inside tests/ in the top-level package directory. Test files should be organized as nested packages and modules under tests/*. Fort this reason, an __init__.py file should be included in tests/ and in each subdirectory including test files. This ensures the discovery of test files as fully-qualified modules, also allowing for test files with the same name in different subdirectories. The __init__.py file can usually remain empty.
Fixtures
Test data is usually stored inside ./tests/fixtures without __init__.py.
Naming Best-Practices
The conventions around naming tests in Python have been somewhat consistent over the years. Nonetheless, there’s a variety of naming conventions, and the ideal choice may depend on the specific circumstances of a project.
PEP 8, the Python Style Guide, doesn’t provide explicit guidelines on naming conventions for tests, but it emphasizes readability and consistency.
The Python standard library’s unittest prescribes that test methods should start with the word test.
In pytest, test discovery is based on file and function names. Specifically, pytest automatically identifies any file named test_*.py or *_test.py as a test file, and any function prefixed with test_ as a test function. It is also advisable to organize test files according to the modules they are testing. For example, if you have a module named calculator.py, the corresponding test file should be named test_calculator.py.
GIVEN-WHEN-THEN Structuring of Tests
Developed by Daniel Terhorst-North and Chris Matts, the “GIVEN-WHEN-THEN” pattern is a method of structuring tests that originated from Behavior-Driven Development (BDD). It is designed to specify a system’s behavior in a readable and understandable format, which can be a useful tool for both writing and understanding tests. This methodology applies mostly to unit testing and end-to-end tests.
- Given: describes the state of the system before the behavior under test is exercised.
- When: describes the action that triggers the behavior to be tested.
- Then: describes the expected outcome or state of the system following the action.
The GIVEN-WHEN-THEN can be particularly beneficial in collaborative or enterprise settings where clarity and readability are crucial.
Here are some examples of proper alignment with the convention. In these examples, the keywords GIVEN, WHEN, and THEN are made explicit in the comments, but this is mainly for showcasing purposes.
def add(a, b):
"""Add two numbers."""
return a + b
def subtract(a, b):
"""Subtract two numbers."""
return a - b
def test_add():
# GIVEN two numbers
a, b = 5, 3
# WHEN the numbers are added
result = add(a, b)
# THEN the result should be the sum of the numbers
assert result == 8
def test_subtract():
# GIVEN two numbers
a, b = 5, 3
# WHEN the numbers are subtracted
result = subtract(a, b)
# THEN the result should be the difference of the numbers
assert result == 2