Python Setup and Ecosystem

Note

You may skip this section if you’re already familiar with Conda, Pixi, and the usage of lock files generated with conda-lock and Pixi.

Installing Python

On many systems, Python is pre-installed, though it might not be the latest stable release, which means you might want to install additional versions or update the existing one. It’s good practice to assign a specific Python version to each specific project, which is handled by using virtual environments, as we’ll see later in this section.

Python normally comes pre-installed on any new version of most operating systems, but you might want to check if it is already installed on your machine. You can verify its presence and version by opening a terminal and running:

python --version

Python 3.10.12  

Note

On certain setups, you might get an error while trying to run python but not python3 from the terminal. In that case, you might want to try both.

If you’re missing Python, an error will be returned when trying to run python. In case you wanted to either install or update your default python version, please follow the platform-specific guidelines available on the official Python website.

Python Packages and Environment Management

Package managers are tools that automate the process of installing, updating, and removing software packages. They simplify the installation of dependencies and increase consistency across different environments. The most widely known package managers for Python are pip and Conda. While both alternatives can install Python packages, Conda is a general-purpose package manager that handles packages from any language, while pip is specific to Python.

Conda can also manage system-level dependencies and binary files, which help seamlessly resolve pure Python dependencies and system-level dependencies within the same environment. On top of that, it comes with a more sophisticated dependency resolution engine that more accurately assesses the compatibility between all packages in an environment.

In enterprise settings, the choice between Conda and pip often depends on the complexity of the projects and the need for managing dependencies across multiple languages: Conda is generally preferred in cases where dependencies include a mix of Python and non-Python packages. More recently, Pixi has been stepping up as a promising and easy-to-migrate-to alternative package management solution to Conda. Similarly to Conda, Pixi supports multiple language ecosystems, and can seamlessly integrate both pip and Conda package dependencies.

Python environments are a way to manage multiple Python versions, projects and sets of dependencies. The main advantage of environments is that they allow one to work on multiple projects with different dependencies and Python versions without conflicts.

There are many different ways to create and manage Python environments. The most widely known are venv, Pipenv, Poetry, uv, and Conda. Recently, Pixi has also been gaining traction.

For the sake of brevity and favoring breadth of adoption, we won’t go over Poetry and will focus on venv and Conda instead. We will also cover Pixi since, although not as widely adopted, it’s a promising tool both for environment management and package management.

Note

When used from within a virtual environment, installation tools like pip will install packages into the virtual environment rather than the base python.

Virtual Environments with venv

The venv module was added to the Python standard library in version 3.3 and allows one to create lightweight virtual environments. To create an environment, you run the following command:

python -m venv .venv

This will create a new directory called .venv in the current working directory. To create an environment with a specific python version, you can run:

python -m venv .venv --python 3.10

You then need to activate the environment:

source .venv/bin/activate

Once activated, you can install a package in the environment with:

pip install <package-name>

To save the environment, you can run:

pip freeze > requirements.txt

This will create a file called requirements.txt in the current working directory, containing the list of all packages installed in the environment. To restore the environment, you can run the following once the environment is activated:

pip install --no-deps -r requirements.txt

Lock Files

The usage of pip freeze as shown in the previous section provides a limited set of functionalities to reproducible environments and dependencies locking by simply creating a snapshot of the installed packages. pip freeze lacks in fact qualities of reproducibility and robustness from employing lock files and dependency managements tools as conda-lock or Pixi, which bring a set of more encompassing features such as cross-platform compatibility, having the full set of “pre-solved” dependencies for environment re-creation, and a better dependency conflicts resolution capability.

A different approach is usually employed to capture the exact state of the environment while avoiding said pitfalls of pip freeze, namely “lock files”. Lock files are generated via different libraries and hereby follow different syntaxes although they all serve the same purpose.

In addition to capturing the exact versions of all dependencies, they often include hashes to verify the integrity of the packages and are platform-specific.

While venv does not support lock files natively, there exist libraries such as Pipenv and Poetry that do. In this guide, we’ll focus on conda-lock and Pixi, which can also be used to generate lock files.

Conda

The best way to install conda correctly is to follow the latest platform-specific official instructions.

The file containing the environment specification is normally named environment.yml and can be created and updated using conda as follows:

conda env export > environment.yml

To restore the environment from the environment file, you can run:

conda env create -f environment.yml

Conda-lock

In this section, we go over the usage of Conda in conjunction with conda-lock for generating lock files for our environments.

Note

The official documentation is available on GitHub. Please refer to the latest setup instructions thereon.

Benefits of conda-lock

From the conda-lock official GitHub page:

Conda lock is a lightweight library that can be used to generate fully reproducible lock files for conda environments.

Conda-lock is able to generate fully reproducible lock files by performing a conda solve for each platform for which you desire lock files.

The benefits of using conda-lock resemble what we’ve introduced prior in the lock files section, namely reproducibility and cross-platform compatibility. Moreover, conda-lock also speeds up dependency resolution, which is notoriously a time-consuming process, especially when dealing with a large number of dependencies that have their own inter-dependencies.
conda-lock simplifies that by resolving all dependencies at the time of creating the lock file. This means that once the lock files have been generated, there’s no need for further resolution upon setting up the environment anew.

Conda-lock Usage

Below is an overview of the most frequent usage of conda-lock:

• To generate a conda-lock file from the environment.yml file: conda-lock -f environment.yml. It’s also possible to specify multiple platforms, for example: conda-lock -f environment.yml -p linux-64 -p osx-64 -p win-64.
• To generate single platform-specific lock files from conda-lock.yml: conda-lock render -p <platform>, for example: conda-lock render -p linux-64.
• To recreate the conda environment from a lock file: conda create --file conda-<platform>.lock -n <env-name>

Pixi

Pixi is a package manager built with Rust that allows to install libraries in a reproducible way. It supports Windows, macOS, and Linux. It is built on on top of the existing Conda ecosystem (although independent from Conda) and supports Python, R, C/C++, Rust and many other languages.

Similarly to Conda and conda-lock, Pixi can handle both the creation and restoring of isolated environments and the management of dependencies. An important difference from the latter two is that Pixi is project-centric rather than environment-centric, thus focusing on managing dependencies and environments on a per-project basis, allowing for more granular control and isolation of project-specific dependencies. Usually all dependencies of a project are stored in a directory inside the project root directory, which is named .pixi by default.

To initialize a Pixi project from scratch, run the following:

pixi init <project-name>
cd <project-name>

which will create the root directory of the project and a .pixi directory in the current working directory.
For an existing project, you can initialize Pixi from the project root directory with:

pixi init

To install a package (or Python), run the following:

pixi add python
pixi add <package-name>

And to run a .py script, you can run:

pixi run python script.py

Pixi can also store scripts in “tasks” but for the scope of this guide, we’ll stick to the basic usage. You can have a look at the official documentation for further information.

Dependencies can also added directly in the pixi.toml file, which can be used to specify the dependencies of the project. The latter has usually the following structure:

[project]
<Metadata about the project>

[tasks]
<task1-name> = "<statement 1>"
<task2-name> = "<statement 2>"

[dependencies]
<...>
<...>

[pypi-dependencies]
<...>
<...>

Where [dependencies] refers to Conda packages.

Pixi Lock File

Similarly to conda-lock, Pixi can generate a lock file for the environment, whose default name is pixi.lock. Here, the singular form is not used by accident: Pixi stores all cross-platform dependencies in a single lock file, in a nested structure. Lock files are conveniently generated when you add a dependency. Moreover, most common statements will trigger an update of the lock file when dependecies change. This means the lock file does not need to be managed at all manually.