Python Virtual Environments: The Why and How

Author(s): Muhammad Arham

Originally published on Towards AI.

This article describes the benefits of Python virtual environments and how they streamline the development processes. Further, we discuss the venv API provided for the usage of virtual environments.

Why use Python Virtual Environments

Consider a scenario. You are a Machine Learning developer and are working on a Python project. The project requires Tensorflow 2.0 and you have installed it using pip. The project works seamlessly and you keep working on the project without any hassle.

However, you are given another task on the side. The new project requires Tensorflow 1.0 for development. Now you have run into a problem. You can not work on both tasks simultaneously, as the project dependencies contradict. Either you can reinstall packages each time you want to work on a specific project, or you can use Python Virtual Environments for ease!

Benefits of Virtual Environments

Isolation

With virtual environments, you can isolate both projects and work independently on both simultaneously. All that will be required is a simple command to change the environment, and changes in one environment will be completely independent of other environments.

Dependency Management

Any packages installed in one project environment will not contradict the packages installed in the other environment. Therefore, you can have Tensorflow 2.0 in one environment and Tensorflow 1.0 in the other environment. Being isolated from each other, there will be no dependency clash, and you will be able to work on both without needing to reinstall packages each time.

Portability

Now extending the same scenario, consider you complete a project and forget about it. After some time, you face an error that needs to be corrected. However, it has been so long that you have forgotten about the dependencies and exact versions of packages that were used during project development.

Python virtual environments can help solve such problems as each environment can be frozen and all dependencies can be saved. Therefore, during or after development, it becomes easier to share code seamlessly. All that needs to be done is share a requirements file that limits setting up a project to a single command. Therefore, reproducing an environment is easier on any development machine.

How to Use Python Virtual Environments

Python virtual environments are limited to a project directory. They are built on an existing base environment. The base environment is the default environment active when you install a Python interpreter. All packages are installed in this environment unless you explicitly change an environment.

Prerequisites

To use virtual environments, you first need to install the virtualenv package.

python -m pip install virtualenv

For Debian/Ubuntu OS, you need to additionally install a python3-venv package. Use the command:

sudo apt install python3.xx-venv

Where “xx” represents the Python version in the base environment. For example, if you have Python version 3.10, use the command:

sudo apt install python3.10-venv

You can check the Python version using:

python -–version

Creating an Environment

Once all dependencies are installed, you can start working with environments.

To create an environment, switch to the project directory. Now considering you want to create an environment named “my_env”, run the command:

python -m venv my_env

python -m venv my_env

This creates a folder name my_env in the current project directory. This represents a lightweight directory that holds all dependencies specific to this environment and project.

Activating an Environment

Although an environment is created, you are still in the base environment and need to explicitly activate the new environment. To activate an environment use:

On Windows PowerShell:

<venv>\Scripts\Activate.ps1

On Linux Bash terminals:

source <venv>/bin/activate

Where <venv> represents the path to the root project directory. If you are in the project directory, you can omit this part and use relative paths. For our current environment example, <venv> should be replaced with my_env or absolute path to my_env directory.

Once you are in an environment, the terminal will highlight the active environment. For example, (my_env) will be visible on the command line. This is used to verify the current environment. Moreover, you can use the command:

pip -V

This will display the pip version with the path of the current environment that can be used to verify the currently active environment.

Once active, all packages will be installed in this current environment and will not affect the base or other virtual environments. So you can work in isolation on the current project.

Freezing Environment Dependencies

Once the project is working and all dependencies are installed correctly, you can share the environment so that the project can be set up by someone on another development machine. This makes it easier to work in collaboration with other developers and share code through GitHub.

To freeze environment dependencies:

pip freeze -r requirements.txt

This will create a requirements.txt file in the current project directory, that will list all the packages installed in the environment. This text file can be used to reproduce an environment using the command:

python -m pip install -r requirements.txt

Deactivating an Environment

Once work is complete or you want to switch to another environment, it is better to deactivate an environment. To do so, run the command:

deactivate

This will switch you back to the base Python environment.

Conclusion

This article introduced the basic nuts and bolts of Python virtual environments. Using the environment is a necessity, and all projects need to be shared and reproduced in development as well as production. They isolate each project’s dependencies and help solve dependency clashes which is a major nuance during development.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI