Package Management in Python

Package Management in Python#

Information	Details
Learning Objectives	• Understand Python package management concepts • Create and manage conda/mamba environments • Install packages using conda and pip
Prerequisites	Basic command line knowledge and Python fundamentals
Estimated Time	60 minutes
Topics	Package management, conda/mamba, pip, environments

Before starting this section, you should be comfortable using the command line as covered in previous sections. You should also understand basic Python concepts and have familiarity with running Python programs.

What is Package Management?#

Python has seen growing popularity in power systems engineering due to its smooth learning curve and various packages for data simulation, data analytics, and optimization. Its widespread adoption stems from both its accessibility and its rich ecosystem of specialized libraries that can be combined to solve complex problems.

When developing applications, you will typically work with multiple specialized packages; some for numerical computations, others for power system modeling, and perhaps others for visualization or data analysis. A package manager helps you install these packages, keep track of which versions you have, update them when needed, and remove them when you don’t need them anymore.

For example, if you want to analyze power systems, you might need packages like pypower. Instead of writing all the code yourself, you can use these pre-written packages to solve complex problems more efficiently.

Why Use Different Environments?#

When developing power system applications, you might work with packages that have specific version requirements. This is where Python environments become essential. An environment is an isolated workspace that allows you to maintain different configurations of packages for different projects.

Think of Python environments like different toolboxes for different jobs. You might need one toolbox for school projects and another for work projects. Each toolbox can have different versions of the same tool.

Consider a scenario where you’re working on two different projects:

A power system stability analysis project that requires an older version of numerical solvers
A machine learning application for load forecasting that needs a newer version of the same solver

Without environments, this would create a conflict since you can only have one version installed system-wide. But with environments, you can create a separate environment for each project, install the specific versions each project needs, and switch between environments as needed.

Package Managers: conda and pip#

Python provides a package manager called pip that makes it easy to install and manage dependencies. While pip works well for many situations, it has limitations when dealing with complex scientific packages that have binary (compiled) dependencies.

For power systems engineering and scientific computing, we recommend using conda (or its faster variant mamba). These tools are more robust in handling binary dependencies, which are often required for scientific and engineering applications. The advantage of conda is that it handles complex scientific packages better, manages Python versions and other dependencies more effectively, and works consistently across Windows, Mac, and Linux.

In the following examples, we’ll use mamba because it’s significantly faster than conda, but the commands are identical for both tools.

Getting Started with mamba#

Installation#

To get started, we’ll install miniforge, which provides both mamba and conda. You can find the installation files at conda-forge/miniforge.

For Linux and WSL (Windows Subsystem for Linux), follow these steps:

First, download the installer script:

wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"

Then, run the installer:

bash Miniforge3-$(uname)-$(uname -m).sh

The installer will ask you to review the license terms. Press Enter to continue, scroll to the end, type yes and press enter to continue. When asked about the installation directory, you can press Enter to accept the default location. At the end, the installer will ask if you want to automatically initialize conda - we recommend typing yes for convenience.

After installation, restart your shell by closing and reopening your terminal. You’ll notice a change in your shell prompt:

(base) User@Host:~$

The (base) indicates that you are now operating in the base environment set by conda. This environment gives you access to the packages installed in that environment, in addition to system-wide packages.

Creating Environments#

Let’s create a new environment specifically for this training course. We’ll name it pct (short for Power Cyber Training) and install Python 3.12 in it:

(base) User@Host:~$ mamba create -n pct python=3.12

The -n flag tells mamba to create a new environment called pct. The python=3.12 specifies that we want to install Python 3.12 in the new environment.

After confirming the installation, mamba will download and install Python 3.12 and its dependencies in the new environment. At the end, you’ll see a message telling you how to activate the environment:

To activate this environment, use

$ mamba activate pct
To deactivate an active environment, use
$ mamba deactivate

Before activating the new environment, let’s check which Python interpreter is being used in the current environment:

(base) User@Host:~$ which python
/home/User/mambaforge/bin/python
(base) User@Host:~$ python --version
Python 3.9.6

Now, let’s activate the new environment:

(base) User@Host:~$ mamba activate pct
(pct) User@Host:~$ which python
/home/User/mambaforge/envs/pct/bin/python
(pct) User@Host:~$ python --version
Python 3.12.3

Notice how the prompt changed from (base) to (pct), indicating that you’re now in the pct environment. Also, the Python interpreter now points to the Python 3.12 version installed in the pct environment.

Avoid Installing to Base Environment

It is good practice to avoid installing packages to the base environment. The base environment is to provide the core libraries and dependencies for a functional conda installation. Installations to base may lead to unexpected behavior.

Instead, create a new environment for each project. Just remember to activate the environment before installing packages or running scripts.

Exercise

Follow the same steps to create a new environment called pct and install Python 3.12 in it.
Activate the environment, and check the Python version.
Deactivate the environment, and check the Python version again.

Installing Packages#

When to use conda and pip#

A Python package may be installable via both conda and pip. The recommended practice is:

If a package is available for conda, install it via conda first. This will satisfy a more complete set of dependencies.
If a package is not available for conda, install it via pip.

Installing Packages with conda#

Here are the commonly used commands for mamba (or conda):

mamba install: Install a package
mamba update: Update a package
mamba remove: Remove a package
mamba list: List installed packages in the current environment

Let’s install the Jupyter notebook package in our pct environment:

(pct) User@Host:~$ mamba install jupyter

You will be prompted to confirm the installation. Type y and press Enter. If you want to skip the confirmation in the future, you can use the --yes flag:

(pct) User@Host:~$ mamba install jupyter --yes

After installation, you can run Jupyter notebook:

(pct) User@Host:~$ jupyter notebook

This will start the Jupyter notebook server and open a web browser with the Jupyter interface. To stop the server, go back to the terminal and press Ctrl+C.

If you deactivate the environment and try to run jupyter notebook again, you’ll likely get an error because the package is only installed in the pct environment, not in the base environment or system-wide.

Installing Packages with pip#

Sometimes, you’ll need to use pip to install packages that aren’t available in conda. In any mamba environment, if Python is installed, pip is also installed.

The basic commands for pip are:

pip install: Install a package
pip uninstall: Remove a package
pip list: List installed packages

Let’s install the pypower package using pip:

(pct) User@Host:~$ pip install pypower

To confirm the installation, you can import the package in Python:

(pct) User@Host:~$ python -c "import pypower; print(pypower)"

Where does pip install packages?

pip is a Python package, so it installs to the same environment that contains the Python executable. This environment is often the active environment.

To confirm, run which python and which pip. Their paths should be the same.

If they are not the same, you will need to deactivate and activate the environment to fix environmental variables.

Managing Your Environment#

You can check what packages are installed in your current environment using:

(pct) User@Host:~$ mamba list

The output will show all installed packages, including dependencies. Here’s part of what you might see:

# packages in environment at /home/User/mambaforge/envs/pct:
#
# Name                    Version                   Build  Channel
[TRUNCATED OUTPUT]
jupyter                   1.0.0             pyhd8ed1ab_10    conda-forge
pypower                   5.1.16                   pypi_0    pypi

Notice how the Channel column shows different values for jupyter and pypower. Since jupyter was installed via mamba, it shows the channel as conda-forge. Since pypower was installed via pip, it shows the channel as pypi.

Common Problems and Solutions#

Package Conflicts#

Package Conflicts

If you try to install a package that conflicts with an existing package, you might get an error. In this case, you have a few options:

Create a new environment for the conflicting package
Try installing a different version of the package
Remove the conflicting package and then install the new one

Finding the Right Package#

If you’re not sure which package to install, you can search for packages using:

(pct) User@Host:~$ mamba search jupyter

This will show all available versions of packages that match the search term.

Package Not Found#

If mamba can’t find a package, try installing it with pip instead:

(pct) User@Host:~$ pip install package-name

Summary#

Managing Python environments and packages is essential for power systems engineering and any complex Python development. By using conda/mamba for environment management, you can maintain separate toolsets for different projects, avoid version conflicts, and ensure reproducibility of your work.

Remember the key principles:

Create a dedicated environment for each project
Use conda/mamba for package installation when possible
Fall back to pip only when necessary
Keep track of your environments and installed packages

With these practices, you’ll be well-equipped to manage complex Python projects efficiently.

Advanced Package Management#

Anaconda Distribution vs Miniconda#

To provide more context, here’s the relationship between the tools we’ve discussed:

Anaconda Inc. is a company that develops the Anaconda Distribution.
Anaconda Distribution is a full collection of packages for Python and R that includes hundreds of pre-installed scientific packages.
Miniconda is a free minimal installer for conda that includes only conda, Python, and a small number of necessary packages.
Miniforge is a community-led alternative to Miniconda that uses conda-forge as the default channel.

The main difference is size and default packages:

Anaconda (~3GB) comes with 250+ pre-installed packages
Miniconda (~400MB) is minimal with just conda and Python
Miniforge (~400MB) is like Miniconda but uses conda-forge by default

More About conda-forge#

conda-forge is a community-led collection of recipes, build infrastructure and distributions for the conda package manager. It’s become the standard channel for most open-source packages.

When you install packages with mamba/conda without specifying a channel, they now typically come from conda-forge by default with miniforge. This simplifies package discovery compared to the historical approach of searching across multiple channels.

When to Mix conda and pip#

Mixing conda and pip

While we recommend using conda/mamba first, there are times when you’ll need to use pip. Here’s the recommended approach:

Create and activate your conda environment
Install as many packages as possible with conda/mamba
Only after that, install additional packages with pip

This order is important because packages installed with pip aren’t tracked by conda, which may lead to dependency conflicts if conda packages are installed later.

If you accidentally install the same package with both package managers, you can check which one is being used:

# Check which package is being used
python -c "import package_name; print(package_name.__file__)"

Package Development#

Installing Packages from Source#

If you’re working with a package that hasn’t been published to PyPI or conda-forge yet, you might need to install it directly from source code:

# For a local package directory with setup.py
pip install /path/to/package/

# For a GitHub repository
pip install git+https://github.com/username/repository.git

Developing a Package#

When you’re developing a package yourself, you’ll want to install it in “development mode” so your changes take effect immediately:

# Navigate to the package directory
cd /path/to/your/package/

# Install in development mode
pip install -e .

The -e flag creates an “editable” installation - any changes to the source code will be reflected immediately without needing to reinstall.

System-wide Package Management#

Besides Python-specific package managers, each Linux distribution comes with its own system package manager.

APT Package Manager#

On Debian-based systems (including Ubuntu and its derivatives), apt is the standard package manager for system libraries and applications:

# Update package lists
sudo apt update

# Install a package
sudo apt install build-essential

# Remove a package
sudo apt remove package-name

Distinguishing Between System and Environment Packages#

It’s important to understand the difference between:

System-wide Python (installed via apt)
conda/mamba environment Python

You can check which Python you’re using with:

which python
python --version

When you activate a conda environment, it changes your PATH to prioritize the environment’s Python over the system Python.

Best Practices for Project Management#

For professional work, consider the following practices.

Environment Files#

Save your environment specification to a file so others can reproduce it:

# Export environment to a file
mamba env export > environment.yml

# Create environment from file
mamba env create -f environment.yml

Requirements Files#

For pip dependencies, use requirements files:

# Generate requirements from installed packages
pip freeze > requirements.txt

# Install packages from requirements
pip install -r requirements.txt

Mini Project#

Here are some exercises to help you practice package management:

Create a new environment called practice with Python 3.11.
Activate the environment and check the Python version.
Install the numpy package using mamba.
Install the pypower package using pip.
List all installed packages to confirm your installations.
Deactivate the environment and check which Python version is active.