C++ feat. Python: Connect, Embed, Install with Ease

Author(s): Dmitry Malishev

Originally published on Towards AI.

C++ enterprise application for Windows executes a Python module. Here’s my journey through the process of this tech fusion, from the first line of code to application delivery.

Intro

Python’s simplicity, extensive package ecosystem, and supportive community make it an attractive choice. Conversely, when tackling complex user interfaces, multi-threaded data flows, and round-the-clock robustness in enterprise solutions, C++ takes center stage. Sometimes, the need arises to amalgamate these two, striving to have the best of both worlds.

Frequently, Python code invokes C++ methods for intricate algorithmic calculations. However, I encountered an opposite scenario where my Machine Learning application urgently required invoking a custom model with Python-based inference code. The prospect of rewriting it in C++ or adopting a corresponding inference engine was unfeasible. My initial thought was simple: “Calling Python from C++ should be a breeze.” Alas, I underestimated the complexity involved!

Expectations vs. Reality

Several readily searchable bridges facilitate the interaction between C++ and Python:

1. Python’s built-in C++ interface accessible via C++ headers and libraries (Python.h, Python.lib) bundled with the standard Python package.
2. Boost.Python, for those comfortable with the Boost library.
3. Pybind11, my preference for its blend of power and ease of integration into projects.

At first glance, the solution appears within reach, and the only thing remains in selecting the most suitable bridge from this trio. But let’s look more closely!

The aforementioned methods excel in their core purpose of enabling Python-C++ communication, sufficient for preliminary trials and experiments. However, when aiming to construct a deliverable solution, we need to think about full software lifecycle. And one of the challenges to solve is distribution. To clarify, the task is to distribute C++ binaries alongside Python components. Even sharing Python code itself can be hard, if taking into account the interpreter installation into OS. Moreover, multiple Python versions might coexist, and every additional package necessitates proper installation, including resolving dependencies. The question becomes: what options are viable for our case?

1. Instruct users to install Python and requisite packages before employing the C++ app.
2. Incorporate the Python installer within the C++ installer to automatically address related issues, including other Python installations on a user system.

Neither seems optimal. Consequently, I embarked on a quest to discover a more dependable and stable approach, which I’m eager to share with you.

Embedded Python

During your last visit to Python’s homepage, did you explore its distribution options? Among them lies a compelling choice: Windows embeddable package. This option piqued my curiosity, presenting a seemingly magical solution to the distribution problem. Delving into the official documentation, I unearthed details about this embedded version – astonishingly, Python offers an embedded version for seamless integration into other applications.

Upon perusing the contents of the embedded version archive, a couple of questions arise: How can extra modules be added into this setup? And how can the C++ application be directed to utilize this embedded variant?

Addressing the first question is relatively straightforward: packages might be just copied from a standard Python installation.

The second question demanded extensive research, as mentioned earlier Python bridges work with the default installation. While changing Python environment variables (PYTHONPATH, PYTHONHOME) can be occasionally helpful, it’s not an ultimate solution to switch to the embedded setup, as Python may put its files to different OS locations.

Python Flags for C++

The simplest Python program may contain only one line:

print(‘Hello, World!’)

Python shields programmers from numerous internal routines to achieve this apparent simplicity. When silently launching this line, the interpreter handles installation parameters, initialization, default modules imports, bytecode compilation, function lookup, and more. However, there’s a way of intervening these secret mechanisms. Quoting the official documentation “Python has variables for the global configuration to control different features and options”. They exist as command line arguments or as flags and methods for C++-code, accessible through already mentioned Python.h. In the present case, we need to override Python’s search path using Py_SetPath() method. Other flags come into play in some unusual cases: Py_NoSiteFlag, Py_NoUserSiteDirectory, Py_IgnoreEnvironmentFlag. These flags refer to skipping loading default module site, or ignoring separate directory with modules in the user’s home directory. Check them out to handle certain cases!

Summary Guide

1. Prepare your C++ project to start integrating Python.
2. Decide on the Python version and install it to Windows as the default installation.
3. Download the same version of Python as an embedded package and extract it to your project directory.
4. Write Python test code that calls all required Python methods it needs for further modules copying and tests.
5. Launch Python test code with the embedded installation (use command line flags to override the search path). You’ll end up in errors of missing modules if some non-default modules are utilized. Copy one-by-one missed modules from the default installation to the embedded installation so that the Python test code is finished successfully.
6. Go to your C++ project. Get pybind11. Add projects that include a search path to it.
7. Add search paths to headers and libs from the default Python installation to your project.
8. In the C++ code use Py_SetPath() to set relative path to Python embedded installation. Use additional flags if needed.
9. If one of the Python packages is located in the user site-package directory, just copy it to the embedded package folder and add a separate path to it in Py_SetPath().
10. In C++ code, use pybind11 API to access Python methods. If the C++ program freezes on calling the Python method, then most probably some Python module cannot be imported — use Python test code from step [5] to solve it.
11. Distribute the C++ binaries with the embedded Python folder as an integral component. You may need to copy some DLLs to your binary location as well, like python3.dll and Windows redistributables.

Final Part

Production-purpose C++ applications can include embedded Python as its part in a way that users won’t even know about it. And that is a great achievement, as we all like to carry out our tasks with the help of simple yet powerful applications rather than puzzle over their installation!

I’m sharing my demo project, which incorporates all insights from the article for your quick start.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI