I spent a solid amount of my engineering time last year getting Halide's pip packaging up to snuff. Distributing binaries that are compatible with manylinux and use heavy, slow-to-build dependencies like LLVM is a particular sort of nightmare1... wiring up dozens of moving parts between GitHub Actions, Docker, cibuildwheel, the pyproject.toml file, scikit-build-core (which is awesome, by the way), CMake, and probably a few I'm forgetting.
But today, I'd like to show you the fruits of this labor. Thanks to uv, I've found a workflow that I will be using for the foreseeable future to create both Python and C++ apps that use Halide.
The first step is to create an example directory somewhere and write down a basic pyproject.toml for it:
$ mkdir example
$ cd example
$ vim pyproject.toml
We start with the required metadata:
[project]
name = "example"
version = "0.1.0"
requires-python = ">=3.9" # The current minimum version required by Halide
Now we can get to the interesting part, which is writing down our dependencies:
[project]
# ... same as above ...
dependencies = [
"halide>=20.0.0.dev0",
]
[tool.uv.sources]
halide = { index = "testpypi" }
[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple"
explicit = true
This fragment ensures we get a nightly build of Halide from the Test PyPI instance, where we publish (ideally) every build of our main branch2. If we only want a particular stable version from the production PyPI instance, we can drop the uv-specific sections like so:
[project]
# ... same as above ...
dependencies = [
"halide==19",
]
We're all set to use the Python bindings. We can write down a very simple pipeline in main.py:
import halide as hl
f = hl.Func("f")
x = hl.Var("x")
f[x] = x
buf = f.realize([10])
print(list(buf))
Running this with uv run will automatically create a virtual environment and install Halide since one doesn't already exist:
% ls -A
main.py pyproject.toml
% uv run main.py
Using CPython 3.9.13 interpreter at: /Library/Frameworks/Python.framework/Versions/3.9/bin/python3
Creating virtual environment at: .venv
Installed 4 packages in 24ms
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
% ls -A
.venv main.py pyproject.toml uv.lock
This prints exactly the result we expect!
Notice that uv run created two new entries in the example directory: .venv, which contains the virtual environment, and uv.lock which contains a list of resolved package versions satisfying the project's dependency constraints. This is called a "lockfile" because the versions are locked at the time the virtual environment is constructed. It is recommended to check this into version control. If it exists, uv will use it directly rather than trying to solve the version constraints anew. This is useful for reproducing errors and carefully tracking dependency upgrades.
Inevitably, you will want to upgrade the dependencies in your virtual environment, but you won't necessarily want to change the constraints (perhaps you just want to pull a new nightly). In this case, the command to run is:
% uv sync --upgrade
The Python package also provides Halide's C++ API and includes the CMake package for easy configuration. To use it, we need to add cmake and (optionally) ninja to our dependencies:
[project]
# ... same as above ...
dependencies = [
"cmake>=3.28", # The minimum version required by Halide
"halide==19", # Use a stable version from PyPI
"ninja>=1.11", # A recent version
]
Now we can run uv sync to update our virtual environment before authoring a basic CMakeLists.txt and main.cpp:
% uv sync
Resolved 9 packages in 41ms
Installed 2 packages in 71ms
+ cmake==4.0.3
+ ninja==1.11.1.4
% vim CMakeLists.txt
We write down a very basic build that links to the Halide::Halide JIT compiler.
cmake_minimum_required(VERSION 3.28)
project(example)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)
find_package(Halide REQUIRED)
add_executable(main main.cpp)
target_link_libraries(main PRIVATE Halide::Halide)
Now we're ready to populate main.cpp with a direct translation of the previous Python example:
#include <Halide.h>
#include <iostream>
int main() {
Halide::Func f{"f"};
Halide::Var x{"x"};
f(x) = x;
Halide::Buffer<int> buf = f.realize({10});
for (int i = 0; i < buf.extent(0); i++) {
std::cout << buf(i) << " ";
}
std::cout << "\n";
return 0;
}
Now we can configure the project with CMake. We will prefix the command with uv run to ensure that the configure step runs inside the virtual environment (where both CMake and Halide are installed):
% uv run cmake -G Ninja -S . -B build -DCMAKE_BUILD_TYPE=Release
-- The C compiler identification is AppleClang 17.0.0.17000013
-- The CXX compiler identification is AppleClang 17.0.0.17000013
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python: /Users/areinking/example/.venv/bin/python3.9 (found suitable version "3.9.13", minimum required is "3") found components: Interpreter
-- Halide 'host' platform triple: arm-64-osx
-- Halide 'cmake' platform triple: arm-64-osx
-- Halide default AOT target: host
-- Found ZLIB: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/libz.tbd (found version "1.2.12")
-- Found PNG: /opt/homebrew/lib/libpng.dylib (found version "1.6.50")
-- Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR)
-- Found Python: /Users/areinking/example/.venv/bin/python3.9 (found suitable version "3.9.13", minimum required is "3") found components: Interpreter Development.Module
-- Configuring done (1.8s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/areinking/example/build
The Halide Python package makes its CMake configuration available when the virtual environment is active3. Finally, we can run the build and the example we wrote:
% uv run cmake --build build
[1/2] Building CXX object CMakeFiles/main.dir/main.cpp.o
[2/2] Linking CXX executable main
% ./build/main
0 1 2 3 4 5 6 7 8 9
I hope you'll agree this workflow is a fast and convenient way to get off the ground with Halide, no matter which API you're using! We'd love to hear about the cool things you're building or help answer your questions. Reach out on GitHub Discussions!
For reference, here is the complete pyproject.toml for using a stable version of Halide (published to PyPI):
[project]
name = "example"
version = "0.1.0"
requires-python = ">=3.9"
dependencies = [
"cmake>=3.28",
"halide==19",
"ninja>=1.11",
]
And here's how to use the nightly version of Halide (published to TestPyPI):
[project]
name = "example"
version = "0.1.0"
requires-python = ">=3.9"
dependencies = [
"cmake>=3.28",
"halide>=20.0.0.dev0",
"ninja>=1.11",
]
[tool.uv.sources]
halide = { index = "testpypi" }
[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple"
explicit = true
By the way... if you're a Python packaging expert and want to help improve Halide's packaging, PRs and constructive issues are very welcome! ↩
We'd love help automating the removal of old dev builds from TestPyPI, which currently requires manual intervention (by me!) due to mandatory 2FA. ↩
This works in any virtual environment, not just those managed by uv! On Windows, the Python virtual environment layout is incompatible with CMake's default search, so you will (unfortunately) need to manually activate the virtual environment and add %VIRTUAL_ENV% to the environment variable CMAKE_PREFIX_PATH. ↩
Unless otherwise stated, all code snippets are licensed under CC-BY-SA 4.0.