Modern python build and packaging

Developing software is not only about code but also about delivering the software to your customers. Sometimes, app stores are the solution, at other times it may be software-as-a-service (Saas) but there are also occasions where installers, compressed archives, system packages oder single file executables are (or fat-jars) are the way to go.

While python as a language and ecosystem has many points in its favour I personally found its build and delivery infrastructure lacking. As soon as you project requires a bunch of dependencies and users expect something like apt update to handle your software, too, things can get messy.

There were multiple options, tools and variants for the different aspects of software delivery:

• setuptools vs. distutils
• venvs, site-packages, pip, pipx, wheels
• packaging tools like py2dsc-deb, bdist, pybuild
• homebrew, anaconda etc.

For non-fulltime python developers like me there was always quite some doubt about the current idiomatic and preferred solution. In my C++ projects, this problem was largely solved by CMake+CPack. For java, maven and gradle provided easy solutions for many delivery and deployment tasks.

From my limited perspective the situation in the python world is more fluid and still in the process of finding a best-of-breed solution and evolving mostly that one option. Maybe we have arrived at that with the “Python Packaging Authority” and the work around pyproject.toml.

Many of our customers use Debian oder Debian-based distributions like Ubuntu and like having packages in their own apt-repository that they can install using system tools. So building deb-packages feels like the natural way to deliver your software to these customers.

Building Debian-Packages

In the past I used several approaches to packaging our python projects as debian packages:

• “python setup.py sdist” in combination with plain dh_make and manual editing of the metadata files
• pybuild/dh_python3
• stdeb, with py2dsc etc.

Nowadays I am moving away from setup.py towards pyproject.toml. I find it quite readable and it provides neat features like executable scripts and single-sourcing the project version and a simple declarative project format. Together with debhelper and pybuild packaging becomes quite easy.

I create the necessary debian metadata-files, e.g. using dh_make or by hand.

The modified rules file may look as simple as:

#!/usr/bin/make -f

%:
	dh $@ --buildsystem=pybuild

# We do not have tests yet, so do not run pytest...
override_dh_auto_test:

My pyproject.toml may look like this:

[build-system]
requires = [
    "setuptools>=61.0.0",
    "wheel"
]
build-backend = "setuptools.build_meta"

[project]
name = "myproject"
dynamic = ["version"]
description = 'My project illustrates packaging'
readme = "README.md"
requires-python = ">=3.10"
maintainers = [
    {name = "Mihael Koep", email = "mihael.koep@softwareschneiderei.de"}
]

[project.scripts]
OmsMaxx = "myproject:start"

[project.urls]
Homepage = "https://gitlab.com/example/python/myproject"
Repository = "https://gitlab.com/example/python/myproject"
Issues = "https://gitlab.com/example/python/myproject/-/work_items"

[tool.setuptools.packages.find]
include = ["myproject"]

[tool.setuptools.dynamic]
version = { attr = "myproject.__version__" }

If the tools python3-build, python3-wheel, dh-python and pybuild-plugin-pyproject are available, building a working package should be as easy as running dpkg-buildpackage like below:

dpkg-buildpackage -b -us -uc

Conclusion

I hope this pybuild/pyproject.toml infrastructure with it extensibility and relative clarity will become (and stay) the de facto standard and improve over time.

This would end my uncertainty and provide a sensible default to shipping and packaging python projects.

Have you other approaches or opinions on this topic?