Advanced Refactoring: Extract Structural Duplication

A friend once told me that when his workday goes sour, he turns to a practice of “therapeutic refactoring”: Making little improvements throughout the source code of a project without an overarching goal. If we dissect this behaviour, we probably find a dopamine-based reward cycle not unfamiliar to those in popular video games like Diablo, only that we slay little flaws in the source code instead of critters. And because we never know what code can be found in the next method and the next class (just like we don’t know which monsters lurk behind the next door in Diablo), we go and explore for a bit, leaving a trail of resolved code smells (or slain monsters) behind.

But somewhere in each level of Diablo, there waits a boss enemy that requires some more hammering. And similar, every now and then you encounter a code area that looks dirty but is not resolved that easily. This blog entry tells the story of one of these encounters and how we refactored it. There is no guarantee that you’ll find the result “better” in any way, the whole point is that it can be done and the mechanic is always the same.

Let’s look at this code snippet, reduced to the relevant parts:

public static void setValueFor(
final PreparedStatement statement,
final ColumnInfo column,
final String value
) throws SQLException {
final int typeNumber = column.type().getVendorTypeNumber();
switch (typeNumber) {
case Types.DATE -> {
statement.setDate(
column.columnIndex(),
new Date(
getDateMillisFrom(
LocalDate.from(parseToDate(value)).atStartOfDay(ZoneOffset.UTC)
)
)
);
}
case Types.TIMESTAMP -> {
statement.setTimestamp(
column.columnIndex(),
new Timestamp(
getDateMillisFrom(
LocalDateTime.from(parseToDateTime(value)).atZone(ZoneOffset.UTC)
)
)
);
}
}
}

A given string value is written into a SQL database, using a prepared statement and adequatly typed setter method calls. The real code is way more exhaustive, but we are only interested in the striking similarity of the DATE and TIMESTAMP case.

The structure resembles something I call a “sandwich” or “burger”: The same bun above and below, but different things in the middle. If it is only “bun, patty, bun”, you can solve it with a simple lambda expression as a method parameter. But in this case, it is a more delicate mix of bread and filling layers. Let’s investigate to plan our refactoring attempt:

  • statement.setDate() and statement.setTimestamp(): Both operate on the prepared statement, but call different methods with a different parameter type. Needs to be configurable in the result.
  • column.columnIndex(): Duplication. Needs to be part of the extracted code.
  • new Date() and new Timestamp(): There is no suitable supertype other than java.util.Date. Since we want the actual parameter types for the final setter call, we shouldn’t abstract to it. We need to make this call configurable, but be able to work with the result without knowing its actual type. A generic parameter is the right choice for it.
  • getDateMillisFrom(): Duplication. Needs to be part of the extracted code.
  • LocalDate.from() and LocalDateTime.from(): They look similar enough to be tempted to unify them into one generic call, but it would refrain from doing it. This feels like a pyrrhic victory trap to me, where the solution is way more complex than the original. I already fear this effect for our initial quest, I don’t need a nested sub-quest right now.
  • .atStartOfDay(ZoneOffset.UTC) and .atZone(ZoneOffset.UTC): Duplication. Needs to be part of the extracted code.

If you don’t see my classification right away, my suggestion is to print the code snippet on paper, grabbing a green and a blue text marker and separating each term by coloring it. I did this for years until I could color the code right in place in my imagination. And with training, I could add more colors…

So we have a structure like: Configure, Extract, Configure, Extract, Configure, Extract. Nice! I’ve counted three configuration parameters and three entities that have their “single source of truth” in the extracted code. We also have six steps in the total story. Let’s plan it:

private static void setTimetype(
ColumnInfo column,
String value,

Ok, this is the foundation. We need the information about the column and the value to work on both. Now on to the configurable parts. I start with the innermost parameter: Something that converts the String value into a LocalDateTime (the time zone aspect is part of our extracted code):

private static void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,

That was the easy part. For the next configurable part, we need to be able to support both Date and Timestamp. Both are created from a long, representing milliseconds since epoch start. This is only easy if we make it easy in the moment:

private static void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, T> creation,

We added a generic type “T” that we need to specify so the compiler doesn’t act up:

private static <T> void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, T> creation,

We specify the generic in the context of the method. Not giving any clue about what types are valid keeps us maneuverable in this dimension. There is one improvement I want to make that is unusual in Java: Any name for a generic type is always one capital letter in all examples. There is no need to be so spartanic with letters. We can just name our “T” like any other type. I like “Timetype” (yes, with lowercase letters, get over it) because the method name talks about a “Timetype” already:

private static <Timetype> void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, Timetype> creation,

If you are unfamiliar with this naming scheme: The type “Timetype” doesn’t actually exist, just like “T” didn’t exist. I just view generic type names as “normal” enough to give them normal names and not just “T”, “X”, “A”, etc.

That was our second configurable parameter and it forced us to make nearly all important decisions. The third one follows suit:

private static <Timetype> void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, Timetype> creation,
Consumer<Integer, Timetype> persistence
)

The outmost calls of both code lines in the original take an integer and the specific newly created type and stores it in the prepared statement. By making everything type specific a parameter, we push all the knowledge about the prepared statement outside our extracted method. All the knowledge, with the exception of exceptions. Unfortunately, the statement.setDate() and statement.setTimestamp() both throw the checked exception SQLException. A java Consumer can’t handle checked exceptions. We need our own type for that:

public static interface TimetypePersister<Timetype> {
void persist(
int columnIndex,
Timetype value
) throws SQLException;
}

This type replaces the Consumer in our signature:

private static <Timetype> void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, Timetype> creation,
TimetypePersister<Timetype> persistence
) throws SQLException {

And now we are ready to fill in the blanks. First we refactor both calls to our method:

case Types.DATE -> {
setTimetype(
column,
value,
t -> LocalDate.from(parseToDate(t)).atStartOfDay(),
Date::new,
statement::setDate
);
}
case Types.TIMESTAMP -> {
setTimetype(
column,
value,
t -> LocalDateTime.from(parseToDateTime(t)),
Timestamp::new,
statement::setTimestamp
);
}

You probably see the nested sub-quest of the LocalDateTime-parsing right away. As I said, I won’t attempt to solve it in this blog entry. The exercise is left to the inclined reader.

Now, we can finally implement our extracted method that retains the structure of the original lines and includes the three “entities of wisdom” that are now de-duplicated:

private static <Timetype> void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, Timetype> creation,
TimetypePersister<Timetype> persistence
) throws SQLException {
LocalDateTime local = parsing.apply(value);
ZonedDateTime zoned = local.atZone(ZoneOffset.UTC);
long millis = getDateMillisFrom(zoned);
Timetype settable = creation.apply(millis);
int index = column.columnIndex();
persistence.persist(
index,
settable
);
}

You can see how we unpacked the original lines in reverse: Our method begins with the last duplicated wisdom, the time zone. Then we turn the ZonedDateTime into the milliseconds since epoch start and finally, we use the column index from the given column. In the same tenor, the calls to the configurable parts are also unrolled from the innermost to the outside.

And there we are! The boss enemy is slain. The complex and intertwined structure is extracted, most duplications are resolved. We can enjoy our dopamine hit and call it a day.

Thank you for reading this far and thinking along. What part made you pause? Maybe I can improve my explanation there for you. Drop me a comment!

Powerful Tools Level the Playing Field, But There’s a Catch (AI impressions, part 5 of 5)

This is the fifth and currently last part of the series “Impressions of Our Current AI Usage”, as outlined by the introduction article.

Software development was (and still is) a peculiar profession. The entry bar for participating was very low for decades now. All you needed was a notebook and a brain (coffee was optional). Since we all had more or less the same notebook, it all boiled down to personal skill – we might call it brain power. Those who honed their skills had a competitive advantage. But market demand for software was so high that virtually anybody with an ability to type straight could find a job or even create one’s own job by going freelance.

With the advancements in AI, the old rules begin to shift: Hardware demand skyrockets to a point where local computing is no longer feasible. Your notebook that was able to do most of the work alone is now demoted to an input/output device for a cloud-based AI service. At the same time, personal skill becomes less pertinent because we now all use the same AI services that turn us into virtually the same developer all over the globe. The demand for software is still high (and probably rising), but now the non-tech people can produce a working piece of software on their own, using the exact same tool as a professional developer would. Software gets commoditized, which in itself is a good thing. And because the output of current AI tools has acceptable quality, we might get rid of some of the low quality code that was the inevitable result of exponential growth in the profession.

So, when everybody can just open an AI session and turn into a run-of-the-mill developer, that’s a good thing, right?

“Yes” to some degree, but there is a big catch that I find underdiscussed for its impact and importance. The foundation of successful work with AI is a powerful inference system. Today, these are LLMs so big that only a few entities in the world can actually create them. The direct consequence is that the means of production are centralized and in direct control of a chosen few. I’m not an economist, but that sounds a lot like the story of the big/rich factory owner with the big machines and the tiny/poor workers that don’t have the resources to build their own factory instead. Which is exactly what software development looks like today: Every team of a few people builds their own software manufacture and achieves competitive results with it.

The catch, to me, is that we give up our economic independence while trying to gain more productivity and convenience. Just like the worker in a shoe factory can handle the machines, but not create a shoe by himself nor replicate the shoe factory. Compare that to a classic shoemaker with his workshop and you see the transformation that we are currently undertaking. The shoemaker won’t scale and might fail his business, but the factory worker is interchangeable and acquires mostly skills that tend to be useless without industry.

The catch is that AI catapults software development as a profession from craftsmanship to industrial work, with all the good and the bad things that accompany such a conversion. If we can see that clearly and govern the process, the good things might overbalance the negatives. If we aren’t aware and caught up in the hype, it’s a game of chance at best.

Can you name articles that talk about this aspect of AI usage? I would like to read them!

Integer Division has no Integrity

Quick question: In the programming language of your choice, what is -7/2 ?
Depending on your training in that programming language, that answer might surprise you or not; but if you have a somewhat mathematically inclined brain, the answer might irritate you.

If your answer is anything other than the following, you might really consider switching, but I imagine, it’s one of these:

  • -3 (as integer or as untyped constant like in Go)
  • -4 (as integer)
  • -3.5 (as float or similar)
  • -7/2 (as rational type like in very mathematically inclined languages)

My Point is: “-7/2” in itself is not a legal entity with a “have you read the fine print” character; its nature is very clear and widely understood.
As in: You eat the half of each of the seven cakes in your overly giant fridge and then need to explain your theory of mental sanity to your significant other. Common scenario.

Now if you have been programming for too long, it might be tempting to rationalize: a division operation on two integers must also be an integer, for consistency reasons or the likes. This doesn’t hold, though, as:

a) it’s fundamentally wrong™, because integer numbers are a Ring, not a Field:
They are not closed under the operation of division.
b) that would violate the Principle of Least Astonishment, as integer data types are modeled after integer numbers in the real world; natural Mathematics is the domain-specific language to which your code has to adjust in order to qualify for Clean Code.

Now, (b) comes with the caveat that the real -3.5 is a real number and floating-point arithmetics like IEEE 754 cannot exactly resemble that, so unless you really do rational 7/2 arithmetics like e.g. in Lisp or Clojure, the deviation is still there; but this is the difference between some stolen finger tips of buttercream and half a cake, so this is no joke.

My point overall is, that if cleanliness and understandability is one of your goals, you should at least think about what your code is doing and whether most of any possible readers have the exact same model in their mind than you do;
if they do, just write the shortest way possible, if they might not, an additional cast / rounding operation might ground their expectations.

And speaking of resembling the real-world mathematics, I am not even sure whether the rounding convention for negative numbers is taught consistently throughout the world; our school back then went with “round away from zero”, i.e. -7/2 would become -4, which is rather seldom found (quick search tells me: Python 2 or Ruby. Truncating to -3 is cheaper on the hardware, I suppose.
I will now resist the investigation between rigid definition vs. cultural contract.

Anyway. Enjoy your cake 🙂

Setting up clangd on Windows

clangd is a language server that allows other programs to access introspection and refactoring capabilities for C++ via the language server protocol. It was originally meant for code editors. For example, VS Code can use it via this extension, and provide functions like “Go to definition”, “rename”. With LLMs and their harnesses, they can give those capabilities to agents which otherwise navigate code-bases via the equivalent of Windows Notepad, e.g. by reading and grep’ing. For example, for pi.dev, there are these extensions that can use clangd.

How it works

clangd is usually run by the host application, which communicates with it by sending and receiving json-rpc fragments via stdin/stdout. It can either be kept running as a continuous service and serve from memory, or keep its preprocessed info persisted to disk and use that from per-request instances/processes.

Install clangd

Go to https://github.com/llvm/llvm-project/releases, and download the Windows installer for LLVM. At the time of this writing, the version is 23.1.2. Install that. By default, the executables will go into C:\Program Files\LLVM\bin.

Add that to your PATH, e.g. by pressing Win+R and running rundll32.exe sysdm.cpl,EditEnvironmentVariables. It should be sufficient to add it for your user.

Setup your project

Make sure the CMake variable CMAKE_EXPORT_COMPILE_COMMANDS is set to ON, which will generate a compile_commands.json in your build folder. This file contains the compiler invocations for your project, which clangd needs to correctly interpret your code.

Note that the CMake documentation specifically mentions that CMAKE_EXPORT_COMPILE_COMMANDS does not work well with unity builds. This is a shame, because they rock.

Drop a file called .clangd in your project root where to find that file. Since I usually use conan and their layout, my builds go in build/Debug or build/Release. Here, I can only pick one at a time:

  CompileFlags:
    CompilationDatabase: "build/Debug"

At this point, you should be able to use clangd on your code. There are many other things you can configure in that file, e.g. whether to store the persistent index I mentioned above.

Try it out!

The VS Code extension I mentioned at the top is a good way to check if everything is working as expected. Hovering over an identifier should give you some nice info about it:

Code snippet in C++ showing the declaration of a logger variable using the spdlog library and error handling in a try-catch block.

Give your workarounds a review date

A dependency has a bug. You investigate, find the offending code and add a small workaround. The application works again. There is a regression test, a comment explaining the problem and perhaps a link to an upstream issue.

That seems like a reasonable place to stop. Except for one small question: Who is going to remove the workaround?

Temporary code tends to stay

A comment saying “remove this when the library is fixed” relies on somebody reading it at the right moment. That moment may arrive six months later, during a dependency update performed by somebody who has never seen this part of the application. The workaround keeps running. The tests stay green. Nobody has a reason to look.

One way to make that reason appear is surprisingly simple: write a test that checks the dependency version.

A test that asks for a review

Suppose an application contains a compatibility patch for version 2.4.1 of a library. Alongside the tests for the patched behavior, add another assertion: the loaded library version must still be 2.4.1. Its failure message should explain the intended action:

The library version has changed. Check whether the compatibility patch is still necessary and compatible. Update the expected version only after this review.

When somebody updates the library to 2.4.2, this test fails – even if the application still works perfectly.

Usually, a test that fails after a harmless update would be a nuisance. Here, that interruption is its purpose. The version records the environment in which somebody last examined the workaround. Changing it means that this review is due again. This matters because a successful regression test can hide an obsolete patch.

Why green tests are not enough

A successful regression test can hide an obsolete patch. Imagine a library method that mishandles empty input. A local override corrects the behavior, and a test verifies the result. Later, the library fixes its implementation. If the override remains active, the regression test may continue exercising the local code. Everything stays green, but the application still owns a piece of code it could have deleted.

Worse, an override may bypass future improvements to the original method. A workaround that once restored compatibility can eventually become the thing that prevents it. The version check gives us a chance to notice.

Make the failure actionable

The failure message deserves some care. “Expected 2.4.1, got 2.4.2” provides very little help. A useful message identifies the patch, explains what needs checking and tells the reader what updating the expectation means. The next developer should be able to start the investigation without first reconstructing why the test exists.

Use the interruption selectively

There is a cost, of course. Exact version checks also interrupt updates that have nothing to do with the workaround. I would use them selectively for local modifications to third-party behavior, especially where internal APIs are involved. Applying them to every dependency would quickly create noise.

Conclusion

Temporary code has a habit of becoming permanent because it keeps working quietly. Giving it a test that eventually asks for attention is a small way to help it leave.

Modern python build and packaging

Developing software is not only about code but also about delivering the software to your customers. Sometimes, app stores are the solution, at other times it may be software-as-a-service (Saas) but there are also occasions where installers, compressed archives, system packages oder single file executables are (or fat-jars) are the way to go.

While python as a language and ecosystem has many points in its favour I personally found its build and delivery infrastructure lacking. As soon as you project requires a bunch of dependencies and users expect something like apt update to handle your software, too, things can get messy.

There were multiple options, tools and variants for the different aspects of software delivery:

  • setuptools vs. distutils
  • venvs, site-packages, pip, pipx, wheels
  • packaging tools like py2dsc-deb, bdist, pybuild
  • homebrew, anaconda etc.

For non-fulltime python developers like me there was always quite some doubt about the current idiomatic and preferred solution. In my C++ projects, this problem was largely solved by CMake+CPack. For java, maven and gradle provided easy solutions for many delivery and deployment tasks.

From my limited perspective the situation in the python world is more fluid and still in the process of finding a best-of-breed solution and evolving mostly that one option. Maybe we have arrived at that with the “Python Packaging Authority” and the work around pyproject.toml.

Many of our customers use Debian oder Debian-based distributions like Ubuntu and like having packages in their own apt-repository that they can install using system tools. So building deb-packages feels like the natural way to deliver your software to these customers.

Building Debian-Packages

In the past I used several approaches to packaging our python projects as debian packages:

  • “python setup.py sdist” in combination with plain dh_make and manual editing of the metadata files
  • pybuild/dh_python3
  • stdeb, with py2dsc etc.

Nowadays I am moving away from setup.py towards pyproject.toml. I find it quite readable and it provides neat features like executable scripts and single-sourcing the project version and a simple declarative project format. Together with debhelper and pybuild packaging becomes quite easy.

I create the necessary debian metadata-files, e.g. using dh_make or by hand.

The modified rules file may look as simple as:

#!/usr/bin/make -f

%:
	dh $@ --buildsystem=pybuild

# We do not have tests yet, so do not run pytest...
override_dh_auto_test:

My pyproject.toml may look like this:

[build-system]
requires = [
    "setuptools>=61.0.0",
    "wheel"
]
build-backend = "setuptools.build_meta"

[project]
name = "myproject"
dynamic = ["version"]
description = 'My project illustrates packaging'
readme = "README.md"
requires-python = ">=3.10"
maintainers = [
    {name = "Mihael Koep", email = "mihael.koep@softwareschneiderei.de"}
]

[project.scripts]
OmsMaxx = "myproject:start"

[project.urls]
Homepage = "https://gitlab.com/example/python/myproject"
Repository = "https://gitlab.com/example/python/myproject"
Issues = "https://gitlab.com/example/python/myproject/-/work_items"

[tool.setuptools.packages.find]
include = ["myproject"]

[tool.setuptools.dynamic]
version = { attr = "myproject.__version__" }

If the tools python3-build, python3-wheel, dh-python and pybuild-plugin-pyproject are available, building a working package should be as easy as running dpkg-buildpackage like below:

dpkg-buildpackage -b -us -uc

Conclusion

I hope this pybuild/pyproject.toml infrastructure with it extensibility and relative clarity will become (and stay) the de facto standard and improve over time.

This would end my uncertainty and provide a sensible default to shipping and packaging python projects.

Have you other approaches or opinions on this topic?

The Number You See Is Not Necessarily the Number You Get

There is a particular class of bugs that is easy to dismiss as a floating-point problem.

For example you have a value in Excel: 25604.88

You open the spreadsheet, and that is exactly what you see. You read the cell with a library, however, and suddenly you get something like:

25604.880000000001

At first this looks like a broken spreadsheet, a broken XML file, or a broken library.

It is none of those.

An .xlsx file is a ZIP archive containing XML files. If you inspect the worksheet XML, you may encounter a value along the lines of:

<c r="A1">
<v>25604.880000000001</v>
</c>

This is not Excel suddenly deciding that the user entered a different number.

The important distinction is that Excel displays numbers according to formatting and its rules for presenting floating-point values. The underlying numeric representation is based on binary floating point. Microsoft documents Excel’s use of IEEE 754 floating-point representation and the resulting precision limitations.

This distinction is worth understanding because it can turn a seemingly harmless value into a surprisingly difficult integration bug.

Why can’t a computer just store 25604.88?

Because computers don’t normally store floating-point numbers as decimal fractions.

A typical double uses the IEEE 754 binary64 representation. Conceptually, the value is represented using a sign, an exponent and a significand. There are 64 bits available in total, with 53 bits of precision in the significand.

This works beautifully for binary fractions.

For example:

0.5 = 1/2 = 0.1b
0.25 = 1/4 = 0.01b
0.75 = 1/2 + 1/4 = 0.11b
0.125 = 1/8 = 0.001b

All of these can be represented exactly in binary.

How to convert with simple Operations:

0.625 * 2 = 1.25 => 1
0.25 * 2 = 0.5 => 0
0.5 * 2 = 1 => 1
=> 0.101b

But other fractions can not be represented by binary fractions exactly like 0.1.

0.1 * 2 = 0,2 => 0
0.2 * 2 = 0.4 => 0
0.4 * 2 = 0.8 => 0
0.8 * 2 = 1.6 => 1
0.6 * 2 = 1.2 => 1
0.2 * 2 = 0.4 => 0 // REPEAT
0.4 * 2 = 0.8 => 0
0.8 * 2 = 1.6 => 1
0.6 * 2 = 1.2 => 1
=> 0.00011001100110011001100110011...b

It is a unending number, limited by memory space for storing the number. When translating back to decimal is won’t be exactly 0.1 again.

0.00011001100110011001100110011b = 0.09999999962747097015d

Or for our example with 25604.88 the decimals 88 will be convert like this:

0.88 * 2 = 1.76 => 1
0.76 * 2 = 1.52 => 1
0.52 * 2 = 1.04 => 1
0.04 * 2 = 0.08 => 0
0.08 * 2 = 0.16 => 0
0.16 * 2 = 0.32 => 0
0.32 * 2 = 0.64 => 0
0.64 * 2 = 1.28 => 1
0.28 * 2 = 0.56 => 0
0.56 * 2 = 1.12 => 1
0.12 * 2 = 0.24 => 0
0.24 * 2 = 0.48 => 0
0.48 * 2 = 0.96 => 0
0.96 * 2 = 1.92 => 1
0.92 * 2 = 1.84 => 1
0.84 * 2 = 1.68 => 1
0.68 * 2 = 1.36 => 1
0.36 * 2 = 0.72 => 0
0.72 * 2 = 1.44 => 1
0.44 * 2 = 0.88 => 0 // REPEAT
=> 0.11100001010001111010...b

What does this mean for programmers?

The important lesson is we need to know what we are dealing with.

When reading numbers from external sources such as Excel, JSON or a database, don’t assume that the value you see is exactly the value your program receives. Check the actual value and the type your library produces.

For many calculations, this is completely fine. But when exact decimal values matter — for example, with money, accounting or other business rules — don’t leave this to chance.

Use a representation that matches the requirement. For example use BigDecimal or BigFraction for exacter decimal values.

You heard it here first: Stay clear of Always-Updating Software

Depending on who you ask, I would be described as one of the most laid-back, laissez-faire people around, or a dangerously megalomaniac, unrealiable weirdo [Author’s Note: this would lead us astray], and so in terms of what-software-to-use, I would consider most people as the world-leading expert in what works best for them personally.

However, one pattern seems to grind my gears, and while I don’t feel that the Schneide Blog is the best place for overbearing rants based on minor inconveniences, I still recognize a pattern that might be worrying because it is, at least, getting more and more common.

There is some software that is ALWAYS, whenever you start it, surprising you with a software update. Sometimes it is forced upon you, sometimes with sleazy sleights of hand, sometimes you even have the right to have a say in that matter, but nevertheless, it’s ALWAYS. Might not be numerical-always, but still noone on Polymarket would bet against you, so, it’s ALWAYS.

In case you would’t know what software I am talking about – I can give you several names from the top of my head – but you probably know lots of examples on your own 🙂

My core message is, that currently, these scream: “I will bring you harm!”

What is going on with software, having a need to constantly disturb my workflow – a thing that is holy for any serious software developer?
(a) maybe the border-radius of the third panel in the About… dialog was slightly off, and the application shutdown could be improved about 200-300µs.
(b) they are now relying purely on AI to write their most critical core features, with no sensitivity for security whatsoever.

And while I can totally understand (a), in current times you should be honest to endorse – unless you know better – it must be (b).

I, myself, am sometimes willing to recognize that I am not the most risk-averse human in the history of the world, maybe ever; and you probably really are more of an expert of what is good for you than my opinion would count; so, I also use some of these regularly. So, do your own what-could-possibly-go-wrong-gambles, but it is our job to at least feel smart about it when shit is going down (in hindsight, naturally).

Point is, I _could_ be bothered to check the release notes of any of these before any update. But because their move is “we demand on this urgency”, and my thought is “I’m in the process of saving my customer’s life (again)”, I will rarely do that, and unless they communicate better, I think there needs to be a rightful place to call these dangerously megalomaniac, unreliable weirdos out.

Docker in Continuous Integration

Docker – or more general containerization – can be applied in several areas of software development to improve many aspects of our work. The most common are:

  • In development to easily setup an environment for running and developing the software without installing every manually on the target OS.
  • For deployment either in docker-stacks (e.g. managed by portainer) or kubernetes only requiring hosts able to run containers without specialized setups.
  • For building software artifacts on your continuous integration (CI) infrastructure with the same benefits as the former two applications.

In essence you trade snowflaking all the involved machines (dev, CI nodes, servers) for some additional complexity around Dockerfiles, images, containers and their orchestration.

In this post I want to focus on the CI part:

Using Docker in CI

Administrators and infrastructure people usually will cheer if they just need to provide docker-capable nodes for developers to build their projects on. No need to provide certain runtimes, libraries, tools and configurations anymore. No need to negotiate with developers about the environment all the time and keeping it updated, appropriate and secure.

Developers on the other hand need to work with another tool encapsulating their build process. While this gives them a lot of freedom in choosing and shaping the environment for their builds (the “inside” of the containers) it adds pitfalls and complexity on the outside.

If your artifact is not (only) another docker image but things like test- and coverage reports, binaries or other files you have to move stuff from inside the containers to the outside aka host. In CI use cases you have essentially the following alternatives, each with different pros and cons:

Bind mounts for a container run

In this approach you build your environment using a normal docker build command and follow it with a docker run, e.g.:

docker build -t build-project .
docker run --rm -u `id -u` -v `pwd`:/build build-project

Pros:

  • This approach feels natural and does automatic container cleanup by using the --rm option on docker run.
  • It also enables dependency caches etc. on the host which can reduce build times, even across projects.
  • Sometimes you do not even need a dockerfile but can use plain docker images without building one yourself, e.g. eclipse-temurin:21-jdk.

Cons:

  • It actually runs a container and has limited file system access to the host.
  • It may leave build artifacts and temporary files on the host.
  • It may create file ownership issues, hence the -u `id -u` arguments to docker run in the example.

Image build and container to copy from

This approach is similar to above in that is consists of several steps but it does not need mounts and does most of the stuff during docker build:

set -eu
docker build -t build-project .
container_id=$(docker create build-project)
trap 'docker rm -f "$container_id" >/dev/null 2>&1 || true' EXIT
docker start $container_id
docker cp $container_id:/buildresults/ ./artifacts/

Pros:

  • Most stuff happens inside docker build.
  • You can use layer caching to improve build times.
  • Cleanup is done using shell mechanismns (trap), can also be performed using different means.
  • What leaves the container is exactly and explicitly controllable

Cons:

  • You need to run a container
  • You need to take care of container cleanup
  • Full benefit of layer caching may require thought/engineering to improve build times

Multi-stage build with scratch-image output

In this approach we never actually run a container and let docker build copy the specified artifacts to the host using a multi-stage build. Most of the interesting bits are in the Dockerfile while the build call looks like:

docker build --output artifacts/ .

The Dockerfile gets a bit more complex:

FROM python:3.14 AS build
WORKDIR /build

COPY . .

RUN pip3 install --no-cache-dir -r requirements.txt
RUN python3 -m build

FROM scratch AS package
COPY --from=build /build/dist/*.whl .

Pros:

  • No need to explicitly cleanup containers or dependency caches
  • Layer caching possible
  • No need to run a container explicitly, building the image is enough
  • Only one simple call in the build pipeline

Cons:

  • More complexity inside the Dockerfile

Conclusion

Using docker for building software has many advantages with the price of an additional tools and its own complexities. Several, easily adaptable approaches exist to facilitate containerization in CI environments. Use the one that fits your requirements and environment best.

Which approach do you like best? What are you using in your build and delivery pipelines?

I would be glad to hear your thoughts and comments.

Dynamic Device Classes in Python

When writing PyTango device servers, it is common to implement one Python class per Tango device class. For small projects, this approach is simple and easy to understand. However, it becomes cumbersome when the set of available devices is not known at development time.

Consider a device server that should be entirely driven by a configuration file. Instead of hard-coding every supported device class, the server reads the device definitions at startup and creates the required Tango device classes automatically.

The goal is to write a generic PyTango server that does not need to be modified whenever a new device class is added. If a new entry appears in the configuration file, the server should simply create the corresponding Tango device class.

Python type

At first glance, this sounds unusual. After all, Python classes are typically defined using the familiar class keyword:

class Motor(Device):
    pass

Most Python developers stop here and never think about how classes are actually created. Under the hood, however, classes are objects themselves, and Python provides a built-in mechanism to construct them dynamically.

The function responsible for this is type().

Most of us use it in its simplest form to inspect the type of an object:

print(type(42))
# <class 'int'>

Less well known is its three-argument form:

type(name, bases, attributes)

The arguments are:

  • name: the name of the new class.
  • bases: a tuple of parent classes.
  • attributes: a dictionary containing class attributes and methods.

This means that the following definition is equivalent to the above class definition.

Motor = type(
    "Motor",
    (Device,),
    {},
)

The resulting object is exactly the same: a Python class that can be instantiated or registered with PyTango.

The third argument of type() becomes particularly interesting when more than just the class name should be configurable. The attributes dictionary allows methods, properties, or other class members to be added dynamically. This can be useful when Tango attributes or commands are also described in the configuration file.

Motor = type(
    "Motor",
    (Device,),
    {
        "some_property": 42,
    },
)

a = Motor()
print(a.some_property)
# 42

Conclusion

For our use case, creating classes dynamically becomes straightforward. Reading the configuration file and creating the required classes can be done in a simple loop. From PyTango’s perspective, there is no difference between a statically defined class and one created dynamically using type(). Both behave like ordinary Python classes.