Setting up clangd on Windows

clangd is a language server that allows other programs to access introspection and refactoring capabilities for C++ via the language server protocol. It was originally meant for code editors. For example, VS Code can use it via this extension, and provide functions like “Go to definition”, “rename”. With LLMs and their harnesses, they can give those capabilities to agents which otherwise navigate code-bases via the equivalent of Windows Notepad, e.g. by reading and grep’ing. For example, for pi.dev, there are these extensions that can use clangd.

How it works

clangd is usually run by the host application, which communicates with it by sending and receiving json-rpc fragments via stdin/stdout. It can either be kept running as a continuous service and serve from memory, or keep its preprocessed info persisted to disk and use that from per-request instances/processes.

Install clangd

Go to https://github.com/llvm/llvm-project/releases, and download the Windows installer for LLVM. At the time of this writing, the version is 23.1.2. Install that. By default, the executables will go into C:\Program Files\LLVM\bin.

Add that to your PATH, e.g. by pressing Win+R and running rundll32.exe sysdm.cpl,EditEnvironmentVariables. It should be sufficient to add it for your user.

Setup your project

Make sure the CMake variable CMAKE_EXPORT_COMPILE_COMMANDS is set to ON, which will generate a compile_commands.json in your build folder. This file contains the compiler invocations for your project, which clangd needs to correctly interpret your code.

Note that the CMake documentation specifically mentions that CMAKE_EXPORT_COMPILE_COMMANDS does not work well with unity builds. This is a shame, because they rock.

Drop a file called .clangd in your project root where to find that file. Since I usually use conan and their layout, my builds go in build/Debug or build/Release. Here, I can only pick one at a time:

  CompileFlags:
    CompilationDatabase: "build/Debug"

At this point, you should be able to use clangd on your code. There are many other things you can configure in that file, e.g. whether to store the persistent index I mentioned above.

Try it out!

The VS Code extension I mentioned at the top is a good way to check if everything is working as expected. Hovering over an identifier should give you some nice info about it:

Code snippet in C++ showing the declaration of a logger variable using the spdlog library and error handling in a try-catch block.

Give your workarounds a review date

A dependency has a bug. You investigate, find the offending code and add a small workaround. The application works again. There is a regression test, a comment explaining the problem and perhaps a link to an upstream issue.

That seems like a reasonable place to stop. Except for one small question: Who is going to remove the workaround?

Temporary code tends to stay

A comment saying “remove this when the library is fixed” relies on somebody reading it at the right moment. That moment may arrive six months later, during a dependency update performed by somebody who has never seen this part of the application. The workaround keeps running. The tests stay green. Nobody has a reason to look.

One way to make that reason appear is surprisingly simple: write a test that checks the dependency version.

A test that asks for a review

Suppose an application contains a compatibility patch for version 2.4.1 of a library. Alongside the tests for the patched behavior, add another assertion: the loaded library version must still be 2.4.1. Its failure message should explain the intended action:

The library version has changed. Check whether the compatibility patch is still necessary and compatible. Update the expected version only after this review.

When somebody updates the library to 2.4.2, this test fails – even if the application still works perfectly.

Usually, a test that fails after a harmless update would be a nuisance. Here, that interruption is its purpose. The version records the environment in which somebody last examined the workaround. Changing it means that this review is due again. This matters because a successful regression test can hide an obsolete patch.

Why green tests are not enough

A successful regression test can hide an obsolete patch. Imagine a library method that mishandles empty input. A local override corrects the behavior, and a test verifies the result. Later, the library fixes its implementation. If the override remains active, the regression test may continue exercising the local code. Everything stays green, but the application still owns a piece of code it could have deleted.

Worse, an override may bypass future improvements to the original method. A workaround that once restored compatibility can eventually become the thing that prevents it. The version check gives us a chance to notice.

Make the failure actionable

The failure message deserves some care. “Expected 2.4.1, got 2.4.2” provides very little help. A useful message identifies the patch, explains what needs checking and tells the reader what updating the expectation means. The next developer should be able to start the investigation without first reconstructing why the test exists.

Use the interruption selectively

There is a cost, of course. Exact version checks also interrupt updates that have nothing to do with the workaround. I would use them selectively for local modifications to third-party behavior, especially where internal APIs are involved. Applying them to every dependency would quickly create noise.

Conclusion

Temporary code has a habit of becoming permanent because it keeps working quietly. Giving it a test that eventually asks for attention is a small way to help it leave.

Modern python build and packaging

Developing software is not only about code but also about delivering the software to your customers. Sometimes, app stores are the solution, at other times it may be software-as-a-service (Saas) but there are also occasions where installers, compressed archives, system packages oder single file executables are (or fat-jars) are the way to go.

While python as a language and ecosystem has many points in its favour I personally found its build and delivery infrastructure lacking. As soon as you project requires a bunch of dependencies and users expect something like apt update to handle your software, too, things can get messy.

There were multiple options, tools and variants for the different aspects of software delivery:

  • setuptools vs. distutils
  • venvs, site-packages, pip, pipx, wheels
  • packaging tools like py2dsc-deb, bdist, pybuild
  • homebrew, anaconda etc.

For non-fulltime python developers like me there was always quite some doubt about the current idiomatic and preferred solution. In my C++ projects, this problem was largely solved by CMake+CPack. For java, maven and gradle provided easy solutions for many delivery and deployment tasks.

From my limited perspective the situation in the python world is more fluid and still in the process of finding a best-of-breed solution and evolving mostly that one option. Maybe we have arrived at that with the “Python Packaging Authority” and the work around pyproject.toml.

Many of our customers use Debian oder Debian-based distributions like Ubuntu and like having packages in their own apt-repository that they can install using system tools. So building deb-packages feels like the natural way to deliver your software to these customers.

Building Debian-Packages

In the past I used several approaches to packaging our python projects as debian packages:

  • “python setup.py sdist” in combination with plain dh_make and manual editing of the metadata files
  • pybuild/dh_python3
  • stdeb, with py2dsc etc.

Nowadays I am moving away from setup.py towards pyproject.toml. I find it quite readable and it provides neat features like executable scripts and single-sourcing the project version and a simple declarative project format. Together with debhelper and pybuild packaging becomes quite easy.

I create the necessary debian metadata-files, e.g. using dh_make or by hand.

The modified rules file may look as simple as:

#!/usr/bin/make -f

%:
	dh $@ --buildsystem=pybuild

# We do not have tests yet, so do not run pytest...
override_dh_auto_test:

My pyproject.toml may look like this:

[build-system]
requires = [
    "setuptools>=61.0.0",
    "wheel"
]
build-backend = "setuptools.build_meta"

[project]
name = "myproject"
dynamic = ["version"]
description = 'My project illustrates packaging'
readme = "README.md"
requires-python = ">=3.10"
maintainers = [
    {name = "Mihael Koep", email = "mihael.koep@softwareschneiderei.de"}
]

[project.scripts]
OmsMaxx = "myproject:start"

[project.urls]
Homepage = "https://gitlab.com/example/python/myproject"
Repository = "https://gitlab.com/example/python/myproject"
Issues = "https://gitlab.com/example/python/myproject/-/work_items"

[tool.setuptools.packages.find]
include = ["myproject"]

[tool.setuptools.dynamic]
version = { attr = "myproject.__version__" }

If the tools python3-build, python3-wheel, dh-python and pybuild-plugin-pyproject are available, building a working package should be as easy as running dpkg-buildpackage like below:

dpkg-buildpackage -b -us -uc

Conclusion

I hope this pybuild/pyproject.toml infrastructure with it extensibility and relative clarity will become (and stay) the de facto standard and improve over time.

This would end my uncertainty and provide a sensible default to shipping and packaging python projects.

Have you other approaches or opinions on this topic?

The day the machines took the fun away (AI impressions, part 4 of 5)

This is the fourth part of the series “Impressions of Our Current AI Usage”, as outlined by the introduction article.

The title of this blog entry is also a continuation of an earlier article I wrote about AI advancements in computer gaming, eight years ago.

And if you are inclined to read words out “loud in your head”, you might read it in Trent Reznor’s voice this time.

There is something weird about human’s sense of achievement. Things seem to be “worth” more when they are harder to reproduce. That’s why we value live music over recordings, idolize top athletes and install examinations and tests to filter for official positions.

In the world of software development, the entry bar hung low for a decade now. Gone were the days of rigorous academic training and selection for a degree. If you wanted to go the freelance route, a decent notebook was all it took to be equipped for the job. After that, you could turn electricity and coffee into source code and source code into money.

The reason to enter the field of programming are manyfold, but two main drivers are common: Passion and money. Let’s put money aside and concentrate on the passion aspect: Many programmers love what they do. They even do it for free, in their spare time. The process of creating source code is a deeply rewarding activity for them. It’s not the finished result that “achieves” something, it’s the journey towards it.

I hope you can easily see the similarity to other mainly passion-driven activities like making music or other forms of art, athletic endeavours like mountain-climbing or scientific discoveries. You probably have a hobby yourself that you are passionate about.

Now imagine that somebody automates the journey. You don’t need to climb a mountain, you can just take the elevator. It’s faster, more convenient – and boring. You don’t need to actually play the guitar, you just need to hold it and it plays the most amazing solos itself. And you don’t need to sing, your voice will be autotuned correctly. You’ll be an amazing musician – like everybody else.

When you automate the journey part of an activity, only the chore parts remain to be done. You still need to prepare for thin air on top of the mountain and no autotune will correct your actual song text, so you still need to learn it.

In software development, creating source code is the journey part of the work. There are many supporting activities that need to be done for the journey to be successful on the long run. Things like modelling, planning iterations, bookkeeping the iterations, code review, functional review and manual testing are only some things that still need to be done, regardless of who actually programs. If you can alternate between the activities and have a suitable chore-passion-balance, things feel natural. If only the chore part is up to you, you are the assistant to the machine.

The problem is, that the support work is still necessary. For a short time, you can live without it, but things will inevitably fall apart without proper management and housekeeping. In larger software development teams, these roles are officially assigned to additional team members, but in reality, everybody needs to participate a bit or the process fails.

So why don’t we focus on the chore parts to be automated? There are attempts and the AI is getting better in that regard. But code creation is by far the easiest activity a machine can learn. It produces an immediately digestable result, follows clear rules and can be verified on correctness in multiple ways. Most of the accompanying work is more ephemeral, leaves less of a trace and is not as teachable as “typing out the right sequence of characters really fast”.

In my blog article about computer games eight years ago, I wrote that human players prefer to team up with bots instead of other humans. This only holds true as long as the speed of the game is adapted to human levels – that is, considerably slowed down. If the bots would play only with themselves, they could speed up by several orders of magnitude. And that’s the problem with integrating AI-driven work units into a human-driven choreography: The speedup is not compatible.

In the software development community, there is a legend about a “10X” developer, a human being that is capable of consistenly solving a problem in software ten times faster than the average fellow, without sacrificing quality or longterm viability of the solution. I can attest that these people exist and I’ve seen a few in my career. If you think about AI as just another programmer behind a chat window, it’s more like a 1000X developer (not to be confused with a hundred 10X developers, a whole other bunch of problems!). There is literally nobody human on this planet that can even closely compete with the speed of an AI agent (unless you slow it down by artificial hurdles).

So, looking at the situation from a money viewpoint, the situation is clear: One source of code can provide 1x to 10x product in a given timeframe, while the other source delivers a hundred times more. Both cost roughly the same. But by buying “passion-free” code, the money-driven decider produces some long-term problems that will lead to diminishing returns:

  • AI code is looped back into the training set. So it will reinforce its own idioms, good and bad, just by multiplying them hundredfold into the existing world code base.
  • If AI code prevented a human developer from nurturing her or his passion, it stunts the human creativity pool and leads to a bleaker learning set.
  • The higher the AI code jenga tower is built, the less possible it will be for human developers to interact with it. Relying on AI will be a self-fulfilling prophecy with a ever-shrinking escape window.

To come to a conclusion here: I like to call AI-generated code “passion-free”. And I think that we concentrate too hard on the two questions “can it be done with AI?” and “will it cost less with AI?” and miss the opportunity for a third question that I find important, too: “will it make our lives more fun to do it with AI?“

The Number You See Is Not Necessarily the Number You Get

There is a particular class of bugs that is easy to dismiss as a floating-point problem.

For example you have a value in Excel: 25604.88

You open the spreadsheet, and that is exactly what you see. You read the cell with a library, however, and suddenly you get something like:

25604.880000000001

At first this looks like a broken spreadsheet, a broken XML file, or a broken library.

It is none of those.

An .xlsx file is a ZIP archive containing XML files. If you inspect the worksheet XML, you may encounter a value along the lines of:

<c r="A1">
    <v>25604.880000000001</v>
</c>

This is not Excel suddenly deciding that the user entered a different number.

The important distinction is that Excel displays numbers according to formatting and its rules for presenting floating-point values. The underlying numeric representation is based on binary floating point. Microsoft documents Excel’s use of IEEE 754 floating-point representation and the resulting precision limitations.

This distinction is worth understanding because it can turn a seemingly harmless value into a surprisingly difficult integration bug.

Why can’t a computer just store 25604.88?

Because computers don’t normally store floating-point numbers as decimal fractions.

A typical double uses the IEEE 754 binary64 representation. Conceptually, the value is represented using a sign, an exponent and a significand. There are 64 bits available in total, with 53 bits of precision in the significand.

This works beautifully for binary fractions.

For example:

0.5   = 1/2       = 0.1b
0.25  = 1/4       = 0.01b
0.75  = 1/2 + 1/4 = 0.11b
0.125 = 1/8       = 0.001b

All of these can be represented exactly in binary.

How to convert with simple Operations:

0.625 * 2 = 1.25 => 1
0.25  * 2 = 0.5  => 0
0.5   * 2 = 1    => 1
=> 0.101b

But other fractions can not be represented by binary fractions exactly like 0.1.

0.1 * 2 = 0,2 => 0
0.2 * 2 = 0.4 => 0
0.4 * 2 = 0.8 => 0
0.8 * 2 = 1.6 => 1
0.6 * 2 = 1.2 => 1 
0.2 * 2 = 0.4 => 0 // REPEAT
0.4 * 2 = 0.8 => 0
0.8 * 2 = 1.6 => 1
0.6 * 2 = 1.2 => 1
=> 0.00011001100110011001100110011...b

It is a unending number, limited by memory space for storing the number. When translating back to decimal is won’t be exactly 0.1 again.

0.00011001100110011001100110011b = 0.09999999962747097015d

Or for our example with 25604.88 the decimals 88 will be convert like this:

0.88 * 2 = 1.76 => 1
0.76 * 2 = 1.52 => 1
0.52 * 2 = 1.04 => 1
0.04 * 2 = 0.08 => 0
0.08 * 2 = 0.16 => 0 
0.16 * 2 = 0.32 => 0 
0.32 * 2 = 0.64 => 0
0.64 * 2 = 1.28 => 1
0.28 * 2 = 0.56 => 0
0.56 * 2 = 1.12 => 1
0.12 * 2 = 0.24 => 0
0.24 * 2 = 0.48 => 0
0.48 * 2 = 0.96 => 0
0.96 * 2 = 1.92 => 1
0.92 * 2 = 1.84 => 1
0.84 * 2 = 1.68 => 1
0.68 * 2 = 1.36 => 1
0.36 * 2 = 0.72 => 0
0.72 * 2 = 1.44 => 1
0.44 * 2 = 0.88 => 0 // REPEAT
=> 0.11100001010001111010...b

What does this mean for programmers?

The important lesson is we need to know what we are dealing with.

When reading numbers from external sources such as Excel, JSON or a database, don’t assume that the value you see is exactly the value your program receives. Check the actual value and the type your library produces.

For many calculations, this is completely fine. But when exact decimal values matter — for example, with money, accounting or other business rules — don’t leave this to chance.

Use a representation that matches the requirement. For example use BigDecimal or BigFraction for exacter decimal values.

You heard it here first: Stay clear of Always-Updating Software

Depending on who you ask, I would be described as one of the most laid-back, laissez-faire people around, or a dangerously megalomaniac, unrealiable weirdo [Author’s Note: this would lead us astray], and so in terms of what-software-to-use, I would consider most people as the world-leading expert in what works best for them personally.

However, one pattern seems to grind my gears, and while I don’t feel that the Schneide Blog is the best place for overbearing rants based on minor inconveniences, I still recognize a pattern that might be worrying because it is, at least, getting more and more common.

There is some software that is ALWAYS, whenever you start it, surprising you with a software update. Sometimes it is forced upon you, sometimes with sleazy sleights of hand, sometimes you even have the right to have a say in that matter, but nevertheless, it’s ALWAYS. Might not be numerical-always, but still noone on Polymarket would bet against you, so, it’s ALWAYS.

In case you would’t know what software I am talking about – I can give you several names from the top of my head – but you probably know lots of examples on your own 🙂

My core message is, that currently, these scream: “I will bring you harm!”

What is going on with software, having a need to constantly disturb my workflow – a thing that is holy for any serious software developer?
(a) maybe the border-radius of the third panel in the About… dialog was slightly off, and the application shutdown could be improved about 200-300µs.
(b) they are now relying purely on AI to write their most critical core features, with no sensitivity for security whatsoever.

And while I can totally understand (a), in current times you should be honest to endorse – unless you know better – it must be (b).

I, myself, am sometimes willing to recognize that I am not the most risk-averse human in the history of the world, maybe ever; and you probably really are more of an expert of what is good for you than my opinion would count; so, I also use some of these regularly. So, do your own what-could-possibly-go-wrong-gambles, but it is our job to at least feel smart about it when shit is going down (in hindsight, naturally).

Point is, I _could_ be bothered to check the release notes of any of these before any update. But because their move is “we demand on this urgency”, and my thought is “I’m in the process of saving my customer’s life (again)”, I will rarely do that, and unless they communicate better, I think there needs to be a rightful place to call these dangerously megalomaniac, unreliable weirdos out.

Largest rectangle in a bitmap

Imagine you have a grid structure with some of the tiles marked as blocked while the others are free. So a grid with zeros (free) and ones (blocked), in other words: a bitmap. Something like this, with blocked tiles marked as blue:

Now the task is to figure out the biggest rectangle you can fit into this bitmap without using any of the blocked tiles, using an interface to the grid that can test single tiles, like bool test(x, y).

The naive version to solve this involves quite a few nested loops:

bool test_rectangle(int x0, int y0, int x1, int y0):
  for y in [y0 .. y1]:
    for x in [x0 .. x1]:
      if (test(x, y)):
        return false
  return true
best_rectangle <- nil
best_area <- 0
for y0 in [0 .. h):
  for x0 in [0 .. w):
    for y1 in [y0 .. h):
      for x1 in [x0 .. w):
        area <- (x1-x0+1)*(y1-y0+1)
        if (area > best_area && test_rectangle(x0, y0, x1, y1))
          best_rectangle <- {(x0,y0) (x1,y1)}

This is in O((w*h)³), so it can be pretty slow for bigger bitmaps. But there is a much more clever algorithm that I only learned about recently and I feel like it is not easily derived but quite interesting, so I want to show it here.

The algorithm

Start by going through the bitmap row-by-row and keep a count for each entry, initializing all to 0. For each bit, check if it is blocked. If yes, set the count to 0, otherwise, increment the value from the previous row.

With the example from above, the 5th row (going top to bottom) would look like this (in red):

Note that we don’t have to store all the numbers, just one row. I’ve kept the other numbers in orange merely for illustration purposes.

The numbers say how much you can go up without touching a blocked bit. Now we can treat each position as a potential “base point”, a point on the bottom edge of the candidate rectangle with the current number as the height. We just need to figure out how for left and right we can extend the candidate, which can easily be solved by going into both directions while the numbers there are at least the current height.

Consider the current row: the two zeros are obviously no candidates. The 4 cannot be extended to the left, because the left neighbor is already a 0. But it can be extended over the two 5s to its right. So the candidates size at the point is 4*3=12. The 5s themselves can only be extended one column, covering the respective other 5. So their candidate sizes are 5*2=10. The 3 can be extended all the way to cover the 4 on the left side and the last column’s 5 on the right, so the size there is 3*5=15. The last column’s 5 can not be extended at all, so it’s just 5*1=5. So the largest rectangle generated from this row is the one with the 3 as the base point. Repeat this for all the rows, and you will eventually find the biggest rectangle in the whole bitmap.

Even faster

You probably noticed that I omitted how to exactly figure out where the next smaller element on the left/right side is. You could just use a find-type loop in both directions and be done with it. That’d only add a factor h to your runtime, which should be sub-linear (in O(sqrt(n))) if you make sure h is not the greater of the two dimensions by transposing the image.

But you can actually do it constant time after a linear preprocess on the row: For each element in the row, store the next smaller element to the right. You do this by building a stack while the numbers are not decreasing, and unwinding that stack when they do. For each unwound number, note the current position as the next lower. The unwind goes on until the top of the stack is actually lower than the current again.

Do that in reverse too, to find the previous smaller element for each entry, and you can lookup the rectangle widths in constant time.

Taking our example row [0, 0, 4, 5, 5, 3, 5] again, we start with:

monotone_stack: []
next_lower: [7, 7, 7, 7, 7, 7, 7] // Past-the-end is the default

On the 5 iterations, not a lot happens, since the numbers are not decreasing. Note that the stack contains the indices, not the values:

monotone_stack: [0, 1, 2, 3, 4]
next_lower: [7, 7, 7, 7, 7, 7, 7] // Nothing changed yet

But at position 5, we need to start unwinding the stack. The 4, 3 and 2, corresponding to values 5, 5 and 4 have to go, and get our current position (also 5) as the next_lower, while the current position is added to stack.

monotone_stack: [0, 1, 5]
next_lower: [7, 7, 5, 5, 5, 7, 7]

The 5 in the last column is again higher than the value referenced by the top of the stack, so it just gets added to the stack and next_lower stays the same. A word on the default value: 7 (=w) works well here, since we can extend the rectangle always to one-minus-the-next-lower. When its 7, this means we can extend right up to 6, which is the last column. When doing this algorithm from right-to-left for the previous-lower, you’d actually use -1 for that.

I’ll leave the steps to you, but the result for prev_lower should be:

prev_lower = [-1, -1, 1, 2, 2, 1, 5]

With those values, the width of the candidate at position x should be next_lower[x] - prev_lower[x] - 1: remove 2 because the lower values cannot actually be included, add 1 back because the interval is inclusive.

With all this, the algorithm actually ends up in O(w*h). The second part of this algorithm is actually called Largest area in histogram.

Deferrable Constraints: When Database Rules Can Wait

Database constraints are usually quite impatient. As soon as a statement violates a unique key, foreign key or another constraint, the database rejects it.

Most of the time this is exactly what we want. But sometimes an intermediate state is invalid even though the final state of the transaction is perfectly valid.

Consider a table that stores the order of items:

CREATE TABLE tasks (
    id       INTEGER PRIMARY KEY,
    position INTEGER NOT NULL,
    CONSTRAINT tasks_position_unique UNIQUE (position)
);

Suppose it contains:

id | position
---+---------
 1 | 1
 2 | 2

Now we want to swap the two positions. The obvious approach does not work:

UPDATE tasks SET position = 2 WHERE id = 1;
UPDATE tasks SET position = 1 WHERE id = 2;

The first statement already violates the unique constraint because task 2 still has position 2. We could work around this by assigning a temporary value:

UPDATE tasks SET position = -1 WHERE id = 1;
UPDATE tasks SET position = 1  WHERE id = 2;
UPDATE tasks SET position = 2  WHERE id = 1;

But this is really an implementation detail leaking into our data manipulation. What we actually want to express is much simpler:

The positions have to be unique when the transaction is finished. We do not care about temporary duplicates while changing them.

This is what deferrable constraints are for.

Deferring a constraint

Both PostgreSQL and Oracle support the SQL keywords DEFERRABLE, INITIALLY IMMEDIATE and INITIALLY DEFERRED. A deferrable constraint can be switched from immediate checking to checking at the end of the transaction.

We can define our constraint like this:

CREATE TABLE tasks (
    id       INTEGER PRIMARY KEY,
    position INTEGER NOT NULL,
    CONSTRAINT tasks_position_unique
        UNIQUE (position)
        DEFERRABLE INITIALLY IMMEDIATE
);

INITIALLY IMMEDIATE means that it behaves like an ordinary constraint by default. The important difference is that a transaction is allowed to defer it explicitly.

Before performing our swap we can write:

SET CONSTRAINTS tasks_position_unique DEFERRED;
UPDATE tasks SET position = 2 WHERE id = 1;
UPDATE tasks SET position = 1 WHERE id = 2;
COMMIT;

After the first UPDATE the table temporarily contains the position 2 twice. This is allowed because the constraint has been deferred. After the second UPDATE all positions are unique again. When the transaction is committed, the constraint succeeds.

If we forgot the second update, the COMMIT would fail instead. So the constraint has not been disabled. Its check has merely been postponed.

Initially immediate or initially deferred?

A deferrable constraint has two useful default modes.

DEFERRABLE INITIALLY IMMEDIATE

means that the database normally checks the constraint immediately, but individual transactions can defer it.

DEFERRABLE INITIALLY DEFERRED

means that the database normally waits until the transaction is committed.

For most application tables INITIALLY IMMEDIATE is probably easier to reason about. Constraint violations still occur close to the statement that caused them, and deferral is explicitly enabled only for operations that need it.

You can also defer all deferrable constraints in a transaction:

SET CONSTRAINTS ALL DEFERRED;

Both PostgreSQL and Oracle support this form.

PostgreSQL and Oracle

The basic mechanism looks remarkably similar in PostgreSQL and Oracle. In both databases, constraints are NOT DEFERRABLE by default. If you want to change their checking mode during a transaction, they have to be created as DEFERRABLE. There are some differences in the details, though.

PostgreSQL currently allows deferral for UNIQUE, PRIMARY KEY, foreign key and EXCLUDE constraints. CHECK and NOT NULL constraints are always checked immediately.

Oracle’s model is somewhat broader. Oracle also supports deferrable constraints such as NOT NULL constraints. Its documentation explicitly describes a deferrable NOT NULL constraint whose violation is detected when the transaction is committed.

Another use case: Foreign keys

Unique constraints are an easy way to demonstrate the problem, but foreign keys are probably the more familiar use case.

Imagine importing a set of objects that reference each other. The final object graph is consistent, but the input format does not guarantee an insertion order that satisfies all foreign keys along the way. Without deferral, the application has to determine the correct ordering itself.

With deferrable foreign keys, all objects can be inserted first and referential integrity can be checked once the transaction is complete. This is particularly useful for bulk imports, cyclic relationships and more complicated restructuring operations.

Deferring vs. Disabling

There is an important conceptual difference between deferring a constraint and disabling it. When a constraint is disabled, invalid data may remain in the database. When a constraint is deferred, invalid data may only exist as an intermediate state inside a transaction.

Before that transaction can successfully finish, the database rules must be satisfied again. That makes deferrable constraints a nice fit for operations where the individual steps temporarily break an invariant, while the operation as a whole preserves it.

Docker in Continuous Integration

Docker – or more general containerization – can be applied in several areas of software development to improve many aspects of our work. The most common are:

  • In development to easily setup an environment for running and developing the software without installing every manually on the target OS.
  • For deployment either in docker-stacks (e.g. managed by portainer) or kubernetes only requiring hosts able to run containers without specialized setups.
  • For building software artifacts on your continuous integration (CI) infrastructure with the same benefits as the former two applications.

In essence you trade snowflaking all the involved machines (dev, CI nodes, servers) for some additional complexity around Dockerfiles, images, containers and their orchestration.

In this post I want to focus on the CI part:

Using Docker in CI

Administrators and infrastructure people usually will cheer if they just need to provide docker-capable nodes for developers to build their projects on. No need to provide certain runtimes, libraries, tools and configurations anymore. No need to negotiate with developers about the environment all the time and keeping it updated, appropriate and secure.

Developers on the other hand need to work with another tool encapsulating their build process. While this gives them a lot of freedom in choosing and shaping the environment for their builds (the “inside” of the containers) it adds pitfalls and complexity on the outside.

If your artifact is not (only) another docker image but things like test- and coverage reports, binaries or other files you have to move stuff from inside the containers to the outside aka host. In CI use cases you have essentially the following alternatives, each with different pros and cons:

Bind mounts for a container run

In this approach you build your environment using a normal docker build command and follow it with a docker run, e.g.:

docker build -t build-project .
docker run --rm -u `id -u` -v `pwd`:/build build-project

Pros:

  • This approach feels natural and does automatic container cleanup by using the --rm option on docker run.
  • It also enables dependency caches etc. on the host which can reduce build times, even across projects.
  • Sometimes you do not even need a dockerfile but can use plain docker images without building one yourself, e.g. eclipse-temurin:21-jdk.

Cons:

  • It actually runs a container and has limited file system access to the host.
  • It may leave build artifacts and temporary files on the host.
  • It may create file ownership issues, hence the -u `id -u` arguments to docker run in the example.

Image build and container to copy from

This approach is similar to above in that is consists of several steps but it does not need mounts and does most of the stuff during docker build:

set -eu
docker build -t build-project .
container_id=$(docker create build-project)
trap 'docker rm -f "$container_id" >/dev/null 2>&1 || true' EXIT
docker start $container_id
docker cp $container_id:/buildresults/ ./artifacts/

Pros:

  • Most stuff happens inside docker build.
  • You can use layer caching to improve build times.
  • Cleanup is done using shell mechanismns (trap), can also be performed using different means.
  • What leaves the container is exactly and explicitly controllable

Cons:

  • You need to run a container
  • You need to take care of container cleanup
  • Full benefit of layer caching may require thought/engineering to improve build times

Multi-stage build with scratch-image output

In this approach we never actually run a container and let docker build copy the specified artifacts to the host using a multi-stage build. Most of the interesting bits are in the Dockerfile while the build call looks like:

docker build --output artifacts/ .

The Dockerfile gets a bit more complex:

FROM python:3.14 AS build
WORKDIR /build

COPY . .

RUN pip3 install --no-cache-dir -r requirements.txt
RUN python3 -m build

FROM scratch AS package
COPY --from=build /build/dist/*.whl .

Pros:

  • No need to explicitly cleanup containers or dependency caches
  • Layer caching possible
  • No need to run a container explicitly, building the image is enough
  • Only one simple call in the build pipeline

Cons:

  • More complexity inside the Dockerfile

Conclusion

Using docker for building software has many advantages with the price of an additional tools and its own complexities. Several, easily adaptable approaches exist to facilitate containerization in CI environments. Use the one that fits your requirements and environment best.

Which approach do you like best? What are you using in your build and delivery pipelines?

I would be glad to hear your thoughts and comments.

We Don’t Program Anymore, We Manage Programming (AI impressions, part 3 of 5)

This is the third part of the series “Impressions of Our Current AI Usage”, as outlined by the introduction article.

In the early years of my career, there was a sudden increase in books about management in the technological fields, especially software development. I didn’t make the connection then, but it was right around the time when the first generation of academically educated software developers were senior enough to climb the corporate ladder one step further and become middle managers. Authors like Tom DeMarco and Timothy Lister wrote books that we discussed fiercly, because their propositions clashed with our observed reality of commercial software development.

“Why are our managers so bad at their job?” we asked. Don’t they see the obvious problems that they themselves created? What we didn’t get was that those managers were very experienced developers, but absolute beginners in their new job – leading people that develop instead of developing things themselves. They knew exactly how they would solve the technical problems, but were forced to concentrate on the very untechnical problems of people under their supervision. Nothing in their previous career prepared them for the challenges that “dealing with people problems” would impose upon them.

I personally know several senior developers that turned down (or reverted) a promotion to a management position, because they might gain some money and status, but lost all the things they were passionate about and good at. Working hard to create the environment where others can have the fun you would want to have yourself, is not the most rewarding job in the world. I fully understand them. And I say that as the founder and boss of a software development company. I’ve found my balance between being a manager and a developer, but it’s a constant negotiation.

The premise of our job as software developer was that customers tell us what they want and we describe it to the computer. In essence, we were translators between the worlds of humans and machines, right down to the languages: Some languages for human interaction, others for machine interaction. Because the computer was dumb as a rock, we needed to craft the solution with immaculate precision and in great detail. If anybody asks why software development took so long and was so demanding, imagine explaining quantum field theory to a first grader, so that the child can work in a laboratory among experienced physicists without screwing everything up.

And now, there is artificial intelligence (currently in the form of LLM-based agents) that produces working source code if prompted roughly right. Back in my days, we would call this role a “junior developer”. Which transforms our role to that of a manager: We bring the requirements from the customer, sketch a solution with our senior developers and architects (in the best case, this is also us) and explain it in necessary detail to the junior developers. Then we wait for them to translate our vision for the computer, review the result and iterate as long as necessary.

The only advantage of this scenario in comparison to the same role a decade ago is that the current junior developers aren’t human anymore. So while we need all the skills a capable manager should have, we don’t have to deal with different personalities and human problems in the workplace.
Except that LLMs tend to exhibit the worst work personality types (charming overconfident dazzlers) and are as ruthless and negligent as human junior developers in their “early ability – peak stupidity” phase. As their manager, you can work with human junior developers on their character journey, using the tools from the “people problems” part of your role, while you can only hope for the next version of LLMs to be more pleasant to deal with.
So, as middle managers between the customer and the juniors, we don’t have to deal with “soft topics” anymore. But we have to endure their effects (good and bad) daily. We are “promoted” to be highly ineffectual managers that supervise a bunch of knowledgable, but otherwise clueless juniors. That’s more or less the experience of previous senior developers that became managers and found themselves overwhelmed and underjoyed. The problem is that our new machine-based junior developers are so fast with their work that we can’t replace them with humans and expect similar performance. And the classic “instruct, oversee, review”-cycle that middle managers go through again and again is sped up from days or weeks to minutes. Which leaves virtually no leeway for typical management activities that still run on human speed and require time.

My hope is that managing AI agents requires a totally different skillset than managing human junior developers. Because otherwise, I can attest most of us experienced developers a pronounced inability and unwillingness to be effective in this new role. And I don’t want to imagine a world where most software development is done by poorly managed AI agents while their managers act and feel like fish out of the water.