When a Method Reference Reuses a Stream

Java method references are often used as a more concise alternative to lambda expressions. In many cases, the two forms are interchangeable and choosing between them is simply a matter of readability.

However, there are situations where replacing a lambda expression with a method reference changes the behaviour of the program. One such case involves Java Streams and the Iterable interface.

Making a Stream Iterable

Java Streams provide an iterator() method but do not implement the Iterable interface. This means they cannot be used directly in an enhanced for loop.

Since Iterable is a functional interface, we can adapt a Stream using a lambda expression:

Iterable<String> values = () -> createStream().iterator();

Alternatively, we can use a method reference:

Iterable<String> values = createStream()::iterator;

Both expressions compile and produce an Iterable<String> that can be used in an enhanced for loop:

for (String value : values) {
System.out.println(value);
}

Despite their similarity, the two implementations behave differently when iterated more than once.

Comparing the Two Approaches

To demonstrate the difference, we use a method that creates a Stream and prints a message whenever it is called:

static Stream<String> createStream() {
System.out.println(" Creating a new stream.");
return Stream.of("Example value");
}

We also need a method that iterates over an Iterable twice:

static void iterateTwice(final Iterable<String> values) {
System.out.println(" First iteration:");
for (final var value : values) {
System.out.println(" " + value);
}
System.out.println(" Second iteration:");
for (final var value : values) {
System.out.println(" " + value);
}
}

Now we can compare the lambda expression and the method reference:

public static void main() {
System.out.println("Lambda:");
final Iterable<String> withLambda =
() -> createStream().iterator();
iterateTwice(withLambda);
System.out.println();
System.out.println("Method reference:");
final Iterable<String> withMethodReference =
createStream()::iterator;
try {
iterateTwice(withMethodReference);
} catch (final IllegalStateException e) {
System.out.println(" Expected exception: " + e);
}
}

Running this code produces the following output:

Lambda:
First iteration:
Creating a new stream.
Example value
Second iteration:
Creating a new stream.
Example value
Method reference:
Creating a new stream.
First iteration:
Example value
Second iteration:
Expected exception: java.lang.IllegalStateException: stream has already been operated upon or closed

The lambda expression allows both iterations to complete successfully. The method reference throws an IllegalStateException during the second iteration.

There is also a difference in when the Stream is created. With the lambda expression, a new Stream is created at the beginning of each iteration. With the method reference, the Stream is created immediately when the Iterable is assigned.

Understanding the Difference

The difference comes from when Java evaluates the expressions involved.

Consider the lambda expression:

() -> createStream().iterator()

The body of the lambda is executed whenever the iterator() method of the Iterable is called. Consequently, every call creates a new Stream and returns its iterator. Its behaviour is equivalent to the following anonymous class:

Iterable<String> values = new Iterable<>() {
@Override
public Iterator<String> iterator() {
return createStream().iterator();
}
};

Each iteration uses an independent Stream instance.

The method reference behaves differently:

createStream()::iterator

This is a method reference to an instance method of a particular object. Java evaluates the expression to the left of :: when the method reference is created. In this case, createStream() is called immediately, and the resulting Stream instance is retained as the receiver of the method reference. The behaviour is therefore equivalent to:

final Stream<String> stream = createStream();
Iterable<String> values = new Iterable<>() {
@Override
public Iterator<String> iterator() {
return stream.iterator();
}
};

Every call to iterator() operates on the same Stream instance.

Streams Can Only Be Consumed Once

Java Streams are designed for a single traversal. Once a Stream has been consumed, it cannot be reused for another traversal. This restriction also applies when consuming a Stream through its iterator.

In the method-reference example, the first iteration consumes the Stream. The second iteration attempts to obtain another iterator from the same Stream instance, which results in an IllegalStateException.

The lambda expression avoids this problem because it creates a new Stream for each call to iterator().

Using a Method Reference Without Reusing the Stream

It is possible to preserve the behaviour of the lambda expression while still using a method reference. Instead of referencing the iterator() method of an existing Stream, we can introduce a static method that creates a new iterator:

private static Iterator<String> createIterator() {
return createStream().iterator();
}

The Iterable can then be defined as:

Iterable<String> values = Main::createIterator;

In this case, the method reference points to a static method. Each invocation of iterator() calls createIterator(), which creates a new Stream. Both iterations now work as expected.

For this particular example, the original lambda expression is arguably simpler:

Iterable<String> values = () -> createStream().iterator();

It also makes the creation of a new Stream for each iteration more explicit.

A General Difference Between Lambdas and Method References

The behaviour demonstrated here is not specific to Streams. An expression of the form

someExpression()::someMethod

evaluates someExpression() immediately when the method reference is created. The resulting object is then used as the receiver whenever the referenced method is invoked.

In contrast, a lambda expression such as

() -> someExpression().someMethod()

evaluates someExpression() each time the lambda is invoked.

This distinction matters when the expression creates a new object, produces side effects, accesses mutable state, or returns a resource that cannot be reused.

It does not apply in the same way to every kind of method reference. For example, a reference to a static method such as SomeClass::someStaticMethod does not capture an instance produced by a receiver expression.

Conclusion

Lambda expressions and method references are not always semantically equivalent, even when they implement the same functional interface.

In this example, replacing

() -> createStream().iterator()

with

createStream()::iterator

changes when the Stream is created and whether a new Stream is available for each iteration.

Both expressions compile successfully, but only the lambda expression produces an Iterable that can be traversed repeatedly.

When refactoring lambda expressions into method references, it is therefore important to consider when the receiver expression is evaluated and whether the resulting object can safely be reused.

Advanced Refactoring: Extract Structural Duplication

A friend once told me that when his workday goes sour, he turns to a practice of “therapeutic refactoring”: Making little improvements throughout the source code of a project without an overarching goal. If we dissect this behaviour, we probably find a dopamine-based reward cycle not unfamiliar to those in popular video games like Diablo, only that we slay little flaws in the source code instead of critters. And because we never know what code can be found in the next method and the next class (just like we don’t know which monsters lurk behind the next door in Diablo), we go and explore for a bit, leaving a trail of resolved code smells (or slain monsters) behind.

But somewhere in each level of Diablo, there waits a boss enemy that requires some more hammering. And similar, every now and then you encounter a code area that looks dirty but is not resolved that easily. This blog entry tells the story of one of these encounters and how we refactored it. There is no guarantee that you’ll find the result “better” in any way, the whole point is that it can be done and the mechanic is always the same.

Let’s look at this code snippet, reduced to the relevant parts:

public static void setValueFor(
final PreparedStatement statement,
final ColumnInfo column,
final String value
) throws SQLException {
final int typeNumber = column.type().getVendorTypeNumber();
switch (typeNumber) {
case Types.DATE -> {
statement.setDate(
column.columnIndex(),
new Date(
getDateMillisFrom(
LocalDate.from(parseToDate(value)).atStartOfDay(ZoneOffset.UTC)
)
)
);
}
case Types.TIMESTAMP -> {
statement.setTimestamp(
column.columnIndex(),
new Timestamp(
getDateMillisFrom(
LocalDateTime.from(parseToDateTime(value)).atZone(ZoneOffset.UTC)
)
)
);
}
}
}

A given string value is written into a SQL database, using a prepared statement and adequatly typed setter method calls. The real code is way more exhaustive, but we are only interested in the striking similarity of the DATE and TIMESTAMP case.

The structure resembles something I call a “sandwich” or “burger”: The same bun above and below, but different things in the middle. If it is only “bun, patty, bun”, you can solve it with a simple lambda expression as a method parameter. But in this case, it is a more delicate mix of bread and filling layers. Let’s investigate to plan our refactoring attempt:

  • statement.setDate() and statement.setTimestamp(): Both operate on the prepared statement, but call different methods with a different parameter type. Needs to be configurable in the result.
  • column.columnIndex(): Duplication. Needs to be part of the extracted code.
  • new Date() and new Timestamp(): There is no suitable supertype other than java.util.Date. Since we want the actual parameter types for the final setter call, we shouldn’t abstract to it. We need to make this call configurable, but be able to work with the result without knowing its actual type. A generic parameter is the right choice for it.
  • getDateMillisFrom(): Duplication. Needs to be part of the extracted code.
  • LocalDate.from() and LocalDateTime.from(): They look similar enough to be tempted to unify them into one generic call, but it would refrain from doing it. This feels like a pyrrhic victory trap to me, where the solution is way more complex than the original. I already fear this effect for our initial quest, I don’t need a nested sub-quest right now.
  • .atStartOfDay(ZoneOffset.UTC) and .atZone(ZoneOffset.UTC): Duplication. Needs to be part of the extracted code.

If you don’t see my classification right away, my suggestion is to print the code snippet on paper, grabbing a green and a blue text marker and separating each term by coloring it. I did this for years until I could color the code right in place in my imagination. And with training, I could add more colors…

So we have a structure like: Configure, Extract, Configure, Extract, Configure, Extract. Nice! I’ve counted three configuration parameters and three entities that have their “single source of truth” in the extracted code. We also have six steps in the total story. Let’s plan it:

private static void setTimetype(
ColumnInfo column,
String value,

Ok, this is the foundation. We need the information about the column and the value to work on both. Now on to the configurable parts. I start with the innermost parameter: Something that converts the String value into a LocalDateTime (the time zone aspect is part of our extracted code):

private static void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,

That was the easy part. For the next configurable part, we need to be able to support both Date and Timestamp. Both are created from a long, representing milliseconds since epoch start. This is only easy if we make it easy in the moment:

private static void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, T> creation,

We added a generic type “T” that we need to specify so the compiler doesn’t act up:

private static <T> void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, T> creation,

We specify the generic in the context of the method. Not giving any clue about what types are valid keeps us maneuverable in this dimension. There is one improvement I want to make that is unusual in Java: Any name for a generic type is always one capital letter in all examples. There is no need to be so spartanic with letters. We can just name our “T” like any other type. I like “Timetype” (yes, with lowercase letters, get over it) because the method name talks about a “Timetype” already:

private static <Timetype> void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, Timetype> creation,

If you are unfamiliar with this naming scheme: The type “Timetype” doesn’t actually exist, just like “T” didn’t exist. I just view generic type names as “normal” enough to give them normal names and not just “T”, “X”, “A”, etc.

That was our second configurable parameter and it forced us to make nearly all important decisions. The third one follows suit:

private static <Timetype> void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, Timetype> creation,
Consumer<Integer, Timetype> persistence
)

The outmost calls of both code lines in the original take an integer and the specific newly created type and stores it in the prepared statement. By making everything type specific a parameter, we push all the knowledge about the prepared statement outside our extracted method. All the knowledge, with the exception of exceptions. Unfortunately, the statement.setDate() and statement.setTimestamp() both throw the checked exception SQLException. A java Consumer can’t handle checked exceptions. We need our own type for that:

public static interface TimetypePersister<Timetype> {
void persist(
int columnIndex,
Timetype value
) throws SQLException;
}

This type replaces the Consumer in our signature:

private static <Timetype> void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, Timetype> creation,
TimetypePersister<Timetype> persistence
) throws SQLException {

And now we are ready to fill in the blanks. First we refactor both calls to our method:

case Types.DATE -> {
setTimetype(
column,
value,
t -> LocalDate.from(parseToDate(t)).atStartOfDay(),
Date::new,
statement::setDate
);
}
case Types.TIMESTAMP -> {
setTimetype(
column,
value,
t -> LocalDateTime.from(parseToDateTime(t)),
Timestamp::new,
statement::setTimestamp
);
}

You probably see the nested sub-quest of the LocalDateTime-parsing right away. As I said, I won’t attempt to solve it in this blog entry. The exercise is left to the inclined reader.

Now, we can finally implement our extracted method that retains the structure of the original lines and includes the three “entities of wisdom” that are now de-duplicated:

private static <Timetype> void setTimetype(
ColumnInfo column,
String value,
Function<String, LocalDateTime> parsing,
Function<Long, Timetype> creation,
TimetypePersister<Timetype> persistence
) throws SQLException {
LocalDateTime local = parsing.apply(value);
ZonedDateTime zoned = local.atZone(ZoneOffset.UTC);
long millis = getDateMillisFrom(zoned);
Timetype settable = creation.apply(millis);
int index = column.columnIndex();
persistence.persist(
index,
settable
);
}

You can see how we unpacked the original lines in reverse: Our method begins with the last duplicated wisdom, the time zone. Then we turn the ZonedDateTime into the milliseconds since epoch start and finally, we use the column index from the given column. In the same tenor, the calls to the configurable parts are also unrolled from the innermost to the outside.

And there we are! The boss enemy is slain. The complex and intertwined structure is extracted, most duplications are resolved. We can enjoy our dopamine hit and call it a day.

Thank you for reading this far and thinking along. What part made you pause? Maybe I can improve my explanation there for you. Drop me a comment!

Setting up clangd on Windows

clangd is a language server that allows other programs to access introspection and refactoring capabilities for C++ via the language server protocol. It was originally meant for code editors. For example, VS Code can use it via this extension, and provide functions like “Go to definition”, “rename”. With LLMs and their harnesses, they can give those capabilities to agents which otherwise navigate code-bases via the equivalent of Windows Notepad, e.g. by reading and grep’ing. For example, for pi.dev, there are these extensions that can use clangd.

How it works

clangd is usually run by the host application, which communicates with it by sending and receiving json-rpc fragments via stdin/stdout. It can either be kept running as a continuous service and serve from memory, or keep its preprocessed info persisted to disk and use that from per-request instances/processes.

Install clangd

Go to https://github.com/llvm/llvm-project/releases, and download the Windows installer for LLVM. At the time of this writing, the version is 23.1.2. Install that. By default, the executables will go into C:\Program Files\LLVM\bin.

Add that to your PATH, e.g. by pressing Win+R and running rundll32.exe sysdm.cpl,EditEnvironmentVariables. It should be sufficient to add it for your user.

Setup your project

Make sure the CMake variable CMAKE_EXPORT_COMPILE_COMMANDS is set to ON, which will generate a compile_commands.json in your build folder. This file contains the compiler invocations for your project, which clangd needs to correctly interpret your code.

Note that the CMake documentation specifically mentions that CMAKE_EXPORT_COMPILE_COMMANDS does not work well with unity builds. This is a shame, because they rock.

Drop a file called .clangd in your project root where to find that file. Since I usually use conan and their layout, my builds go in build/Debug or build/Release. Here, I can only pick one at a time:

  CompileFlags:
    CompilationDatabase: "build/Debug"

At this point, you should be able to use clangd on your code. There are many other things you can configure in that file, e.g. whether to store the persistent index I mentioned above.

Try it out!

The VS Code extension I mentioned at the top is a good way to check if everything is working as expected. Hovering over an identifier should give you some nice info about it:

Code snippet in C++ showing the declaration of a logger variable using the spdlog library and error handling in a try-catch block.

Modern python build and packaging

Developing software is not only about code but also about delivering the software to your customers. Sometimes, app stores are the solution, at other times it may be software-as-a-service (Saas) but there are also occasions where installers, compressed archives, system packages oder single file executables are (or fat-jars) are the way to go.

While python as a language and ecosystem has many points in its favour I personally found its build and delivery infrastructure lacking. As soon as you project requires a bunch of dependencies and users expect something like apt update to handle your software, too, things can get messy.

There were multiple options, tools and variants for the different aspects of software delivery:

  • setuptools vs. distutils
  • venvs, site-packages, pip, pipx, wheels
  • packaging tools like py2dsc-deb, bdist, pybuild
  • homebrew, anaconda etc.

For non-fulltime python developers like me there was always quite some doubt about the current idiomatic and preferred solution. In my C++ projects, this problem was largely solved by CMake+CPack. For java, maven and gradle provided easy solutions for many delivery and deployment tasks.

From my limited perspective the situation in the python world is more fluid and still in the process of finding a best-of-breed solution and evolving mostly that one option. Maybe we have arrived at that with the “Python Packaging Authority” and the work around pyproject.toml.

Many of our customers use Debian oder Debian-based distributions like Ubuntu and like having packages in their own apt-repository that they can install using system tools. So building deb-packages feels like the natural way to deliver your software to these customers.

Building Debian-Packages

In the past I used several approaches to packaging our python projects as debian packages:

  • “python setup.py sdist” in combination with plain dh_make and manual editing of the metadata files
  • pybuild/dh_python3
  • stdeb, with py2dsc etc.

Nowadays I am moving away from setup.py towards pyproject.toml. I find it quite readable and it provides neat features like executable scripts and single-sourcing the project version and a simple declarative project format. Together with debhelper and pybuild packaging becomes quite easy.

I create the necessary debian metadata-files, e.g. using dh_make or by hand.

The modified rules file may look as simple as:

#!/usr/bin/make -f

%:
	dh $@ --buildsystem=pybuild

# We do not have tests yet, so do not run pytest...
override_dh_auto_test:

My pyproject.toml may look like this:

[build-system]
requires = [
    "setuptools>=61.0.0",
    "wheel"
]
build-backend = "setuptools.build_meta"

[project]
name = "myproject"
dynamic = ["version"]
description = 'My project illustrates packaging'
readme = "README.md"
requires-python = ">=3.10"
maintainers = [
    {name = "Mihael Koep", email = "mihael.koep@softwareschneiderei.de"}
]

[project.scripts]
OmsMaxx = "myproject:start"

[project.urls]
Homepage = "https://gitlab.com/example/python/myproject"
Repository = "https://gitlab.com/example/python/myproject"
Issues = "https://gitlab.com/example/python/myproject/-/work_items"

[tool.setuptools.packages.find]
include = ["myproject"]

[tool.setuptools.dynamic]
version = { attr = "myproject.__version__" }

If the tools python3-build, python3-wheel, dh-python and pybuild-plugin-pyproject are available, building a working package should be as easy as running dpkg-buildpackage like below:

dpkg-buildpackage -b -us -uc

Conclusion

I hope this pybuild/pyproject.toml infrastructure with it extensibility and relative clarity will become (and stay) the de facto standard and improve over time.

This would end my uncertainty and provide a sensible default to shipping and packaging python projects.

Have you other approaches or opinions on this topic?

Dynamic Device Classes in Python

When writing PyTango device servers, it is common to implement one Python class per Tango device class. For small projects, this approach is simple and easy to understand. However, it becomes cumbersome when the set of available devices is not known at development time.

Consider a device server that should be entirely driven by a configuration file. Instead of hard-coding every supported device class, the server reads the device definitions at startup and creates the required Tango device classes automatically.

The goal is to write a generic PyTango server that does not need to be modified whenever a new device class is added. If a new entry appears in the configuration file, the server should simply create the corresponding Tango device class.

Python type

At first glance, this sounds unusual. After all, Python classes are typically defined using the familiar class keyword:

class Motor(Device):
    pass

Most Python developers stop here and never think about how classes are actually created. Under the hood, however, classes are objects themselves, and Python provides a built-in mechanism to construct them dynamically.

The function responsible for this is type().

Most of us use it in its simplest form to inspect the type of an object:

print(type(42))
# <class 'int'>

Less well known is its three-argument form:

type(name, bases, attributes)

The arguments are:

  • name: the name of the new class.
  • bases: a tuple of parent classes.
  • attributes: a dictionary containing class attributes and methods.

This means that the following definition is equivalent to the above class definition.

Motor = type(
    "Motor",
    (Device,),
    {},
)

The resulting object is exactly the same: a Python class that can be instantiated or registered with PyTango.

The third argument of type() becomes particularly interesting when more than just the class name should be configurable. The attributes dictionary allows methods, properties, or other class members to be added dynamically. This can be useful when Tango attributes or commands are also described in the configuration file.

Motor = type(
    "Motor",
    (Device,),
    {
        "some_property": 42,
    },
)

a = Motor()
print(a.some_property)
# 42

Conclusion

For our use case, creating classes dynamically becomes straightforward. Reading the configuration file and creating the required classes can be done in a simple loop. From PyTango’s perspective, there is no difference between a statically defined class and one created dynamically using type(). Both behave like ordinary Python classes.

AI Code Won’t Be for Humans Much Longer (AI impressions, part 2 of 5)

This is the second part of the series “Impressions of Our Current AI Usage”, as outlined by the introduction article.

In the first years of software development, the word “source code” didn’t exist, because code was just that: encoded machine instructions. How they were encoded changed rapidely, from flipping bits in the RAM directly by mechanical switches over feeding paper tapes with punched holes to magnetic storage. But for a long time, we worked with none or little abstraction over the actual machine code. I remember assembler code listings that had two columns of text: the first column for the computer, the second one just translating the first column into human-readable text.
And even with this little bit of clarification what the code actually does, we already needed additional software that took our source code and translated it for the machine.
With the adoption of higher-level programming languages, the additional software stack grew in depth until the distance between the source code and the actual machine instructions was big enough to warrant an intermediate layer of representation. Programming languages like Java or C# put a “byte-code layer” between our textual source code and the binary machine code. The machine we program against is no longer a real computer, but a “virtual machine” or in better words, a model of a machine. As long as we write source code that works correct with the model, we can assume that all the translation layers will find a way to run it correctly on the real computing substrate.
We are used to this kind of programming. We describe our goals using the machine model and a sophisticated machinery of software and hardware parts make it happen.

Forward to today and we use artificial intelligence (or inference using another kind of “model”) to produce source code in our favorite programming languages by describing our goals in even broader terms than before. We might mistake our prompts for natural language and think that we are able to produce source code by just saying what we want.

The question that poses itself nearly instantly is: If we invented a device that transforms natural language into machine behaviour from scratch today, would we include all the intermediate layers into its inner works? Is it really a good idea to transform natural language into higher-level source code, compile the source code into byte code and do all the weird magic to come up with a sequence of machine instructions that resemble the byte code? Isn’t it more efficient to teach the inference how the actual machine works and let it program directly?
Or, asking from the other side, who is the target audience of the generated source code if nobody reads it and the compiler only parses it once? Why does the inference invent all the variable and method names when the compiler throws them away again in the first step of its processing? Of course, right now the inference only imitates our way of working. But we work like we do because we write source code for other humans. If the human at the helm can’t read any layer of code anyway, why not jump directly to the most obscure representation of code and skip all the readability requirements?

As soon as the inference doesn’t imitate but actually learns about the target computing substrate, it will produce working code that is undecipherable for human readers, but optimal for the machine. (If you want to experience this effect in a tiny dose, I encourage you to play the “TIS-100” programming game). And because most inference users don’t need the readable code anyway, they won’t miss anything and get faster solutions with less hassle.

So my guess is that today’s source code will be a dying art, invented for humans and ignored by the machines because it doesn’t provide anything useful for them. The source code of the future will be less readable, more enigmatic and probably more efficient for the machine. Which means that human intervention or even just participation in the software production process will become more cumbersome and therefore even more limited.

My sorrow is that this distancing of the programming process from actual human oversight might provide a hard depedency on inference work alone. It would mean that humans aren’t just scales slower than the inference, but actually incapable of doing its work by hand anymore.

Quotes Are Not Part of the Argument

Recently, we upgraded a project from an older Java version to a newer one. Many of the changes were routine: update dependencies, replace deprecated APIs, fix a few compiler errors and run the test suite.

One of these changes concerned the invocation of an external Windows program.

The application used the deprecated overload:

Runtime.getRuntime().exec(command);

The command was assembled as one long string. The external program accepted command-line parameters of the following form:

/a="value of A" /b="value of B" /c="value of C"

Because the parameter values could contain spaces, we enclosed them in double quotes. We even had unit tests that explicitly verified the quoting. It was an important detail, or so we thought.

As part of the upgrade, we switched to the recommended overload that accepts the executable and its arguments separately:

Runtime.getRuntime().exec(commandArray);

The migration seemed straightforward. Instead of joining the executable and all parameters into one command string, we put them into a string array:

String[] command = {
"program",
"/a=\"value of A\"",
"/b=\"value of B\"",
"/c=\"value of C\""
};
Runtime.getRuntime().exec(command);

The unit tests were still green. The quotes were still present. Everything looked correct.

Then the application was deployed.

Invalid switch

In production, the external program stopped accepting our invocation. Its only diagnostic message was:

Invalid switch

This was not particularly helpful.

We inspected the parameters. All switches were present. Their spelling was correct. Their order was correct. The values were correct. The quote characters were exactly where our tests expected them to be.

Even more confusingly, the command worked perfectly when entered manually in a Windows command prompt:

program /a="value of A" /b="value of B" /c="value of C"

The executable clearly supported these parameters. The shell command clearly worked. And our Java code appeared to produce the same command.

But it did not.

Asking the receiving program

After spending some time comparing strings and staring at quote characters, we decided to stop reasoning about what the external program ought to receive. Instead, we wrote a small program that showed us what it actually received:

void main(String[] args) {
for (var i = 0; i < args.length; i++) {
System.out.println("args[" + i + "]: '" + args[i] + "'");
}
}

We packaged it as a JAR and invoked it from cmd.exe using the same parameter structure:

java -jar exec-experiment.jar /a="value of A" /b="value of B" /c="value of C"

The output was:

args[0]: '/a=value of A'
args[1]: '/b=value of B'
args[2]: '/c=value of C'

The double quotes were gone.

This was the missing piece.

The quotes in a shell command are not necessarily characters intended for the receiving program. They are instructions to the command-line parser. They tell it that a sequence containing spaces belongs to one argument.

The value

/a="value of A"

does not mean that the program receives an argument containing two quote characters. It means that the program receives one argument rather than three:

/a=value of A

The quote characters control parsing. They are not part of the resulting argument.

Quotes can appear in surprising places

To verify this interpretation, we performed a slightly more unusual experiment:

java -jar exec-experiment.jar /"a="va"lue of A" /b="value of B" /c="value of C"

This command is certainly not how anybody would normally write the parameters. Nevertheless, its output was unchanged:

args[0]: '/a=value of A'
args[1]: '/b=value of B'
args[2]: '/c=value of C'

The quotes can surround different portions of a token. Their purpose is to influence how the command line is divided into arguments. Once parsing is complete, they disappear.

This distinction is easy to overlook because a command line is usually presented as a string. It looks as though this string is passed to the program. In reality, there are two different representations involved:

program /a="value of A"

is a textual command line that still needs to be parsed.

By contrast,

new String[] {
"program",
"/a=value of A"
}

already describes the result of that parsing: an executable followed by one complete argument.

We had moved the parsing boundary

With Runtime.exec(String[]), every array element already represents one argument. Spaces inside an element do not split it into additional arguments.

By retaining the quotes, we had changed their meaning. They were no longer syntax interpreted by a shell-like parser. They had become literal characters inside the argument:

/a="value of A"

That was not what the external program expected. It expected:

/a=value of A

The error message “Invalid switch” was therefore accurate, but not very illuminating. The switch looked correct in our logs because we were looking at its command-line representation rather than at the argument format expected by the program.

The fix was simple:

String[] command = {
"program",
"/a=value of A",
"/b=value of B",
"/c=value of C"
};
Runtime.getRuntime().exec(command);

Or, preferably, using ProcessBuilder:

Process process = new ProcessBuilder(
"program",
"/a=value of A",
"/b=value of B",
"/c=value of C"
).start();

After removing the quote characters, the external program worked again.

Ironically, the quotes that our old implementation and its tests had treated as essential were exactly what broke the new implementation.

The takeaway

A command line and an argument array are not interchangeable representations.

When constructing a command line, quoting may be required to preserve spaces during parsing. When constructing an argument array, parsing has already happened conceptually. Each element is one argument, spaces included.

Do not ask:

How would I type this command in a shell?

Ask:

What exact strings should the receiving program find in its argument array?

The answer to the second question is what belongs in Runtime.exec(String[]) or ProcessBuilder.

Sometimes an API migration changes more than a method signature. It moves a boundary – in this case, the boundary between formatting a command line and supplying already separated arguments.

And when that boundary moves, yesterday’s carefully tested solution can become today’s bug.

When Optional sounds too optional: opt for more expressive types

So, I have one PyQt application which not only is quite data-heavy, but also has significant real-time requirements, as well as multiple windows. This construct brings some absolutely horrifying highly intellectually inspiring quests with it, and Python turned out to be kind of a good decision for that project, because that is one of the languages where, when you think about your structure a bit, you might get to write very natural-sounding like code.

Of course, the following idea is actually language-agnostic, I will just use fictive Python examples close to problems-based-on-a-true-story.

This in itself is not only a matter of aesthetics, but because real-time demands are quite tricky to reliably be covered by unit tests alone, the actual code has to read itself so clearly that one does not need to second-guess what any of this does. Think of a bedtime story, which usually would not, coming to think of it, contain clauses – or paragraphs, for that matter – requiring, under circumstances not even trivial to the human eye, one kind of meticulous gymnastics, easily negating twice, or thrice, and relying on Python’s borderline criminal degrees of freedom in duck typing, or canards even– you see — your toddler will now not go to sleep anytime soon. Or trust you with another story, for that matter.

Now I found out: While Qt is somewhat mature, one cannot even trust their way of doing things – i.e. turns out, the signals/slots system is not particularly designed for performance. Neither did I feel inclined to put my faith into even another state management solution like e.g. python-statemachine package, because – as capable as that sounds, it might be overkill, and distracting with its own idiosyncrasies (as also: I would not recommend Redux for a web project anymore, especially in TypeScript, except for you really know from the start that this is a good fit).

But, so, I have some tricky interplays between

  • Data consistency / single-source-ness demands that e.g. between two windows, there should only be primitive data exchanged, say str/int identifiers, and both have access to their repositories; not throwing loaded data sets around my memory in order to go stale at times
  • Comprehension, most significantly Single Level of Abstraction, or other indicators of mental load like how many levels of intendation / return paths are mixed within sight (and also, Type Annotations do help a lot in Python, even though not mandatory, i.e. the complete opposite of fighting Redux-TypeScript-chimaeras – but I digress…
  • Robustness, where I would believe that my user (me) has virtually no chance of even seeing this and that window when their data is maybe still loading somewhere – but I still check these cases, because this bedtime story has no business in leaving you an hopeful-to-anxious pile of nerves
  • Traceability of your state, for troubleshooting and useful UI feedback (as you’d guess, real-time event based stuff is not easily debugged by break points or logging alone).

So over months in that project, I grew annoyed of code like (Symbolbild)


class Editor:
    # ...
  
    def load_editor(self, params: Optional[EditorParams]):
        if params and (self._entity is not None or
                       self._entity.id != params.id):
            if entity := repository.load_entity(params.id):
                self._entity = entity
            else:
                raise ValueError("repository needs some alone time :(")
            self._entity.other_stuff = other_repo.check_stuff()
        elif params is None:
            raise TypeError(
                "sounds Optional in our signature, but actually is not"
            )
        elif self._entity.id == params.
            self.adjust_more_stuff(self._entity, params.stuff)
            # ...

Because encountering any single block of these drags you down, I have currently accustomed myself to write these as (one can argue whether the names like “Supplier” are the best here, but they’re not the worst, I believe)

@dataclass(frozen=True)
class LoadedEntity:
id: str
entity: Optional[Entity]
stuff: Optional[OtherStuff]
@property
def is_unusable(self):
return self.entity is None
@property
def missing_stuff(self):
if self.is_unusuable:
return True
else:
return self.stuff is None
class EntitySupplier:
_current: LoadedEntity
_entity_repo: EntityRepository
_stuff_repo: OtherStuffRepo
# __init__ etc. hereby left out as boilerplate
def load_params(self, params: Any):
# do all your checks in here
if (... very bad ...):
self._current = LoadedEntity(params.id, None)
return
entity = self._entity_repo.get(params.id)
stuff = self._stuff_repo.get(params.stuff).for(entity)
return LoadedEntity(
params.id,
entity,
stuff
)
@property
def entity(self):
return self._current.entity
def expecting(self, stuff: bool = False) -> Optional[LoadedEntity]:
if stuff and self._current.missing_stuff:
return None
return self._current
class Editor:
_supply: EntitySupplier
_logger: SomeLogger
def __init__(self, **kwargs):
self._supply = EntitySupplier(**kwargs)
self._logger = BlaBlaLogger()
def load(self, params):
self._supply.load_params(params)
if entity := self._supply.entity:
self.update_ui(entity)
else:
self._logger.error("Outsmarted, eh? %s | %s", str(params), stack_trace())
return
if supply := self._supply.expecting(stuff=True):
self.initiate_stuff_from(supply)
else:
self._logger.info("Entity %s is ready, Stuff is not | %s", str(entity), stack_trace())

So, the LoadedEntity serves like a concatenation of several Optional types, but it wraps the logic (i.e. there’s no sense in having stuff when you don’t have entity first) instead of just shruggingly claming “well, this entity here is optional – and that other stuff is, too”. Now, LoadedEntity is not a pretty name at all (have a better one?), but it sure beats having two straightaway lies.

I like that pattern because it allows me to stash the EntitySupplier and LoadedEntity somewhere on their own (I do strictly not believe that every class needs its own file, but some of the “Single …” ideas (Responsibility, Level of Abstraction, you name it) do also apply here; and the Editor.load(…) itself does read somewhat like a short story. It has quite linear structure and can early-return, and/or log, on demand, and while naming is still hard (consistently voted one half of famous Hard Things), I could even have some fun in designing that language while preserving the idea, that future-me can arrive in a few weeks (read: hours) and still trust in some of the entites and stuff.

The quintessence here is: Checking for None (which is Python’s NULL, and the typing Optional[T] is identically equal to T | None) is still a thing in 2026 due to its sheer practicality, but if you design some some structure around that and keep these checks in something like LoadedEntity, you can keep the abyss from staring back into you.

Rails Strict Locals: Giving Partials an Explicit Interface

Rails partials are a great way to reuse view code, but they have traditionally suffered from one weakness: their interface is implicit.

When opening a partial written by another developer, it is often unclear which locals are required, which are optional, and whether all of them are still used. IDEs typically cannot help much either, often showing warnings about unresolved variables because they cannot determine where the values come from.

The problem becomes even more apparent as an application grows and partials are rendered from multiple places.

If a local is forgotten by call, the error only appears when the template is rendered:

undefined local variable or method `missing_local'

If an extra local is passed, Rails traditionally ignores it.

Over time this creates a situation where the real API of the partial exists only in the heads of the developers maintaining it.

Rails Strict Locals

Rails provides a feature called strict locals that allows a partial to declare its expected interface:

<%# locals: (title:, highlight: false) %>

The declaration resembles Ruby keyword arguments and is placed at the top of the template.

A local like title without a default value is required. Locals like highlight with default values become optional

The partial now documents and enforce its own API. If a required local is missing, Rails raises an exception instead of failing later when the variable is accessed. Likewise, if a caller provides a local that is not declared, Rails rejects it.

Conclusion

Strict locals do not fundamentally change how partials work, but they make them easier to understand and maintain.

By declaring the expected locals directly in the template, partials become self-documenting and gain an explicit contract with their callers. Missing locals are detected early, obsolete locals are rejected, and developers no longer have to search through controllers, parent templates, and render calls to understand where variables come from.

An additional benefit is improved tooling support. Once the interface of a partial is explicit, IDEs can understand the available variables much better. Your IDE becomes a helpful companion again rather than a source of noise.

A doubly linked list for entt components

I recently implemented a small CRTP template to group entities in entt. It turned out quite nicely, so let me share it here:

template <class T> class doubly_linked_component
{
public:
using node_type = doubly_linked_component<T>;
static void on_construct(entt::registry& entities,
const entt::entity e)
{
auto& that = entities.get<T>(e);
that.next_ = that.prev_ = e;
}
static void on_destroy(entt::registry& entities,
const entt::entity e)
{
auto& that = entities.get<T>(e);
// List has only this element?
if (that.next_ == e)
{
return;
}
auto next = that.next_;
auto prev = that.prev_;
auto& next_node = static_cast<node_type&>(entities.get<T>(next));
auto& prev_node = static_cast<node_type&>(entities.get<T>(prev));
prev_node.next_ = next;
next_node.prev_ = prev;
}
static void merge(entt::registry& entities,
entt::entity lhs, entt::entity rhs)
{
auto& lhs_node = static_cast<node_type&>(entities.get_or_emplace<T>(lhs));
auto& rhs_node = static_cast<node_type&>(entities.get_or_emplace<T>(rhs));
// The end of left (which is left.prev_) needs to point to right
auto lhs_end = lhs_node.prev_;
auto rhs_end = rhs_node.prev_;
auto& lhs_end_node = static_cast<node_type&>(entities.get<T>(lhs_end));
auto& rhs_end_node = static_cast<node_type&>(entities.get<T>(rhs_end));
lhs_end_node.next_ = rhs;
rhs_node.prev_ = lhs_end;
rhs_end_node.next_ = lhs;
lhs_node.prev_ = rhs_end;
}
static std::generator<entt::entity> enumerate(entt::registry& entities,
entt::entity e)
{
// By default, entities are their own lists
if (!entities.any_of<T>(e))
{
co_yield e;
}
else
{
auto current = e;
while (true)
{
co_yield current;
current = entities.get<T>(current).next_;
if (current == e)
break;
}
}
}
private:
entt::entity prev_ = entt::null;
entt::entity next_ = entt::null;
};

You can use it like this:

struct cool_group : doubly_linked_component<cool_group> {};

By default, each entity represents its own 1-sized group. To merge two groups:

cool_group::merge(entities, left, right);

To iterate over all entities, I am using the new std::generator and coroutines. You can use it like this:

for (auto entity : cool_group::enumerate(entities, head))
do_something(entity);

entt will automatically call into on_construct and on_destroy.

People nowadays usually avoid linked lists because of all the pointer chasing required to actually use them. The pointer chasing is not the problem though, the non-locality is. If you make sure your nodes are all allocated in memory close to each other, there is hardly a penalty. entt will usually do this if nodes are also allocated close in time, which is often the case if you want to group things, so this works nicely in that regard, too. Feel free to use this code under CC0.