Category: Software Development

Caching Conan Dependencies in Docker for Faster Builds

One problem I often have when dockerizing my C++ Jenkins CI projects is handling incremental builds, for both our own code and the dependencies. Starting builds from scratch can take tens of minutes, too long for my taste.

My build stack is usually conan as a dependency manager and CMake/Ninja for building. Conan will usually try to download precompiled dependencies, but often enough, those are not available for my specific combination of compiler settings and flags, so it’ll build them on demand with the --build=missing flag. That usually takes the bulk of the time needed for a full build. So it makes sense to keep the dependencies cached, once they are built. However, since we use Docker to setup the build environment, they are all lost by default.

Who Owns What?

The obvious solution is to mount a folder on the build host to keep the conan cache using the -v / –volume option for docker run. This can be done by setting the CONAN_HOME environment variable, and I usually use one cache per build folder, which seems like a good compromise between speed and isolation.

But that causes other problems: docker will create all the files for the user inside the container, which is root by default, creating a whole bunch of files that the CI host user cannot delete, e.g. when a branch gets deleted. This breaks the CI setup to a point where manual intervention is required. A somewhat simple clutch is the -u user:group option to docker run, which will execute the build with the given user. The problem I was having with that, however, was that this user did not have access to user-scoped tool installations like conan via pipx.

User-specific Images

My current strategy to deal with this is to inject the host CI user and group into the docker ‘builder’ image, and then do all the building in the container using that user, as if using the CI host user on the metal. The Dockerfile looks like this:

FROM gcc:14.1-bookworm
RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get -y dist-upgrade
RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get -y install \
  cmake \
  debhelper \
  ninja-build \
  python-is-python3 \
  python3-pip \
  pipx
ARG HOST_USER_ID
ARG HOST_GROUP_ID
RUN groupadd -g ${HOST_GROUP_ID} hostgroup && \
  useradd hostuser -u ${HOST_USER_ID} -g ${HOST_GROUP_ID} -m -s /bin/bash && \
  mkdir /conan_home && chown hostuser:hostgroup /conan_home
USER hostuser
ENV PATH="$PATH:/home/hostuser/.local/bin"
RUN pipx install conan
ENV CONAN_HOME=/conan_home
WORKDIR /build_root
# Build the viewer deb package
CMD ["/source_root/build.sh"]

After doing the user-independent setup, this declares two ARGs for retrieving the user and group IDs, and then sets up a user with those in the docker image, calling it hostuser:hostgroup internally. Note that the names will not leak out of the container, only the IDs do.

It installs conan via pipx as that user and makes sure it is in the PATH for the build later. This is the real advantage of passing the user into the image creation: user specific things can be installed!

In our Jenkinsfile, I build the image from that while injecting the current user via the –build-arg option:

docker build . --iidfile docker_image_id \
  --build-arg HOST_USER_ID=`id -u` \
  --build-arg HOST_GROUP_ID=`id -g`

This expects three folders to be mounted: /source_root for the sources/repository, /build_root for the out-of-source build, and /conan_home for the conan cache. Important: make sure these folders are created by the CI user before passing them to docker, or it will create them with the wrong owner. I’m only creating the latter two, since the first one is obviously created by Jenkins.

mkdir -p docker/build docker/conan

Once the folders are set up and the image is built, I run the actual build in a container via:

docker run --rm \
  -v `pwd`:/source_root:ro \
  -v `pwd`/docker/conan:/conan_home \
  -v `pwd`/docker/build:/build_root \
  `cat docker_image_id`

That should run the actual build and populate the conan cache. After that I extract the artifacts I need and remove the docker image and ID file with:

docker image rm `cat docker_image_id` && rm docker_image_id

And we’re done!

The future of Grails

Many long-term readers of our blog may have noticed a post about Grails Framework topics every now and then. We are using Grails for more than 15 year both in customer projects and internal ones.

Sometimes using the framework was fun, productive and bliss. At other times it could be frustrating upgrading, chasing bugs or finding workarounds. Occasionally, performance could be a problem. Most of the time it was a solid framework with solid output and customer value.

The more recent past

Ownership/stewardship of Grails changed several times over the years from one company to another. Updates were very infrequent, the general direction was very unclear and the future of the framework extremely uncertain.

Because all of the above we did not start new projects using Grails and did not recommend it to potential customers but instead used other frameworks like Micronaut, Javalin or .NET.

But suddenly there was light at the end of the tunnel: Grails was handed over to the Apache Software Foundation (ASF) in the middle of 2025 and became a top-level project there. That by itself may not be a complete turnaround and rescue for the framework, but it certainly sparked a bit of hope into the whole situation.

The presence and future

Since the adoption of Grails by the ASF a lot has changed. Tons of work has been going on in the background to streamline future development, work on reproducible builds (as required by the ASF) and to enable frequent releases, updates and improvements.

Grails 7 was born and release under ASF stewardship. The community is more open than ever and it seems to be growing again after years of stagnation or even decline.

The roadmap and plans for the next months is clear and the project is moving with a steady pace towards the goals. Of course, there is a lot of work to do, like reviving and porting several plugins to enable all users to migrate to Grails 7 and beyond – but it is happening.

Conclusion

If all the positive changes under the stewardship of the ASF continue, the future of Grails can be bright for the years to come. At least, it does not pose a liability or risk for its users and their customers anymore.

In my opinion, two things besides all the technical stuff are most important:

  • The strong commitment of the ASF to further develop and enhance the project provides safety for developers and customers investments
  • The mindshift from an exotic framework to a productive, JVM-based framework leveraging standard technologies from the java/spring/hibernate ecosystem provides familiarity and stability

Imho, this quote of the Project Management Committee (PMC) chairman James Fredley says it best:

Grails is NOT an exotic outlier. It’s PRODUCTIVITY LAYERS on top of SPRING BOOT.

An Indicator That You’re Leaving the Realm of Unit Tests

Automated unit tests are the grassroot foundation of a healthy test suite. But they aren’t the only type of automated tests that we need to write in order to test a system thoroughly enough to be confident about its production readiness.

There are things like end-to-end or even GUI based tests that have completely different testing mechanics that unit tests. It is clear just from looking at the test code that they aren’t unit tests.

But for the wide range of integration tests, there is a subtle and nearly impercetible transition from unit test to integration test that is hard to explain. It doesn’t really matter on which side of the diving line between the two test types you are as long as you are close to it. But as tests evolve, you need to apply different advancement strategies to the different types of tests. One goal is to keep unit tests from becoming integration tests over time, which is prevalent when factoring out system parts that were small at first.

When things are hard to explain, we search for indicators that can serve as objective counselors and help with making the decision. For the distinction between unit and integration tests, one such indicator is the distance between motor point and reaction point. Let me explain the concepts:

Let’s pretend we need to test the implementation of a baker (or a baking machine):

@Test
void can_produce_bread() {
	Baker target = new Baker();
	
	Bread actual = target.bake();
	
	Bread expected = new Bread();
	assertEquals(
		expected,
		actual
	);
}

This is a straight-forward unit test in the AAA (arrange, act, assert) structure:

  • Arrange: We build the “test world” or the slice of the system that should be tested. We call it the “target” (some call it the “cut”, from “code under test”, which corresponds nicely with the “slice of the system”).
  • The target contains the motor point, the specific entry point where the code under test is “irritated” by calling a method. It is this irritation that causes the code under test to exhibit a certain behaviour that produces an observable result. The point where this result can be observed is the reaction point.
  • Act: We enable the motor point by calling the bake() method on our target baker. The code under test works its magic and gives us the result, which we call “actual”. The return value of the bake() method is the reaction point. It has two roles in the context of our test:
    1. It provides the observable result of the code under test.
    2. It serves as the last step of the code under test. The test framework leaves the code under test by returning the result. The exit point and the reaction point of the code under test are at the same spot (the distance between them is zero).
  • Assert: We compare the actual result of the code under test with our expected result. In our case, that’s a bit silly because we just want to have a bread, without any further attributes to it. But this blog post is not about the art of assertion, so we keep it simple and silly.

Let’s review the positions of the three named points:

If you read from top to bottom and left to right, the reaction point seems to be placed before the motor point. If you read it like a programmer should, you see that the point are positioned in their execution order: motor point, exit point, reaction point.

You also see that the distance between the points is very small and in the case of exit and reaction point only distinguishable if you look very closely.

That’s the indicator for writing an unit test: If your entrance to the code under test (the motor point) is effectively the same position as your exit from the code under test and the place where you get your actual result (the reaction point), you are unequivocally writing an unit test.

If the distances between the three points get larger, you are drifting away from unit tests and entering the big realm of integration tests. That is not necessarily a bad thing, sometimes it’s a necessity, but it should be a deliberate decision on your part and not an unnoticed accident.

Let’s look at an example where the distances between the points are larger:

@Test
void can_sell_prepared_goods() {
	Baker given = new Baker();
	Bakery target = new Bakery(
		given
	);
	
	target.prepareGoods(1);
	
	assertEquals(
		Optional.of(new Bread()),
		target.sell()
	);
	assertEquals(
		Optional.empty(),
		target.sell()
	);
}

In this case, our baker now owns his own bakery where he can sell his breads to make a living. But baking breads “just in time” a customer requests one is not a sustainable business model, so the bakery has to prepare in advance and sell from the supply.

To test that we can fill up the supply and it gets emptied correctly, this test (in combination with other tests not shown here) does the AAA structure again:

We arrange our test world by inventing a baker and giving him to the bakery, which is the target in our case. We want to test the functionality of the bakery and a baker is required to do so. We already asserted that the baker knows his trade.

Then we act on our target. This is the motor point moment: We call the code under test to elicit a behaviour. But as you can see, we don’t receive a result right away. The effect seems to happen internally and we need to observe it from a different angle. Our reaction point has moved away from the motor point. And we have several exit points on our test journey. This is getting complicated!

In order to assert that the bakery’s supply holds one bread when told to prepare only one, we just buy two breads consecutively and see what happens. If there is only one bread in supply, we should get a bread the first time and nothing for our second purchase. The reaction point is now the sell() method, a good distance away from the prepareGoods() method we used as the motor point. Both points are (hopefully) connected by internal machinery in the bakery. We don’t want to assert the internal machinery, we want to assert its outcome. This requires the distance between motor point (“pressing a button up here”) and reaction point (“getting a product down here”).

You might argue that this example is still an unit test and I would agree. But we already see mechanics that occur predominantly in integration tests:

  • Elaborate arrange steps
  • act step without a return value (“actual” is missing)
  • Multiple assertions, telling a story with their order

When you imagine that the breads need to be of different kinds (dark bread, wholemeal bread, the whole german bread culture), you can probably see how the small unit test we just wrote kind of explodes with secondary complexity.

A realiable indicator that an automated test is going to be complicated is the distance between motor point and reaction point. Once you know about the concept, you can incorporate it into your testing intuition.

I hope it helps you write better tests or write good tests more deliberately. If you have thoughts about the concept, share them in a comment!

Out of Memory when loading Records in Rails

Recently I ran into a problem that only showed up outside the development environment.

I had a small script that needed to iterate over all records in the database and load blobs.

Document.all.each do |doc|
  process(doc.blob)
end

With a small dataset everything worked as expected.
With production-sized data, however, the job was terminated by the runtime with an out-of-memory error.

This behaviour is not surprising once you look at what all.each actually does.

How all.each works

When calling all.each ActiveRecord is loading the complete result set into memory before the iteration starts.
For large tables this means that thousands or even millions of Ruby objects are instantiated at once.

If each record also references additional data — for example blobs, attachments, or associations — the memory usage grows quickly.

Loading Records with find_each

ActiveRecord provides find_each for exactly this scenario:

Document.find_each do |doc|
  process(doc.blob)
end

In contrast to each, this method does not load all records at once.
Instead, records are fetched in batches and yielded one by one.

Conceptually the process looks like this:

  1. Load a limited number of records
  2. Yield them to the block
  3. Discard them
  4. Load the next batch

By default, find_each loads records in batches of 1000.
The batch size can be configured:

Document.find_each(batch_size: 100) do |doc|
  process(doc.blob)
end

find_each always iterates in primary key order. This means the model must have a primary key that is orderable like integer or string. Any explicit ordering will be ignored.

If more control is required, find_in_batches can be used instead. It requires manual iteration over the batches.

Conclusion

Iterating over large tables with all.each is easy to write but can lead to excessive memory usage once the dataset grows.

For batch processing tasks, find_each is usually the safer default because it limits the number of instantiated records and keeps memory usage predictable.

Writing Integration Tests – Heuristics of what to Aim for

When software has grown over some years, or split into multiple moving parts, or both, chances are that a customer reports erroneous behaviour that cannot be represented in unit tests. You might then need an integration test, and while these have the advantage of resembling more the story of what your customer wanted to do and expected to see – one can feel a larger degree of freedom, or rather, a larger question mark, in what they actually should know or assume.

And especially when you are not the original author of said code base, it might be especially tricky to strike the right chords; so with this post, I try to specify a few heuristics to find the right scope, the right aim for such a test.

Now first of all, try not to aim too needlessly large. As in – when possible, prefer unit tests, and if you can’t, be intentional about writing an integration test. As Daniel laid out in a series of blog posts some years ago, you can think of a unit test as a theater play, and using these metaphors helps in making your test clearly convey what it is about. The largest impact is in identifying what the “target” is. It has to contain the code under test (“cut”, also), and it should be very clear who that is (think of a simple stage play, not some avantgarde David Lynch mystery thriller). To perpetuate that idea – unit tests are these that define the character of your target; who is our hero, in their inner core, regardless of the context?

As you knew beforehand, totally™ – writing the test in advance gives you a clearer overall understanding of your actualy problem. Any kind of test. If not done, for whatever reason, you might either now switch to a fresh branch and do so – or, at least, treat any test code as it could have been written before.
I do stress this because it can be too tempting to focus on specific implementation details, but if some interna are rightfully encapsulated, hidden from other production code, then your tests should also not see these things.

Do not change any field or method visibility in order to make your test work.

More “freedom” is deceptive. Integration tests tell the behaviour of your target-hero within a given backstory, but how much backstory do you need? And can you keep up the storytelling – imagine yourself to be a cliche Gen Z TikTok user that ran short of Methylphenidate – would you still follow the story, after describing that one tree over here, these cars over there, who owns them, and the general idea of a planet where all that takes place?

But in said codebase grown-over-years, it might still be hard to aim right. A “quick” way out (speaking in time of writing that test) would be to try to mock away everything you need to construct your target; but you shouldn’t do so on first sight.

For every argument, any dependency that your situation needs, get at least a coarse understanding of what it is used for, and why not a version of that entity could exist without that dependency. Take that time. This can feel like a detour, but either it reveals to you how the backdrop actually looks, or it might even give you a hint of needless coupling in that existing code base.

Either way, does every single line of the test reveal its purpose?
(you should apply that question to any line of production code as well, but the cases where it’s just not possible is larger there than it should be within tests.)

And then there’s the point of deduplication. As stated, tests can exist that have a bit of scaffolding to them. It’s not illegal. But consider that any behaviour covered by an integration test probably has some variations – stuff happens in a different order – and when several tests have the same setup, this might encourage outsourcing a “setup helper” method. You even should do so (imo) if you clearly want to say “in this scene, everything is the same as yesterday, BUT THEN…” but if you do so, keep one thing in mind: that setup function needs to show the same clarity that is required of the tests themselves.

If your setup functions exceed any trivial logic that makes it feel like you require an extra test just to prove that your test setups works, stop and rethink.

So this list surely is incomplete, and as stated, they are rough “heuristics”.
But if you have a nontrivial case to test, and do so in a project that is not completely in your brain right now, but just might want to somehow draw the line between “what should I (or the test) need to know” and “what is to be mocked or otherwise left out” – maybe these thoughts can help you too.

And if you disagree, I will be very glad to consider your stage plays.

C# is very strict about modify-while-iterating

Today I stumbled upon some behavior in C#’s List<> that I found very surprising. I had forgotten about RemoveAll() and basically implemented it myself:

var target = 0;
foreach (var each in list)
{
  if (!predicate(each))
    list[target++] = each;
}
list.RemoveRange(target, list.Count - target);

Apparently, this is not allowed. You cannot assign to any element in the List<> while you are iterating/enumerating it: The List<> implementation holds a ‘version’ number that is incremented any time a change is made, including assignments. When the Enumerator is advanced via MoveNext it checks for this version and throws the dreaded ‘Collection was modified’ exception.

Except that there shouldn’t really be a problem here, and the modification checking code is basically being too coarse. This code ‘compacts’ the list, copying elements where the predicate evaluates to true to the front of the list, and then cutting off the rest of the elements. There’s never really any doubt what each references. In fact, in other languages, this approach is even considered idiomatic to remove elements while iterating. In ancient C++, this is known as the remove/erase idiom, see also std::erase.

So why did the library designers of C# consider setting a value while iterating a problem? I don’t know, but at least now I have a story to remind of of RemoveAll()‘s existence.

Hybrid Python packaging for Debian/Ubuntu

Writing software in Python often is a pleasure and can lead to great products with limited costs because of its expressiveness and rich ecosystem.

One area where imho Python falls a bit short is deployment and packaging. On Linux many users and customers expect packages for their platform so they can manage the software installation and updates using the standard tools.

This is where the pain often starts. Depending on the dependencies of your python project it may be simple or rather hard to provide a decent experience for the people managing your software.

I want to present several ways of providing a decent deployment experience to your customer specifically for Debian-based linux distributions.

The simple case

If all the dependencies of your project are available in usable versions for the target distribution, it is quite easy to package a python project as a .deb. My preferred way is to just use stdeb like below:

python3 -m build --sdist --no-isolation
py2dsc-deb --with-python3=True --debian-version 1 ./dist/my_project.tar.gz

This will built a simple debian package installable on a matching destination platform. For simple cases this often is enough.

If only one or a few dependencies are missing, you could consider packaging them too using this approach and allowing your project to take this same route.

Not using packages at all

If some dependencies are not available on the target platform through Debian packages it may be easiest to just provide a tarball with an installation script. This script would essentially perform the following steps

  1. Unpack the source to a nice destination directory
  2. Create a venv there
  3. Install the dependencies in the venv
  4. Provide some startscript and/or service definition to launch the software using the venv

This is simple and usually scales to bigger projects but does not provide nice and clean integration into the system tools. Administrators have to manage the software this way and not the package manager way they may expect and be comfortable with.

A hybrid Debian package approach

My hybrid approach is a blend of the two above:

It builds a normal debian package containing the project itself along with version and dependency metadata. In the postinst-script of the package however, it creates a venv and installs the dependencies unavailable or unusable (e.g. wrong version) on the target platform.

First we create the debian packaging files using 

python3 setup.py sdist
dh_make -p my-project_1.0.0 -f dist/my-project-1.0.0.tar.gz

This creates a debian/ directory containing all the packaging metadata files. You should mainly edit the control, copyright and changelog files and then craft the postinst file for our hybrid packaging approach:

#!/bin/sh

set -e

case "$1" in
    configure)
      python3 -m venv /opt/my-project/venv
      . /opt/my-project/venv/bin/activate && pip install PyQt5 pytango==9.5.1 taurus pyepics
    ;;

    abort-upgrade|abort-remove|abort-deconfigure)
    ;;

    *)
        echo "postinst called with unknown argument '$1'" >&2
        exit 1
    ;;
esac

exit 0

For correct removal we need a modified postrm script too:

#!/bin/sh

set -e

case "$1" in
    purge|remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear)
      rm -rf /opt/my-project/venv/
    ;;

    *)
        echo "postrm called with unknown argument '$1'" >&2
        exit 1
    ;;
esac
exit 0

Using a final dpkg-buildpackage -b -us -uc we get a debian package that builds its own venv on the target machine using the dependencies we actually need and not what the system offers.

For us and our customers this is a perfect compromise:

It allows us to define the dependencies and their versions exactly and mostly independent from what the target system offers while coming as a normal debian package managed using system tools.

Finding the culprit in massive-components-interactions web apps

So, one of the biggest software developer joys is, when a customer, after you developed some software for them a few years back, is returning to you with change requests and it turns out that the reason you didn’t hear much from them was not complete abandonment of their project, or utter mistrust in your capabilites, or – you name it – but just that the software worked without any problems. It just did its job.

But one of the next biggest joys is, then, to actually experience how your software behaves after their staff – also just doing their job – growed their database up to a huge number of records and now you see your system under pressure, with all parts playing each other in large scales.

Now, with interactions like on-the-fly-updates during mouse gestures like drag’n’drop, falling behind real-time can be more than a minor nuisance; this might go against a core purpose of such a gesture. But how do you troubleshoot an application which is behaving smoothly most of the time, but then have very costly computations during short periods of time?

Naively, one would just put logging statements in every part that can possibly want to update, but this very quickly meets its limit when your interface just has so many components on display, and you can’t just check that one interaction on one singular part?

This is where I thought of this small device, which I now wanted to share. In my case, this was a React application, but we just use a single object, defined globally, which is modified by a single function like this


const debugBatch = {
    seen: [],
    since: null,
    timeoutId: null
};

const logBatched = (id)=> {
    if (debugBatch.timeoutId) {
        debugBatch.seen.push(id);
    } else {
        debugBatch.since = new Date();
        debugBatch.seen = [];
        debugBatch.timeoutId = setTimeout(() => {
            console.log("[UPDATED],", debugBatch);
            debugBatch.timeoutId = null;
        }, 1000);
    }
};

const SomeComplicatedComponent = ({id, ...}) => {
    ...

    logBatched(id);

    return <>...</>;
};

and I do feel compelled to emphasize that this completely goes outside the React way of thinking, taking care with every state to follow the framework-internal logic. Not some global object, i.e. shared memory that any component can modify at will. But while for production code, I would hardly advise doing so, our debugging profits from that ghostly setup of “every component just raises the I-was-updated-information to somewhere out of its realm”.

It just uses one global timeout, giving a short time frame during which every repeated function call of logBatched(...); raises one entry in the “seen” field of our global object; and when that collection period is over, you get one batched output containing all the information. And you can easily extend that by passing more information along, or maybe, replacing that seen: [] with a new Set() if registering multiple updates of the same component is not what you want. (Also, the timestamp is there just out of habit, it’s not magically required or so).

Note that you can do all additional processing of “how do I need to prepare my debugging statements in order to actually see who the real culprits are” after collecting, and done in an extra task, can even be as expensive as you want it to be without blocking your rendering. As in, having debugging code that significantly affects the actual problem that you are trying to understand, is especially probable in such real-time user interactions; which means you are prone to chasing phantoms.

I like this thing because of its simplicity, and particularly, because it employs a way of thinking that would instinctively make me doubt the qualification of anyone who would give me such a piece to review (😉) but for that specific use case, I’d say, does the job pretty well.

Breaking WebGL Performance

A couple of weeks ago, I ported my game You Are Circle to the browser using Emscripten. Using the conan EMSDK toolchain, this was surprisingly easy to do. The largest engineering effort went into turning the “procedurally generate the level” background thread into a coroutine that I could execute in the main thread while showing the loading screen, since threads are not super well supported (and require enabling a beta feature on itch.io). I already had a renderer abstraction targeting OpenGL 4.1, which is roughly on feature parity with OpenGL ES 3.0, which is what you see WebGL2 as from Emscripten. And that just worked out of the box and things were fine for a while. Until they weren’t.

60FPS to 2FPS

Last week I finally released a new feature: breakable rocks. These are attached to the level walls and can be destroyed for power-ups. I tested this on my machine and everything seemed to be working fine. But some people soon started complaining about unplayable performance in the web build, in the range of 1-2 FPS, coming from a smooth 60 FPS on the same machine. So I took out my laptop and tested it there, and lo and behold, it was very slow indeed. On chrome, even the background music was stuttering. I did some other ‘optimization’ work in that patch, but after ruling that out as the culprit via bisection, I quickly narrowed it down to the rendering of the new breakable rocks.

The rocks are circled in red in this screenshot:

As you can see, everything is very low-poly. The rocks are rendered in two parts, a black background hiding the normal level background and the white outline. If I removed the background rendering, everything was fine (except for the ‘transparent’ rocks).

Now it’s important to know that the rendering is all but optimized at this point. I often use the most basic thing given my infrastructure that I can get away with until I see a problem. In this case, I have some thingies internally that let me draw in a pretty immediate mode way: Just upload the geometry to the GPU and render it. Every frame. At the moment, I do this with almost all the geometry, visible or not, every frame. That was fast enough, and makes it trivial to change what’s being rendered when the rock is broken. The white outline is actually more geometry generated by my line mesher than the rock-background. But that was not causing any catastrophic slow-downs, while the background was. So what was the difference? The line geometry was batched on the CPU, while I was issuing a separate draw-call for each of those rocks. To give some numbers: there were about 100 of those rocks, with each of with up to 11 triangles.

Suspecting the draw call overhead, I tried batching, e.g. merging, all the rock geometry into a single mesh and rendering it with a single draw call. That seemed to work well enough. And that is the version currently released.

Deep Dive

But the problem kept nagging at me after I released the fix. Yes, draw calls can have a lot of overhead, especially in Emscripten. But going from 60FPS to 2FPS still seemed pretty steep, and I did not fully understand why it was so extremely bad. After trying Firefox’s Gecko Profiler, which was recommended in the Emscripten docs, I finally got an idea what was causing the problem. The graphics thread was indeed very busy, and showing a lot of time in MaxForRange<>. That profiler is actually pretty cool, and you can jump directly into the Firefox source code from there to get an idea what’s going on.

Geometry is often specified via one level of indirection: The actual ‘per Point’- a.k.a. Vertex-Array and a list of indices pointing into that, with each triplet defining a triangle. This is a form of compression, and can also help the CPU avoid duplicate work by caching. But it also means that the indices can be invalid, e.g. there can be out-of-bounds indices. And browsers cannot allow that fore safety reasons, so they check the validity before actually issuing a rendering command on the GPU. MaxForRange<> is part of the machinery to do just that via its caller GetIndexedFetchMaxVert. It determines the max index of a section of an index buffer. When issuing a draw call, that max-index is checked against the size of the per-point-data to avoid out-of-range accesses.

This employs a caching scheme: For a given range in the indices, the result is cached, so it doesn’t have to be evaluated again for repeated calls on the same buffer. Also, I suspect to make this cache ‘hit’ more often, the max-index is first retrieved for the whole of the current index buffer, and only if that cannot guarantee valid access, is the subrange even checked. See the calls to GetIndexedFetchMaxVert in WebGLContext::DrawElementsInstanced. When something in the index list is changed from the user side, this cache is completely evicted.

The way that I stream my geometry data in my renderer is by using “big” (=4mb) per-frame buffers for vertex and index data that I gradually fill in some kind of emulated “immediate mode”. In the specific instance for the rocks, this looks like this:

for (auto& each : rock)
{
  auto vertex_source = per_frame_buffer.transfer_vertex_data(each.vertex_data);
  auto index_source = per_frame_buffer.transfer_index_data(each.index_data);
  device.draw(vertex_source, index_source, ...);
}

The combination of all that turned out to be deadly for performance, and again shows why caching is one of the two hard things in IT. The code essentially becomes:

for (auto& each : rock)
{
  auto vertex_source = per_frame_buffer.transfer_vertex_data(each.vertex_data);
  invalidate_per_frame_index_buffer_cache();
  auto index_source = per_frame_buffer.transfer_index_data(each.index_data);
  fill_cache_again_for_the_whole_big_index_buffer();
  device.draw(vertex_source, index_source, ...);
}

So for my 100 or so small rocks, the whole loop went through about 400mb of extra data per frame, or ~24gb per second. That’s quite something.

That also explains why merging the geometry helped, as it drastically reduced the amount of cache invalidations/refills. But now that the problem was understood, another option became apparent. Reorder the streamed buffer updates and draw calls, so that all the updates happen before all the draw calls.

Open questions

I am still not sure what the optimal way to stream geometry in WebGL is, but I suspect reordering the updates/draws and keeping the index buffer as small as possible might prove useful. So if you have any proven idea, I’d love to hear it.

I am also not entirely sure why I did not notice this catastrophic slow-down on my developer machine. I suspect it’s just because my CPU has big L2 and L3 caches that made the extra index scans very fast. I suspect I will see the performance problem in the profiler.

Digitalization is hard (especially in Germany)

Digitalization in this context means transforming traditional paper based processes to computer-based digital ones. For existing organisations and administrations – both private and public – such a transformation requires a lot of thought, work and time.

There are mostly functioning albeit sometimes inefficient processes in place providing services that do not allow interruptions or unavailabilities for extensive periods of time. That means the transition has to be as smooth as possible often requiring running multiple solutions in parallel or providing several ingestion methods and output formats.

Process evolution in general

Nevertheless I see a general pattern when business processes are transformed from traditional to fully digital:

I have observed and performed such transformations both privately as a client or customer and professionally implementing or supporting them.

Status quo

The current state in many organisations in Germany is “Digital Documents” and that is where it often stops. The processes themselves remain largely unchanged and opportunities and improvements remain lost.

Unfortunately this is the step where a lot of potential could be uncovered: Just by using proper collaboration tools one could assign assign tasks to specific people in a process associated to digital documents, track the progress and inform watchers. In many cases this results in much tighter processes, shorter resolution times and hugely improved documentation and traceability.

Going even further

The next step is where service providers like us are often brought to the table to extend, improve or replace the existing solution with custom- and purpose-build software to maximise efficiency, usability and power of the digital world.

Using general tools for certain processes and a certain time often shows the shortcomings and lets you destill a clearer picture of what is actually needed. Using that knowledge helps building better solutions.

Requirements for success

For this whole transformation to be successful one has to be very careful with the transition. It is seldom as easy as shutting down the old way ™ and firing up the new stuff.

Often we need to keep several ingestion points open – imaging snail mail, e-mail, texting, voice mail, web interface, app etc. as possible input media. At different points in the process several people may want to use their own way of interating with the process/documents/associated people. In the end the output may still be a paper document or a digital document as the end artifact. But maybe in addition other output like digital certificates, codes or tokens may benefit the whole experience and process.

So imho the key besides digitalisation and a good process analysis is keeping the process flexible and approachable using different means.

Some examples we all know:

  • Paying at a store often offers cash, bank card, credit card and sometimes even instant payment systems like Paypal or Wero
  • Document management with tools like Paperless-ngx office allows ingestion by scan, e-mail, direct upload etc. in different formats like PDF, JPG, PNG and hybrid storage digitally and optionally in a filing cabinet using file folders.
  • Sick notices may be sent in using phone, e-mail, web forms, in-app and be delivered by the means the recipient likes most.

The possibilities are endless and the potential improvement of efficiency, speed and comfort is huge. Just look around you and you will begin to see a lot of processes that could easily be improve and cause many win-win situations for both, service providers and their clients.