Multi-Page TIFFs with C++

If you are dealing with high-speed cameras or other imaging equipment capable of producing many images in a short time you may find it handy to put many images into a single file. There are several reasons to do so:

  • Dealing with thousands of files in a single directory or spreading them over a directory hierarchy may be slow and cumbersome depending on the tools.
  • Storing many images together may communicate better that they belong together, e.g. to the same scan.
  • Handling and transmitting fewer files is often easier than juggling with many.

TIFF is a wide-spread lossless image format capable of handling many individual images in a single file. Many image viewers and image manipulation tools are able to work with multi-page TIFF files so you are quite flexible in working with such files.

But how do you produce these files from your programs? I found some solutions with different strengths and weaknesses:

Using Magick++

Magick++ is the C++ API for ImageMagick – a powerful image manipulation library. If you can hold all the images for one file in memory the code is easy and straightforward:

#include <string>
#include <Magick++.h>

class TiffWriter
{
public:
    TiffWriter(std::string filename);
    TiffWriter(const TiffWriter&) = delete;
    TiffWriter& operator=(const TiffWriter&) = delete;
    ~TiffWriter();

    void write(const unsigned char* buffer, int width, int height);

private:
    std::vector<Magick::Image> imageList;
    std::string filename;
};

TiffWriter::TiffWriter(std::string filename) : filename(filename) {}

// for example for a 8 bit gray image buffer
void TiffWriter::write(const unsigned char* buffer, int width, int height)
{
    Magick::Blob gray8Blob(buffer, width * height);
    Magick::Geometry size(width, height);
    Magick::Image gray8Image(gray8Blob, size, 8, "GRAY");
    imageList.push_back(gray8Image);
}

TiffWriter::~TiffWriter()
{
    Magick::writeImages(imageList.begin(), imageList.end(), filename);
}



The caveat is that you need to  hold all your images in memory before writing it to the file on disk. I did not manage to add and persist images on the fly to disk.


In our environment it was absolutely necessary to do so because of the amount of data and the I/O required to persist all image in time. So I had to implement a slightly more low-level solution using libtiff and its C API.


Using libtiff


#include <string>
#include <tiffio.h>

class TiffWriter
{
public:
    TiffWriter(std::string filename, bool multiPage);
    TiffWriter(const TiffWriter&) = delete;
    TiffWriter& operator=(const TiffWriter&) = delete;
    ~TiffWriter();

    void write(const unsigned char* buffer, int width, int height);

private:
    TIFF* tiff;
    bool multiPage;
    unsigned int page;
};

TiffWriter::TiffWriter(std::string filename, bool multiPage) : page(0), multiPage(multiPage)
{
    tiff = TIFFOpen(filename.c_str(), "w");
}

void TiffWriter::write(const unsigned char* buffer, int width, int height)
{
    if (multiPage) {
        /*
         * I seriously don't know if this is supposed to be supported by the format,
         * but it's the only we way can write the page number without knowing the
         * final number of pages in advance.
         */
        TIFFSetField(tiff, TIFFTAG_PAGENUMBER, page, page);
        TIFFSetField(tiff, TIFFTAG_SUBFILETYPE, FILETYPE_PAGE);
    }
    TIFFSetField(tiff, TIFFTAG_PLANARCONFIG, PLANARCONFIG_CONTIG);
    TIFFSetField(tiff, TIFFTAG_IMAGEWIDTH, width);
    TIFFSetField(tiff, TIFFTAG_IMAGELENGTH, height);
    TIFFSetField(tiff, TIFFTAG_SAMPLEFORMAT, SAMPLEFORMAT_UINT);
    TIFFSetField(tiff, TIFFTAG_ROWSPERSTRIP, TIFFDefaultStripSize(tiff, (unsigned int) - 1));

    unsigned int samples_per_pixel = 1;
    unsigned int bits_per_sample = 8;
    TIFFSetField(tiff, TIFFTAG_BITSPERSAMPLE, bits_per_sample);
    TIFFSetField(tiff, TIFFTAG_SAMPLESPERPIXEL, samples_per_pixel);

    std::size_t stride = width;
    for (unsigned int y = 0; y < height; ++y) {
        TIFFWriteScanline(tiff, buffer + y * stride, y, 0);
    }

    TIFFWriteDirectory(tiff);
    page++;
}

TiffWriter::~TiffWriter()
{
    TIFFClose(tiff);
}



Note that line 14 is needed if you do not know the number of images to store in the file in advance!

Learning UX design: where do I start?

Where do I start? A typical question when trying to step into a new field. So many resources, so many definitions, concepts, opinions.

Where do I start? A typical question when trying to step into a new field. So many resources, so many definitions, concepts, opinions. A needle in a haystack. Most of the beginner articles for UX are tailored for design students. Many of the resources for teaching design to developers aim at getting better at visual design or interaction design. But what if your goal is to make life better for users of the software you develop ?
A simple question. ‘It depends’ I hear a lot. I think there are two things everybody can do or learn to do that have a profound effect on the UX of your products.

Think

Increasingly – since beginning to focus on UX – I get the following feedback by our customers and users: “You surely put much thought into it.” Much thought. What does that even mean? Why do they say that?
Common wisdom says you should think like a user in order to create a better experience for him. In my view you should think for the user. Think about what he wants and what he needs. The goals he want to reach and the information and guidance he needs.
Don’t stop at the what. As developers we often only consider what needs to be done. The user stories. The functional specifications. What functionality needs to be implemented. Fixed scope. We do not need to create exactly what the spec says.
Remember our goal? Make life better for our users. We need to understand what this means. Besides understanding the processes, methods and concepts of our user’s domain, we need to find out what the goals are and why the user wants to reach it.
Nobody wants to use a writing application. Nobody wants to write a letter. We need to dig deeper. Maybe he wants to cancel a magazine subscription or a contract. Or he wants to express his feelings to a loved one. That’s better. But what is his goal? In case of the cancellation letter: To save money? To reduce waste?
The why is important to the user and therefore it should be important to us. Not only our focus is changed from the things we need to implement to the goals the user wants to reach, we also have more freedom and a guiding post at the same time. We can find solutions which are outside of the initial user story or even outside the computer. And on the other hand we can evaluate decisions we need to make against helping our users reach their goals.
Think of our work as a bridge. The user wants to reach the other side of the river. Our bridge should be the most efficient and pleasurable way to get there.

Pleasurable

This is where the feelings from the journey of our user comes in. As a developer we try to find a balance between the goals of the business and the technological constraints. When we consider making the lives of our users better, the goals and needs of the users add a new force to be weighted. This is the primary task of the UX designer. We need to find the underlying goals and intentions on the one side and the needs on the other. All of them need to be balanced with the goals of the business.
So how do I find these intentions and needs?

Listen

Really, really listen to your users. The notion that users do not know what they want is poisonous. They don’t need to tell you directly what they want. You need to listen. You need to observe. Start with a beginner’s mind. Let the user explain. Do not assume. I tend to think ahead, to formulate ideas and solutions while listening. Now I am rather naive. I ask why. Why is it this way. Why after why. Do not settle but also do not stress your users. Get to the goals. Discover the needs from the current office environment of the user or the difficulty of the task. Learn what is important for the user and what not. Learn to listen for specific naming and phrasing. Human needs stem from our basic nature, look at Maslow’s hierarchy of needs.
After you collected all sorts of information you need to resolve conflicts, balance the trade offs, reach consensus. You need to construct a whole from the parts of your listening. Therefore you need to think. Prioritize. Sort out. Reflect. Again: you should not assume. Use your whys.

Start with thinking and listening

UX design is a broad field with a simple goal: making life better for our users. In order to reach this we need to think. We need to listen. We need to care. No tool or method will change that. As developers we like to learn tools, languages and have recipes and methods. We would love to have a top ten resources list. The books to read. The course to learn. But all that does not save us from thinking. UX design is even more so: thinking for the user is the core of UX design.

Our voyage to service separation – Part I

We transformed our evolutionary grown IT landscape to a planned setup. Here is what we learned on the way.

What you need to know

We are a small software development company with a home-grown IT infrastructure. The euphemism for such a state is “evolutionary grown”, denoting a process that was shaped by the most elementary forces, often implicitely. One such implicit force is laziness: If there is a quick way to do things, it will be done this way. Why invest effort if everything works just fine?
During an internal safety review, we identified our IT landscape as a risk factor. It was designed to meet yesterday’s and perhaps today’s demands, but in no way aligned to our strategic vision. We decided to invest in our IT to bring it to a planned state that we are confident will sustain our demands of tomorrow – or be easily adaptable.

Where we started

Our starting point was a room full of servers that were bought at some point to serve a specific need like “be the build box”. Every server started with a good reason to exist and evolved from there. Some gathered more and more services, some were repurposed and some idled along. We identified only two servers that were essential for the company: one was the continuous integration server master and one hosted nearly all mission-critical services at once. The latter server was also our oldest machine in production usage. It was secured against data loss, but not against outages. So everytime this server went down, our company essentially came to a stop because all services were offline. Luckily, it went down very infrequently, but it still identified as a clear single point of failure.

Where we wanted to go

containersIn April 2014, the heartbleed vulnerability was published. We luckily weren’t affected on a large scale, but took it as a wake-up call to review our IT setup and to develop a strategy to mitigate the effects of disasters similar in scope to heartbleed while we still have time. We wanted to have our IT in a condition where we actually choose which risks we take instead of just hoping for the best. So we sat before a whiteboard and outlined the goals: We wanted to separate and self-contain every essential service, so that the compromising or outage of one service doesn’t affect the others. That means one machine (or container) per service. To gain flexibility, we also planned to separate our IT landscape into two layers: The “metal layer” provides the computation power, while the “appliance layer” realizes the services. We wanted to be able to implement the appliance layer nearly independtly to the metal layer, which means to use some sort of virtualization. In modern words, we wanted to have a “cloud platform” to deploy our service applications on. We just don’t wanted it out on the internet but in our computer center. To sum up, we wanted to separate hardware and software and move every service in its own compartment.

What technologies we chose

We thought about fitting technology for a long time but settled for a small-scale, bottom-up approach: Start with just a few metal machines (hosts) and use a familiar virtualization product. In our case, this meant two standard servers, Linux and Oracle’s VirtualBox to run the virtual computers. There sure are more professional and powerful virtualization products out there, but we had years of experience (and sometimes frustration) with VirtualBox and didn’t want to rely on an unknown technology. It’s not exciting, but works well enough for our use case – and we knew that beforehands.
We decided against any fancy cloud or grid software to combine the hosts to a pool and just planned the hosting of the virtual machines (VMs) statically by hand. This might mean that one host gets bored while another host cannot handle the pressure anymore. It will be our responsibility to take that problem into account. This approach primarily achieves one thing: it keeps everything rather simple. Each host has a list of VMs and that’s it. If we want to migrate a VM to another host, we have to do it manually.
To create the VMs, we used Vagrant, which turned out fine for three-quarters of our machines, but proved toxic for the remaining ones. Vagrant is a very handy tool for developers to quickly launch a VM, but it makes a lot of assumptions that might not match your specific requirements. We essentially abandoned Vagrant after the initial phase.
During the migration phase of our services, we adopted another tool to solve the problem of scaling effects in maintainance. It’s another story to maintain 20+ servers instead of the handful we had beforehands. Luckily, Ansible proved useful to automate most of our normal administration tasks. This transition from manual to automated administration wasn’t part of the original plan, but is one of the biggest payoffs. But that’s stuff for the next blog entry.

What’s next?

In this first part of our story to regain control of our IT landscape, we described the starting point, the plan and the tools. In the next part, you’ll hear about the migration and where we ended up. We will also point out our experiences along the way and hopefully give some useful tips if you think of reshaping your services, too: Click here to read part two of the series.

Meet my Expectations!

A while ago I came across a particulary irritating piece of code in a somewhat harmlessly looking mathematical vector class. C++’s rare feature of operator overloading makes it a good fit for multi-dimensional calculations, so vector classes are common and I had already seen quite a few of them in my career. It looked something like this:

template <typename T>
class vec2
{
public:
  /* A few member functions.. */
  bool operator==(vec2 const& rhs) const;

  T x;
  T y;
};

Not many surprises here, except that maybe the operator==() should be a free-function instead. Whether the data members of the class are an array or named individually is often a point of difference between vector implementations. Both certainly have their merits. But I digress…
What really threw me off was the implementation of the operator==(). How would you implement it? Intuitively, I would have expected pretty much this code:

template <typename T>
bool vec2<T>::operator==(vec2 const& rhs) const
{
  return x==rhs.x && y==rhs.y;
}

However, what I found instead was this:

template <typename T>
bool vec2<T>::operator==(vec2 const& rhs) const
{
  if (x!=rhs.x || y!=rhs.y)
  {
    return false;
  }

  return true;
}

What is wrong with this code?

Think about that for a moment! Can you swiftly verify whether this boolean logic is correct? You actually need to apply De Morgan’s laws to get to the expression from the first implementation!
This code was not technically wrong. In fact, for all its technical purposes, it was working fine. And it seems functionally identical to the first version! Still, I think it is wrong on at least two levels.

Different relations

Firstly, it bases its equality on the inequality of its contained type, T. I found this quite surprising, so this already violated the POLA for me. I immediately asked myself: Why did the author choose to implement this based on operator!=(), and not on operator==()? After all, supplying equality for relations is common in templated C++, while inequality is inferred. In a way, this is more intuitive. Inequality already has the negation in its name, while equality is something “original”! Not only that, but why base the equality on a different relation of the contained type instead of the same? This can actually be a problem when the vector is instantiated on a type that supplies operator==(), but not operator!=() – thought that would be equally surprising. It turned out that the vector was only used on built-in types, so those particular concerns were futile. At least, until it is later used with a custom type.

Too many negations

Secondly, there’s the case of immediately returning a boolean after a condition. This alone is often considered a code-smell. It could be argued that this is more readable, but I don’t want to argue in favor of pure brevity. I want to argue in favor of clarity! In this case, that construct is basically used to negate the boolean expression, further obscuring the result of the whole function.
So basically, the function does a double negation (not un-equal) to express a positive concept (equal). And negations are a big source of errors and often lead to confusion.

Conclusion

You need to make sure to make the code as simple and clear as possible and avoids any surprises, especially when dealing with the relatively unconstrained context of C++ templates.  In other words, you need to make sure to meet the expectations of the naive reader as well as possible!

At your service, master!

One of the most important lessons that I had to learn in my job is that you have to be aware of the client. As a service provider, it is my duty to satisfy my client’s needs – and without knowing him, I will not be able to succeed. In this blog post I describe some insights that helped me to gain a better understanding of my clients.

The main connection between the service provider and his client is the communication between them. In an ideal world, the two parties would be able to understand each other perfectly, however, humans and their language are fallible and to me, this seems to be the root of most problems. Of course, both parties are responsible, nevertheless, the service provider should not only deal with his own defeciencies, but also with his client’s, in order to attract and keep clients. Next, I will list five instructions that can reduce or sometimes even prevent the incomprehension in the communication.

Be prepared

This is maybe the clearest rule: Before meeting a client, you should know the basics of the domain he is working in and of the problem he wants to solve. It is not the client’s job to explain his request, but rather the service provider’s job to comprehend it. Besides, if a client feels understood, he will also feel that you can solve his problem – and at that stage this matters more than whether you can actually solve his problem.

Be attentive

You and your client are different persons and, as a result, have a different understanding of the same things. Your client might quickly slur over some little details in a software he wants, so you could assume that they are of no importance – but you will be unpleasantly surprised when it turns out to be a critical aspect of the program. And this is not necessarily a flaw in the customer’s communication: Maybe to a domain expert – and your customer might be one – the importance of these details is totally obvious.

Furthermore, sometimes even language will lead you nowhere. For example, people do not always realize why a system is hard to use or where they make mistakes and hence cannot tell you about it, but by watching them you might find the problems. In such a situation, it is crucial to grasp not only the words the client is saying, but also other signals he is emitting.

Be without bias

As soon as I start listening to a client’s problem, sometimes I can literally watch myself constructing a solution in my head. I create a mental model composed of the components the customer is talking about, think about their relationships – and suddenly, I find myself thrown a curve because the client added a thought that objects my conception.

Of course, a model can improve the understanding of a client’s demands, however, one has to constantly question the validity of the model and – in case it is disproved – one must drop it without hesitation. Do not become attached to a model just because it is so elegant – in most cases, you will be betrayed. In contrast, it will probably become easier to adapt your mental models if you stay open-minded.

And even if your view seems to suit the customer’s requirements perfectly, you should hesitate to present it to him, you should not ask for confirmation early on. In fact, the better the concept seems, the more careful you should be: You might lead your client into thinking that it is a adequate solution, and by focusing on the conformity between the concept and his problem, you and your client may fail to see flaws.

Instead, you should try to ditch your assumptions, try to listen without bias. You still have to prepare yourself before you meet your client, but you should be willing to scrutinize your knowledge and to discard incorrect information.

Be concrete

The human language is a wonderful medium, but unfortunately terribly inaccurate. If, instead of writing, you can talk with your customer, you should usually choose the latter. Even better, if you can meet him in person, do it – there are so many more options to communicate if you are in one room that you will almost surely benefit from it.

For instance, if your client wishes a feature with some user interface, you can sketch it or build a paper prototype; you could even prepare a real prototype consisting only of the user interface. This allows your customer to play with it and facilitates the communication. And do not be abstract, do not fill your widgets with texts as “Lorem ipsum” – it does not matter if the content is made-up, but it should be realistic.

User interface design is a neat example since it is graphical, nevertheless, you can apply this principle to other tasks. It does not matter if you talk about a processes, some architecture, domain models or other structures: Even though most of them have no inherent graphical representation, it is usually easier to describe them graphically then by using text.

Seek for the why, not the what

Often, I tend to ask my clients about the problem they wish to solve – I ask what they wish to solve, not why. Usually, this is sufficient; he knows his situation and is able to express his needs. Unfortunately, it also happens that albeit the customer’s problem is solved according to his description, his wants are not satisfied – and the reason for this is that even he did not know what he actually needed. Even worse, sometimes I get caught by the “how”, that is, I quickly find a nice solution for some parts of a client’s problem, so I stick to it, maybe even implement it – and in the end I realize that it actually prevents me from solving the complete problem.

Hence, it is not only important to find out what your client wants to achieve, but also why he wants to achieve it, you have to understand his motivation. This can enable you to correct your client’s mistakes and to lead him to the question he actually wants to answer. Furthermore, this is a great handle to control the effort of a project: It becomes easier to identify indispensable core functionality and to find features whose usefulness is questionable, and hence, one can communicate with the client if some of the latters might be dropped. Simon Sinek gave an interesting TED talk to a similar topic found here.

Conclusion

Understanding your customers is difficult, but not impossible. I think that actively directing the attention at your counterpart, being open for input and questioning your assumptions and knowledge can strongly improve the communication with your clients.

Transferring commits via Git bundles

Sometimes you want to send (e.g. by e-mail) a set of new Git commits to someone else who has the same repository at an older state, without transferring the whole repository and without sharing a common remote repository.

One feature that might come to your mind are Git patches. Patches, however, don’t work when there are branches and merge commits in the commit history: git format-patch creates patches for the commits across the various branches in the order of their commit times and doesn’t create patches for merge commits.

Git bundles

The solution to the problem are Git bundles. Git bundles contain a partial excerpt of a Git repository in a single file.

This is how to create a bundle, including branches, merge commits and tags:

$ git bundle create my.bundle <base commit>..HEAD --branches --tags

<base commit> must be replaced with the last commit (i.e. commit hash or tag), which was included in the old state of the repository.

A Git bundle can be imported into a repository via git pull:

$ git pull /path/to/my.bundle

Get it up and running

West Side-project story
West Side-project story

I have seen quite a few projects that spent a ton of calendar/developer time and bugdet in building components and frameworks instead of getting something running. Running in this sense does not mean being able to show an application started from a developers IDE on her development notebook. With running I mean versioned artifacts deployed on some sort of staging infrastructure the client/customer has access to. Let me elaborate on that:

Walking skeleton

I really like the notion of the walking skeleton coined by Steve Freeman and Nat Pryce in “Growing Object-Oriented Software, Guided by Tests“. While I do not want to emphasize TDD and/or automated end-to-end testing I see great benefits in producing a walking skeleton that touches all important parts of a system and is working with a minimal set of functionality. For me that means that all of the following elements are in place, albeit in a primitve way which can be refined on the way:

  • The code is hosted in a source code repository accessible to the project members
  • A build system is chosen and able to produce a runnable artifact
  • A continuous integration server (CI) is triggered on changes in the repository and produces the runnable artifact
  • The artifact is easily installable on the target machines and/or installed on a staging system that resembles the target system as closely as reasonable
  • If there are components they talk to another using minimal requests and stubbed replies that can be refined over time

I usually aim for that walking skeleton in the first few hours into a project after the base technologies and requirements allow starting with coding. That may take up to several days when the system is more complex but it should not take weeks. Connect different parts of the system as early as possible even if responses are minimal, hardcoded or “wrong”. It shows that the parts are able to communicate and mismatches will become visible either someway along the build process or at least when running application. Why should you choose such an approach?

Benefits

  • Most people I know are better at evolving and improving existing stuff than at creating new stuff in empty space in a focused and efficient way. If you have a skeleton of the system it is much easier to talk about the interfaces and responsibilites of the different parts of the system.
  • You get to define APIs which you can evolve over time instead of specifing the complete API and experience implementation and mismatch problems much later when the components need to be integrated.
  • Similarily it is much more difficult to package a complex system after it is finished than to package a simple minimal system and evolve all aspects – building, implemenation and packaging/deployment, maintainance – when necessary.
  • You see if things may work like expected or if there is some inherent problem in the whole design much earlier. Essentially you evolve your proof-of-concept into something with real value for your clients.
  • As soon as your system provides some value you can deploy the working stuff long before the whole project is finished.
  • It is much easier to discuss a working system than to reason about a system not yet existing.
  • You are eating your own dogfood from early on and can address pain points in development, user experience, deployment and running the application.
  • You have something to show more or less right from the beginning and progress will be visible throughout the project.
  • No “Works on my machine!” syndrome.

Potential Problems

Of course there is the challenge of continuously refactoring and extending existing stuff. Later on data migration or migration of configuration may be additional tasks. But hey, you are skilled, agile developers that embrace the idea of changing requirements and the ability to move fast. You have to pay attention not to accumulate too much technical debt as it will slow you down and hurt you in the long run.

Summary

Running systems providing value are what your clients often care about the most. If you can something like that early on communication with your stakeholders tends to be more relaxed as they have better possibilities of steering in the right direction and they steadily see progress. Running software should be the primary goal.

Thinking in immutability

The way I learned programming is dictated by objects and states. Besides advantages and on going efforts in the industry I couldn’t help but thinking: immutability is nice. I can use it in some cases and keep it quietly stored in the corner. But it didn’t remain silent.

The way I learned programming is dictated by objects and states. According to my thinking data is packed into objects which are later modified to reflect the changes over time. State and modification are a central modelling technique. For me programming and OOP in particular resolved around this common theme. Mutating objects pervade my thinking even beyond the code into the database and even the architecture of the whole system.
Besides advantages and on going efforts in the industry I couldn’t help but thinking: immutability is nice. I can use it in some cases and keep it quietly stored in the corner.
But it didn’t remain silent.
So I asked myself: How do you construct programs that build upon immutability? How do you (mostly) avoid mutable objects? How do you think in immutability?
The first step was to unlearn. No updates. No modifications. Read, create, copy. That’s about it. No more CRUD only CR. No more SQL updates, only inserts.

Events and logs

To illustrate I use a simple example. Creating, moving, translating and deleting a point. In the traditional OO way it looks like this:

Point p = new Point(40, 30);
p.translateXBy(5);
p.moveTo(10, 20);
p.delete();

Or using SQL might be something like this (omitting primary keys and where clauses here):

insert into points (x, y) values (40, 30)
update points p set p.x = p.x + 5
update points p set p.x = 10, p.y = 20
delete from points

In our memory (or database if we use one) every line updates our point:

Point p = new Point(40, 30); // p = {x: 40, y: 30}
p.translateXBy(5); // p = {x: 45, y: 30}
p.moveTo(10, 20); // p = {x: 10, y: 20}
p.delete(); // p = ?

But what if we do not store the results of the operations but the operations themselves? The events.
Imagine your state changes as a series of events. Just imagine.

new PointCreated(40, 30); // pointEvents = [{created[x: 40, y:30]}]
new PointTranslatedXBy(5); // pointEvents = [{created[x: 40, y:30]}, {translated[x: 5]}]
new PointMovedTo(10, 20); // pointEvents = [{created[x: 40, y:30]}, {translated[x: 5]}, {moved:[x: 10, y:20]}]
new PointDeleted(); // pointEvents = [{created[x: 40, y:30]}, {translated[x: 5]}, {moved:[x: 10, y:20]}, {deleted}]

Even in the database we would just use inserts, no more updates and no more deletes. The events are stored in a log (ironically the database does this the same way). A log is a fully ordered, append only queue. Once we use and store events we have some extras besides immutability: an audit trail, an undo stack, recovering, …
We could externalize the event stream in a message queue and could monitor it, replay it to reproduce bugs, distribute it. The possibilities are endless.

But. That’s all nice and fine. I have one more question: what’s the current state? A user should see the current state and other parts of the system also (not mentioning that I – coming from a mutable state kind of thinking – would also feel better seeing it).

So what’s the current state?

All events applied in order.

OK. But isn’t this expensive doing this all the time?

Yes!

Here another concept from databases helps us: materialized views. We can easily translate in our mind between the new immutable event driven way and the old in place update way. It is just the same data in different representations (if we only are interested in the current state). If we store the current state as a materialized view (or cache) besides the event log we can have both.
Every part of the program which needs the current state gets an immutable copy of it. If this part needs to know when something changes, it can observe the events and act accordingly. This way mutability is pushed to the borders, to the parts where the current state is shown (like the UI layer).

My motto: Make it visible

To have a clear principle that guides your actions and inventions is a very powerful thing. In this short blog entry, I describe my principle of making information visible.

Nearly ten years ago, I read the wonderful book “Behind Closed Doors. Secrets of Great Management” by Johanna Rothman and Esther Derby. They shared a lot of valuable insights and tipps for my management career, but more important, gave a name to a trend I was pursuing much longer. In their book, they introduce the central aspect of the “Big Visible Chart”, a whiteboard that contains all the important work. This term combined several lines of thought that lingered in my head at the time without myself being able to fully express them. Let me reiterate some of them:

  • Extreme Feedback Devices (XFD) were a new concept back in the days. The aspect of physical interaction with a purely virtual software project thrilled me. Given a sensible choice of the feedback device, it represents project state in a intuitive manner.
  • Scrum and Kanban Boards got popular around the same time. I always rationally regarded them as poor man’s issue tracker, but the ability to really move things around instead of just clicking had something in itself.
  • My father always mentioned his Project Cockpit that he used in his company to maintain an overview of all upcoming and present projects. This cockpit is essentially a Scrum Board on project granularity. We use our variation with great success.
  • A lot of small everyday aspects required my attention much too often. Things like if the dishwasher in a shared appartment contains dirty or clean dishes always needed careful examination.

It was about time to weave all these motivations into one overarching motto that could guide my progress. The “Big Visible Chart” was the first step to this motto, but not the last. A big chart is really just a big information radiator and totally unsuited for the dishwasher use case. The motto needed to contain even more than “put all information on a central whiteboard”. I wasn’t able to word my motto until Bret Victor came along and held his talk “Inventing On Principle” (if you don’t know it, go and watch it now, I’ll be waiting). He talks about the personal mission statement that you should find to arrange your actions around it. That was the magical moment when everything fell into place for me. I knew my motto all along, but couldn’t spell it. And then, it was clear: “Make it visible”. My personal mission is to make things visible.

Let me try to give you a few examples where I applied my principle of making information visible:

  • I built a lot of Extreme Feedback Devices that range from single lamps over multi-colored displays to speech synthesis and even a little waterfall that gets switched on if things are “in a state of flux”, like being built on the CI server. All the devices are clearly perceivable and express information that would otherwise need to be actively pulled from different sources. I even wrote a book chapter about this topic and talk about it on conferences.
  • A lot of recurring tasks in my team are handled by paper tokens that get passed on when the job is done. Examples are the blog token (yes, it’s currently on my desk) for blog entries or the backup token as a reminder to bring in the remotely stored backup device and sync it. These tokens not only remind the next owner of his duty, but also act as a sign that you’ve accomplished your job, just like with task cards on the Scrum board.
  • If we need to work directly on a client server, we put on our “live server hat“ so that we are reminded to be extra careful (in german, there’s the idiom of “auf der hut sein”). But the hat is also a plain visible sign to everybody else to be a tad more silent and refrain from disturbing. Don’t talk to the hat! A lesser grade of “do not disturb” sign is the fully applied headphone.
  • Of course I built my own variation of my father’s Project Cockpit. It’s a great tracking device to never forget about any project, how sparse the actual activity might be.
  • And I solved the dishwasher case: The last action when clearing the dishes should be to already apply the next dishwasher tab. That way, whenever you open the dishwasher door, there are two possible states: if the tab case is empty, the dishes are clean (or somebody forgot to re-arm). If the tab is closed, you can be sure to have dirty dishes in the machine. The case gets re-opened during the next washing cycle.
  • An extra example might be the date of opening we write on the milk and juice cartons so you’ll know how long it has been open already.

All of these examples make information visible in place that would otherwise require you to collect it by sampling, measuring or asking around. Information radiators are typically big objects that typically do that job for you and present you the result. I’ve come to find that information radiators can be as little as a dishwasher tab in the right spot. The important aspect is to think about a way to make the information visible without much effort.

So if you repeatedly invest effort to gather all necessary data for an information, ask yourself: how could you automate or just formalize things so that you don’t have to gather the data, but have the information right before your eyes whenever you need it? It’s as simple as a little indicator on your mailbox that gets raised by the mailman or as complicated as a multi-colored LED in your faucet indicating the water temperature. The overarching principle is always to make information visible. It’s a very powerful motto to live by.

Quantities in C++ and User Defined Literals

Some weeks ago one of my colleagues wrote about the use and implementation of physical quantities in C#. If you are writing an application in the technical or scientific domain chances are high that you should adhere to his advice and use a suitable representation of physical quantities instead of plain primitive values. Good news is that you can easily port/implement quantities to modern C++ or use existing libraries like Boost.Units.

With C++11 you can go one step further adding the so called User-defined literals. This feature allows definition of suffices for integer, floating-point, character and string literals to produce objects of the desired (quantity) type. While there is nothing wrong with using the multiplication operator to produce quantity instances user-defined literals provide just a little bit more syntactic sugar:

// Your quantity classes...
class Angle;

// operators for user-defined literals
constexpr Angle operator "" _deg(long double deg)
{
    return deg * degrees;
}

constexpr Angle operator "" _deg(unsigned long long int deg)
{
    return deg * degrees;
}

constexpr Angle operator "" _rad(long double rad)
{
    return (rad * 180 / M_PI) * degrees;
}

// add more if needed

This allows you to write code like:

Angle rightAngle = 90_deg;
Angle halfCircle = 3.141_rad;
Angle fullCircle = 4 * 90_deg;

In many cases this looks a tad simpler and cleaner than using the multiplication operator in conjunction with a unit especially in more complex formulas. There are a few things about quantities and user-defined literals in C++ I find noteworthy:

  • These literals are only supported for the built-in literal types. If exact calculation and better than floating-point precision is needed, raw literals (instead of the explained cooked) and decimal libraries have to be used. For raw literals you have to parse the characters of the literal yourself.
  • User-defined literals need to be prefixed with _ to avoid namespace clashes with current and future standard library literals. There are for example some nice literals for durations in the <chrono>-date and time standard library.
  • If you implement your literal operators as constexpr they will be evaluated at compile time meaning slightly increased compile times and zero runtime overhead.

For some more in-depth discussion of user-defined literals have a look at the blog series from Andrzej Krzemieński.