Category: Author: Daniel Lindner

The inevitable emergence of domain events

If you’ve ever encountered code that cannot be modified anymore because business relies on a specific side effect of it, you’ve encountered an implicit domain event.

Even if you’ve read the original Domain Driven Design book by Eric Evans, you’ve probably still not heard about domain events (or DDD Domain Events), as he didn’t include them in the book. He talked about it a lot since then, for example in this talk in 2009 in the first 30 minutes.

Domain Events

In short, domain events are occurrences of “something that domain experts care about”. You should always be on the lookout for these events, because they are integral parts of the interface between the technical world and the domain world. In your source code, both worlds condense as the same things, so it isn’t easy (or downright impossible) to tell them apart. But if you are familiar with the concept of “pure fabrication”, you probably know that a single line of code can clearly belong to the technical fabric and still be relevant for the domain. Domain events are one possibility to separate the belongings again. But you have to listen to your domain experts, and they probably still don’t tell you the full story about what they care about.

Revealed by Refactoring

To underline my point, I want to tell you a story about a software project in a big organization. The software is already in production when my consulting job brings me into contact with the source code. We talked about a specific part of the code that screamed “pure fabrication” with just a few lines of domain code in between. Our goal was to refactor the code into two parts, one for the domain code and the other, bigger one for the technical part. In the technical part, some texts get logged into the logfile, like “item successfully written to the database” and “database connection closed”. It were clearly technical aspects of the code that got logged.

One of the texts had a spelling error in it and I reached out to correct it. A developer stopped me: “Don’t do that! They filter for that exact phrase.”. That surprised me. Nothing in the code indicated the relevance of that log statement, least of all the necessity of that typo. And I didn’t know who “they” were and that the logfiles got searched. So I asked a lot of questions and finally understood the situation:

Implicit Domain Events

The developers implemented the requirements of the domain experts as given in the specification documents. Nothing in there specified the exact text or even presence of logfile entries. But after the software was done and in production, the business side (including the domain experts) needed to know how many items were added to the system in a given period. And because they needed the information right away and couldn’t wait for the next development cycle, they contacted the operation department (that is separated from the development department) and got copies of the logfiles. They scanned the logfiles with some crude regular expression magic for the entries (like “item written to the database”) and got their result. The question was answered, the problem solved and the solution even worked a second time – and a third time, and so on. The one-time makeshift script was used permanently and repeatedly, in fact, it ran every hour and scanned for new items, because it became apparent that the business not only needed the statistics, but wanted to start a business process for each new item (like an editorial review of sorts) in a timely manner.

Pinned Code

Over the course of a few weeks, the purely technical logfile entry line in the source code got pinned and converted to a crucial domain requirement without any formal specification or even notification. Nothing in the source code hinted at the importance of this line or its typo. No test, no comment, no code structure. The line looked exactly the same as before, but suddenly played in another league. Every modification at this place or its surrounding code could hamper the business. Performing a well-intended refactoring could be seen as direct sabotage. The code was sacred, but in the unspoken kind. The code became a minefield.

Extracting the Domain Event

The whole hostage situation could be resolved by revealing the domain event and making it explicit. Let’s say that we model an “item added” domain event and post it in addition to the logfile entry. How we post it is up to the requirement or capabilities of the business department. An HTTP request to another service for every new item is a simple and viable solution. A line of text in a dedicated event log file would be another option. Even an e-mail sent to an human recipient could be discussed. Anything that separates the technical logfile from the business view on the system is better than forbidden code. After the separation, we can refactor the technical parts to our liking and still have tests in place that ensure that the domain event gets posted.

Domain Events are important

These domain events are important parts of your system, because they represent things (or actions) that the business cares about. Even if the business only remembers them after the fact, try to incorporate them in an explicit manner into your code. Not only will you be able to tell domain code and technical code apart easily, but you’ll also get this precious interface between business and tech. Make a list of all your domain events and you’ll know how your system is seen in the domain expert world. Everything else is more or less just details.

What story about implicit domain events comes to your mind? Tell us in a comment or write a blog entry about it. We want to hear from you!

How to approach big tasks

If you see a problem in your current project, don’t delay it – tackle it by going risk first.

In the heart of software development lies “the system”. The system is always complicated enough that you cannot fully grasp it and it is built by stacking parts on top of another that are just a tad too big to be called simple. The life of a software developer is an ongoing series of isolated projects that are at the threshold of his or her capabilities. We call these projects “epics”, “stories” or just “issues”. The sum of these projects is a system.

Don’t get me wrong – there a tons of issues that just require an hour, a cup of coffee and a few lines of code. This is the green zone of software development. You cannot possibly fail these issues. If you require twice the time, it’s still way before lunchtime. And even if you fail them, a colleague will have your back.
I’m talking about those issues that appear on your to-do list and behave like roadblocks. You dread them from far away and you know that this isn’t smooth sailing for an hour, this will be tough work for several days. This isn’t just an issue, it is an issue by itself for you. You are definitely unsure if you can make it.

Typical small project management

How do you approach such a project? It isn’t an issue anymore, as soon as you get emotionally involved, it becomes a project. Even if your emotion is just dread or fear, it is still involvement. Even if your management style is evasion, it is still project management. Sure, you can reassign a few of these icebergs, but they will always be there. You need to learn to navigate and to tackle them. Hitting an iceberg in the “frontal collision”-style isn’t a good idea.

On closer inspection, every project consists of numerous parts that you already know a solution for – typical one-hour issues – and just a few parts that you cannot estimate because you don’t know how to even start. Many developers in this situation take the route of least resistance and start with the known pieces. It’s obvious, it feels good (you are making good progress, after all!) and it defers failure into an uncertain future (aka next work week). Right now, the project is under control and on its way. We can report 80% finished because we’ve done all the known parts. How hard can the unknown parts be anyway? Until they strike hard and wreck your estimates with “unforeseen challenges” and “sudden hardships”. At least this is what you tell your manager.

Risk first!

My preferred way to approach those projects is to reveal the whole map, to estimate all parts before I delve into the details. I already know most of the easy parts, but what about the unknown and/or hard parts? I don’t know their solution so I cannot reliably estimate their size. So I sit down and try to extract the core problem that I don’t know how to solve yet. This is the thing that prohibits an estimate. This is the white area on my map. This is the “here be dragons” area. If I spend my resources doing all the work other than this, I will succeed until I stand on the border of this area and see the dragon. And I will not have sufficient resources left. My allies (like my manager and colleagues) will grow weary. I will have to fight my hardest battle in the most inconvenient setting.

My approach is to take the risk upfront. Tackle the core problem and fail. Get up and tackle it again. Fail once more. And again. If you succeed with your task, the war is won. Your project will still require work, but it’s the easy kind of work (“just work”). You can estimate the remaining tasks and even if you’ve overspent in your first battle, you reliably know how much more resources you will need.

Fail fast

And if you don’t succeed? Well, then you know it with the least damage done. Your project will enter crisis mode, but in a position when there is still time and resources left. This is the concept of “fail fast”. To be able to fail fast, you need choose the “risk first” approach of task selection. To tackle the risk first, you need to be able to quantify the “risk” of your upcoming work.

Assessing risk

There are whole books about risk assessment that are interesting and helpful, but as a starter, you only need to listen to your stomach. If your stomach tells you that you are unsure about a specific part of your project, put that part on the “risky” list. If you don’t have a reliable stomach, try to estimate the part’s size. Do the estimation game with your colleagues. Planning poker, for example, is a great tool to uncover uncertainty because the estimates will differ. Just remember: Risk isn’t correlated with size. Just because a part is big doesn’t mean it is risky, too. Your crucial part can maybe be developed in an hour or two, given an inspiration and a cup of coffee.

Failing late means you’re out of options. Failing fast means you’ve eliminated an option and moved on.

What Dwarf Fortress taught me about motivation in software development (part I)

Dwarf Fortress is a complex simulation of a magical world where you cannot give direct commands to the inhabitants, but have to resort to suggestions. And you need to keep your dwarves motivated, which taught me a lot about motivation. Here’s part one.

Dwarf Fortress is a peculiar game. It is free to play, developed by two guys that very much depend on donations. It looks like the last 30 years of advancement in computer graphics just didn’t happen, using raw ASCII graphics and an user interface that would have been horrible even in the 1980s.
In Dwarf Fortress, you try to build up a colony of dwarves without giving direct commands to them. You see your dwarves (represented by ASCII characters 0x01 and 0x02 in codepage 850) from above, in a three-dimensional environment consisting of blocks of material like stone or wood. The world is dynamic and simulated with strange, but comprehensible physics. You cannot grow a tree on a stone patch. You can pour water over dirt and get mud. Water flows downwards and will result in FUN if unsupervised. I’ve written fun in capital letters to differentiate the FUN of dwarf fortress from the fun of other games. It really is different.

You can try to imagine Dwarf Fortress being a weird crossover of Minecraft, the Sims and Rogue. Why the Sims? Because each dwarf isn’t just an action figure, but a complex individual with its own beliefs, value system, preferences and aversions. Each dwarf has its own skills and abilities and interacts with other dwarves in a social manner. Dwarf Fortress has a detailed simulation of nature and a detailed simulation of dwarves, down to their individual toes and teeths. It is very possible that one dwarf detests another dwarf so much that he pushes the victim over a cliff if nobody else is around. If the victim survives, you’ll have drama (aka FUN) in your fortress for years.

How can such a game give insights about motivation? Well, let me present you one more aspect of the game: the production system. Our dwarves need food to survive. They need clothes, tools and furniture. Most need some kind of art or decoration. One thing they all can agree to is that they need alcohol. All dwarves are addicted to alcohol so much, they will go crazy without it. And crazy dwarves result in immediate FUN.
But it is our task to govern the dwarves to actually produce these products in sufficient amounts. And this is where the complex production system hits us. In order to produce alcohol, you need to have fermentable plants, a brewery and an empty pot or barrel. To obtain the plants, you can suggest to your dwarves to raise them on farm plots (remember, you cannot give commands) or go out into the wilderness and gather them. Most dwarves really don’t like being outside and will get very unhappy if they are caught in the rain or cold. Yes, the weather is simulated in great detail, too. Water, for example, freezes in the winter.
So, to only have alcohol for your colony, you need one dwarf to prepare the field, one to plant the seeds, one to harvest, one to carry the harvest into the brewery, one to actually brew – and then you discover that you have no pots, so nothing gets stored. You also need to have one dwarf to gather wood or stone and one to produce a pot out of it. This can only be done at the Craftsdwarf’s workshop, so you need to have on built, too.
Did I tell you that dwarves have preferences? If you only grow wheat, you’ll get the finest dwarven beer, but all your wine gourmet dwarves will be unhappy (on a side note: Don’t let them fool you, they drink way too much wine to be called a “gourmet” anymore). You need to produce a variety of alcoholic drinks to give everybody their favorites.

Let’s review the production system one more time. Every dwarf wants to have clothing. Their own clothing! The game simulates clothing down to the left and right sock. Each sock has a quality and can show wear. To produce a sock, you need to obtain some specific plants, process them to obtain threads, weave the threads to cloth, dye the cloth to some color (the dye is the product of another production chain) and then tailor the sock in the Clothier’s shop. The tailor is probably a dwarf that enjoys clothesmaking and is very skilled doing it. He produces socks day in, day out. Some of them are of high quality, maybe even masterpieces (there is the very rare legendary sock that has in-game songs and poems written about it). Others are poor quality, mere trash from the beginning. The tailor knows about the quality of his products and gets a little amount of happiness for each well-done sock and a little amount of unhappiness if the sock was trash.

And here is the first insight about motivation: Motivation doesn’t only come from skill and preferences, but also from good results. If a dwarf is able to produce a good result regularly, he stays happy and motivated. Give him a task where he cannot succeed, no matter the effort, and he will get unhappy. Which will ultimately result in FUN, because the dwarf will try to compensate, maybe by going on a wine gourmet rampage. In the first fortress, a happy tailor produces quality socks for everybody. In the second fortress, a very drunk, unhappy tailor wastes your precious cloths while everybody else walks barefoot and gets unhappy if their toes hurt because of it.

So, motivation is not primarily about performing a task, but achieving a result. A result like crafting a product or, in our case as software developers, releasing a new version of the software with additional features.

Have your dwarves, I mean, your team, produce good results regularly. This is one thing that agile software development processes (and particularly SCRUM) get right: There is a public result at the end of each iteration – if the developers are skilled enough. With lesser skilled developers, you essentially signal them a failure every other week. They are probably trying very hard, but cannot come up with a good enough sock each sprint yet. Make the sprints longer or change your definition of a good result to something more attainable.

Without a clear result, something to hold onto, motivation will fade, too. That’s because uncertainty is stressful for most people. They will compensate for this stress by doing all kind of work, but not the necessary one. If you take a look at the work they do – it gives them measurable results. It may not contribute to the big result at the end of the cycle in any meaningful way, but it contributed to their motivation during the journey.

As a manager, you cannot give your developers direct orders. Or at least, you shouldn’t. If you use suggestions and a setup that facilitates self-organization of the team so that their preferences and needs align with or at least support the goal of the project, you’ll get a highly motivated team that doesn’t fear recurring result examinations, but looks forward to them – because they validate their efforts and give greater meaning to their work. Work that in itself already aligns with their own skills and preferences.

To sum it up: Dwarf Fortress taught me that direct orders are not the way to motivate teams. Creating an environment that anticipates public results often and making sure that the team is skilled enough to meet the expectations (or adjusting the expectations) are key factors to ongoing motivation.

Bending the Java syntax until it breaks

Did you ever bend a programming language until it breaks? This blog entry explores some rather unknown and silly regions of Java and connects them with upcoming language features.

Java is a peculiar programming language. It is used in lots of professional business cases and yet regarded as an easy language suitable for beginner studies. Java’s syntax in particular is criticized as bloated and overly strict and on the next blog praised as lenient and powerful. Well, lets have a look at a correct, runnable Java program that I like to show my students:

class $ {
	{
		System.out.println("hello world");
	}

	static {
		System.out.println("hello, too");
	}

	$() {
		http://www.softwareschneiderei.de
		while ($()) {
			break http;
		}
	}

	static boolean $() {
		return true;
	}

	public static void main(String[] _$) {
		System.out.println($.$());
	}
}

This Java code compiles, perhaps with some warnings about unlucky naming and can be run just fine. You might want to guess what its output to the console is. Notice that the code is quiet simple. No shenanigans with Java’s generics or the dark corners of lambda expressions and method handles. In fact, this code compiles and runs correctly since more than 20 years.

Lets dissect the code into its pieces. To really know what’s going on, you need to look into Java’s Language Specification, a magnificent compendium about everything that can be known about Java on the syntax level. If you want to dive even deeper, the Java Virtual Machine Specification might be your cup of tea. But be warned! Nobody on this planet understands everything in it completely. Even the much easier Java Language Specification contains chapters of pure magic, according to Joshua Bloch (you might want to watch the whole presentation, the statement in question is around the 6 minute mark). And in the code example above, we’ve used some of the magic, even if they are beginner level tricks.

What about the money?

The first glaring anomaly in the code is the strange names that are dollar signs or underscores. These are valid names, according to chapter 3.8 about Identifiers. And we’ve done great by choosing them. Let me quote the relevant sentence from this chapter:

“The “Java letters” include uppercase and lowercase ASCII Latin letters A-Z (\u0041-\u005a), and a-z (\u0061-\u007a), and, for historical reasons, the ASCII dollar sign ($, or \u0024) and underscore (_, or \u005f). The dollar sign should be used […]”

Oh, and by the way: Java identifiers are of unlimited length. You could go and write valid Java code that never terminates. We’ve gone the other way and made our names as short as possible – one character. Since identifiers are used as class names, method names, variable names and (implicitly) constructor names, we can name them all alike.

The variable name of the arguments in the main method used to be just an underscore, but somebody at Oracle changed this section of the Language Specification and added the following sentence:

“The underscore may be used in identifiers formed of two or more characters, but it cannot be used as a one-character identifier due to being a keyword.”

This change happened in Java 9. You can rename the variable “_$” to just “_” in Java 8 and it will work (with a warning).

URLs as first-class citizens?

The next thing that probably caught your eye is the URL in the first line of the constructor. It just stands there. And as I told you, the code compiles. This line is actually a combination of two things: A labeled statement and a comment. You already know about end-of-line comments that are started with a double slash. The rather unknown thing is the labeled statement before it, ending with a colon. This is one of the darker regions of the Language Specification, because it essentially introduces a poor man’s goto statement. And they knew it, because they explicitly talk about it:

“Unlike C and C++, the Java programming language has no goto statement; identifier statement labels are used with break…”

And this explains the weird line in the while loop: “break http” doesn’t command Java to do harm to the computer’s Internet connection, but to leave and complete the labeled statement, in our case the while loop. This spares us from the looming infinite loop, but raises another question: What names are allowed as labels? You’ve guessed it, it’s a Java identifier. We could have named our label “$:” instead of “http:” and chuckled about the line “break $”.

So, Java has a goto statement, but it isn’t called as such and it’s crippled to the point of being useless in practice. In my 20+ years of Java programming, I’ve seen it used just once in the wild.

What about it all?

This example of Java code serves me as a reminder that a programming language is what we make out of it. Our Java programs could easily all look like this if we wanted to. It’s not Java’s merit that our code is readable. And it’s not Java’s fault that we write bloated code. Both are results of our choices as programmers, not an inevitableness from the language we program in.

I sometimes venture to the darker regions of programming languages to see what the language could look and feel like if somebody makes the wrong decisions. This code example is from one of those little journeys several years ago. And it proved its worth once again when I tried to compile it with Java 9. Remember the addition in the Language Specification that made the single underscore a keyword? It wasn’t random. Java’s authors want to improve the lambda expressions in Project Amber, specifically in the JEP 302 (Lambda Leftovers). This JDK Enhancement Proposal (JEP) was planned for Java 10, is still not included in Java 11 and has no clear release date yet. My code gave me the motivation to dig into the topic and made me watch presentations like the one from Brian Goetz at Devoxx 2017 that’s really interesting and a bit unsettling.

Bending things until they break is one way to learn about their limits. What are your ways to learn about programming languages? Do you always stay in the middle lane? Leave a comment on your journeys.

Developer Experience means nothing

Always choose User Experience over Developer Experience.

You’ve probably already heard about the concept of User Experience (UX) and the matching virtue of User Experience Design. If not, you might want to go check it out. I would suggest you start with the excellent book “The Design of Everyday Things” from Donald Norman. It has nothing to do with software development, so you won’t mix up User Experience Design with Graphical User Interface Design or even Graphics Design.

In short, User Experience is what an user of your software will experience while using your software. If your software makes them smile, you’re some kind of UX god. If your software makes them curse repeatedly, you’re probably not. You can try to improve the experiences of your users by making changes to the software. This is called User Experience Design and is an important part of software development. Most developers know nothing about User Experience Design.

What developers naturally know about is Developer Experience (DX), a concept that isn’t really defined in the literature. I made it up to explain my point to you. Every software developer has a favorite programming language and IDE. Remember, the source codes of the same problem in different programming languages will ultimately yield the same machine code. The machine is totally agnostic about your preferences for a certain programming language. Your choice of a programming language for a certain problem is more about your Developer Experience than anything else. Developer Experience is everything you feel and think during software development. A bad Developer Experience lets you swear about the tools, the code and everything and everybody else around you. A good Developer Experience gives you a sense of accomplishment and safety while coding and makes your work not harder than necessary.

Because you can’t read elsewhere about the concept of Developer Experience, I want to give two more examples and show how it affects the User Experience. The first example is a big, long-lasting software system that is in production and still developed further. If you have a software system in production, everything requires an additional thought about the matching upgrade strategy. You can’t just modify the database structure, you need to provide migrations. You can’t just add a configuration entry, you need to make it optional or consider the least harmful default value. In the project of our example, the developers had three possible approaches to a new configuration entry:

  • They could add it to the code but leave the configuration files unchanged. This required the code to handle the “absent from configuration” case in a useful manner. It required the developer to make effort in the code. The user would not know about the new configuration entry if not stated in some external documentation.
  • They could add it to the code and write a configuration migration script that added it to the existing configuration files on the customer’s installation. The code could now expect the entry to be present, but the developer had to write and test the migration script code. The user would see the new configuration entry with the default value.
  • They could introduce a new configuration file to the system and place the new configuration entry in it. The code could expect the entry to be present, because new configuration files were added to an existing installation during the upgrade process. The user would see the new configuration file and the new configuration entry with the default value.

You can probably guess which of the three options got used so excessively that users complained about the configuration being all over the place, in a myriad of little one-liner configuration files with ominous names. The developers chose the best option for them and, in short view, for the users. But on the long run, the User Experience declined.

The second example is from a computer game in the mid 2000s. It was a massive multiplayer online shooter with a decent implementation ahead of its time. But one thing was still from the last century: After each update of the game, the key bindings were reset to the default. And every other aspect of local modification to the settings, too. After each update, you had to configure your video, audio, controls and everything else like your in-game equipment again and again. The game didn’t offer any means to copy or reload your settings. It was up to you to maintain a recent backup of the settings files just in case. And if the file format was changed, you needed to combine their changes with yours. WinMerge is a decent tool for that task. But the problem is clear: The game developers couldn’t be bothered to think about how their upgrade strategy would affect their users. They ignored the problem and let the users figure it out. They chose better Developer Experience, free from complexities like user-side modifications over good User Experience like a game that can be upgraded without drawbacks for the gamers.

Sadly, this is an usual formula in software development: Developer Experience is more important than User Experience. Just look at it from an utilitarian point of view: If you burden thousands or even millions of users with just an easy one-minute task that you could fix during development, you have a budget of two workdays up to 10 person-years. Do the math yourself: One million users, each spending one minute of work on your software, equal to over 2000 workdays lost on a thing you could probably fix for everybody in an hour or two.

And this brings me to my central statement: Developer Experience, as opposed to User Experience, means nothing. It’s just not important. At least for all the users. It is important to the small group of developers and should not be forgotten. But never should a decision lead to better Developer Experience on the expense of User Experience. It’s a small inconvenience for us developers to think about smooth upgrades, meaningful and consistent control element titles or an easy installation. It’s a whole new game for our users.

Always choose User Experience over Developer Experience.

Call to action: If you have another good example of developers being lazy on the expense of their users, please share it as a comment.

Call to action 2: The initial picture is linked from the excellent website badhtml.com. If you ever feel like an imposter for your latest design, go and visit this website.

The day the machines took gaming away

Just a few years ago, bots in computer games were a liability for the team. Now they are the preferred teammate. What happened?

August 5th, 2018 was a noteworthy day in the history of mankind. It was a Sunday and had Europe aching in unusual heat and drought. But more important, it was the day when the machines gently asserted their dominance in the field of gaming. It was the day when our most skilled players lost a tournament of Dota 2 against a bunch of self-learned bots.

“Bot” used to be a vilification

How did we end up in this situation? Let’s look back at what “bot” used to mean in gaming. Twenty years ago, we were thrilled about games like Starcraft where you control plenty of aggressive, but otherwise dumb units in a battle against another player that also controls plenty of those units. The resulting brawls were bloody, chaotic and ultimately overwhelming with their number of necessary tasks (so-called micromanagement) and the amount of information that needed to be processed at once to react to the opponent. In a human versus human (or pvp for player versus player) game, those battles were usually constrained to a certain area and executed with a certain laissez-faire attitude. Only the best players could stage two or more geographically independent attacks and control every unit to their full potential. We admired those players like astronauts or rockstars.

If you could not play against another human, you would start a game against a bot. A bot usually had four things that worked in their advantage and a lot of things stacked agaist them. In their favor, they had minimal delay in their reactions, ultimate precision in their commands and full information about everything on the gamefield. And more often than not, they received more game resources and other invisible cheats because they didn’t stand a chance against even moderately skilled humans otherwise. Often, their game was defined by a fixed algorithm that couldn’t adapt to human strategy and situational specifics. A very simple war of attrition was enough to defeat them if their resource supply wasn’t unlimited. They didn’t learn from their experience and didn’t cooperate, not with other bots or allied humans. These early bots relied on numbers and reaction speed to overwhelm their human counterparts. They played against our natural biological restrictions because the programmers that taught them knew about these restrictions very well.

Barely tolerated fill-ins

Those bots were so dumb and one-dimensional that playing with them against other opponents was even more of a challenge because you always had to protect them from running in the most obvious traps. They weren’t allies, they were a liability that dictated a certain game style. Everybody preferred human allies even if they made mistakes and reacted slower.

The turning point

Then, a magical thing happened. An artificial intelligence had trained itself the rules of Go, a rather simple game with only two players taking turns on a rather static gamefield. This AI played Go against itself so excessively, it mastered the game on a level that even experts could not grasp easily. In the first half of the year 2017, the machines took the game Go out of our hands and continued to play against themselves. It got so bad that an AI that was named AlphaGo Zero taught itself Go from scratch in three days and outclassed the original bot that outclassed mankind. And it seemed to play more like a human than the other bots.

So we got from dumb bots that were inferior stand-ins for real humans to overly powerful bots that even make it seem as if humans are playing in just a few years.

The present days

It should be no surprise to you anymore that on that first Sunday of August 2018, a group of bots beat our best players in Dota 2. There are a few noteworty differences to the victories in Go:

  • Dota 2 is a game where five players battle against five other players, not one versus one. It wasn’t one bot playing five characters, it was five bots cooperating with only in-game communication against humans cooperating with a speech side-channel.
  • Go is an open map game. Bot opponents see every detail of the gamefield and have the same level of information. In Dota 2, your line of sight is actually pretty limited. The bots did not see the whole gamefield and needed to reconnoiter just like their human opponents.
  • In Go, the gamefield is static if nobody changes it. In Dota 2, there a lots of units moving independently on the gamefield all the time. This fluidity of the scenario requires a lot of intuition from human players and bots alike.
  • The rules of Go are simple, but the game turns out to be complex. The rules of Dota 2 are very complex, but the game refuses to be simple, because the possibilities to combine all the special cases are endless.
  • Go is mostly about logic, while Dota 2 has an added timing aspect. Your perfect next move is only effective in a certain time window, after that, you should reconsider it.

Just a year after the machines took logic games from us (go and read about AlphaZero if you want to be depressed how fast they evolve), they have their foot in the real-time strategy sector, too. Within a few years, there is probably no computer game left without a machine player at the top of the ladder. Turns out the machines are better at leisure activities, too.

The future?

But there is a strange side-note to the story. The Go players reported that at first, the bots played like aliens. Later versions (the purely self-learned ones) had a more human-like style. In Dota 2, if you mix bots with humans in one team, the humans actually prefer the cooperation with the bots. It seems that bots could be the preferred opponent and teammate of the future. And then, it’s no longer a game of humans with a few bots as fill-in, but a game between machines, slowed down so that humans can participate and do their part – as a tolerated inferior fill-in.

The Pure Fabrication Tax

Last week, I attended the Maexle event of the C++ user group in Karlsruhe. The Maexle event is basically a programming contest where your program plays the Mia dice game against other programs. You have to implement a simple network protocol to join games, announce rolls and call bluffs. Your program earns points for every game it has participated and not lost. So there is a strong emphasis on starting early and staying in the game, even if your program doesn’t perform the best.

Since it was an event of the C++ user group, the programming language to be used was C++. I’m certainly no C++ hero and knew I couldn’t compete, so I joined the fun with an espionage role and programmed an observing bot that doesn’t play, but gathers data on the players. I chose Java for the task. My observer was online after two minutes, the first real player joined the server after 20 minutes. It turned out to be written in Python. The first real C++ bot was online after 35 minutes, the last one played its first round after two hours.

I listened closely to the problems the teams around me tackled and noticed something strange: Nobody talked about the actual game (Maexle/Mia). Every task was a technical one. Let’s talk about why that’s a problem.

Three Definitions

Before I dive into the subject, I want to define some terms that I’ll use to help you understand my point. It’s entirely possible to look at the story above and see a bunch of engineers having fun with some engineering tasks.

  • First, I value the economics of my customer. In this case, the customer is a lonely server on the LAN that wants to host some games of Maexle for bots. Like, lots of games. Thousands of games. The customer gives points for early market entry, so time to market is an economic factor (or a key performance indicator, some might say). You can roughly say that being online early means you can make bucks longer. The second key performance indicator is uptime. You want to stay in the game as long as you don’t lose all the time. There are some more KPI, but the two I’ve listed should have a major impact in your programming approach – if you value the economics.
  • Second, I don’t care about tools. A programming language is a tool. A compiler is a tool. Your IDE or text editor is a tool. Use your preferred tooling as long as it suits your needs. That means explicitely as long as it doesn’t actively work against your other values like the customer’s economics. This blog post is definitely not about Java or Python being “better” or “better suited” than C++. They aren’t. The first two bots (observer and player) were programmed by participants that had prior experience with the event. It wasn’t the tool that made them fast, it was the absence of rookie errors in the domain and its technical structure.
  • Third, I will explain my point with the concept of “pure fabrication“. Pure fabrication is everything that is not specified by the customer, but necessary to fulfill the specification. It’s the code you write because your customer wants to persist some data. He never ordered you to write SQL statements or “open a connection to the database”, maybe he didn’t even know what a database was. Your customer wanted the data stored somehow. The code that enables you to actually program the storage is “pure fabrication” in terms of the domain. Think of it as a scaffolding holding your domain code in place. If you hire a painter to color your house, he will scaffold the walls to reach every spot with ease. You didn’t hire him to set those structures up, they are just necessary for the task. The difference to most of our code is that the painter removes the scaffolding afterwards.

Pure Fabrication vs. Domain

So, if I would have been a customer on the Maexle event, paying for a competitive Maexle bot, I would be very surprised about the actual construction process. Up to two hours into a three-hours event, my programmer would solve apparently hard and important problems, but not my problems. In fact, I wouldn’t even understand the relation between the attempted problems and my required solution. And I would have to have blind faith for more than half my money that something useable will come out of this.

This is the effect of too much pure fabrication in the programming approach. I’m all for solving hard programming problems, but I’m not interested in solving them over and over again. After some iterations, they become solved problems or, essentially, tools. And I don’t care about tools as long as they get their job done. If your domain problem requires a better tool, then we can put the programming problem on our todo list again. Otherwise, we are not valueing our customer’s economics, we are showing off to our peers.

If you program a simple game of Maexle with a heavy emphasis on time to market and even after the initial ramp up aren’t able to reason about your code using language from the domain (like game, dice, roll, bluff, double and, of course, mia), you are staying in pure fabrication land. That’s the level of programming where it matters if you used an integer or freed that memory. That’s when you pay the Pure Fabrication Tax to the fullest. Because your code now does something valueable in the domain, but the distance between your customer’s language and your code’s language is an hindrance. And this distance will demand its tax with every new feature, every change request and every bug.

Bugs are another area where the distance is measureable. If you can’t explain your bugs to the customer, you’ve made them in the pure fabrication part of your code. If you can never explain your bugs, your domain code is hidden between lines and lines of source code with lots of special characters, brackets and magic numbers. Just imagine your hired painter tries to tell you why your house is now pink instead of white or yellow: “It was a small mishap in the way we constructed the scaffolding, we used an E5 steel beam instead of a rail clamp and forgot to perform a hammer check on it”. The last part is totally made up, but I’m sure that’s how we sound for non-programmers.

Exemptions from the Tax?

What solution would I suggest? I don’t think there is a definite solution to the problem. You can’t go full Domain Driven Design on a three-hour Maexle event. By the time you’ve built your fancy Domain Specific Language to write code with the customer besides you, everybody else has gathered their game points and gone home. If you switch to a language that has a string tokenizer in its standard library, you can speed up your programming, but maybe just produce a bigger heap of slightly less low-leveled pure fabrication code.

I don’t want to advocate a solution. My attempt is to highlight the problem: The Pure Fabrication Tax. Given the right (or wrong) amount of extrinsic (or intrinsic) motivation, we are able to produce a mess in just a few hours without really connecting to the domain we produce the mess for. If we didn’t program a Maexle bot that night, but a poker bot or a chat bot, most if not all of the problems and bugs would have been the same. This is not a domain-specific problem. It’s our problem. We probably just like to pay the tax.

What are your thoughts on the Pure Fabrication Tax? Can you see it? Do you have an idea for a solution approach? Leave your comment below!

Disclaimer

And to counter everybody who thinks I’m just bashing the other participants on the event: I was the first one online on the server, with a task that requires virtually no effort and doesn’t even participate directly in the competition, with tools that solved nearly all my pure fabrication problems and still managed to create a program that contained less than five domain terms and was useless for its intended purpose. I said I value the economics of my customer (even if there was none), so I know that I failed hardest on the event. And I had prior knowledge. There was just nobody to compare my mess to.

Putting toilet books into practice

A toilet book is a book with a special structure that facilitates short-span context-free reading. In this blog post, I’ll recommend some good toilet books for IT topics.

I’m reading a lot of books and based on my profession and interests, my list includes many software development and IT books. I want to share how I manage my reading and give some recommendations for a special type of book that I call “toilet book”.

Three books at once

The human mind is a peculiar thing. You’ve probably experienced the effect of getting up to perform some minor task in an other room only to arrive there with no recollection about what you wanted to do. Between the thoughts of “ok, let’s do this now!” and “why did I go here?”, just a few seconds have passed, but another aspect has changed dramatically: your geographic position. As a side note: If you don’t know what I’m talking about, consider yourself lucky. Our memory is often bound to the geographic position and changes when we move. If you want to remember what your forgotten task was, try returning to your original location. You’ll often see me walking around the same way twice within seconds. That’s when I have to rewind my location-based memory.

A particular use case where I leverage my location-based memory is when I read books. I often read three books at once, but strictly separated by location:

  • The first book is the “leisure book“: I will only read it at comfortable locations like the couch, in the sun on the balcony or in the bathtub. This book is often fiction or has at least nothing to do with IT.
  • The second book is the “travel book“: You’ll seldom see me travelling without a book and just a few minutes of tram are sufficient to read some pages. This book is often IT-based, because I read it on my commute to and from work and sometimes in my lunch break.
  • The third book is the “toilet book“: You’ll never see me reading this book, because it is stored besides my toilet and is exclusively read there. Books that are suitable for this task often have a special structure that aligns with the circumstances. More on this in a moment.

By having a clear separation by location for the three books, I’m able to keep their content separated and switch from one reading context to the next without effort. It happens naturally if I refrain from reading my travel book at home or taking my leisure book on the train.

The structure of a toilet book

A good toilet book has a special structure that accommodates for the special timing of a toilet visit. If you spend two minutes on the toilet, the book should have chapters or at least paragraphs that can be read in two minute intervals. Ideally, the book is specifically designed to contain short chapters on different topics that have no strong over-arching story. A typical example of a good non-IT toilet book are comic books like Calvin & Hobbes, The Peanuts or any other comic series that has small self-contained comic strips. You read one or two strips, are amused and interrupt again without having to memorize a complex context. Good toilet books allow for short, context-free reading sessions.

A collection of worthwhile toilet books

Over the years, I’ve read some toilet books with IT and software development topics and want to share my list of books that I enjoyed reading in this fashion:

In short, for me, calling a book a “toilet book” is not a derogatory taunt, but a neutral description that this book is structured in a way to support repeated short-time reading sessions. For me, these books are a good choice for a tertiary reading track.

A call for proposals

Right now, my reading list of good IT toilet books is rather short. If you happen to know a book that fits my description, I would be thankful for a hint in the comments. Thank you!

Ten books that shaped me as a software developer – Part II (Books 5 to 9)

I was asked what books shaped me as a software developer the most. Here is the second part of my answer.

In the first part of my answer (books 0 to 4), I highlighted five books that influenced my career as a software developer. The list is not ordered, so the next five books aren’t inferior or better than the first ones. Every book on the complete list made a significant contribution to my knowledge and work ethic.

Clean Code

If we were to choose the holy book of software development, we probably couldn’t agree on one or even a dozen titles. And that is a good thing, because there is no one true way of software development. Clean Code by Robert C. Martin would maybe show up in the late contenders. But if we were to choose the most preachy book of software development, well, I have a favorite. This book is so loud that you cannot ignore it. And it is so opinionated that you’re either nodding your head like a heavy metal fan or writhing in averseness. That’s a good thing, too. Because it forces you to think. Your immediate emotional answer needs support by rational arguments and this book will provide you with ample opportunity to gather arguments for your consent or rejection. What this book probably won’t do is leave you unaffected. When it came out in 2008, it was an instant classic. You could spice up any gathering of software developers by making a statement about this book, be it pro or contra. And even today, ten years later, I would say that even if the loudness is deafening, the clarity of the messages makes this book a worthwhile read for every software developer. My grief with it is foremost that for a book called “Clean Code”, some examples of actual code are quite dirty or even plain wrong. Read it with an active mind and it will be a cornerstone of your professional career. But be careful, it seems that currently printed instances have physical quality problems.

Growing Object-Oriented Software, Guided by Tests

Ever since Extreme Programming hit the (european) scene in 1999, I was curious about Test Driven Development (TDD). I tried automated testing and unit tests whenever I could, read books and later watched videos about the topic. But I never grokked it. It just didn’t work for me and I didn’t even know why. My most feared trap was the one-two-everything syndrome, where you write two simple tests and then have to implement the whole algorithm to fulfill the third test. It was always the third test that broke my rhythm. I tried to exchange experience with TDD practitioners, but their own examples were mostly trivial and my examples always led nowhere (for reference: Try a simple Game of Life in TDD style). I felt dumb and inadequate. When Robert C. Martin (the author of Clean Code) told the developer world that you are either “TDD or not professional” (read the original from 2007 behind this paywall or the reprise from 2014 here or, even better, watch this discussion from 2012), that didn’t make me feel exactly great, too. But imagine my surprise when I started to read a book by two authors I hadn’t heard much of before with a title that reveals its intent only after a comma: “Growing Object-Oriented Software, Guided by Tests” (henceforth called the GOOS book). The book spoke clearly to me. Every step was actionable, even more so, the book acted it out right before my eyes. It was as if Steve Freeman and Nat Pryce, the two authors, were sitting left and right at my table and discussing actual code with me. It didn’t help that I read this book during a summer beach holiday. The beach and even the sun didn’t see much of me that year. I was busy learning about Acceptance Test Driven Development (ATDD), ports and adapters and all the other great content in this book. And the best thing was: it wasn’t theoretical, the examples in the books could be followed one a line-to-line basis. My experience with this book was unique and still is. It’s the best book about actual software development that I’ve read. You might enjoy it, too.

Domain Driven Design

Some years after the GOOS experience, another summer beach holiday was due and as usual, I included a software development book in my luggage. “Domain Driven Design” by Eric Evans came out in 2003 and was praised by some and ignored by most, including me. It took me ten years to finally read it and when I did, it hit me hard. Since my early days as a programmer, I tried to build a meaningful data model with actual types for each program I developed. But it occurred to me that I did it half-heartedly all the time. It shouldn’t stop at a data model, it should be a complete domain model. And for that to work, you need to grok the domain. I review a lot of my code before that insight and always find it funny how I invested effort in my models but more often than not stayed in the technical realm. I cannot say that my programming has changed much from the book, as most concepts meandered through the community since 2003 and were picked up by me mostly under different names. But my software development approach has changed dramatically. I don’t start my thinking from the technical side anymore. And that helps with “business alignment” and all the other magic words that finally have real tangible benefit. And I can now pinpoint when that alignment loosens and employ counter-measures instead of ending up in a special case hell. The best thing was that this book doesn’t require a laptop so I got to sit on the beach that summer with the book in my hands and my head in the clouds. It might be old, but it’s still gold.

Clean Architecture

I anxiously waited for this book to be printed. Not because I pre-ordered, but because I held talks, workshops and lectures about the topic before the book was available. And I wanted to make sure that I’m not telling nonsense. But Robert C. Martin took his time and delayed the deadline month after month. Then, nearly a year later, the book reached the stores in late 2017. So I would have to wait for my winter holiday to read it. I couldn’t wait and began right away. The book is a slow burner and feels like a long introduction. By the time the central proposition is revealed (and yes, it reads like good unagitated spy thriller at times), you’ve probably already figured it out yourself. And that’s a good thing in my mind, because it feels as if it was your idea and Uncle Bob is just there to nod and congratulate you for your intellect. This book is so many times less preachy than “Clean Code”. If we compare spy thriller literature, this is a John le Carré while Clean Code would be an Ian Fleming (James Bond). “Clean Architecture” is not about programming, it talks about software architecture, a topic that I missed greatly in my early developer years. I liked this book so much I even wrote a full review about it.

The Inmates Are Running the Asylum

All the other books talk about different aspects of programming, software development or related technical topics. But what about a book that raises a simple question: “Why is IT technology so complicated?”. And gives the answer: “Because we want it this way.”. That’s actually true. In a world without most of the restrictions of the physical world, we were unable to build solutions that actually helped us and came up with machines and software that overwhelmed most people. It needed a whole new generation of “digital natives” until concepts like internal operation modes (e.g. insert vs. overwrite) were intuitively understood. Not because they became simpler, we were just used to the complexity. Alan Cooper described the problem and gave at least hints for solutions in 1999, nearly 20 years ago. That’s the timespan of a generation. This book made me think hard about the status quo I silently had accepted with technology. It just was like it was, what else could there be? If I reveal a tiny bit of different approaches I can think of now, I’m often confronted with incomprehension. Not because I’m particularly clever and everyone else is dumb, but because there seems to be no problem if you’ve grown accustomed to it. If you want to see some of the pain other (older) people feel when interacting with technology and software, read this book. It is an eye-opener to common problems no software developer ever had. It is the first step into the world of UX (user experience), where it’s not as important if the developer feels alright but if the user feels at least adequate. It might be a classic and feel a bit outdated and weak on the solution side, but to understand the problem properly is the first step to appreciate possible answers. And Alan Cooper didn’t stop there. Read his ongoing series “About Face” (current version: 4.0) for lots of solution ideas.

Epilogue

And that’s it. These are the ten books I recommend everybody who wants to read good books about software development. And just a few days ago, another student asked me if I’m seriously recommending twenty years old books about topics that change fundamentally every five years. I am serious. If you read just one book of this list and judge afterwards, you’ll see what I mean when I say that there are timeless topics even in an ever-changing field like software development. Maybe you want to begin with “Refactoring” and compare it to the second edition (Java vs. JavaScript). The underlying concepts stay the same, no matter the syntax.
I hope you enjoyed this list. And I hope the student who originally asked the question got his answer. Are there books you want to recommend? Drop a comment below or blog about them! The average software developer reads less than one book per year. Maybe our insistence can change that a bit.

Ten books that shaped me as a software developer – Part I (Books 0 to 4)

I was asked what books shaped me as a software developer the most. Here is my answer, or the first part of it.

Last week, I’ve done a question and answers event with students when the question came up what the most influential books were that I have read as a software developer. I couldn’t answer the question right away but promised to compile the list with short descriptions of the book’s influence. And here it is – my list of books that left a big mark in my day-to-day work. Others have done the list of books thing before me, and most lists contain the same books over and over again. I take it as an indicator that my list isn’t too far off.

Prologue

Before I start the list, I want to say a few things. The list isn’t ordered or ranked. I describe the effects of each book from my current standpoint, sometimes 20 years after the fact. I read a lot more good, interesting and inspiring books in the last 20 years and they all added to my work personality. But with all the books on my list, I felt enlightened and vibrant with new ideas. They didn’t just inspire me, they elevated my thinking. And because of this criteria of immediate improvement, one book is missing from the list. It’s the first “serious” software development book I’ve ever read in 1998: “Design Patterns“. The book was just too much for me (and my study group peers) to handle such early in our careers. We were in our first year of study and had a lot of other battles to fight. I crossed it from my reading list and moved on. Years later, I re-read it and saw so much insight I plainly missed the first time, but gathered elsewhere since. If you want to read this classic, don’t hesitate! If you “only” want to know about design patterns, there’s a better book for that: “Head First Design Patterns“.

The Pragmatic Programmer

https://images-na.ssl-images-amazon.com/images/I/41BKx1AxQWL._SX396_BO1,204,203,200_.jpgMore by chance, my co-founder stumbled upon “The Pragmatic Programmer” in 1999 and devoured it. Then he gave the book to me and it shattered me to my core. I thought I was a decent software developer and here are Dave Thomas and Andy Hunt and talk about things I didn’t even knew existed. A healthy dose of Dunning-Kruger effect is crucial in everybody’s upbringing, but this book ended my overestimation once and for all and gave my studies a focus and direction I wouldn’t have thought to be possible before. I own my whole career to this book, at least in terms of work ethics. I cannot fathom how my professional life would have played out otherwise.

Refactoring

https://images-na.ssl-images-amazon.com/images/I/51K-M5hR8qL._SX392_BO1,204,203,200_.jpg

Also in 1999, Martin Fowler wrote his instant classic “Refactoring“. We bought this book at the first chance we got and raced through the pages. I was a Java developer back then and with most of the examples being in Java, the book needed no explanation nor translation. It was directly applicable knowledge that gave me years of experience virtually for free. This book is a must-read even 20 years later, and has just recently had the second edition announced, this time with code examples in JavaScript. I thought it was a joke first, but I guess it makes sense.

Working Effectively With Legacy Code

https://images-na.ssl-images-amazon.com/images/I/51EgCCLOWxL._SX376_BO1,204,203,200_.jpgIn 2004, Michael Feathers wrote a book that contains his 20+ years of experience with software development and named it “Working Effectively With Legacy Code“. Well, joke’s on you – I don’t write legacy code, my code is perfect. That wasn’t my attitude since 1999 (see list entry #1) and I took this book everywhere. It’s a heavy one, but I read it in the tram, right before the movie starts in cinema, during breakfast, lunch and dinner and virtually any other circumstance. I realized that reading this book will gain me experience a lot faster than actually writing code, so I just stopped for a few weeks. This book answered a lot of mysteries in the form of “is there really no better way to do this?” for me. And it introduced the concept of code seams for me that permeates my work ever since. I can clearly remember the day when I looked at my existing code again and saw the seams for the first time. It was truly eye-opening for me.

Analysis Patterns

https://images-na.ssl-images-amazon.com/images/I/41uNHkTq8NL._SX378_BO1,204,203,200_.jpgMartin Fowler was a very productive author in the late nineties. I’ve read most of his books from this period, if maybe with a few years delay. “Analysis Patterns” from 1996 arrived in my bookshelf in the early 2000’s and was my wake-up call to seeing models instead of actualities. I’ve given this book to many peers, but haven’t received the reactions that I had with this book: Being taught a language (with a graphical notation) that can express actual problems in terms of an overarching solution. Since then, I’ve seen the same solutions applied in many different forms, with many different names and a lot of different special requirements. But they all derive from the same model. This effect was promised by the “Design Patterns” book, but for me, delivered by “Analysis Patterns”. Even Martin Fowler admits that the book is showing its age, but for me, its timeless.

Peopleware

https://images-na.ssl-images-amazon.com/images/I/51MlUgcSICL._SX331_BO1,204,203,200_.jpgSince the late 80’s, Tom DeMarco and Timothy Lister wrote one book after the other. Each book describes a common business-oriented problem and at least one working solution for it. And yet, the very same problems still persist in the business world. It’s as if nobody reads books. “Peopleware” was written in 1987, 30 years ago, and discovered by me and my peers in the late 1990’s. We talked about this book a lot, as it described a (business) world where we didn’t want to work in. We wanted to do better. In a way, this book was a spark to found our own company and don’t repeat the mistakes that seemed to be prevalent in our industry. If you’ve ever shaken your head about “the management”, do yourself a favor and read this book. It will pinpoint the precise problem you’ve felt and give you the words to describe it. And if you’ve read “Peopleware”, liked it and want more, there is good news: There is a whole series waiting for you (not just Vienna).

Epilogue to Part I

These are the first five books from my list, with the last entry being more of a catch-all for a whole series. Remember that this isn’t a generic “go and read these books if you want to call yourself a professional software developer” list. I’m not gatekeeping and it would be useless to even try to do so. These books helped me further my career in the last 20 years, they won’t necessarily help you for the next 20 years. Good books are published every year, you just have to read them.

I’m looking forward to share the second part of my list in the next blog post of this series. Stay tuned!