Books and talks that shaped my mind as a developer

Over the years I’ve read many books and watched many talks but a few stand out (at least for me) that influenced me in my development career.

The inmates are running the asylum by Alan Cooper
This book opened my eyes that I approached software development completely from the wrong standpoint: the software should serve the user not vice versa.

Design Patterns by Erich Gamma et al
Oh others use the same patterns as me and what? you can even talk about it without explaining every detail…

Refactoring by Martin Fowler
This book taught me that you can change the structure and the design of the software without changing its function. Cool.

Inventing on Principle by Bret Victor
Seeing a new way of interacting with your software in development blew my mind. Think WYSIWYG on steroids.

Getting real by 37signals
Getting to the core of what is essential and what really needs to be done in software/product development is laid out here so clear and stripped down that it struck me.

Information visualization by Edward Tufte
Another book which reduces its topic (this time: presenting information) to the core and by this identifying so much unnecessary practice that it hurts.

Start with why by Simon Sinek
Purpose. Why do you develop software? Why do I arrange an UI or the architecture of an application? This is what design is about.

Only openings by Frank Chimero
Do I try to eliminate failures and therefore options or do I leave the possibility to the user to choose…

Web design is 95% typography by Oliver Reichenstein
Concentrate on the main part, the bigger part, the 95%. If you get them right the rest isn’t so important after all.

Discount usability by Jakob Nielsen
Do what you can do with what you have.

The day the machines took gaming away

August 5th, 2018 was a noteworthy day in the history of mankind. It was a Sunday and had Europe aching in unusual heat and drought. But more important, it was the day when the machines gently asserted their dominance in the field of gaming. It was the day when our most skilled players lost a tournament of Dota 2 against a bunch of self-learned bots.

“Bot” used to be a vilification

How did we end up in this situation? Let’s look back at what “bot” used to mean in gaming. Twenty years ago, we were thrilled about games like Starcraft where you control plenty of aggressive, but otherwise dumb units in a battle against another player that also controls plenty of those units. The resulting brawls were bloody, chaotic and ultimately overwhelming with their number of necessary tasks (so-called micromanagement) and the amount of information that needed to be processed at once to react to the opponent. In a human versus human (or pvp for player versus player) game, those battles were usually constrained to a certain area and executed with a certain laissez-faire attitude. Only the best players could stage two or more geographically independent attacks and control every unit to their full potential. We admired those players like astronauts or rockstars.

If you could not play against another human, you would start a game against a bot. A bot usually had four things that worked in their advantage and a lot of things stacked agaist them. In their favor, they had minimal delay in their reactions, ultimate precision in their commands and full information about everything on the gamefield. And more often than not, they received more game resources and other invisible cheats because they didn’t stand a chance against even moderately skilled humans otherwise. Often, their game was defined by a fixed algorithm that couldn’t adapt to human strategy and situational specifics. A very simple war of attrition was enough to defeat them if their resource supply wasn’t unlimited. They didn’t learn from their experience and didn’t cooperate, not with other bots or allied humans. These early bots relied on numbers and reaction speed to overwhelm their human counterparts. They played against our natural biological restrictions because the programmers that taught them knew about these restrictions very well.

Barely tolerated fill-ins

Those bots were so dumb and one-dimensional that playing with them against other opponents was even more of a challenge because you always had to protect them from running in the most obvious traps. They weren’t allies, they were a liability that dictated a certain game style. Everybody preferred human allies even if they made mistakes and reacted slower.

The turning point

Then, a magical thing happened. An artificial intelligence had trained itself the rules of Go, a rather simple game with only two players taking turns on a rather static gamefield. This AI played Go against itself so excessively, it mastered the game on a level that even experts could not grasp easily. In the first half of the year 2017, the machines took the game Go out of our hands and continued to play against themselves. It got so bad that an AI that was named AlphaGo Zero taught itself Go from scratch in three days and outclassed the original bot that outclassed mankind. And it seemed to play more like a human than the other bots.

So we got from dumb bots that were inferior stand-ins for real humans to overly powerful bots that even make it seem as if humans are playing in just a few years.

The present days

It should be no surprise to you anymore that on that first Sunday of August 2018, a group of bots beat our best players in Dota 2. There are a few noteworty differences to the victories in Go:

  • Dota 2 is a game where five players battle against five other players, not one versus one. It wasn’t one bot playing five characters, it was five bots cooperating with only in-game communication against humans cooperating with a speech side-channel.
  • Go is an open map game. Bot opponents see every detail of the gamefield and have the same level of information. In Dota 2, your line of sight is actually pretty limited. The bots did not see the whole gamefield and needed to reconnoiter just like their human opponents.
  • In Go, the gamefield is static if nobody changes it. In Dota 2, there a lots of units moving independently on the gamefield all the time. This fluidity of the scenario requires a lot of intuition from human players and bots alike.
  • The rules of Go are simple, but the game turns out to be complex. The rules of Dota 2 are very complex, but the game refuses to be simple, because the possibilities to combine all the special cases are endless.
  • Go is mostly about logic, while Dota 2 has an added timing aspect. Your perfect next move is only effective in a certain time window, after that, you should reconsider it.

Just a year after the machines took logic games from us (go and read about AlphaZero if you want to be depressed how fast they evolve), they have their foot in the real-time strategy sector, too. Within a few years, there is probably no computer game left without a machine player at the top of the ladder. Turns out the machines are better at leisure activities, too.

The future?

But there is a strange side-note to the story. The Go players reported that at first, the bots played like aliens. Later versions (the purely self-learned ones) had a more human-like style. In Dota 2, if you mix bots with humans in one team, the humans actually prefer the cooperation with the bots. It seems that bots could be the preferred opponent and teammate of the future. And then, it’s no longer a game of humans with a few bots as fill-in, but a game between machines, slowed down so that humans can participate and do their part – as a tolerated inferior fill-in.

Decoding non-utf8 server responses using the Fetch API

The new Javascript Fetch API is really nice addition to the language and my preferable, and in fact the only bearable, way to do server requests.
The Promise based API is a lot nicer than older, purely callback-based, approaches.

The usual approach to get a text response from a server using the Fetch API looks like this:

let request = fetch(url)
  .then(response => response.text())
  .then(handleText);

But this has one subtle problem:

I was building a client application that reads weather data from a small embedded device. We did not have direct access to changing the functionality of that device, but we could upload static pages to it, and use its existing HTML API to query the amount of registered rainfall and lightning strikes.

Using the fetch API, I quickly got the data and extracted it from the HTML but some of the identifiers had some screwed up characters that looked like decoding problems. So I checked whether the HTTP Content-Type was set correctly in the response. To my surprise it was correctly set as Content-Type: text/html; charset=iso-8859-1.

So why did my Javascript Application not get that? After some digging, it turned out that Response’s text() function always decodes the payload as utf-8. The mismatch between that and the charset explained the problem!

Obviously, I had to do the decoding myself. The solution I picked was to use the TextDecoder class. It can decode an ArrayBuffer with a given encoding. Luckily, that is easy to get from the response:

let request = fetch(url)
  .then(response => response.arrayBuffer())
  .then(buffer => {
    let decoder = new TextDecoder("iso-8859-1");
    let text = decoder.decode(buffer);
    handleText(text);
  });

Since I only had to support that single encoding, that worked well for me. Keep in mind that the TextDecoder is still experimental Technology. However, we had a specific browser as a target and it works there. Lucky us!

Selecting all columns of a database table with an SQL GROUP BY expression

Suppose we have an SQL database table named “temperatures” with the following contents:

LOCATION  TIME        CELSIUS
inside    2018-08-01  24
inside    2018-08-02  28
inside    2018-08-03  21
inside    2018-08-04  28
outside   2018-08-01  29
outside   2018-08-02  31
outside   2018-08-03  25
outside   2018-08-04  30

We want to find the highest temperature for each location. We use the MAX aggregate function and a GROUP BY expression:

SELECT location, MAX(celsius) celsius
FROM temperatures
GROUP BY location;

As expected, the result is:

LOCATION  CELSIUS
outside   31
inside    28

Now suppose we also want to know when each of these extreme temperatures occured. Naively, we try the following query:

SELECT location, time, MAX(celsius) celsius
FROM temperatures
GROUP BY location;

The response is an error: “not a GROUP BY expression”. In a GROUP BY expression all selected columns must be either part of the GROUP BY clause or an aggregate.

To achieve what we want we can use a JOIN:

SELECT
  t.location, t.time, t.celsius
FROM
  temperatures t
JOIN (SELECT location, MAX(celsius) celsius
      FROM temperatures
      GROUP BY location) tmax
ON
  t.location=tmax.location AND t.celsius=tmax.celsius;

This query results in multiple rows per location if the maximum temperature was recorded at different times:

LOCATION  TIME        CELSIUS
outside   2018-08-02  31
inside    2018-08-04  28
inside    2018-08-02  28

If we are only interested in the first occurrence of the maximum temperature per location, we can use the following query:

SELECT
  location,
  MIN(time) KEEP (DENSE_RANK LAST ORDER BY celsius) time,
  MAX(celsius) celsius
FROM
  temperatures
GROUP BY
  location;
LOCATION  TIME        CELSIUS
inside    2018-08-02  28
outside   2018-08-02  31

Here we don’t need a JOIN anymore, because select clause for the time column is an aggregate as well.

For what the javascript!

The setting

We are developing and maintaining an important web application for one of our clients. Our application scrapes a web page and embeds our own content into that page frame.

One day our client told us of an additional block of elements at the bottom of each page. The block had a heading “Image Credits” and a broken image link strangely labeled “inArray”. We did not change anything on our side and the new blocks were not part of the HTML code of the pages.

Ok, so some new Javascript code must be the source of these strange elements on our pages.

The investigation

I started the investigation using the development tools of the browser (using F12). A search for the string “Image Credits” instantly brought me to the right place: A Javascript function called on document.ready(). The code was basically getting all images with a copyright attribute and put the findings in an array with the text as the key and the image url as the value. Then it would iterate over the array and add the copyright information at the bottom of each page.

But wait! Our array was empty and we had no images with copyright attributes. Still the block would be put out. I verified all this using the debugger in the browser and was a bit puzzled at first, especially by the strange name “inArray” that sounded more like code than some copyright information.

The cause

Then I looked at the iteration and it struck me like lightning: The code used for (name in copyrightArray) to iterate over the elements. Sounds correct, but it is not! Now we have to elaborate a bit, especially for all you folks without a special degree in Javascript coding:

In Javascript there is no distinct notion of associative arrays but you can access all enumerable properties of an object using square brackets (taken from Mozillas Javascript docs):

var string1 = "";
var object1 = {a: 1, b: 2, c: 3};

for (var property1 in object1) {
  string1 = string1 + object1[property1];
}

console.log(string1);
// expected output: "123"

In the case of an array the indices are “just enumerable properties with integer names and are otherwise identical to general object properties“.

So in our case we had an array object with a length of 0 and a property called inArray. Where did that come from? Digging further revealed that one of our third-party libraries added a function to the array prototype like so:

Array.prototype.inArray = function (value) {
  var i;
  for (i = 0; i < this.length; i++) {
    if (this[i] === value) {
      return true;
    }
  }
  return false;
};

The solution

Usually you would iterate over an array using the integer index (old school) or better using the more modern and readable for…of (which also works on other iterable types). In this case that does not work because we do not use integer indices but string properties. So you have to use Object.keys().forEach() or check with hasOwnProperty() if in your for…in loop if the property is inherited or not to avoid getting unwanted properties of prototypes.

The takeaways

Iteration in Javascript is hard! See this lengthy discussion…The different constructs are named quite similar an all have subtle differences in behaviour. In addition, libraries can mess with the objects you think you know. So finally some advice from me:

  • Arrays are only true arrays with positive integer indices/property names!
  • Do not mess with the prototypes of well known objects, like our third-party library did…
  • Use for…of to iterate over true arrays or other iterables
  • Do not use associative arrays if other options are available. If you do, make sure to check if the properties are own properties and enumerable.

 

UX tips: charts

Continuing our series about UX tips for small areas we take a look at charts this time.

  • always label your axis – this one should be obvious the reader needs to know at what data he looks
  • use units – is this meter or yards, data can only be understood if the scale is clear
  • use distinctive colors – different data, different color, and don’t forget the color blind
  • use color with purpose – same data, same color, e.g. having different values of the same source should use the same color or tint of color
  • for bar charts always start at zero – the distortion often used for exaggerating differences is immense
  • leave out decoration – decoration distracts from the data
  • no 3d effect – data speaks for itself, please do not try to make it prettier
  • use them for lots of data – if you just have a few numbers a table of data might be more appropriate
  • use a title – what aspect or slice of data are you showing me, a summary helps to understand perspective and intention
  • use a big enough font – this and the appropriate contrast should be obvious for all UI work
  • label tick marks – use them at meaningful intervals, do not just slice the axis into equidistant intervals
  • include time – even if it isn’t a time series chart, tell the reader from when the data is
  • size 2d shapes by area – do not use diameter or radius for sizing 2d shapes (circles, bubbles…), unless you use bar charts (where the length is compared) the proportion gets way out of hand
  • use consistent rounding – first round your numbers and do this consistently
  • for comparisons use the same scale – do not distort your data by presenting it at different scale

The Pure Fabrication Tax

Last week, I attended the Maexle event of the C++ user group in Karlsruhe. The Maexle event is basically a programming contest where your program plays the Mia dice game against other programs. You have to implement a simple network protocol to join games, announce rolls and call bluffs. Your program earns points for every game it has participated and not lost. So there is a strong emphasis on starting early and staying in the game, even if your program doesn’t perform the best.

Since it was an event of the C++ user group, the programming language to be used was C++. I’m certainly no C++ hero and knew I couldn’t compete, so I joined the fun with an espionage role and programmed an observing bot that doesn’t play, but gathers data on the players. I chose Java for the task. My observer was online after two minutes, the first real player joined the server after 20 minutes. It turned out to be written in Python. The first real C++ bot was online after 35 minutes, the last one played its first round after two hours.

I listened closely to the problems the teams around me tackled and noticed something strange: Nobody talked about the actual game (Maexle/Mia). Every task was a technical one. Let’s talk about why that’s a problem.

Three Definitions

Before I dive into the subject, I want to define some terms that I’ll use to help you understand my point. It’s entirely possible to look at the story above and see a bunch of engineers having fun with some engineering tasks.

  • First, I value the economics of my customer. In this case, the customer is a lonely server on the LAN that wants to host some games of Maexle for bots. Like, lots of games. Thousands of games. The customer gives points for early market entry, so time to market is an economic factor (or a key performance indicator, some might say). You can roughly say that being online early means you can make bucks longer. The second key performance indicator is uptime. You want to stay in the game as long as you don’t lose all the time. There are some more KPI, but the two I’ve listed should have a major impact in your programming approach – if you value the economics.
  • Second, I don’t care about tools. A programming language is a tool. A compiler is a tool. Your IDE or text editor is a tool. Use your preferred tooling as long as it suits your needs. That means explicitely as long as it doesn’t actively work against your other values like the customer’s economics. This blog post is definitely not about Java or Python being “better” or “better suited” than C++. They aren’t. The first two bots (observer and player) were programmed by participants that had prior experience with the event. It wasn’t the tool that made them fast, it was the absence of rookie errors in the domain and its technical structure.
  • Third, I will explain my point with the concept of “pure fabrication. Pure fabrication is everything that is not specified by the customer, but necessary to fulfill the specification. It’s the code you write because your customer wants to persist some data. He never ordered you to write SQL statements or “open a connection to the database”, maybe he didn’t even know what a database was. Your customer wanted the data stored somehow. The code that enables you to actually program the storage is “pure fabrication” in terms of the domain. Think of it as a scaffolding holding your domain code in place. If you hire a painter to color your house, he will scaffold the walls to reach every spot with ease. You didn’t hire him to set those structures up, they are just necessary for the task. The difference to most of our code is that the painter removes the scaffolding afterwards.

Pure Fabrication vs. Domain

So, if I would have been a customer on the Maexle event, paying for a competitive Maexle bot, I would be very surprised about the actual construction process. Up to two hours into a three-hours event, my programmer would solve apparently hard and important problems, but not my problems. In fact, I wouldn’t even understand the relation between the attempted problems and my required solution. And I would have to have blind faith for more than half my money that something useable will come out of this.

This is the effect of too much pure fabrication in the programming approach. I’m all for solving hard programming problems, but I’m not interested in solving them over and over again. After some iterations, they become solved problems or, essentially, tools. And I don’t care about tools as long as they get their job done. If your domain problem requires a better tool, then we can put the programming problem on our todo list again. Otherwise, we are not valueing our customer’s economics, we are showing off to our peers.

If you program a simple game of Maexle with a heavy emphasis on time to market and even after the initial ramp up aren’t able to reason about your code using language from the domain (like game, dice, roll, bluff, double and, of course, mia), you are staying in pure fabrication land. That’s the level of programming where it matters if you used an integer or freed that memory. That’s when you pay the Pure Fabrication Tax to the fullest. Because your code now does something valueable in the domain, but the distance between your customer’s language and your code’s language is an hindrance. And this distance will demand its tax with every new feature, every change request and every bug.

Bugs are another area where the distance is measureable. If you can’t explain your bugs to the customer, you’ve made them in the pure fabrication part of your code. If you can never explain your bugs, your domain code is hidden between lines and lines of source code with lots of special characters, brackets and magic numbers. Just imagine your hired painter tries to tell you why your house is now pink instead of white or yellow: “It was a small mishap in the way we constructed the scaffolding, we used an E5 steel beam instead of a rail clamp and forgot to perform a hammer check on it”. The last part is totally made up, but I’m sure that’s how we sound for non-programmers.

Exemptions from the Tax?

What solution would I suggest? I don’t think there is a definite solution to the problem. You can’t go full Domain Driven Design on a three-hour Maexle event. By the time you’ve built your fancy Domain Specific Language to write code with the customer besides you, everybody else has gathered their game points and gone home. If you switch to a language that has a string tokenizer in its standard library, you can speed up your programming, but maybe just produce a bigger heap of slightly less low-leveled pure fabrication code.

I don’t want to advocate a solution. My attempt is to highlight the problem: The Pure Fabrication Tax. Given the right (or wrong) amount of extrinsic (or intrinsic) motivation, we are able to produce a mess in just a few hours without really connecting to the domain we produce the mess for. If we didn’t program a Maexle bot that night, but a poker bot or a chat bot, most if not all of the problems and bugs would have been the same. This is not a domain-specific problem. It’s our problem. We probably just like to pay the tax.

What are your thoughts on the Pure Fabrication Tax? Can you see it? Do you have an idea for a solution approach? Leave your comment below!

Disclaimer

And to counter everybody who thinks I’m just bashing the other participants on the event: I was the first one online on the server, with a task that requires virtually no effort and doesn’t even participate directly in the competition, with tools that solved nearly all my pure fabrication problems and still managed to create a program that contained less than five domain terms and was useless for its intended purpose. I said I value the economics of my customer (even if there was none), so I know that I failed hardest on the event. And I had prior knowledge. There was just nobody to compare my mess to.