A short story about priorities

When you mix your own priorities with the ones of your customers, embarrassing things might happen.

Before I can tell you the story, I have to add this disclaimer: The story is true, but it wasn’t us. To protect the identities, I’ve changed nearly all the details, hopefully without watering down the essence of the story.

The assignment

A few years ago, when developing software for mobile devices wasn’t as ordinary as it is today, a freelance developer got a call from a loyal customer. He should develop a little web application that could also be used on a smartphone. The functionality of the application was minimal: Show a neat image containing two buttons to download a song teaser in two different file formats, probably MP3 and WMA. That’s all. The whole thing was a marketing app that should be used a few weeks or months and then be mothballed. Being a marketing app, the payment was very decent.

The development

The freelancer was excited to say the least. This was his chance to enter the emerging field of mobile device software development. And because it should be a web application, he would try out all these new fancy mobile web frameworks and settle for the best. The image of the website should rotate and adjust to the device screen if the user tilted his phone. And because phone screens weren’t that well equipped with pixels as today and mobile internet was expensive, the image should be optimized for the best possible tradeoff between details and file size.

Some mobile web frameworks provided access to the device type, so there were all kinds of javascript magic tricks to be potentially applied. Well, they weren’t applied, but very well could have been. The freelancer prototyped most of them, out of curiosity.

The web application itself was completed in one session once the song files and the master image were provided by the customer. The freelancer installed everything on a staging web server and communicated back success. The customer was pleased by the quick reaction time and appointed a meeting to inspect the results.

The presentation

The customer gave the freelancer a notebook attached to a beamer to present the application. The notebook had a screen resolution many times the envisioned target platform, so the intro image was very pixellated and far from appealing. The freelancer used his smartphone to present the image on a smaller screen, but the customer only shook his head: “You should develop a web application for both phones and PCs. Most of our customers will use their PC to visit the site.” The freelancer apologized and promised to add another browser switch to present a full resolution image to PC users.

Now, the freelancer continued to present the web application on his phone. He invited the customer to tilt the device and was satisfied when the web site adjusted perfectly. The customer was pleased, too. Then, he tapped on one of the buttons. Nothing happened. The whole application consisted only of two buttons and one wasn’t working. The freelancer frantically tried to figure out what had gone wrong, when the customer tapped on the other button. No reaction from that one, too. “But I’ve uploaded both song files to the right location with the right access rights.”, the freelancer just said to himself when it dawned him: He forgot to insert the link tags in the HTML file. The buttons were just images. He never actually clicked on one of the buttons during development.

The moral of the story

Recapitulating, the freelancer was asked to develop a web application with an image and two download buttons, but he managed to cripple the image for the larger part of the anticipated users and never provide any download functionality at all. The customer’s requirements somehow got lost along the way.

This isn’t so much a story about a confused freelancer or improper requirement analysis. It’s a story about priorities. The customer expressed his priorities through the requirements. The freelancer superimposed his own priorities very early in the process (without telling anybody) and never returned to the original set until the presentation. And while it is granted to have a secondary agenda as a service provider, it should never interfere with the primary agenda – the one set by the customer.

Don’t fool yourself into thinking that this could never happen to you. It’s not always as obvious as in this story. Some, if not most of your customer’s priorities are unintentionally (or even intentionally) kept secret. They can only be traced during live exercises, which is why early prototyping and using the prototype in real scenarios is a good way to reveal them.

When it comes to multithreading, better be safe than sorry

Writing multithreaded applications in Java is hard. Here are five problems and how to avoid them without much effort (mostly).

Recently, I attended a code review of the core parts of a web application, written in Java. The application is used by a large customer base and occassionally, there are error reports and exceptions in the log files. Some of these exceptions are the dreaded ConcurrentModificationExceptions, indicating conflicting read/write access on an unsynchronized collection data structure. In the code review, we found several threading flaws, but not after an exhaustive reading of the whole module. Here, I want to present the flaws and give some advice on how to avoid them:

The public lock

In some parts of the code, methods were defined as synchronized through the method declaration keyword:

public synchronized String getLastReservation() { [...]

While there is nothing wrong with this approach in itself, it can be highly dangerous in combination with synchronized blocks. The code above effectively wraps a synchronized block using the object instance (this) as a lock. No information of an object is more publicly visible as the object reference (this), so you have to check all direct or indirect clients of this object if they synchronize on this instance, too. If they do, you have chained two code blocks together, probably without proper mentioning of this fact. The least harmful defect will be performance losses because your code isn’t locked as fine grained as it could be.

The easiest way to avoid these situations it to always hide the locks. Try not to share one object’s locks with other objects. If you choose publicly accessible locks, you can never be sure about that.

The subtle lock change

In one class, there were both instance and class (static) methods, using the synchronized keyword:

public synchronized String getOrderNumberOf(String customerID) { [...]
public  synchronized static int getTotalPendingOrders() { [...]

And while they were both accessing the same collection data structure (a static hashmap), they were using different locks. The lock of the instance method is the instance itself, while the lock of the static method is the class object of the type. This is very dangerous, as it can be easily missed when writing or altering the code.

The best way to prevent this problem it to avoid the synchronized modifier for methods completely. State your locks explicitely, all the time.

Partial locking

In a few classes, collection datatypes like lists were indeed synchronized by internal synchronized-blocks in the methods, using the private collection instance as lock. The synchronized blocks were applied to the altering methods like putX(), removeX() and getX(). But the toString() method, building a comma-separated list of the textual list entries, wasn’t synchronized to the list. The method contained the following code:

public String toString() {
    StringBuilder result = new StringBuilder();
    for (String entry : this.list) {
        result.append(entry);
        result.append(",");
    }
    [...]
    return result.toString();
}

I’ve left out some details and special cases, as they aren’t revelant here. The problem with the foreach loop is that an anonymous Iterator over the list is used and it will relentlessly monitor the list for any changes and throw a ConcurrentModificationException as soon as one of the properly synchronized sections changes it. The toString() method was used to store the list to a session dependent data storage. Every once in a while, the foreach loop threw an exception and failed to properly persist the list data, resulting in data loss.

The most straight-forward solution to this problem might be to add the missing synchronization block in the toString() method. If you don’t want to block the user session while writing to disk, you might traverse the list without an Iterator (and be careful with your assumptions about valid indices) or work on a copy of the list, given that an in-memory copy of the list would be cheap. In an ACID system scenario, you should probably choose to complete your synchronized block guards.

Locking loophole

Another problem was a collection that was synchronized internally, but could be accessed through a getter method. No client could safely modify or traverse the collection, because they had the collection, but not the lock object (that happened to be the collection, too, but who can really be sure about that in the future?). It would be ridiculous to also provide a getter for the lock object (always hide your locks, remember?), the better solution is to refactor the client code to a “tell, don’t ask” style.

To prevent a scenario when a client can access a data structure but not its lock, you shouldn’t be able to gain access to the data structure, but pass “command objects” to the data structure. This is a perfect use case for closures. Effectively, you’ll end up with something like Function or Operation instances that are applied to every element of the collection within a synchronized block and perform your functionality on them. Have a look at op4j for inspirational syntax.

Local locking

This was the worst of all problems and the final reason for this blog entry: In some methods, the lock objects were local variables. In summary, these methods looked like this:

public String getData() {
    Object lock = new Object();
    synchronized (lock) {
        [...]
    }
}

Of course, it wasn’t that obvious. The lock objects were propagated to other methods, stored in datastructures, removed from them, etc. But in the end, each caller of the method got his own lock and could henceforth wreck havoc in code that appeared very well synchronized on first look. The error in its clarity is too stupid to be widespread. The problem was the obfuscation around it. It took us some time to really understand what is going on and where all that lock objects really come from.

My final advice is: If you have to deal with multithreading, don’t outsmart yourself and the next fellow programmer by building complex code structures or implicit relationships. Be as concise and explicit as you can be. Less clutter is more when dealing with threads. The core problem is the all-or-none law of thread synchronization: Either you’ve got it all right or you’ve got it all wrong – you just don’t know yet.

Hide your locks, name your locks explicitely, reduce the scope of necessary locking so that you can survey it easily, never hand out your locked data, and, most important, remove all clutter around your locking structures. This might make the difference between “just works” and endless ominous bug reports.