Category: Uncategorized

Tests may remember the spec better than the customer or yourself

We have an application in maintenance mode for some years now. One part of the app displays messages in a certain format. They contained %-characters which have a special meaning. Both we and our customer thought they were about encoding line endings or some such. One day our customer reported missing parts within these messages. We dove down into the issue, analysed the raw messages containing a few %-signs and noticed some weird looking code:

public String parse(String message) {
    StringChunker tok = new StringChunker(message, Text.PERCENT);
    DirectChunkBuffer result = new DirectChunkBuffer(Text.NEWLINE);
    if (tok.hasMoreChunks()) {
        result.add(tok.getNextChunk());
    }
    return result.toString();
}

The if-statement feels unusual here as most would expect a while loop essentially splitting the original message by % and putting it together again with newlines in between. Almost immediately we thought of a bug that never until now occurred in production triggered by malformed raw messages.

But our unit tests documented clearly the current behaviour as correct. So we decided to talk again with our customer. He then asked his experts and they confirmed the behaviour and explained their workflow. The %-characters were used as comment characters to hide text blocks the expert workers used as templates. Nothing after the first %-character should be displayed. They also confirmed that the displayed message was correct and the whole error report was indeed some kind of communication problem somewhere in the organisation.

The tests saved us from breaking specified and correctly working behaviour.

After the clarification by the experts and we improved the situation by refactoring the code to communicate its intent clearer. We also documented the message format in the javadocs and a wikipage in addition to the tests.

Prepare for the unexpected

In most larger projects there are many details which cannot be foreseen by the development team. Specifications turn out to be wrong, incomplete or not precise enough for your implementation to work without further adjustments. New features have to work with production data that may not be available in your development or testing environment.

The result I often observed is that everything works fine in your environment including great automated tests but fails nevertheless when deployed to production systems. Sometimes it is minor differences in the operating system version or configuration, the locale for example, may cause your software to fail. Another common problem is  real production data containing unexpected characters, inconsistencies in the data (sometimes due to bugs) or its sheer size.

What can we do to better prepare for unexpected issues after deployment?

The thing is to expect such issues and to implement certain countermeasures to better cope with them. This may conflict with the KISS principle but usually is worth a bit of added complexity. I want to provide some advice which proved useful for us in the past and may help you in the future too:

  1. Provide good, detailed and persistent debug output for certain features: Once we added a complex rule system which operated on existing domain objects. To check every possible combination of domain object states would have been a ton of work, so we wrote tests for the common cases and difficult cases we could think of. Since the correctness of the functionality was not critical we decided to rather display slightly incorrect information instead of failing and thus breaking the feature for the user. We did however provide extensive and detailed logs whenever our rule system detected a problem.
  2. Make certain parts of your communication interface to third party systems configurable: Often your system communicates to different kinds of users and other systems. Common examples are import/export functionality, web service APIs or text protocols. Even if most of the time details like date and number formats, data separators, line endings, character encoding and so forth are specified it often proves valuable to make them configurable. Many times the specification changes or is incorrect, some communication partner implements the protocol slightly different or a format deviates from your assumption breaking your application. It is great if you can change that with a smile in front of your client and make the whole thing work in minutes instead of walking home frustrated to fix the issues.

The above does not mean building applications with ultimate flexibility and configurability and ignoring automated tests or realistic test environments. It just means that there are typical aspects of an application where you can prepare for otherwise unexpected deviations of theory and praxis.

The Great Divide

There is a great divide in the C++ developer community between “normal” developers that use only basic language features and very savvy ones that know every little corner of the language. The upcoming C++ standard deepens this divide even more.

Recently, I had two very contrary conversations about C++ which show very good the great divide in C++ developer community.

The first was with the technical lead of a team that writes and maintains drivers and control software for a scientific institution. These systems run 24/7 and have to be very stable and reliable.

I had discovered that they use a self-written toolbox library containing classes like SharedPtr<T>, and Thread and suspected immediately a classical NIH-syndrome. I asked him about it and why they don’t use well established libraries like boost. He told me that they indeed are only using the standard library and their own toolbox.

The reason he gave was that despite boost being most elegant C++ library out there, it required very good knowledge about the most advanced C++ mechanisms, and that his team was not on this level … I should probably mention here that his team does a very good job in running their systems. So, apparently, they get along very well with using only basic  C++ features and no “fancy” boost stuff.

The other conversation was with a friend of mine with whom I chat regularly about all sorts of programming related stuff. This time the topic was the upcoming  C++ standard and all its  exciting new stuff. He has lot’s of experience with C++ and knows the language very well. But even someone like him had a hard time to really understand what rvalue references are all about. I had not looked at them in detail, yet,  so he tried to explain them to me. During our discussion I was thinking about if teams like the one introduced before will ever use rvalue references, or other C++0X stuff in their production code, other than maybe the auto keyword for type inference, or constructor delegation.

Honestly, I don’t think stuff like  rvalue refs will become a feature that is often used by “standard industry” teams, because it adds a lot of complexity to an already complex language. Even easy-to-get stuff like the new keywords override, constexpr and final, or additional initialization means like std::initializer_list<T> will take a lot of time to get used regularly by most C++ teams.

Instead, most of C++0X will greatly increase the divide between “normal” C++ developers who get along well with using only basic language features, and experts that know every little corner of the language. And this is simply because there is so much more to know with C++0X.

But don’t let us paint this picture overly black. I, for one, am looking forward to the new standard and I will certainly spread the word about the new possibilities and features in every C++ team I work with.

Using Groovy? Prepare for the unexpected!

Disclaimer

This post is not intended as some kind of Groovy-bashing. It rather points out some of the common problems and maybe differences in the mindset between Java and Groovy developers.

Groovy powers Grails and Grails empowers us to build modern web application conveniently and fast. We have come to love many of the features of Groovy and Grails. Sadly, every once in a while you stand puzzled before some bizarre error message. Often it is our own fault, but sometimes it is a real problem in the software stack we are using. Grails was quite buggy some time ago (pre 1.2) but we are more and more happy with its development. Groovy itself has been quite solid for a long time but things like dynamic dispatch with null values, boolean truth  and its  NullObjects (e.g. null + null == "nullnull"!) may yield some surprises.

A few days ago we received an failure report from a client and found the corresponding exception in our logs:

java.lang.UnsupportedOperationException
at $Proxy12.toString(Unknown Source)
at java.lang.Throwable.getLocalizedMessage(Throwable.java:267)
at java.lang.Throwable.toString(Throwable.java:343)
at java.lang.String.valueOf(String.java:2827)
at org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:211)
at org.apache.commons.logging.Log$error$0.call(Unknown Source)
at UserGroupController.callSendMail(UserGroupController.groovy:117)

You may be able to see that the UnsupportOperationException is thrown in an attempt to log a Throwable. Yeah right, toString() of some generated proxy object throws an exception. This is not exactly what you would expect from a call to the toString()-method and violates the principle of least astonishment. Equally bad in this case is he fact that it shadows the real cause of the problem. Some research revealed that this is a bug in Groovy itself.

This leads me to my highly speculative and opionated “mindset” hypothesis. In the java world people try to avoid the unexpected at the cost of clunkyness and verbosity. Examples for this mindset are static typing and checked exceptions. Sources of uncertainty, like reflection and downcasts, are minimized and put in defined areas of code often documented and annotated. Everyone tries to strictly define everything and relies on these definition. All this sums up to quite a burden and may feel like a cage.

Enter Groovy which removes a lot of the burden by its more liberal syntax and other language features. Suddenly, you are free and do not have to define everything and life feels a lot easier at first. You do not need well designed class hierarchies, just use duck typing. Everything is dynamic and you can cope with things at runtime. Not only the above bug (it is still a bug, not intended behaviour!) but also the handling of boolean values or null may make the seasoned java developer shout “WTF!” from time to time.

So what is the bottom line now? Both worlds have their pros and cons. In Groovy fundamental things can surprise you at runtime so there is an even greater need for good test coverage and a thorough knowledge of the language. Groovy (and Grails for that part) are not as easy as they seem in smaller projects when going big. The Java camp has a solid base but should strive to make life more convenient for the developer. It is mostly the verbosity and some missing features like type inference or closures that are driving people away from Java. Imho both languages and the Java platform with its great community and eco-system have a bright future ahead but there are some bumps on the way ahead.

A big benefit of Convention over Configuration

Convention over Configuration helps a lot getting a fast start on an existing project.

Recently I joined a long running Grails project and had to complete some issues in a short time. After a quick introduction I was ready to dive in and instantly could chunk out code. Convention over Configuration helped a lot getting a fast start:

  • Which classes are persisted? Look into the domain folder.
  • What controller handles xyz? It is named after xyz.
  • What is the page that is served for URL u? Look into the corresponding folder and find the view named accordingly.

IMHO this is a benefit that many conventions are determined by the framework. If you know the framework, you know the layout of the project. The opposite holds also true: if the project departs from these “standards” you have to look closely into the configuration.
So think twice before you do something your way. Sometimes you have no choice like when you are using Oracle and have to cut all table and column names to 30 characters. But normally you should keep the defaults.

Story about bogus error messages

Most computer users know the situation where some system or service that worked for months without problems suddenly stops working. I want to tell you of a small war story which happened some weeks ago to us.

We are maintaining an own mail server with imap access for our employees. That allows for relativly easy serverside spam protection, mailing list management and archiving and so on. We use the trusted combination of postfix, cyrus and mailman for the task and everything works very reliably. Then suddenly we got the error message ssl_error_rx_record_too_long in our e-mail clients. Nothing had changed software-wise. Googleing on the internet brought up all kinds of different obscure reasons for this error but no explanation why something like that would happen out of thin air.

Fortunately, looking in cyrus’ log files quickly showed the reason: the hard drive was full! A two days before the mail system failing there was a larger upload to the server for sharing stuff with a colleague. This upload fitted almost exactly onto the free disk space and some mails later the disk was full. It was really a murphy’s law situation because some kilo bytes less free space would have made the file sharing fail with a sensible error message. But it worked and made the mail system fail suddenly without immediate connection some changes to the server.

There are some lessons to be learned here:

  • Aside from file managers most applications assume memory and disk space are unlimited. If they do hit such a limit they usually fail miserably with complete bogus errors.
  • Monitor critical resources on important systems to receive warnings ahead of time before important service fail. Tools like Nagios can help here.
  • Try to be aware of side effects of your actions. Separating services to different machines may help to reduce unexpected side effects on seemingly unrelated stuff. We used the server to run many different unrelated services.

GORM-Performance with Collections

The other day I was looking to improve the performance of specific parts of our Grails application. I quickly found the typical bottleneck in database centric Grails apps: Too many queries were executed because GORM hides away database queries by its built-in persistence methods for domain objects and the extremely nice dynamic finders. In search for improvements and places to use GORM/Hibernate caching I stumbled upon a very good and helpful presentation on GORM-performance in general and especially collection usage. Burt Beckwith presents some common problems and good patterns to overcome them in his SpringOne 2GX talk. I highly recommend having a thorough look at his presentation.

Nevertheless, I want to summarize his bottom line here: GORM does provide a nice abstraction from relational databases but this abstraction is leaky at times. So you have to know exactly how the stuff in your domain classes is mapped. Be especially careful it collections tend to become “large” because performance will suffer extremely. We already observed a significant performance degradation for some dozen elements; your mileage may vary. For many simple modifications on a collection all its elements have to be loaded from the database!
Instead of using hasMany/belongsTo just add a back reference to the domain object your object belongs to. With the collection you lose cascading delete and some GORM functionality but you can still use dynamic finders and put the functionality to manage associations yourself into respective classes. This may be a large gain in specific cases!

Looping in C++

What is “the best” way to loop over collections in C++?

One recurring discussion point in one of our customers C++ project team is the following:

What is “the best” way to loop over collections?

In a typical scenario there is a standard container like std::list, or some equivalent collection, and the task is to do something with every element in the collection. The straight forward way would be like this:

std::list<std::string> mylist;
for (std::list<std::string>::iterator iter = mylist.begin(); iter != mylist.end(); iter++)
{
   ...
}

This code is correct and readable. But my guess is that most of you instantly see at least two possible improvements:

  1. the call to mylist.end() occurs in every loop an can be expensive e.g. in case of long std::lists
  2. iter++ creates one unnecessary intermediate object on the stack

So this

for (std::list<std::string>::iterator iter = mylist.begin(), end = mylist.end(); iter != end; ++iter)
{
   ...
}

would be much better but can already be seen as a little less readable.

Using BOOST_FOREACH can save you much of this still tedious code but has one nasty pitfall when it comes to std::maps.

In some places of the code base std::for_each is used together with a function, or function object.  The downside of this is that the function/function object code is not located where the loop occurs. However, this can be made “readable enough” when the function, or function object does only one thing and has a telling name.

Looping is sometimes done to create other collections of objects for each element. What to do there? Define the new collection use a for-loop of BOOST_FOREACH like above, or use std::transform with the same downside as std::for_each?

The other day one team member suggested to use boost::lambda expressions in loops. The initial usage examples where very promising but let me tell you – readability can drop dramatically very fast if you don’t be careful. It is very easy to get carried away with boost’s lambdas. I happened that we found ourselves having spent the last hour to carve out a super crisp lambda expression that takes anybody else another hour to read.

So the initial question remains undecided and will most likely stay like that. As for everything else in programming, there doesn’t seem to be a silver bullet for this task.

How do you go about looping in C++? Do you have some kind of coding style in place? Do you use std::for_each, BOOST_FOREACH, or some other means?

Looking forward to some feedback.

Simple code, subtle bugs: write unit tests!

We developed an applet some years ago that was supposed to run 24/7. Basically, it fetches some values periodically and plots them in a chart. It always shows the last 24 hours. Everything seemed to work fine but every few weeks it crashed. The problems were clouded by hardware instabilities and the use of some obscure JVM. After quite a bit of analysis it became clear that it was a bug in our applet: a memory leak. Here is the troublemaker code:

for (int i = 0; i < dataSeries.getItemCount(); i++) {
    XYDataItem dataItem = dataSeries.getDataItem(i);
    if (dataItem.getX().longValue() < startDate.getTime().getTime()) {
        dataSeries.remove(i);
    } else {
        break;
    }
}
dataSeries.add(newDataPoint);

This simple and innocently looking piece of code essentially removes old items from a list by index. Many of you may already have spotted the main problem leading to the leak. Removing items by index means that following items shift one place towards the head of the list. As the index is incremented one element is skipped by the next iteration. This added up over time and lead to an OutOfMemoryError after some weeks.

Now, even if the code is not great it does relatively look straightforward on first sight, yet it is not and is contains errors. Making things worse, the code was buried somewhere in some tangled logic. This leads me to the point of my post:

Write unit tests even for relatively simple code.

Most likely writing a unit test for this cleanup work would have led to a class that manages the dataSeries and nothing more. Quite easy to understand and test in isolation. The problem would never have slipped into production and caused months of  investigating different stability problems.

The programmer may not have been aware of iterators but he should have made sure that this block works as intended. The best way to do this is automated unit tests. They make sure that your building blocks are as solid as you expect them to be. Use acceptance testing to make sure you have put your building blocks together the right way. Together unit and acceptance tests will save countless hours hunting regressions.

Podcasts

Podcasts are a very good means to shorten your commute, to keep you entertained during otherwise boring house-keeping activities, or, if you’re into sports, during your training sessions. Here is a list of some of my favourite shows.

This Developer’s Life

Rob Conery and Scott Hanselman interview developers and other IT professionals who share their stories. Very interesting, very well edited and flavoured with some nice pieces of music.

TechZing

Basically, TechZing are two guys, Jason Roberts and Justin Vincent, who discuss different topics concerning their lives as freelance web developers and startup bootstrappers. They enjoy themselves very much just talking to each other which is very entertaining already. The occasional interview and panel shows are then the icing on the cake.

It’s impossible to give a clear range of  topics since they consist of technical stuff like ‘how to store images in web applications’, SEO, NoSQL, JavaScript and iPhone development, but also non IT stuff like Pioneer One, geological challenges, and the Luck-Surface-Area. Edutainment at its best! Highly recommended!

Software Engineering Radio

This is purely an interview show which addresses all sorts of topics of interest for professional software developers: languages, platforms, technologies, methodologies, etc. Very informative, high profile guests and very competent hosts. Unfortunately, the output rate has gone down a lot in the last year.

Software ArchitekTOUR Podcast

This german (with little bits of swabian) speaking podcast is mostly concerned with topics around software architecture (as the name already suggests). DSLs, NoSQL databases and REST have been some of the latest topics.

FLOSS Weekly

Randal Schwartz (mostly) and other hosts are talking about Free Libre Open Source Software projects, ranging from whole OSes like CentOS to smaller niche projects like Ledger. Great show if you want to know what’s going on in the Open Source world.

Security Now

Steve Gibson and Leo Laporte talk about everything related to IT security. This will keep you informed about the latest browser vulnerabilities, Adobe Flash updates and Windows patches. But you will also learn e.g. how SSL works, the details of Stuxnet and everything about BitCoins. Don’t miss the all-time favourite episode 248: The Portable Dog Killer.

What are your favourite shows?