Shrink your dependency list with POCO

POCO is a nice set of C++ libraries which provides elegant solutions for day-to-day tasks.

When you write C++ applications of any sort you are very likely to need support libraries in addition to what comes with C++ (which is not much, btw). Of course, this holds true for any other language. But with Java and its rich JDK for example this need is not so imminent.

Starting at the very beginning, let’s see how fast the need for support arises.

int main(int argc, char** argv)
{
// parsing command line arguments
...

How to parse those command line arguments in a simple and easy way? How about a little help output when the program is called with -h or –help? Ok, we got boost::program_options for this.

Going further in your program you may want to have some sort of logging capability. Unfortunately, as of boost version 1.45 there is nothing to be found there. So you add a nice logging library.

And so on.

But wait! You don’t want to depend on too many 3rd party libraries because, among other things, they add deployment complexity.

Not even Qt, as one of the major players in the C++ framework world, provides solutions to both previous examples. As of version 4.7, no logging and not much support with command line arguments. And you end-up having to use QString, one of many non-std::string classes in C++ frameworks, which can get annoying at times (of course there are reasons why those exist).

I could go on with the list of smaller or larger concerns for which you either roll your own implementation or include yet another library in your project.

Instead I would like to point you to POCO, a nice set of C++ libraries which provide easy solutions for many basic and/or advanced day-to-day tasks. From their website:

Modern, powerful open source C++ class libraries and frameworks for building network- and internet-based applications that run on desktop, server and embedded systems

Besides very basic stuff like logging, date/time handling, threads, memory management, UTF-8, etc. they also provide lots of higher level classes for things like SMTP, POP3, SQL database access and HTTP. They even have a so called C++ Server Page Compiler which is basically something like JSP or Active Server Pages.

And they have no own string class! Yay! Instead they provide lots of functions classes and streams to do string manuipulation on good old std::string.

One thing I like most about POCO, though, is its clean, well-documented and apparently very high quality code. Although it is not overly functional or template-heavy, like you see it in in boost very often, it still provides elegant solutions.

Check it out and shrink your dependency list.

Diving into Hibernate’s Query Cache behaviour

Hibernate is a very sophisticated OR-Mapper and as such has some overhead for certain usage patterns or raw queries. Through proper usage of caches (hibernates featured a L1, L2 cache and a query cache) you can get both performance and convenience if everything fits together. When trying to get more of our persistence layer we performed some tests with the query cache to be able to decide if it is worth using for us. We were puzzled by the behaviour in our test case: Despite everything configured properly we never had any cache hits into our query cache using the following query-sequence:

  1. Transaction start
  2. Execute query
  3. Update a table touched by query
  4. Execute query
  5. Execute query
  6. Transaction end

We would expect that step 5 would be a cache hit but in our case it was not. So we dived into the source of the used hibernate release (the 3.3.1 bundled with grails 1.3.5) and browsed the hibernate issue tracker. We found the issue and correlated it to the issues HHH-3339 and HHH-5210. Since the fix was simpler than upgrading grails to a new hibernate release we fixed the issue and replaced the jar in our environment. So far, so good, but in our test step 5 still refused to produce a cache hit. Using the debugger strangely enough provided us a cache hit when analyzing the state of the cache and everything. After some more brooding and some println()'s and sleep()‘s we found the reason for the observed behaviour in the UpdateTimestampsCache (yes, yet another cache!):

	public synchronized void preinvalidate(Serializable[] spaces) throws CacheException {
		//TODO: to handle concurrent writes correctly, this should return a Lock to the client
		Long ts = new Long( region.nextTimestamp() + region.getTimeout() );
		for ( int i=0; i
			if ( log.isDebugEnabled() ) {
				log.debug( "Pre-invalidating space [" + spaces[i] + "]" );
			}
			//put() has nowait semantics, is this really appropriate?
			//note that it needs to be async replication, never local or sync
			region.put( spaces[i], ts );
		}
		//TODO: return new Lock(ts);
	}

The innocently looking statement region.nextTimestamp() + region.getTimeout() essentially means that the query cache for a certain “region” (e.g. a table in simple cases) is “invalid” (read: disabled) for some “timeout” period or until the end of the transaction. This period defaults to 60 seconds (yes, one minute!) and renders the query cache useless within a transaction. For many use cases this may not be a problem but our write heavy application really suffers because it works on very few different tables and thus query caching has no effect. We are still discussing ways to leverage hibernates caches to improve the performance of our app.

Open Source Love Day October 2010

Our Open Source Love Day for October 2010 brought love for the cmake hudson plugin. Other issues were addressed but not finished. If you like to type fast and accurate, we suggest you check out typeracer.

On Friday two weeks ago, we held our Open Source Love Day for October 2010. This day was special in several ways. We strayed very far from the usual schedule for this day, there were several internal tasks that couldn’t be delayed and we introduced a “fun practice” event. But we eventually produced something valuable this day.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

The distractions

  • A regular project needed an urgent cost estimation by the whole team. This was the last opportunity because of an upcoming parental leave to have the team together for a long time.
  • Another regular project needed an urgent problem solved. This turned out to be so obscure that one of our developers had to be on-site. You can read about it in this blog entry now.
  • We received several shipments of office furniture and computer parts. They had to be checked and placed in.
  • We had a fun practice event. We discovered the online “game” typeracer and practiced our raw typing skill against each other for some time. Pro tip for beginners: don’t look at the highscores!

On this OSLD, we accomplished the following tasks:

  • A new version 1.8 of the cmakebuilder hudson plugin implements several feature requests. You can now choose to NOT clean your workspace before building and set different paths for the cmake installation for every job or node (hudson slaves). The latter option can be applied using an environment variable.

On this OSLD, we also tried the following tasks:

  • We keep an eye on Scala and its associated web framework Lift as a promising technology. One issue with Lift that bugs us is the use of “sun bastard format” properties for internationalization. We tried to teach Lift to accept UTF-8 encoded property files. After a lot of “downloading the internet” (you can always tell which project uses maven by their initial setup delay), we quickly implemented our own ResourceBundle.Control. But the Lift framework itself could not be built: “Error occurred during initialization of VM: Could not reserve enough space for object heap”. We ran out of time and will investigate in this issue on the next OSLD.
  • Grails is another web framework we use in projects. There are some bugs that really annoy us, and the OSLD is the perfect time to fix them. One of these bugs is GRAILS-6475, which we tried to reproduce with the latest code base. After writing a test case that would go green unexpectedly, we tried to provoke the error by setting up a sample project. The bug didn’t show up there, too. We left a comment in the issuetracker and ceased development.

What were our lessons learnt today?

  • You can’t tear off massive amounts of time from the OSLD and expect it to still be working. An OSLD doesn’t scale down apparently.
  • Most issues that can’t be done fail with the project’s build. The build process of a foreign project is the most crucial phase in your decision on commitment. If it fails, your participation in the project is at risk. We’ve seen many brittle, undocumented and incredible complex build processes now. And we can state one thing: It doesn’t stop with throwing maven at a project, you still have to “think the build”.

Retrospective of the OSLD

This OSLD was special in the amount of non-OSLD work done. The remaining efforts weren’t as successful as we wished. This has been an ongoing issue with our OSLD for the last months now and we are looking forward to adapt our workstyle to yield better results in the future. The distraction by typeracer was fun, though.

An advent of unconditional quality code

A four-week experiment dealing with conditional statements and how to avoid or replace them. Starting at the first advent, the experiment runs until christmas. We invite you to join and share your experiences.

This blog entry invites you to an experiment in code. It’s an experiment that runs four weeks and can be performed secretly even at your workplace. It might improve the way you think about conditional statements in an object oriented programming language. You don’t need any special hardware or setup, just the will to change your coding style a bit each week.

The experiment

Beginning with this year’s advent (a season of the christian religion), you are asked to omit one type of conditional statement each week while programming your regular code. The omitted statements add up, so that you have to spare four different statements in the week before christmas. There is no relation to christmas (or religion) other than it’s a four week period at the end of the year, which is the perfect timeframe for the experiment. And you might buy yourself a little present for christmas if you succeeded at the experiment (idea: a new programming book).

The four stages

For every stage, you are asked to write your normal code without a specific statement. It is perfectly valid to use semantically equivalent code constructs to achieve the same goal. This experiment is even more successful if you are creative and diversified in your variations of the original statement. Remember that the stages add up. On the fourth stage, you are asked to use none of the statements mentioned below.

  • Stage 1 (first week): Don’t use “else”
  • Stage 2 (second week): Don’t use the conditional operator “?:”
  • Stage 3 (third week): Don’t use “switch”
  • Stage 4 (fourth week): Don’t use “if”

You are not asked to change existing code to conform to these restrictions, except you need to work on the lines that contain the prohibited statements. You should apply the rules to your new code rigorously, though.

Explanation of stage 1 (Don’t use “else”)

This rule bans all the different occurrences of the else-branch to your if-statements. It includes every “else if” or “elsif” your programming language might provide. The rationale behind the rule can be found in the Object Calisthenics, rule #2 by Jeff Bay. Here is an explanation of it by Being Cellfish.

Explanation of stage 2 (Don’t use the conditional operator “?:”)

Elvis is dead. Let this resemblance to his hairdo rest for a week, too. It contains a hidden else statement that is restricted since stage 1. Another rationale is that the conditional operator isn’t very easy to read/grasp if stretched out a long line.

Explanation of stage 3 (Don’t use “switch”)

A switch (or case, or select) statement is nothing but a big if-else cascade. It’s handy sometimes, but can be replaced by a lookup table (like a hashmap) virtually everytime . In Martin Fowler’s book “Refactoring”, the switch statement counts as its own code smell category. You should try to live without it for a week. If you need inspiration, try this article on how to avoid it.

Explanation of stage 4 (Don’t use “if”)

Yes, you didn’t misread. There is a whole campaign that tries to avoid the if-statement altogether. Read their website for inspiration on how to survive this week. Maybe you might make new friends with polymorphism and some other implicit conditional structures. Remember, this is a short week just before christmas. Try it, you might be surprised how easy it looks with hindsight.

Ready, steady, go!

This experiment starts with the first advent at Sunday, 28.11.2010. Every stage lasts for one week and adds up to the previous stages. The experiment ends at christmas.

Good luck! And if you’re done with it, drop us a comment with your experiences.

Bug hunting fun with std::sort

Small errors in custom comparison functions used with std::sort can lead to hard-to-find bugs.

The other day I came across a nice little C++-shoot-yourself-in-the-foot at one of our customers. Let’s see how fast you can spot the problem. The following code crashes with segmentation fault sometime, somewhere in the sort call (line 31).

#include <iostream>
#include <vector>
#include <boost/shared_ptr.hpp>
#include <boost/bind.hpp>

using namespace std;
using namespace boost;

enum SORT_ORDER
{
  SORT_ORDER_ASCENDING,
  SORT_ORDER_DESCENDING
};

bool compareValues(const std::string& valueLeft,
                   const std::string& valueRight,
                   SORT_ORDER order)
{
  const bool compareResult = (valueLeft < valueRight);
  if (order == SORT_ORDER_DESCENDING) {
    return !compareResult;
  }
  return compareResult;
}

int main(int argc, char *argv[])
{
  std::vector<std::string> strValues(300);
  std::fill(strValues.begin(), strValues.end(),
            "Hallo");
  std::sort(strValues.begin(), strValues.end(),
            bind(compareValues, _1, _2, SORT_ORDER_DESCENDING));
  return EXIT_SUCCESS;
}

Any ideas? The tricky thing about this bug is that the stacktrace output in the debugger gives absolutely no hint at all about its cause. And this is a simplified version of the real code which has to sort boost::shared_ptrs instead of strings. Believe me, you don’t want to see that stacktrace. Because of the use of boost::bind together with boost::shared_ptrs it looks, well, let’s say intimidating.

Still no idea?

I’ll give you a hint. If the SORT_ORDER is set to SORT_ORDER_ASCENDING everything is fine. …

Ok, the problem is that std::sort algorithm must be given a comparison function (object) that defines a strict weak ordering on the elements that are to be sorted. In other words the comparison function object must implement the ‘<‘ (less than) relationship on the elements.

Unfortunately, lines 20 to 22 break this ordering when SORT_ORDER_DESCENDING is given. The initial idea of this code was that, well, if compareResult gets returned on ascending sort order, lets just return the negation of it when the “negation” of acscending order is requested. This, of course, destroys the strict weak ordering requirement because whenever valueLeft == valueRight, the function returns true, meaning instead that valueLeft < valueRight. And this somehow wreaks havoc inside std::sort.

A better version of the function could be:

...
bool compareValues(const std::string& valueLeft,
                   const std::string& valueRight,
                   SORT_ORDER order)
{
  // solution: return false independent of sort order
  // whenever valueLeft == valueRight
  if (valueLeft == valueRight) {
    return false;
  }
  const bool compareResult = (valueLeft < valueRight);
  if (order == SORT_ORDER_DESCENDING) {
    return !compareResult;
  }
  return compareResult;
}
...

The really annoying thing about this whole issue is that std::sort just randomly crashes with a stack trace that shows nothing but some weird memory corruption going on. After the initial shock, this sends you down the complete wrong bug hunting road where you start looking for spots where memory could be overwritten or the like.

So beware of custom comparison functions or function objects. They might look innocent and easy, but they can give you lot’s of headaches.

Follow-up to our Dev Brunch October 2010

A follow-up to our October 2010 Dev Brunch, summarizing the talks and providing bonus material.

Last Sunday , we held our Dev Brunch for October 2010. We gathered inside (no more roof garden sessions for this year) and had a good time with lots of chatter besides the topics listed below.

The Dev Brunch

If you want to know more about the meaning of the term “Dev Brunch” or how we implement it, have a look at the follow-up posting of the brunch in October 2009. We continue to allow presence over topics. Our topics for the brunch were:

  • Beyond Scrum – The first-hand tale of a local team that transformed their process to do Scrum and failed for several reasons. They finally admitted failure and search for alternatives since. Great stories of mistakes you don’t have to make yourself to learn the lessons now. We decided to transform at least some aspects of the whole story in an essay, as it’s too valuable to not be published.
  • Code Camp experiences – We already blogged about it, but this talk gave away more details and more insight from the trainer’s perspective. The speaker guided a two-day developer code camp in the spirit of code retreats with an experienced team and draw several conclusions from the event. In short: It’s well worth the time and you will see your team differently afterwards. Other attendees added their experiences with team games that reveal social structures and behaviour even quicker.
  • Local dev gossip – Yes, this is a rather unusual topic for the offical topic list, but we exchanged so much gossip talk this time that it qualifies as a topic on its own behalf. The best summarization of this topic is that there’s a lot of moving around in the local developer community, at least from our point of view. We look forward to a very exciting next year.

As there was no dev brunch in September (due to several reasons), we needed to talk about the news and rumours of two months at once. And there are a lot of things going on around here in the moment. A great brunch with lots of useful information.

Statement against public fields in Java

Every once in a while I talk to people about coding style and sooner or later there is discussion about public fields and getters/setters in Java. I would like to elaborate my opinion on this issue in addition to other quite well balanced articles to a broader audience.

First I want to differentiate properties of a class from other fields/member variables. Properties are fields, whose values are useful and important to clients of the class. We consciously decide to break encapsulation here and provide this data to our clients. The size of a collection may serve as a nice example. Fields on the other hand store state or dependencies our class needs to be fully operational. Datastructures like arrays, data access objects (DAOs) or some kind of notification service may serve as examples here.

The internal implementation of both, properties and fields, should never be exposed because this truely breaks encapsulation and takes away the freedom of the class implementor to change their implementation. At a later time you may decide to compute a value or read it from a database instead of storing it directly . On the other hand properties themselves may well be public and belong to the API of our class.

Now on to Java. There is no native property support in the Java language as it does not support the uniform access principle using language constructs. In other languages like Python, Ruby, Groovy or Scala you can change from direct field access to accessor methods without changing the clients, so it is no problem to expose fields (or more precisely properties) and thus make them public or protected. To gain the same degree of freedom in Java you have to emulate properties by using the getter/setter convention of Java Beans. You have to trade conciseness of public fields against this freedom and you really should do it. An IDE can generate the accessors and fold the methods away from your sight. The cost of getters/setters is really negligible.

Now we can derive the conclusions for Java programers. With each member variable you introduce you have to decide if it is a property or just some internal field. For properties you may provide getters and/or setters with appropriate visibility when needed. That means you should not provide accessor methods for all of your fields. In general you should never expose fields directly and all instance variables should be private. Not doing so will remove the freedom to change class internals without affecting the clients. Once a class with exposed internals is published as part of an API it is almost impossible to change internal design decisions.

Code Camp Experiences

Experiences gained when performing a two day code camp with a team.

Some weeks ago, I conducted a code camp with a team of twelve developers that build a software product together for years now. The team had already introduced sporadic code reviews (in the team vs. author review style), so the main emphasize of the meeting was to improve team coherence by writing code together while generally having some time off project. In this article, I describe what I had planned, what happened and what the effects are so far.

A plan for the code camp

The code camp was scheduled for two consecutive days when the whole team gathers in one room with one computer for each pair. We would switch pairs (and seats) after each iteration, with one iteration being 45 minutes coding time followed by 2-3 minutes presentation of the achievement to the team. With a recreation break of 20-30 minutes, this means one iteration every two hours.

Every iteration starts from scratch, without access to previous code fragments (see also the code retreat concept). This had several reasons: I wanted the iterations to be comparable. Some of the insights I wanted to share are dependent on directly assimilable experiences. The iterations should also be independent, without ballast from previous sessions.

On the first day, we started with a given code resembling a little puzzle game in Java Swing. The code worked, but had some bugs and was written in an awful manner. It was unknown code for the team with no emotional attachments. The assignment for each iteration was to refactor the code to something equivalently working, but much “better”. How this “better” is defined was up to the teams.

On the second day, we started with a blank editor and had the task to code the same little puzzle game (in Java Swing) we refactored the day before. Even with some practice, it was nearly impossible to finish within time, so concentration on the most important key aspects of the code was crucial. The main lesson here was to “create from scratch” rather than “fix the existing”.

What really happened

Day one

The camp started with the usual delay for setting up all the computers in a uniform manner. This couldn’t be prepared beforehands, as the computers were in use by another group. When we installed our software, we found the hotkey configuration of the whole system severely flawed (for example, Ctrl+1 was defined as “set keyboard layout to traditional chinese”).

To warm up for the first iteration, I presented the existing code and explained its structure in detail. The code was slightly too much to remember it all in one pass, so only a rough understanding remained. Every team had to examine the code again during their work.

After the warm up, the first iteration started, with everybody buzzing over the code. The 45 minutes went by really fast and the first presentations focussed on local improvements. No team had restructured the code in any meaningful way, but every solution was perceived as “better” than the original. One team failed to get their refactored code to work again.

The second iteration held the biggest surprise of the whole camp. The 45 minutes flew by and the presentations showed the difference. One team failed to work on the assignment, but every other team presented a solution that was far superior of the original code. Some teams restructured the code to an extend where the original structure wasn’t recognizable anymore. The distinction between the first and the second iteration was so great that everybody was baffled by what could be achieved in 45 minutes when you do it for the second time.

The third iteration added some interesting twists on the best solutions of the second iteration, but didn’t produce the massive boost in productivity and code quality. Everybody felt worn out afterwards, so we decided to close the coding part of this day.

While working with the Java Swing code, nobody on the team noticed the threading flaws in the code. When I pointed this out, I was asked to explain the mechanics of the Swing threading model. The team develops a web application and hasn’t much exposure to desktop application development, let alone with Java Swing. So we ended the day with a lecture about the EDT, the EventQueue and the SwingWorker.

The whole team strolled to a bar to share some beers afterwards.

Day two

After a short night, we gathered early in the morning to continue the coding part of the camp. I explained the task (develop the game from scratch) and off we went.

The first iteration yielded only very rudimentary results. One team started with an UML diagram of the application structure and had to stop after setting up the outline of every method in code. Most other teams started with the domain model and failed to attach the GUI part of the application. All solutions had similar concepts in mind, no team used test driven development or other “advanced” techniques.

As a result of the poor performance, we decided to change the rules. Instead of scrapping the whole code, every new team could take over the code base of any other team, as long as it wasn’t the own. In the second iteration, we completed the drafted solutions of the previous team. This didn’t work out, too. The teams were frustrated by their lack of results.

We changed the plan again and held a prolonged review discussion of the code camp instead of a third iteration. This was by far the better choice with hindsight.

The effects

The code camp was perceived very positively by the attendees. The main goal of the camp was not about coding, but about team coherence and team focus. We learnt a lot about the personal style and abilities of each team member because the code samples shown in the code review were directly comparable. And we revealed team problems we weren’t aware of yet, but some problems we thought would arise did not. This was a very healthy process, because some of these issues can be addressed directly now.

The side benefit of the camp, as stated by one programmer was the increased awareness that “throwing away your code and starting over isn’t as hurtful as I thought”.

Every attendee stated that they want another code camp soon.

Personal summary

The code camp greatly improved my sense for the team and for the individual team members. By sharing a common code base and performing the same tasks, I could directly see (and feel) their thoughts and abilities. The camp is a powerful way to get to really know a team. If you have to mentor a whole new team, consider performing a code camp to get in touch with them.

Developing Grails Apps – Some Dark Sides

Most of the time, developing Grails apps is a nice experience. But there are also dark sides. One of which is when bugs do appear or do not appear depending on how you started your app.

Usually, I try to avoid it but this time a Disclaimer is in order: This is not a Grails rant. Most of the time developing Grails projects is fast and smooth. Using Grails brought many advantages for us. But there are also dark sides…

My main criticism is that Grails abstractions are more than leaky! In every list of examples for the definition of the term Leaky Abstraction Grails should be top. As soon as you leave the tutorial/scaffolding/helloworld level you have to know a lot about the underlying stack. And with Hibernate and Spring neither of the words small, easy and lightweight do apply.

GORM, too, is only easy to use at first sight. The very informative blog series about GORM gotchas should absolutely become part of the user guide or the refence docs.

And there are those times where it gets really unpleasant. This is e.g. when a bug does appear in your grails application running in a servlet container (packaged in a .war)  but does not appear when the application is started from within the IDE. Our last one of those was a naming conflict in a .gsp file. The controller handed a model like this to the .gsp:

...
return [fieldValue: 'THE_VALUE', ...]

The model entry ‘fieldValue’ was used in the .gsp to set the value of a combo box. Unfortunately, ‘fieldValue’ is also the name of a built-in Grails tag …

Admittedly, ‘fieldValue’ was not the wisest choice of names and I would certainly expect to get scolded loudly by Grails for that – ideally with a nice descriptive exception. But what happend instead led to a loud scolding of Grails from us. And to some big question marks: What is the difference between executing Grails from the IDE and within a servlet container with respect to naming resolution? Why is there a difference, at all?

We had a hard time figuring out this one, not least because the error message was not very telling. And since this was not the first of those works-in-the-IDE-but-not-in-a-real-environment bugs there is always this slightly uneasy feeling…

As I said in the beginning, most of the time developing Grails applications is nice and shiny. I would not support their slogan, though. My personal search for the best web development tool is definitively not over.

How about your search?