Dancing the TANGO

One of our customers is an administration department at a research center, which is responsible to operate and maintain a synchrotron light source. They are in charge of a whole bunch of “normal” IT infrastructure as well as a wide variety of electronic devices which are used in all kinds of experiment settings. These can be cameras, electronic motors, detectors of all sorts, etc. One of their main day-to-day challenges is to integrate all those devices such that they can be controlled in a uniform way with standard measurement and control tools.

In order to provide a common solution to this task the TANGO platform has been developed in a collaborative effort of some the the main European synchrotron institutes. TANGO is an object-oriented distributed control system in which every device is represented in an abstract way by a so-called Device Server. A device server provides access to a given piece of hardware by exposing its attributes, properties, states, events and supported commands in a uniform way. CORBA is used as middleware which shows that it is still popular in real-time and embedded environments. Device server instances are registered at a central database and can be accessed and controlled using a variety of TANGO tools.

The typical TANGO development process is as follows: Each device comes with some vendor provided driver library and corresponding interface documentation (C interfaces in many cases). Starting with that information, all attributes, states and supported commands are defined using a tool called POGO. The resulting model of the device is then used to generate skeleton code for the device server. Right now, POGO supports C++, Java and Python. Then, the device server skeleton code is completed by accessing the actual device using the driver library.

For example, one of our latest projects was an X-ray detector which is roughly like a CCD camera for X-rays. As such it has read-only TANGO attributes Width and Height which corresponds to the width and height of the CCD chip. Furthermore it has a read-write attribute called ROI (region of interest) which is an array of four integer values (X0, Y0, X1, X2), Exposure Time, an integer value in milliseconds and a variety of other attributes. One obvious TANGO command is Start which tells the camera to start exposure and store resulting images.

So, if you happen to have a synchrotron light source in your garage (or of course any other bunch of hardware that you want to integrate), consider dancing the TANGO.

Hudson for C++/CMake/CppUnit Revised

A few months ago, in order to use Hudson as CI for your C++/CMake/CppUnit projects you had to do quite a lot of shell scripting. By now the situation has very much improved as some very useful plugins came into existence. To cover the situation described in my previous post you can now use a combination of the CMake plugin and the CppUnit plugin.

With these extensions Hudson gets more and more useful for C/C++ developers. Yet another new plugin that supports this trend is the CCCC Plugin which uses the CCCC tool to generate trend reports for various software metrics including cyclomatic complexity.

Structuring CppUnit Tests

How to structure cppunit tests in non-trivial software systems so that they can be easily executed selectively during code-compile-test cycle and at the same time are easy to execute as a whole by your continuous integration system.

While unit testing in Java is dominated by JUnit, C++ developers can choose between a variety of frameworks. See here for a comprehensive list. Here you can find a nice comparison of the biggest players in the game.

Being probably one of the oldest frameworks CppUnit sure has some usability issues but is still widely used. It is criticised mostly because you have to do a lot of boilerplate typing to add new tests. In the following I will not repeat how tests can be written in CppUnit as this is described already exhaustively (e.g. here or here). Instead I will concentrate on the task of how to structure CppUnit tests in bigger projects. “Bigger” in this case means at least a few architectually independent parts which are compiled independently, i.e. into different libraries.

Having independently compiled parts in your project means that you want to compile their unit tests independently, too. The goal is then to structure the tests so that they can easily be executed selectively during development time by the programmer and at the same time are easy to execute as a whole during CI time (CI meaning Continuous Integration, of course).

As C++ has no reflection or other meta programming elements like the Java Annotations, things like automatic test discovery and how to add new tests become a whole topic of its own. See the CppUnit cookbook for how to do that with CppUnit . In my projects I only use the TestFactoryRegistry approach because it provides the most automatics in this regard.

Let’s begin with a simplest setup, the Link-Time Trap (see example source code): Test runner and result reporter are setup in the “main” function that is compiled into an executable. The actual unit tests are compiled in separate libraries and are all linked to the executable that contains the main function. While this solution works well for small projects it does not scale. This is simply because every time you change something during the code-compile-test cycle the unit test executable has to be relinked, which can take a considerable amount of time the bigger the project gets. You fall into the Link Time Trap!

The solution I use in many projects is as follows: Like in the simple approach, there is one test main function which is compiled into a test executable. All unit tests are compiled into libraries according to their place in the system architecture. To avoid the Link-Time-Trap, they are not linked to the test executable but instead are automatically discovered and loaded during test execution.

1. Automatic Discovery

Applying a little convention-over-configuration all testing libraries end with the suffix “_tests.so”. The testing main function can then simply walk over the directory tree of the project and find all shared libraries that contain unit test classes.

2. Loading

If a “.._test.so” library has been found, it simply gets loaded using dlopen (under Unix/Linux). When the library is loaded the unit tests are automatically registered with the TestFactoryRegistry.

3. Execution

After all unit test libraries has been found and loaded text execution is the same as in the simple approach above.

Here my enhanced testmain.cpp (see example source code).

#include ... 

using namespace boost::filesystem; 
using namespace std; 

void loadPlugins(const std::string& rootPath) 
  directory_iterator end_itr; 
  for (directory_iterator itr(rootPath); itr != end_itr; ++itr) { 
    if (is_directory(*itr)) {
      string leaf = (*itr).leaf(); 
      if (leaf[0] != '.') { 
    const string fileName = (*itr).string();
    if (fileName.find("_tests.so") == string::npos) { 
    void * handle = 
      dlopen (fileName.c_str(), RTLD_NOW | RTLD_GLOBAL); 
    cout << "Opening : " << fileName.c_str() << endl; 
    if (!handle) { 
      cout << "Error: " << dlerror() << endl; 
      exit (1); 

int main ( int argc, char ** argv ) { 
  string rootPath = "./"; 
  if (argc > 1) { 
    rootPath = static_cast<const char*>(argv[1]); 
  cout << "Loading all test libs under " << rootPath << endl; 
  string runArg = std::string ( "All Tests" ); 
  // get registry 
  CppUnit::TestFactoryRegistry& registry = 
  // Create the event manager and test controller 
  CppUnit::TestResult controller; 

  // Add a listener that collects test result 
  CppUnit::TestResultCollector result; 
  controller.addListener ( &result ); 
  CppUnit::TextUi::TestRunner *runner = 
    new CppUnit::TextUi::TestRunner; 

  std::ofstream xmlout ( "testresultout.xml" ); 
  CppUnit::XmlOutputter xmlOutputter ( &result, xmlout ); 
  CppUnit::TextOutputter consoleOutputter ( &result, std::cout ); 

  runner->addTest ( registry.makeTest() ); 
  runner->run ( controller, runArg.c_str() ); 


  return result.wasSuccessful() ? 0 : 1; 

As you can see the loadPlugins function uses the Boost.Filesystem library to walk over the directory tree.

It also takes a rootPath argument which you can give as parameter when you call the test main executable. This solves our goal stated above. When you want to execute unit tests selectively during development you can give the path of the corresponding testing library as parameter. Like so:

./testmain path/to/specific/testing/library

In your CI environment on the other hand you can execute all tests at once by giving the root path of the project, or the path where all testing libraries have been installed to.

./testmain project/root

CMake Builder Plugin for Hudson

Update: Check out my post introducing the newest version of the plugin.

Today I’m pleased to announce the first version of the cmakebuilder plugin for Hudson. It can be used to build cmake based projects without having to write a shell script (see my previous blog post). Using the scratch-my-own-itch approach I started out implementing only those features that I needed for my cmake projects which are mostly Linux/g++ based so far.

Let’s do a quick walk through the configuration:

1. CMake Path:
If the cmake executable is not in your $PATH variable you can set its path in the global Hudson configuration page.

2. Build Configuration:

To use the cmake builder in your Free-style project, just add “CMake Build” to your build steps. The configuration is pretty straight forward. You just have to set some basic directories and the build type.

cmakebuilder demo config
cmakebuilder demo config

The demo config above results in the following behavior (shell pseudocode):

if $WORKSPACE/build_dir does not exist
   mkdir $WORKSPACE/build_dir
end if

cd $WORKSPACE/build_dir
make install

That’s it. Feedback is very much appreciated!!

Originally the plan was to have the plugin downloadable from the hudson plugins site by now but I still have some publishing problems to overcome. So if you are interested, make sure to check out the plugins site again in a few days. I will also post an update here as soon as the plugin can be downloaded.

Update: After fixing some maven settings I was finally able to publish the plugin. Check it out!

Make friends with your compiler

Suppose you are a C++ programmer on a project and you have the best intentions to write really good code. The one fellow that you better make friends with is your compiler. He will support you in your efforts whenever he can. Unless you don’t let him. One sure way to reject his help is to switch off all compiler warnings. I know it should be well-known by now that compiling at high warning levels is something to do always and anytime but it seems that many people just don’t do it.
Taking g++ as example, high warning levels do not mean just having “-Wall” switched on. Even if its name suggests otherwise, “-Wall” is just the minimum there. If you just spend like 5 minutes or so to look at the man page of g++ you find many many more helpful and reasonable -W… switches. For example (g++-4.3.2):

-Wctor-dtor-privacy: Warn when a class seems unusable because all the constructors or destructors in that class are private, and it has neither friends nor public static member functions.

Cool stuff! Let’s what else is there:

-Woverloaded-virtual: Warn when a function declaration hides virtual functions from a base class. Example:

class Base
virtual void myFunction();

class Subclass : public Base
void myFunction() const;

I would certainly like to be warned about that, but may be that’s just me.

-Weffc++: Warn about violations of the following style guidelines from Scott Meyers’ Effective C++ book

This is certainly one of the most “effective” weapons in your fight for bug-free software. It causes the compiler to spit out warnings about code issues that can lead to subtle and hard-to-find bugs but also about things that are considered good programming practice.

So suppose you read the g++ man page, you enabled all warning switches additional to “-Wall” that seem reasonable to you and you plan to compile your project cleanly without warnings. Unfortunately, chances are quite high that your effort will be instantly thwarted by third-party libraries that your code includes. Because even if your code is clean and shiny, header files of third-pary library may be not. Especially with “-Weffc++” this could result in a massive amount of warning messages in code that you have no control of. Even with the otherwise powerful, easy-to-use and supposedly mature Qt library you run into that problem. Compiling code that includes Qt headers like <QComboBox>with “-Weffc++” switched on is just unbearable.

Leaving aside the fact that my confidence in Qt has declined considerably since I noticed this, the question remains what to do ignore the shortcomings of other peoples code. With GCC you can for example add pragmas around offending includes as desribed here. Or you can create wrapper headers for third-party includes that contain

#pragma GCC system_header

AFAIK Microsoft’s compilers have similar pragmas:

#pragma warning(push, 1)
#pragma warning(pop)

Warning switches are a powerful tool to increase your code quality. You just have to use them!

Industry Standard C++

The other day I was browsing through the C++ API code of a third-party library. I was not much surprised to see stuff like

#define MAX(a, b) ( (a) >= (b) ? (a) : (b))
#define MIN(a, b) ( (a) <= (b) ? (a) : (b))

because despite the fact that std::min, std::max together with the rest of the C++ standard library is around for quite a while now, you still come across old fashioned code like above frequently. But things got worse:

#define FALSE 0
#define TRUE 1

and later:

bool someVariable = TRUE;

As if they learned only half the story about the bool type in C++. But there was more to come:

class ListItem
   ListItem* next;
   ListItem* previous;

class List : private ListItem

Yes, that’s right, the API guys created their own linked-list implementation. And a pretty weird one, too, mixing templates with void* pointers to hold the contents. Now, why on earth would you do that when you could just use std::list or std::vector? Makes you wonder about the quality of the rest of the code. Especially with C++ where there are so many little pitfalls and details which can burn you. Hey, if you have no clue about the very basics of a language, leave it alone!

Unfortunately, the above example is not exceptional in industry software. It seems that the C++ world these days is actually split into two worlds. In one, people like Andrei Alexandrescu write great books about Modern C++ design, Scott Meyers gives talks about Effective C++ and the boost guys introduce the next library using even more creative operator overloading that in the spirit library (which is pretty cool stuff, btw).

In the other world, you could easily call it industry reality, people barely know the STL, don’t use templates at all, or fall for misleading and dangerous c++ features like the throw() clause in method signatures. Or they ban certain c++ features because they are supposedly not easy to understand for the new guy on the project or are less readable in general. Take for example the Google C++ Style Guide. They don’t even allow exceptions, or the use of std::auto_ptr. Their take on the boost library is that “some of the libraries encourage … an excessively “functional” style of programming”. What exactly is bad about piece of functional programming used as the right tool in the right place? And what communicates ownership issues better than e.g. returning a heap allocated object using a std::auto_ptr?

The no-exceptions rule is also only partly understandable. Sure enough, exceptions increase code complexity in C++ more than in other languages (read Items 18 and 19 of Herb Sutter’s Exceptional C++ as an eye-opener. Or look here). But IMHO their advantages still outweigh their downsides.

With the upcoming new C++0X standard my guess is that the situation will not get any better, to put it mildly. Most likely, things like type inference with the new auto keyword will sell big because they save typing effort. Same thing with the long overdue feature of constructor delegation. But why would people who find functional programming less readable start to use lambda functions? As little known as the explicit keyword is now, how many people will know about or actually use the new “= delete” keyword, let alone “= default“? Maybe I’m a little too pessimistic here but I will certainly put a mark in my calender on the day I encounter the first concept definition in some piece of industry C++ software.

Update: Concepts have been removed from C++0X so that mark in my calender will not come any time soon…

Observer/Listener structures in C++ with boost’s smart pointers

Whenever you are developing sufficiently large complex programs in languages like C++ or Java you have to deal with memory issues. This holds true especially when your program is supposed to run 24/7 or close to that. Because these kinds of issues can be hard to get right Java has this nice little helper, the garbage collector. But as Java solves all memory problems, or maybe not? points out, you can still easily shoot yourself in foot or even blow your whole leg away.  One of the problems stated there is that memory leaks can easily occur due to incorrect listener relations. Whenever a listener is not removed properly, which is either a large object itself or has references to such objects,  it’s only a matter of time until your program dies with “OutOfMemoryError” as its last words.  One of the proposed solutions is to use Java weak pointers for listener management.  Let’s see how this translates to C++.

Observer/listener management in C++ is often done using pointers to listener objects. Pointers are pretty weak by default. They can be :

  • null
  • pointing to a valid object
  • pointing to an invalid memory address

In listener relationships especially the latter can be a problem. For example, simple listener management could look like this:

   class SimpleListenerManagement
      void addListener(MyListener* listener);
      void removeListener(MyListener* listener);
      void notifyListeners();
      std::list<MyListener*> listeners_;

   void SimpleListenerManagement::notifyListeners()
      // call notify on all listeners
      for (std::list<MyListener*>::iterator iter = listeners_.begin();
          iter != listeners_.end();
         (*iter)->notify(); // may be a bad idea!

In notifyListeners(), the pointer is used trusting that it still points to a valid object. But if it doesn’t, for instance because the object was deleted but the client forgot to removed it from the listener management, well, too bad.

Obviously, the situation would be much better if we didn’t use raw pointers but some kind of wrapper objects instead.  A first improvement would be to use boost::shared_ptr in the listener management:

   typedef boost::shared_ptr<MyListener> MyListenerPtr;

   class SimpleListenerManagement
      void addListener(MyListenerPtr listener);
      void removeListener(MyListenerPtr listener);
      void notifyListeners();
      std::list<MyListenerPtr> listeners_;

Provided that the given MyListenerPtr instance was created correctly by the client we can be sure now that all listeners exist when we call notify() on them.  Seems much better now. But wait! Using boost::shared_ptr, we now hold  strong references in our listeners list and are therefore kind of in the same situation as described in the post mentioned above. If the client forgets to remove its MyListenerPtr instance it never gets deleted and may be in a invalid state next time notify() is called.

A solution that works well in most cases is to use boost::weak_ptr to hold the listeners. If you see boost::shared_ptr on a level with normal Java references, boost::weak_ptrs are roughly the same as Java’ s weak references. Our listener management class would then look like this:

   typedef boost::shared_ptr<MyListener> MyListenerPtr;
   typedef boost::weak_ptr<MyListener> MyListenerWeakPtr;

   class SimpleListenerManagement
      void addListener(MyListenerPtr listener);
      void removeListener(MyListenerPtr listener);
      void notifyListeners();
      std::list<MyListenerWeakPtr> listeners_; // using weak_ptr

Note that addListener and removeListener still use MyListenerPtr as parameter. This ensures that the client provides valid listener objects.  The interesting stuff happens in notifyListeners():

   void SimpleListenerManagement::notifyListeners()
      std::list<MyListenerWeakPtr>::iterator iter = listeners_.begin();
      while(iter != listeners_.end())
         if ((*iter).expired())
            iter = listeners_.erase(iter);
            MyListenerPtr listener = (*iter).lock(); // create a shared_ptr from the weak_ptr

Each weak_ptr can now be checked if its object still exists before using it. If the weak_ptr is expired, it can simply be removed from the listeners list. With this implementation the removeListener method becomes optional and can as well be omitted. The client only has to make sure that the shared_ptr holding the listener gets deleted somehow.

Award your Customer

Recently, we successfully finished a web app project that had many specialties we never had before. Major issues were very tight budget and time constraints (about 3 months) including an absolutely unpostponable deadline. However, the bigger concern for us was the diversity of our customer. Although we had one or two main reference persons, for the project to be successful we depended on the collaboration of a total of 8 departments.

As a first step to meet those challenges we decided on one-week iteration cycles – the shortest ever for us. At the kick-off meeting, where delegates of all departments were assembled, we presented our strategy and tried to make clear that communication and collaboration would be essential for the project to succeed. We also invited everyone to come to iteration meetings even when the agenda is not exactly about their specific requirements. After the meeting we hoped for the best.

With (almost) all departments it went like this: We did one requirements gathering appointment with one or two delegates and they either showed up once or twice on following meetings or they approved our implementation based on emailed screen shots. With most departments, email response time was good, with some, well, let’s just say holiday season didn’t really help. But altogether it was sufficient to keep the project well on track.

But wait! Did I say all departments? Not exactly! One single department actually managed it to sent at least one delegate to every single iteration meeting. And they not only enjoyed coffee and cookies but contributed a great deal every time. This was very helpful for us especially because after every iteration, we were a little bit more confident that we were still on the right track. Towards the end of the project, when success was foreseeable, we had the idea that their outstanding performance had to be rewarded somehow. So at the last iteration meeting, again with people from every department, we presented them with the Continuous Collaboration Award. ccaward They were very delighted and for the others it was a good laugh. And with the help of a little champagne and some snacks it became a very nice last iteration meeting.

As many of you know, good understanding between customer and developer can never be taken for granted. This is why agile methods always put great emphasis on extensive customer communication. A-Story-of-Project-Failure-Mitch-Lacey shows that even agile-by-the-book projects can fail basically due to lack of understanding on customer side. So do it like us and, if they deserve it, show your appreciation to your customer once in a while in a more creative way. And if you use a cup, make sure that there is also champagne around to fill it.

Using Hudson for C++/CMake/CppUnit

Update: Hudson for C++/CMake/CppUnit Revised

As a follow-up to Using grails projects in Hudson, here is another not-so-standard usage of Hudson: C++ projects with CMake and CppUnit. Let’s see how that works out.

As long as you have Java/Ant/JUnit based projects, a fine tool that it is, configuration of Hudson is pretty straight forward. But if you have a C++ project with CMake as build system and CppUnit for your unit testing, you have to dig a little deeper. Fortunately, Hudson provides the possibility to execute arbitrary shell commands. So in order to build the project and execute the tests, we can simply put a shell script to work:

   # define build and installation directories

   # we want to have a clean build
   rm -Rf $BUILD_DIR
   mkdir $BUILD_DIR
   cd $BUILD_DIR

   # initializing the build system

   # fire-up the compiler
   make install

Environment variable WORKSPACE is defined by Hudson. Other useful variables are e.g. BUILD_NUMBER, BUILD_TAG and CVS_BRANCH.

But what about those unit tests? Hudson understands JUnit test result files out-of-the-box. So all we have to do is make CppUnit spit out an xml report and then translate it to JUnit form. To help us with that, we need a little xslt transformation. But first, let’s see how we can make CppUnit generate xml results (a little simplified):

#include <cppunit/necessary/CppUnitIncludes/>

using namespace std;
using namespace CppUnit;

int main(int argc, char** argv)
   TestResult    controller;
   TestResultCollector result;

   CppUnit::TextUi::TestRunner runner;
   runner.addTest( TestFactoryRegistry::getRegistry().makeTest() );

   // important stuff happens next
   ofstream xmlFileOut("cpptestresults.xml");
   XmlOutputter xmlOut(&result, xmlFileOut);

The assumption here is that your unit tests are built into libraries that are linked with the main function above. To execute the unit tests we add the following to out shell script:

   export PATH=$INSTALL_DIR/bin:$PATH

   # call the cppunit executable

This results in CppUnit generating file $WORKSPACE/cpptestresults.xml. Now, with the help of a little program called xsltproc and the following little piece of XSLT code, we can translate cpptestresults.xml to testresults.xml in JUnit format.

 <?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="xml" indent="yes"/>
    <xsl:template match="/">
            <xsl:attribute name="errors"><xsl:value-of select="TestRun/Statistics/Errors"/></xsl:attribute>
            <xsl:attribute name="failures">
                <xsl:value-of select="TestRun/Statistics/Failures"/>
            <xsl:attribute name="tests">
                <xsl:value-of select="TestRun/Statistics/Tests"/>
            <xsl:attribute name="name">from cppunit</xsl:attribute>
    <xsl:template match="/TestRun/SuccessfulTests/Test">
            <xsl:attribute name="classname" ><xsl:value-of select="substring-before(Name, '::')"/></xsl:attribute>
            <xsl:attribute name="name"><xsl:value-of select="substring-after(Name, '::')"/></xsl:attribute>
    <xsl:template match="/TestRun/FailedTests/FailedTest">
            <xsl:attribute name="classname" ><xsl:value-of select="substring-before(Name, '::')"/></xsl:attribute>
            <xsl:attribute name="name"><xsl:value-of select="substring-after(Name, '::')"/></xsl:attribute>
                <xsl:attribute name="message">
                    <xsl:value-of select=" normalize-space(Message)"/>
                <xsl:attribute name="type">
                    <xsl:value-of select="FailureType"/>
                <xsl:value-of select="Message"/>
                File:<xsl:value-of select="Location/File"/>
                Line:<xsl:value-of select="Location/Line"/>
    <xsl:template match="text()|@*"/>

The following call goes into our shell script:

xsltproc cppunit2junit.xsl $WORKSPACE/cpptestresults.xml > $WORKSPACE/testresults.xml

In the configuration page we can now check “Display JUnit test results” and give testresults.xml as result file. As a last step, we can package everything in $WORKSPACE/install_dir into a .tgz file and have Hudson to store it as build artifact. That’s it!

As always, there is room for improvements. One would be to wrap the shell script code above in a separate bash script and have Hudson simply call that script. The only advantage of the approach above is that you can see what’s going on directly on the configuration page. If your project is bigger, you might have more than one CppUnit executable. In this case, you can for example generate all testresult.xml files into a separate directory and tell Hudson to take into account all .xml files there.

Update: For the CMake related part of the above shell script I recently published the first version of a cmakebuilder plugin for Hudson. Check out my corresponding blog post.