Category: 2010

Open Source Love Day May 2010

Our Open Source Love Day for May 2010 brought love for Netty, Launch4j and RXTX. Everything went smooth again this time.

Last friday, we held our Open Source Love Day for May 2010. After the excessive yak shaving of last month’s OSLD, this one was more productive. We got one feature right nearly instantly and several others throughout the day. To keep us from feeling depressed, we allocated dedicated “research time” to get to know a project before starting to develop on it. So one point on our current list of accomplishments is only research work that should be counted as “thought investment” for further improvements.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

  • Netty is a java framework to perform network communication, transparently using different transport mechanisms. We use it in a customer project environment and it worked mostly well. But we noticed some issues that might have to do with the framework, so we decided to put it under development. But before hopping in right away, we invested some hours to really anatomize it. Our achievement isn’t noticeable yet, but might be in the future.
  • Launch4j is a java application launcher for Windows, handling all the stuff a startup script (the infamous start.bat) would do, too. One thing that’s easy to perform in a batch script is to restart the application if it went down with an error. We wanted this feature in Launch4j and we succeeded. Before releasing the changes to the project, we need some rethinking of our approach, as it is very special to our case, but if you are interested, we send you the patch on request.
  • RXTX is a project to perform serial (RS232) communication. Over the course of the last OSLDs, we worked on an issue that appears in combination with serial converters. See the OSLD posting for march 2010 for details. This issue seems to be completely fixed now. We will run some longterm stability tests with it and then release a patch to the issue tracker of RXTX, issue #144.
  • Our internal tool for business administration was greatly improved this time. The tool isn’t open sourced yet, but we are not afraid to publish it in the future. The new features will directly decrease our administrative overhead and give us more time to perform our main duty – developing software.

A sidenote regarding the yaks (see last month’s OSLD posting to understand this one)

Our monthly “Homepage Comittee” meeting went smooth this time. The setup pain of last month is now paying out and we got our homepage updated. It seems the yaks haven’t grown beards again yet.

What were our lessons learnt today?

  • Pure C code tends to produce endless code blocks. Simple refactorings like extract method/function take a long time and aren’t supported by the IDE. The sentence “there is no boolean!” left some of us impressed, too.
  • The C/C++ Eclipse workbench (in fact, a bunch of plugins for Eclipse) was a joke instead of a help. It didn’t exactly prevent us from doing our work, but that’s all it did. Launch4j came along with project files for the Bloodshed Dev-C++ IDE, which – besides its bloody name – seems to be up to its tasks. We decided to use Dev-C++ over Eclipse for development on Launch4j.
  • Hpricot is a very versatile HTML parser/manipulator. It is just pure joy to work with. (Ok, we knew that one before)

After the fruitless OSLD of april, this one was a great relief for us. The new restart feature for Launch4j worked out instantly and will solve many of our current immediate problems (and some of yours, hopefully). The solution for RXTX got us out of trouble, too. All in all, this was a successful OSLD.

Follow-up to our Dev Brunch May 2010

A follow-up to our May 2010 Dev Brunch, summarizing the talks and providing bonus material.

Last sunday, we held our Dev Brunch for May 2010. It was a small group of developers brunching together this time. One reason was a communication failure on my part, as a crucial email announcing a change in the appointment didn’t reach everyone interested in participating. Sorry for this one again! The other reason was more of a pleasure: Two of our regular brunch attendees are turning into parents. But a core group brunched and talked in the office roof garden, discussing the topics listed below.

The Dev Brunch

If you want to know more about the meaning of the term “Dev Brunch” or how we implement it, have a look at the follow-up posting of the brunch in October 2009. We continue to allow presence over topics. Our topics for the brunch were:

  • Options in Scala – If you are used to traditional programming languages like Java, you might be surprised that you can’t nullify a reference in Scala. There is no concept of “null” in Scala, therefore avoiding the so-called billion dollar mistake (you might want to listen to the guy who invented the mistake, too). If you want to use “uncertain” references, you should look into the Option type of Scala, which was the main topic of this talk. With some code examples, this was a decent introduction into the concept.
  • Summary of the Sensor+Test trade fair – This talk was a short report of the recently visited german Sensor+Test measurement fair in Nuremburg. Details may follow on this blog, but the overall summary is that the measurement industry in Germany and neighbours is mostly self-confident and down-to-earth.
  • The Modbus protocol – This talk was a short introduction to the ancient (but still useful) Modbus protocol and the possibilities to access it in Java. There are at least two projects that provide full coverage of the protocol: jamod and modbus4j. The protocol itself is rather low-level, but sufficient for simple control and data query tasks on an embedded device. The possibility to be mostly agnostic over the physical transport layer is a strength of the Modbus protocol and its implementations.
  • Usage patterns for mocks – Most of us had to confess: we are mockists. But there are many different flavors of using mocks in tests. Inspired by the book “Growing Object-Oriented Software, Guided by Tests”, this talk was an open discussion round about personal mock usage preferences.

As usual, the topics ranged from first-hand experiences and impressions to literature reviews and research. For additional information provided by the talk authors, check out the comment section (or leave a comment to request further content). Comments and resources might be in german language.

Retrospection of the brunch

The usual brunch setup is nearly perfect. What was lacking this time was the coordination of the appointment. To further improve on that point, we introduced a new mailing list, containing everone that is currently interested in participating the brunch.

Improved Version of CMake Builder for Hudson

Introducing version 1.5 of cmake builder plugin for Hudson.

Today I just want to give a small round-up of the improvements made on the cmake builder plugin since my last blog post. Back then, version 1.2 was released to support master/slave configurations. As of yesterday, we are at version 1.5 which contains the following improvements/bug-fixes:

  • Bug: The drop-down box for selecting the build type didn’t remember its value. This was fixed with a patch by Atte Timonen.
  • Improvement: Also included in Atte’s patch was the propagation of environment variables to the cmake command which now allows to do parameterized builds. A big thank’s to Atte!
  • Improvement: The install command gets only executed when install directory and install command is given. Before, the build was either broken or $WORKSPACE was used automatically as install directory. Thanks to Dat Chu for his feedback.
  • Improvement: The one-line ‘Other CMake Arguments’ field can get full pretty quickly, so it was changed to a multi-line text-area.

Thank’s again for the feedback, and have fun with the new version!

Step-by-step tutorials and manuals are priceless

Writing documentation in and outside of code annoys most developers. I am no different but after several years in the software development business I have come to value one type of documentation a lot: step-by-step tutorials and manuals. Surprisingly, they are often missing. I want to depict some common use cases where such documents have proven valueable many times:

when learning new tools and frameworks

Most people I know learn best by example. A tutorial with step-by-step explanation of how to setup the tool and then some good examples brings you up to speed quickly. The Lift framework has such a thing and gets you a webapp running in minutes. But when you dive deeper you may find much documentation and how-tos are missing. Books may help but most of the time it is faster and easier to look for examples on the web and in project wikis.

Also beware, that bad examples may cause a lot of frustration (typos are poison for newbies) and teach bad style or obsolete techniques. Keep the tutorials up-to-date (I know, thats one problem of documentation….) and as accurate as possible and many people will have a better time working with the tools and frameworks.

The same is true for developers in your team: Many will look for examples in the existing code base and learn from them be they good or bad. So keep the code clean and full of good examples. Let your seniors spread them in the teams. Do not forget to provide good examples and best practices in your wiki or other documentation system.

when performing manual tasks

We often have to manage and deploy production systems for our customers. Even though many tasks are automated by scripts and other tools you sometimes need to perform manual tasks. Especially in an stress situation at customer site or when otherwise dealing with production systems it is extremely helpful to have a simple and precise guide for the task at hand. There are enough unknowns and problems that may need brain power so the basic tasks and procedures should be well documented. That way you have one less thing to worry about and are ready to face potential upcoming problems. A good guide gives you much needed confidence.

when trying to work on an open source project

We regularily (see our OSLDs) dive into some open source project to improve it. Nothing is more frustrating than spending more than half a day with setting up the development environment. Overboarding dependencies, works-on-my-machine-style build scripts and missing documentation and tutorials prevent a productive experience. That hurts the themselves projects by driving away potential contributors. We had several such experiences but also many positive ones like Hudson plugin development or EGit where you can get up to work in minutes and perform your first monkey see – monkey do experiments.

Conclusion

Much documentation generated or written nowadays is not worth 2 cents. API-Docs which list the classes, methods and fields and no additional info and descriptions provide no use and just waste the time of people trying to find information there. Highlevel blabla or theoretical dissertations are all nice but do not help much getting the job done (at least not in the beginning). But small guides and tutorials written to the point really do make a difference regardless if they are written for inhouse work and development or for openly available projects. Choose the kind of documentation you write wisely.

Responsibility reduces waste

Most of the waste comes from being irresponsible. Over time waste becomes even harder to remove. So be responsible now.

Recently we participated in a local effort (site in German) to help making our environment cleaner by removing waste which was left by other people. When you take a look at your environment you may come to the conclusion that many people are irresponsible. Waste on the streets and in the parks, prohibition signs everywhere which name things and actions you wouldn’t even think of and when did people forget to flush the toilet?. And it doesn’t stop in the material world, you even find waste in your code. Allowing waste and not removing or preventing it leads to two effects:

  • even more waste (according to the broken window theory)
  • over time the waste becomes more and more intertwined with the environment

Imagine a plastic cup in a forest: When first thrown there it is clearly distinguishable from the mud, the leaves and its surroundings. Easy to see and easy to remove. But over time it is trampled over, crushed, hidden under leaves, wash over with mud, … in the end you may not even spot it when you look at the place where it was left.
The same happens to your code: You start with a small clearly defined part of bad smelly code and leave it alone. Now the first additional features come in, you add code, there and elsewhere. The surroundings change. You refactor. You move code. And in the end the once good known waste is littered all over your code and hard to remove.
So be responsible now! And don’t wait until the waste is hard or impossible to remove. Collective ownership (of code or of your environment) does not mean nobody is repsonsible, it means you are responsible.

Keeping it simple is hard indeed

A true story I was reminded of when I read Chad LaVigne’s essay on simplicity from the book “97 Things Every Software Architect Should Know”.

When I read the great book (or collection of essays) “97 Things Every Software Architect Should Know”, there was a story I had experienced first-hand for nearly every chapter. But it was only after I read about it in the well-placed words of somebody with greater in-depth experience than me that it appeared clear to me. The essence of my own experience was written down so I could iterate over it. Here is a story about simplicity that I hope will help somebody out there iterate over the same thought sometimes.

The problem

In the early days, we were asked to provide custom software for a nearly robot-like machine that performs measurements. Our software had to control some engines that could move sensors and actuators around. The whole machine was built by another company that only had eyes for the mechanical and electrical aspects. As a result, the communication protocol between the microcontrollers of the machine and our software was horribly awkward and unpleasant in regard of the software engineering side of the project.

One engine was the so-called “vertical engine”, because it could move an array of sensors up and down. There were four actual positions where the engine would stop automatically upon contact. It was the job of our software to send the correct commands in the correct order at the correct time to reach the destination. Effectively, we would “hop” from position to position, either moving the sensors up or down. If only the communication protocol would’ve allowed that. What we got was two commands: moving one position down and moving up to the topmost position. The movement options are depicted here:

The first solution

But what would a developer’s life be without a few challenges? Our team quickly came up with several possible solutions, from finite state machines to object graphs of “position” instances that offered several “movement options” to an agent-like object travelling inside the graph. In the brainstorming session, even more sophisticated possibilities were discussed. The solution we finally implemented was complex in terms of clarity. We needed quite a few unit and integration tests to proof the whole thing correct. During all this process, nobody ever questioned the complexity of the problem.

The essay

The chapter I read in the book “97 Things Every Software Architect Should Know” when this story came back to me was “Make Sure the Simple Stuff Is Simple” written by Chad LaVigne. Let me cite some sentences to show the message of his essay:

“People who design software are smart – really smart. The simple problem-complex solution trap can be an easy one to fall into because we like to demonstrate our knowledge. If you find yourself designing a solution so clever that it may become self-aware, stop and think. Does the solution fit the problem?”

The essay hit the nail on the head for me. Our solutions for this “vertical engine” were all clever and solved the problem, but a much more complex problem than the one that was really needed.

The second solution

Our first version of the software went into production and worked a treat. Years later, the hardware (specifically the microcontrollers) fell apart and was replaced by more sophisticated electronics. We needed to rewrite parts of our software, especially the engine controls. But this time, we had real movement patterns of all engines, collected in the logfiles. Some regular expression magic later (remember, we are smart people and like to show that!), it was clear that the vertical engine had never used two of the possible movement options. The real usage pattern of the engine is depicted in the picture below:

The real problem we should have solved from the beginning is much simpler than the original one. Every engine position has only one successor, there is no need to choose from several options or to calculate the shortest path to the destination. It’s the most boring finite state machine you’ve ever seen and there is no need for recursive structures.

It was our own cleverness that made the problem appear more complex than it really was, just because we could handle it. Some investigation on the problem domain instead of the solution domain would have brought us to the same conclusion that the log file analysis finally revealed.

The lesson learnt

The whole story isn’t about failure. Both solutions worked for the customer. In fact, he never noticed a difference when the second solution was going live. This story is about accidental complexity and about being smarter than the requirement. It’s in our nature to accept a problem as “given” and come up with an elegant solution. It might be much more effective to question the problem instead.

Thank you, Chad, for your great essay!

Open Source Love Day April 2010

Our Open Source Love Day for April 2010 didn’t yield the desired results. Instead, we went to give our yaks a shave.

Today, we held our Open Source Love Day for April 2010. Well, to be honest, we tried to. We didn’t achieve that much to present publically and felt really depressed at the end of the day. At least, we can try to explain what went wrong. It all has to do with our own breed of yaks that needed to be shaved today. But first, lets explain the basics.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

Today, we reached out to perform the following tasks:

  • RXTX is a fine java library to access the (rather legacy) serial port to send and receive data over the RS232. On our last Open Source Love Day, we proposed a patch for a problem with disappearing emulated COM ports. This happens a lot if you use USB or ethernet converters and unplug them accidentally or because your intranet got shut down. The patch works well, but not perfectly. We noticed that the IOException we introduced gets thrown much too often. Today, we wanted to fix our fix. We only came to the conclusion that the problem might be a lot more complex than we initially thought.
  • Launch4j is another java library/tool we use to deliver production-ready software bundles. It’s a launcher that increases the user experience at the critical “first contact” for Windows users. The project is mature and useful, but out of development for nearly two years now. We want some features in it, so we thought we might build them today. But the yaks didn’t let us.
  • We have an internal tool for our business administration that desperately needs some hugs. So even if this isn’t strictly Open Source yet, we allocated some time to add a few features. But they outgrew their timebox, too.

At the end of the day, we didn’t accomplish anything ready to be released. That’s frustrating to us and has no value to anybody else. We promise to do better next time.

How the yaks got shaved

Every organization has a secret stable where yaks are raised just for the sake of raising yaks. Once a year or so, they need a shave, so you’ll find yourself shaving yaks before you know what’s happening. That’s what happened to us today.

The term “yak shaving” isn’t our invention. It describes the result of a process when in order to accomplish task A, you need to do task B first. Task B can’t be done before tasks C and D both are completed. Task D, you figure it already, is dependent on task E, which in turn… Soon and inevitably, you’ll find yourself shaving yaks because it is what needs to be done to accomplish your todo list.

This is our story: On every Open Source Love Day, we also gather to have a short “Homepage Comittee” meeting. This is our formal approval process to publish content on our website. One improvement we wanted to implement today was an automatic publishing process. It copies the content of our internal Content Management System to our webserver. Simple as that. This was the starting point of an odyssee.

To copy the content, we used wget. But wget in the version of our distribution had a bug (and missed a feature), so we needed to update wget… Soon afterwards, we found ourselves upgrading our main PostgreSQL database (which went incredibly smooth but took forever) and wrestling with certain dependencies of other parts of the system. We realized we are now officially shaving yaks when we got the error message:

You need to have ruby gems 1.3.2 installed. Your version is 1.3.1!

Remember we started with taking a snapshot of our CMS using wget. Clearly, separating our services to more machines would have helped a lot. But that would’ve taken more than one day to do. It’s on our wish list for the next Admin Day.

Let’s conclude this blog post with the insight that we need to timebox our work better when working for the OSLD. And our yaks are shaved and ready for the summertime.

Wrestling with Qt’s Model/View API – Filtering in Tree Models

Qt4’s model/view API can be kind of a challenge sometimes. Well, prepare for a even harder fight when sorting and filtering come into play.

As I described in one of my last posts, Qt4’s model/view API can be kind of a challenge sometimes. Well, prepare for a even harder fight when sorting and filtering come into play.

Let’s say you finally managed to get the data into your model and to provide correct implementations of the required methods in order for the attached views to display it properly. One of your next assignments after that is very likely something like implementing some kind of sorting and filtering of the model data. Qt provides a simple-at-first-sight proxy architecture for this with API class QSortFilterProxyModel as main ingredient.

Small preliminary side note: Last time I checked it was good OO practice to have only one responsibility for a given class. And wasn’t that even more important for good API design? Well, let’s not distract us with such minor details.

With my model implementation, none of the standard filtering mechanisms, like setting a filter regexp, were applicable, so I had to override method

QVariant filterAcceptsRow ( int source_row, const QModelIndex& sourceParent ) const

in order to make it work. Well, the rows disappeared as they should, but unfortunately so did all the columns except the first one. So what to do now? One small part of the documentation of QSortFilterProxyModel made me a little uneasy:

“… This simple proxy mechanism may need to be overridden for source models with more complex behavior; for example, if the source model provides a custom hasChildren() implementation you should also provide one in the proxy model.”

What on earth should I do with that? “… may need to be overridden“? “… for example.. hasChildren()…” Why can’t they just say clearly what methods must be overridden in which cases???

After a lot more trial and error I found that for whatever reason,

int columnCount ( const QModelIndex& parent ) const

had to be overridden in order for the columns to reappear. The implementation looks like what I had thought the proxy model would do already:

int MyFilter::columnCount ( const QModelIndex& parent ) const
{
   return sourceModel()->columnCount(parent);
}

So beware of QSortFilterProxyModel! It’s not as easy to use as it looks, and with that kind of fuzzy documentation it is even harder.

Follow-up to our Dev Brunch April 2010

A follow-up to our April 2010 Dev Brunch, summarizing the talks and providing bonus material.

Last sunday, we held our Dev Brunch for April 2010. It was the start of the open air season, introducing our new office roof garden. We brunched under a clear, sunny sky (no clouds, even no vapor trails from airplanes, as they aren’t allowed to fly because of some distant volcano ash emission) and talked about agile processes and books. A major part of the brunch was graded as “informal chatter”, just as it should be like.

The Dev Brunch

If you want to know more about the meaning of the term “Dev Brunch” or how we implement it, have a look at the follow-up posting of the brunch in October 2009. We continue to allow presence over topics. Our topics for the brunch were:

  • Agile life planning – Your software development daily routine is to incrementally build software from a backlog and release it after a fixed timebox. What might happen if you transfer this process to your private life (often called “real life”)? You’ll release a week’s life every week and have a week planning meeting on sunday, filling your week/sprint backlog with the most important items of your life’s backlog. Telling from this first hand experience of about a year, it works exceptionally well, improving the leisure time quality and making “progress” visible even at harder times. The only remaining question is who acts as the product owner.
  • Converting to Scrum – First hand experience of a team that, after years of “alarm call”-style development, successfully implemented Scrum as their primary process. The effects were at least less overtime, improved progress tracking, improved code quality (though Scrum is  only a management process!) and less pressure in the project. The customer even adjusted their wording when talking about new features: “next sprint” instead of “immediately, now”. The implemented process isn’t vanilla Scrum, but works nevertheless.
  • Defending Continuous Integration – What if, after an initial phase of excitement over the new tool, the continuous integration server really reveals flaws in your project? There seems to be the tendency to kill the messenger: Shut down the CI server and everything’s fine again. This talk was about the reasoning of both sides and some basic insights gained about machine sharing. Tip of the talk: Reserve your CI server exclusively for this task.
  • Book recommendations – A random collection of technical books read in the last half year, presented with a short summary and personal rating. Titles included were Coders At Work, Founders At Work, 97 Things Every Software Architect Should Know, 97 Things Every Project Manager Should Know, 97 Things Every Programmer Should Know, Hiring the Best Knowledge Workers, Techies & Nerds and Manage Your Project Portfolio. Yes, there are some patterns visible in the book titles. And it’s a good idea to keep some checklist of read articles for most of these books.
  • Dreyfus Model of Skill Acquisition – The Dreyfus Model differentiates five to six different stages of learning some arbitrary skill. It makes assumptions how the members of the stages work, how they process feedback information and what they need to get better. See the comment section for more information and bonus material.

As usual, the topics ranged from first-hand experiences to literature research. For additional information, check out the comment sections. Comments and resources might be in german language.

Retropection of the brunch

Holding the brunch in the bright morning sun, surrounded by rooftops and birds, really is a huge gain for the ambience factor. We even found a solution to produce fresh coffee up there. This will be a fun summer for the Dev Brunch.

About PrintStream and Exceptions

Several of our projects deal with sensor hardware of different types often connected via the good old™ serial port. That is fine most of the time because most protocols are simple and RXTX provides a nice cross-platform library for most of your serial port needs. But many new computers do not feature the old RS232 serial ports anymore or other contraints prevent the use of a plain RS232 serial port. Here come serial converters like the Advantech ADAM 4570 (serial-to ethernet) or usb-to-serial converters into play. Usually this works fine.

Now one of our customers had a test system using an unreliable converter with sensor hardware. The hardware problems uncovered a robustness issue in our software which crashed the JVM when the virtual serial port of the converter disappeared and our app tried to write to it. Despite the faulty hardware our software had to be robust because it manages many more devices other than just that one sensor over serial. Looking at the problem we discovered that the crash occurred somewhere in the native part of RXTX. So we decided to scratch our own itch (and the one of the customer) and set out to fix the issue in RXTX at a Open Source Love Day (OSLD) . So we fixed the problem and submitted the patch to the bugtracker of the RXTX project. Our sample program now worked flawlessly and threw an IOException when the serial port failed in some way.

Happy to have fixed the problem we incorporated the patch RXTX in our production software but it still crashed and no IOException appeared anywhere in the logs. After another bughunting session we spotted the subtle difference of sample and production program: the use of OutputStream insted of PrintStream. PrintStream silently swallows all exceptions which proved fatal in our use case with the unreliable stream carrier. So the final fix was essentially replacing our PrintStream code

RXTXPort port = new RXTXPort("COM6");
PrintStream p = new PrintStream(port.getOutputStream(), true, "iso8859");
p.print("command");

with using OutputStream directly:

RXTXPort port = new RXTXPort("COM6");
OutputStream o = port.getOutputStream();
o.write("command".getBytes("iso8859"));

Conclusion

Be careful when using PrintStream with unreliable stream carriers it swallows exceptions! That may shadow problems which you may want to know of. Often PrintStreams behaviour will not be a problem but in certain cases like the one depicted above it causes a lot of headaches.