Author: daniel.lindner

A blind spot of Continuous Integration

Imagine you haven’t changed a continuously integrated project for months when suddenly a unit test breaks. Here’s why CI has a blind spot with flawed tests.

In the early days of April 2008, we updated our hudson continuous integration (CI) server to a new version. This was no unusual action, as there was a new version every day back then, bringing new features in a rapid rate. What was unusual after the upgrade was that one of the surveilled projects failed to build all of a sudden.

Sudden (test) failure without a change

The build was started manually, without a code change. The project itself was inactive back then, meaning that no changes were made for months. And suddenly, a unit test broke. The test was in the project for two whole years without ever going off. What happened?

Good unit tests

There are rules for good unit tests. A basic set are the A-TRIP rules formulated in the excellent beginner’s book “Pragmatic Unit Testing” by Andy Hunt and Dave Thomas. The failed test clearly disobeyed the “repeatable” rule (the R in A-TRIP): It didn’t result in the same result as before while the code under test didn’t change.

Write repeatable tests or your CI will be blind (partially)

The cause of the failed test was putting the clocks back because the daylight-saving time part of the year began. The unit test secured some date calculations by taking the current date and comparing it to future and past dates that got calculated. The calculation went wrong when the daylight-saving mode changes in the calculated period, which was a bug. Repeatability of the unit test was lost when “the current date” entered the code – whether on the unit test or productive code side.

Two years of blindness

How could this bug survive two years without being noticed? The project was under CI surveillance since the beginning, the unit test to detect the bug was present along with the bugged code. The answer is: We never programmed for this project around the weeks of the year when the clock is adjusted and the bug occurs. This was a coincidence influenced by the customer’s schedule. So every time CI (or we) ran the unit test, it passed. Until that day right after putting the clocks back.

How to avoid this blind spot

There are two things you can do to avoid this scenario:

  • Always inject a fixed “the current date” into your code when dealing with date calculations. Only use absolute dates in your unit tests. Time isn’t a healer for your tests, it’s a beast to be tamed.
  • Set up a nightly build for your project that runs once a day even when no changes have been made. It would have caught this bug one and a half year earlier.

To sum it up:

CI only spots bugs when they move (aka the code is changed). Nightly builds provide a (fuzzy) security layer against non-repeatable unit tests. And unit tests with flaws provide only delusory security.

Additional background information

After fully understanding the circumstances, we were curious why the customer didn’t notice the bug and asked him about it. The answer was delightful: “Our computers don’t adjust their clocks. Daylight-saving time only causes trouble.” What a wise decision!

For a good comparison of CI vs. Nightly Builds see this blog entry.

Blog harvest, Christmas 2009

Some noteworthy blog articles, harvested for Christmas 2009. Merry Christmas, everyone.

This is the Christmas special edition of our blog harvest. Granted, there is nothing really “special” about it other than the release date. But some of the collected articles are too good to wait until 2010. Be sure to check out the video link at the end, it’s a great presentation.

  • 7 Computer Books I Enjoyed in 2009 – Even if it isn’t my list, I’m glad for this summary of enjoyable books by Freddy Daoud. The first entry on the list is my current programming language book, so I might be somewhat biased.
  • For a fistful of dollars: quantifying the benefits of TDD – A very balanced article about test driven development, backed up by studies. What do we need more? Oh yes, literature! See the links and the last paragraph for this. And grant John Ferguson Smart some self-marketing for his wonderful posting.
  • Remote Pair Programming – Pair programming, like TDD, is a valueable practice. But pairing always requires physical presence. Keith Brophy explored the possibilities of modern software to nullify this requirement. The comments, mentioning Crucible, are also noteworthy.
  • Java 7 chapter 1: The Virtual Machine – The last few years, there was this announceware, nicknamed “Dolphin”. Every few months, blog entries pop up to tell us about the latest and greatest feature of Java 7. Harals Kl. wrote a mini-series that sums it up really nice and doesn’t conceal the dropped features like the Swing Application Framework. Be sure to read all articles of the series.
  • Learn to Let Go: How Success Killed Duke Nukem – While the context to place this article right here is purely coincidential and this isn’t exactly hard computer science, it’s a worthwhile story about a failed project (and the most successful vaporware ever). We grew up with Duke Nukem 3D, it’s a hard loss for us (ok, we will get over it).
  • The Law of Unintended Consequences – This article talks about coincidence, too. Justin Etheredge describes sources of negative side-effects and what software developers can do against it. The introductory example of heatless traffic lights won’t do for developers, because we all know the joke: “How many software developers do you need to change a light bulb?” (*).
  • A Pictorial Guide to avoiding Camera Loss – As this already tends to be humorous: Andrew McDonald talks about another hardware problem – losing a digital camera and how to regain it. Let me warn you about the 16th picture of the guide.

This was the article side of this harvesting. Let’s continue to have fun by watching the promised video and reading about a real misadventurer:

Thank you for reading through this last blog harvest in 2009. See you again in 2010!

(*) “How many software developers do you need to change a light bulb?”: None, it’s a hardware problem.

http://www.stripesbook.com/blog/index.php?/archives/46-7-Computer-Books-I-Enjoyed-in-2009.html

Open Source Love Day December 2009

Our Open Source Love Day for November 2009 brought love for EGit and several hudson plugins. We got slightly frustrated over KDevelop4.

On Tuesday, we had our last regular working day for 2009. We celebrated this circumstance by having our fourth Open Source Love Day (OSLD). The day was successful, you can review our list of today’s achievements below.

We introduced a monthly Open Source Love Day to show our appreciation to the Open Source software ecosystem and to donate back We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

Participate at our OSLD by using the features we’ve built today:

  • Our campfire plugin for hudson was updated to version 1.1. The new version contains the improvements Mark Woods suggested (global configuration and login recovery). Thank you, Mark!
  • The campfire plugin also switched the communication model from webpage scraping to the brand new campfire API. This should improve the stability of the plugin.
  • Some of the EGit (git plugin for eclipse) patches we sent in at the last OSLD needed some rework and polishing. You can review the details in EGit’s code review system gerrit: change 121 and change 122.
  • Our cmake hudson plugin was updated to version 1.1. The new version checks the environment (installed cmake version, etc.) before delegating the call and provides better error messages.
  • We started working on a feature of KDevelop4 that was present in KDevelop3 and is now missing: “Compile file”. The progress was slowed down by some problems. See below for details.
  • Hudson got a new major version of the IRC plugin from Christoph Kutzinksi. The plugin was in a rather desolate state before. We had used a private fork with specific additions to control our infrastructure. The plugin was on our list of OSLD patients, when Christoph merged it with the hudson instant-messaging plugin and introduced a multitude of cool new features. We beta-tested the new version and it was great. The only drawback was the complete alteration in message syntax that broke our infrastructure. So in order to scratch our own itch, we programmed a little API to parse hudson IRC plugin messages of the new 2.X version stream. Our code is published on github, have a look if you are interested and drop us a line if you found it useful.

What were our lessons learnt today?

  • If maven decides to work properly, everything is really cool.
  • Just because you use JGit/EGit on top of Eclipse, all three being platform independent, doesn’t save you from slash vs. backslash issues. EGit’s initial user experience is better on unixoid platforms than on windows systems. The patch #141 helped us beyond the showstopper of unrecognized local repositories.
  • We acquired an additional share of eclipse plugin development knowledge when polishing our EGit features.
  • Working with git and gerrit is challenging on first encounter. We are constantly learning in this area.
  • Bugzilla fails to present open issues in a manner where you can quickly pick an issue of interest. If you really want to use it for your open source project, think of a scraped website that only lists the “low hanging fruits” for newbie developers.
  • KDevelop4 has outdated documentation, the projects kdevplatform and kdevelop were moved inside the repository.
  • If you encounter a rather erratic error stating that “KDE4Workspace not found”, try excluding the debuggers/gdb subproject from your build.
  • Most of us used the waiting delays of one project (“oh, maven is downloading the internet again”) to switch over to a secondary task. So this event trains our multitasking abilities right along.

In summary, this OSLD was a fun way to end a workyear on heavy duty. We will continue to celebrate OSLDs in 2010, as it’s a fun way to peek into foreign projects, learn a lot in short time and contribute to the community.

The fallacy of “the right tool”

There is a fallacy around Polyglot Programming, especially the term “the right tool for the job”: Programming languages aren’t tools.

Let me start this blog post with a disclaimer: I’m really convinced of the value of multilingual programming and also think that applying the “right tool for the job” is a good thing. But there is a fallacy around this concept in programming that i want to point out here. The fallacy doesn’t invalidate the concept, keep that in mind.

Polyglot cineasts

Let me start with an odd thought: What if there was a movie, a complicated international thriller around a political intrigue, playing in over half a dozen countries. The actors of each country speak their native tongue and no subtitles are provided. Who would be able to follow the plot? Only a chosen few of really polyglot cineasts would ever appreciate the movie. Most of us wouldn’t want to see it.

Polyglot programming

Our last web application project was comprised of that half a dozen languages (Groovy, Java, HTML, CSS, HQL/SQL, Ant). We could easily include more programming languages if we feel the need to do it. Adding Clojure, Scala or Ruby/JRuby doesn’t sound absurd to us. A programmer capable of knowing and switching between numerous programming languages is called a “Polyglot Programmer“.

The main justification for heterogeneous (polyglot) projects often is the concept of “using the right tool for the job”. The job often is a subtask of the whole project, like building the project, accessing the database, implementing the ever-changing business logic. For each subtask, some other language might outshine the competitors. Besides some reasonable doubt concerning the hidden cost of this approach, there is a misconception of the term “tool”.

Programming languages aren’t tools

If you use a tool in (basic or advanced) engineering, let’s say a hammer to drive some nails into a wooden plate or a screwdriver to decompose your computer, you’ll put the tool aside as soon as “the job” is finished. The resulting product (a new wooden cabinet or a collection of circuit boards) doesn’t include the tool. Most of the times, your job is really finished, without “change requests” to the product.

If your tool happens to be a programming language, you’ll produce source code bound to the tool. Without the tool, the product isn’t working at all. If you regard your compiled binaries as “the product”, you can’t deal with “change requests”, a concept that programmers learn early and painful. The product of a programmer obviously is source code. And programming languages don’t act as tools, but as materials in this respect. Tools go away, materials stick.

Programming languages are materials

As source code is tied to its programming language, they form a conceptional union. So I suggest to change the term to “using the right material for the job” when speaking about programming languages. This is a more profound decision to make in comparision to choosing between a Phillips style or a TORX screwdriver. Materials need to outlast when the tools are long put aside.

But there are tools, too

In my web application example above, we used a lot of tools. Grails is our framework of choice, Jetty our web container to deploy to, the Spring Framework provides mighty utilities and we used IDEA to bolt it all together. We could easily exchange Tomcat for Jetty or IDEA with Eclipse without changing the source code (the example doesn’t work that easy for Grails and Spring, though). Tools need to be replaceable or even disposable.

Summary

The term “the right tool for the job” cannot easily be applied to programming languages, as they aren’t tools, but materials. This is why polyglot programming is dangerous to when used heavily in a single project. It’s easy to end up with a tangled “amalgam project”.

Two more disclaimers:

  • If chosen right, “composite construction” is a powerful concept that unifies the advantages of two materials instead of adding up their drawbacks.
  • Being multilingual is advantageous for a programmer. Just don’t show it all off in one project.

Follow-up to our Dev Brunch December 2009

A follow-up to our December 2009 Dev Brunch which was omitted because of really bad weather and bad health.

Yesterday, we had an appointment for our Dev Brunch meeting in December 2009. We tried to adapt to the christmas time and scheduled the “brunch” to be in the late evening, replacing coffee with mulled wine and toast with ginger bread. We even called it a “drunch” as a combination of “drunk” and “brunch”. It didn’t help.

The Dev Drunch omitted

It was the coldest and most snowy weekend in years. Most participants called in sick, others refused to go outside. Two brave remainders accomplished to reach our company but decided to concentrate on the mulled wine and the heating radiator instead of talking about software development. We chattered about a lot of topics, but none of it is worth to be reported. When the wine ran out, we went home again.

This was the last chance in 2009 to attend a Schneide Dev Brunch. See you all in 2010 and get well soon.

Blog harvest, December 2009

Some noteworthy blog articles, harvested for early December 2009

Today’s blog harvest spans a lot of topics that i’ve found noteworthy in the last weeks. As an added bonus, there’s a watchworthy video link at the end. I hope you enjoy reading the articles as much as I did. If you have thoughts one the articles, feel free to comment them here.

This was the article side of this harvesting. Let’s have some fun by watching a video and relieving our conscience:

  • Living with 1000 Open Source Projects – It might get crowded on your disk! Nic Williams shares his secrets of mastering open source heavy lifting. The video runs a short half hour and has its funniest minute between 11:20 and 12:20. Brilliant!
  • The Bad Code Offset – Guilty of writing bad code? Well, remember the last entry of the list above? You’ve probably created a new job. If not, you can find absolution by buying some “Bad Code Offsets”. Think of it as the Carbon offset of the software industry.

Follow-up to our Dev Brunch November 2009

A follow-up to our November 2009 Dev Brunch, summarizing the talks and providing bonus material.

Today we held our Dev Brunch meeting for November 2009. It was the last possible date for this month, but we were affected by absences nonetheless. This is the follow-up posting for this rather small gathering, summarizing the topics and providing additional information.

The Dev Brunch

If you want to know more about the meaning of the term “Dev Brunch” or how we realize it, have a look at the follow-up posting of October’s brunch. This time, no notebook was needed.

The November 2009 Dev Brunch

The topics of this session were:

  • Object Calisthenics by example – Experiences gained while programming a small project following the Object Calisthenics rules while practicing Test Driven Development, too.
  • Object Calisthenics inspected – Observations and insights gained when explaining Object Calisthenics to several teams, programmers and student courses.

As you can immediately see, the meeting was small, but surprisingly consistent. We didn’t agree upon the topic beforehands, but it was a perfect match. Everybody who missed this brunch definitely missed some very interesting first-hand experiences on Object Calisthenics, too. To ease this lack a bit, let me rephrase the content a bit.

Object Calisthenics

You might have heard about Object Calisthenics before, on this blog or other resources on the net. Perhaps you’ve read the original article, which is highly advised. In short, Object Calisthenics are a set of inspiring, if not irritating programming rules that should lead to better programming style through excercise. You should consult the links above for specifics.

Object Calisthenics by example

When applying the rules to a domain class model, some new techniques arose to compensate the “train wreck line”-programming style (see rule 4) and to introduce first class collections (rule eight) and avoid getters and setters (rule 9). This techniques included the use of the Visitor design pattern, which wasn’t the author’s first choice beforehands. Test Driven Development alone wouldn’t have led to this solution, but the solution works well for the given use case.

The author softened some rules for his example and found valid explanations for doing so. This might be the content of an additional blog posting that still needs to be written. It will be announced in the comments when published.

Test Driven Development and Object Calisthenics do not interfere with each other. They both aim for better code and design, but through different means. They could be regarded as complements in a programmer’s toolbox.

Object Calisthenics inspected

When teaching the nine rules, some effects occurred repeatedly. The first observation was that the rules follow a dramatic composition that orders them from “most obvious and immediate code improvement” to “hardest to achieve code improvement” and in the same order from “easiest to acknowledge” to “most controversial”. At the end of the list, the audience rioted most of the time. But if you reject the last few rules, you’ve silently agreed to the first ones, the ones with the greatest potential for immediate improvement.

Another observation is that the rules stick. Even if you reject them on first notion, it creeps into your thinking, whispering that “it might be possible right now with this code“. It’s a learning catalyst for those of us that aren’t born as programming super-heros. To speak in terms Kent Beck coined: Object Calisthenics provide some handy practices that might eventually lead to a better understanding of their underlying principles. Even beginners can follow the practices and review their code on compliance. When they fully get to know the principles (like Law Of Demeter, for example), they are already halfway there.

The third observation was that most experienced programmers intuitively revealed the principles behind the rules before I could even try to explain. Some even found very interesting associations with other principles that weren’t so obvious.

At last, Object Calisthenics, if performed as a group exercise, can be a team solder. You can rant over code together without regrets – the rules were made elsewhere. And you can discuss different solutions without feeling pointless – fulfilling the rules is the common goal for a short time.

The Dev Brunch retrospected

This brunch was small both in attendee and topic count. That created a very productive discussion. We’ll try to grow the insights gained today into additional blog entries. Stay tuned.

Open Source Love Day November 2009

Our Open Source Love Day for November 2009 brought love for EGit, the Eclipse git plugin and some frustration over lacking documentation.

Today, we celebrated our third Open Source Love Day (OSLD). It was slightly degraded by a company-wide illness outbreak, but we tried to make the best out of the situation.

Open Source Love Days are our way to show our appreciation and care to the Open Source software ecosystem. Our work wouldn’t be as fun or just not possible without professional Open Source software. You can read more about our motivation and specifica in our first OSLD blog posting.

You can participate at our OSLD by using the features we’ve built today:

  • We think git is an enrichment to the SCM market, as is Eclipse to the IDE market. Improving the quality of Eclipse’s git plugin is the next logical step. The ability to diff the content of two revisions in EGit was committed today. As a bonus, the name of the committer shows up in the right manner, too. See the screenshots to get the idea:

When developing EGit, we were already using it to pull the sources. Unfortunately, the repository URL changed bigtimes since our checkout without us noticing. This got us into trouble trying to follow the contributor guide. The command line version of git isn’t that communicative yet. But after all, this is a great time to learn about the real world problems when using git. The EGit contributor guide itself is a fantastic way for a project to show initial appreciation to volunteer efforts. Thanks for caring, guys! If you are interested to review our changes yourself, fetch the patch.

  • Another part of today’s work was on the KDevelop project. We tried to fix some outstanding little features or bugs, whatever is on the list of KDevelop 4. But we spent our day fixing our development machines instead. The Ubuntu linux operating system (8.10) was way too old to get useful results and KDE needs to be up-to-date to develop KDevelop. Besides our sluggishness to keep our virtual machines on the bleeding edge, the checkout experience of KDevelop was rather sleek. What bothers us a bit is the ominous entanglement between KDevelop and KDE. It seems you can’t have one without the other and need to master both to make a stand.
  • As a third part, we wanted to contribute to the TANGO project (not the useful icon collection, but the useful control system). They migrated their main repository from CVS to SVN lately, but the migration seems unfinished still. At least, the migration effort lacks public documentation for the occasional contributor. That’s a real showstopper, because you never get beyond the very first step: setting up a working project. We won’t give up and email the project leads on this topic, but it didn’t fit into this OSLD.

What were our lessons learnt today?

  • Just having a possibility to view or download the source code doesn’t make an Open Source project. The key to success is the ability of complete strangers to hop in and perform useful work. Having terse, but accurate documentation helps a lot. The EGit contributor guide is a good example of a single document that makes the difference. If you own an open source project and want to attract occassional contributors (like us), write such a document and watch us (and others) drop you a patch. That said, we come to belief that the person that writes technical documentation for the developers is one of the most important roles on a project. Perhaps we join some projects in the future to fill that role.

To sum it up, this OSLD was limited from the beginning by developer availibility. With lacking documentation, we nearly grinded to a halt. We look forward to our next OSLD in December.

Speed up your buildbox, Part III: Memory

This is the third part of a series on how to boost your build box without much effort. This episode talks about the effects of faster and more RAM.

© Friedberg - Fotolia.comIn the first and second part of our effort to speed up our buildbox, we replaced the harddisk with a RAM disk and swapped in a bigger CPU. This brought the build time down from 03:30 minutes to 02:00 minutes.

Boosting the memory

When we began the journey, we wanted to undercut the 02:00 minutes threshold. The last component that directly impacts performance of our box was the memory. We started out with 4 GB of DDR2-800 modules. To have a feeling for the effects, we upgraded to 4 GB of DDR2-1066 first and then added another 4 GB, resulting in 8 GB of RAM. We expected the performance gain to be small, but noticeable. The RAM disk, for example, is directly affected by memory speed.

As much, but faster

The first upgrade brought the first surprise: Upgrading from DDR2-800 to DDR2-1066 modules didn’t change anything. It’s not that the mainboard or CPU doesn’t support the faster RAM, it just seems to be fast enough, despite the data bus clock rate. Our build process still took 02:00 minutes, reproducible and without exception.

Filling all the banks

The mainboard can load up to 16 GB of RAM, but our budget just allowed to buy 8 GB of DDR2-1066 RAM. We installed it and ran the same 32 bit Ubuntu Linux as before. The build process took 02:00 minutes, which was expected now.

Changing to 64bit

We changed to boot harddisk, installed a 64 bit Ubuntu Linux and ran the build again. Still 02:00 minutes. The switch to 64 bit wasn’t a big deal with Java, but some of the included native libraries complained about the change. Recompiling them solved the issue.

Finally reaching the target

As a last measure, we increased the maximum memory of the build JVM to the biggest value it would accept. This was -Xmx2600m, a surplus of 600 MB to the original setting. This sped up the build process by five seconds, it took 01:55 minutes now.

Conclusion and perspective

We’ve reached our anticipated target of less than two minutes build time. We exceeded our original budget of 500 EUR, but bought some parts that finally weren’t used in the build box, but elsewhere. The two parts that made the whole difference were the CPU and some more memory to spend it on the RAM disk.

If you want to speed up your single build box, aim for the CPU/RAM combo and try to install a RAM disk to perform all the work on.

This leads me to the perspective of the next part of the series: If you plugged in the most expensive CPU and enormous amounts of RAM to speed up your buildbox, you still aren’t done. You should invest some time to look into distributed builds. Hudson as our continuous integration server provides nearly instant “build slave” support. With this feature, you can set up a whole build farm to further increase your build throughput.

Stay tuned for “Part IV: Beyond the box”

Blog harvest, October II

Some interesting blog articles, harvested for late October 2009

harvest64A great way to stay up to date with current musings and hypes of our industry is to follow other people’s blogs. We do this regularly – everybody scans his RSS feeds and roams the internet. But to have a pool of shared knowledge, we pick our favorite recent blog articles and usually write an email titled “blog harvest” to the rest of the company.

Then, the idea came up to replace the internal email by a public blog post. So here it is, the first entry of a new category called “blog harvests”. You’ll read more harvests in the future. They will be categorized and tagged appropriate and have the harvest icon nearby.

Second Blog harvest for October 2009

There are four main blog entries I want to share:

  • 8 Signs your code sucks – Let’s assume we all read Martin Fowler’s classic “Refactoring” book, then these eight signs are a mere starter. But as the follow-up post indicates, it got quite a few people started and upset for the “comments are code smells” line. Well, we heartfully agree with the premise that comments are clutter and code should be the comment. /* TODO: Add a joke using comments here */
  • ORMs are a thing of the past – Another opinion that might get in the way of hibernate fanboys. We’ve had our share of hibernate “experiences”. It’s a useful tool if you know how to use it – and when not to. Replies followed instantly, here are two noteworthy ones by Scot Mcphee and by Jens Schauder.
  • The Case for Clojure – Clojure is functional programming on the Java VM (think LISP). Stay tuned for our own book review on this topic. You can argue that Clojure isn’t pure, though.
  • Bad Programmers Create Jobs – As is already is a controversy harvesting, lets add some more, written by Mohammad Azam. Side note: Half of our work was initially created by “bad” programmers, so I think Mohammad hit the nail on the head. And remember that you’ve produced legacy code today.

Then there is a bit of (future) knowledge you shouldn’t miss:

That’s it for now. My harvest format has changed for the blog, i’ll evolve it further in the next months, Thanks for your attention, stay tuned.