Twelve years a bug

Recently, a friend recommended the talk “Love your bugs” by Allison Kaptur to me. It’s a good talk with a powerful message: Every bug is a chance to learn. The only prerequisite for this chance: You need to be aware of the bug. You need to see it and then understand and fix it.

What if you don’t see a bug for over a decade?

You can still learn from it – a lot. Here is a story about a bug that lingered in my code for twelve years and had impact on the software results without anybody, including me, noticing.

Let’s say the software is a measurement system that records physical measurement values that cannot be reenacted easily. The samples are measured in rapid succession and then stored away. The measurement results act as a quality quantifier as in “the sample was this good at that time”. The sample’s quality diminishes over time, even with perfect storage conditions. And just to be sure that the measurement is accurate, it isn’t performed once, but twice in quick succession: The measurement device traverses the sample in one direction, recording the values and uses the return path for a second measurement. This results in two measurement streaks that should, in theory, be very similar.

The raw measurement values are aggregated in different ways. In the final report, the maximum value over both measurement streaks is the most prominent and most important aspect. There is nothing exciting or error-prone about finding the maximum value in a value series, even twelve years ago. But I tested the code nonetheless. The whole value aggregation code was under test and had a good test coverage. All tests showed their thumbs up.

But twelve years ago, at the end of a workweek, on a Friday at 17 o’clock, I made a small and easy refactoring to the code. I know this in such detail because of the wonders of version control. Without version control, I probably would have learnt a lot less from this bug.

The code before the refactoring contained two nearly identical sections for the measurement streaks, making it duplicated code. There were only two differences: The first section of code counted from 0 to 99 and stored the values in the first streak’s array. The second section counted from 99 to 0, because the measurement device travels backwards over the sample, and stored the values into the second array.

My refactoring brought both pieces of code together: A new method with two parameters was introduced and called at both places. The first parameter specified a series of positions (0 to 99 or backwards), the second parameter specified the array to store into. All automated tests approved the changes. My manual testing showed no differences. The refactoring was going live.

Twelve years later, while working on a new engine for the measurement device, the customer took the raw values from the journal and performed some manual calculations. The maximum value calculation was wrong. It wasn’t wrong all of the time, but also not correct for all measurements. When it went wrong, it selected the second greatest value or, seldom, the third greatest value as the maximum value.

All the tests still insisted that everything is correct – as it was for twelve long years. There was no other change to the code that could affect the aggregation in any way.

There are two different ways that lead me to find the bug’s origin. The first way was to inspect the trail of commits in the area of code that performs the aggregation. My findings were that the abovementioned refactoring was the most likely culprit. The second way was to write more tests to ensure that yes, the maximum value calculation was indeed correct if given the correct values. Using the examples my customer had examined, I could prove with enough certainty that, given the input of all 200 values, the correct maximum value was returned in all cases. The aggregation therefore wasn’t given all values!

And that lead directly to the bug: The first measurement streak used the same array to store the values as the second streak. During the refactoring, I must have copied and pasted the call to the new method and forgotten to change the second parameter. Of 200 measured values, only 100 got stored permanently. The first 100 values were stored and promptly overwritten as the measurement device returned to its park position. Change the calls to store in both arrays and everything works like intended and like before the refactoring.

How did no test, automated or manual, catch this bug? It turns out that most automated tests were too focussed to indicate a problem. All unit tests that secured the maximum value calculation used given sets of values, they didn’t care about the origin of these value sets. The integration tests that covered the whole measurement process should have raised objections to the refactoring. But they used given sets of measurement values, too. And by chance, the given maximum value was in the second measurement streak. In fact, the given set of measurement values was produced by a loop that just increased a value. The greatest value was always the last one.

The manual tests had the same problem: The simulated measurement device produced measurement values that were either fixed or random. If your whole measurement uses the same fixed values, you don’t see it if half of them went missing. And if your measurement uses random values, you’ll have to pay close attention to a detail that isn’t in focus because it is unchanged. Except that one time when it was changed recently. Remember the change date? Friday afternoon, only minutes before the weekend? Not the best time to manually test a change that is a simple standard refactoring, after all.

So, what have I learnt? First: automated tests, even with great test coverage, aren’t enough. There is so much leeway in the setup of these tests that they will have blind spots without your knowledge. Second: A code review by another human (or even by the same human, some days later) might have caught the bug. It was painfully obvious in hindsight. The problem? “Might have” is an heuristics, just like your automated tests are.

My guess is that mutation testing would have shown the blind spot of the existing tests – among several hundred others. The heuristics is now your trained eye that sifts through the results and separates false positives from true positives.

Right now, I’ve fixed the bug, kept all additional tests and added one more: An integration test that performs a measurement with fixed values and checks nothing else but if all 200 values are stored at the end of the measurement. It’s oddly specific, but conveys this story in an automated fashion.

Oh, and I made another refactoring: I’ve replaced the arrays with collections. Hopefully, I won’t regret this one twelve years in the future. I’ve made it on a Monday.

A minimal set of skills for software development contractors

“Our company is specialized in providing professional software development for our customers”. That’s a nice statement to inspire your customers with. The only problem with it is: every contractor claims to be professional. You wouldn’t even get a project if you admitted to be “unprofessional”. But how can a customer, mostly unaware of the subtleties in the field of software development, decide if his contractor really works professionally? A lot of money currently spent on projects doomed from the beginning could be saved if the answer was that easy. But there’s a lower limit of skills that have to be present to pass the most minimal litmus test on developer professionality. This blog article gives you an overview about the things you should ask from your next software development contractor.

First a disclaimer: I’ve compiled this list of skills with the best intentions. It is definitely possible to develop software without some or even any of these skills. The development can even be performed in a very professional manner. So the absence of a skill doesn’t reveal an unprofessional contractor without fail. And on the other side, the clear presence of all skills doesn’t lead to glorious projects. The list is a rule of thumb to distinguish the “better” contractor from the “worse”. It’s a starting ground for the inexperienced customer to ask the right questions and get hopefully insightful answers.

Let’s assume you are a customer on the lookout for a suitable software development contractor, maybe a freelancer or a company. You might take this list and just ask your potential developer about every item on it. Listen to their answers and let them show you their implementation of the skill. In my opinion, the last point is the most crucial one: Don’t just talk about it, let them demonstrate their abilities. You won’t be able to differentiate the best from the most trivial implementation at first, but that’s part of the learning process. The thing is: if the developer can readily demonstrate something, chances are he really knows what he is talking about.

The minimal skills

The list is sorted by their direct impact on the overall development quality. This includes the quality perceived by you (the customer), the end user and the next developer who inherits the source code once the original developer bails out. This doesn’t mean that the topics mentioned later are “optional” in the long run.

Source code management system

This tool has many different names: source code management (SCM), revision control system (RCS) and version control system (VCS) are just a few of them. It is used to track the changes in the code over time. With this tool, the developer is able to tell you exactly which change happened when, for what version and by whom. It is even possible to undo the change later on. If your developer mentions specific tool names like Git, Subversion, Perforce or Mercurial, you are mostly settled here. Let him show you a typical sync-edit-commit cycle and try to comprehend what he’s telling you. Most developers love to brag about their sophisticated use of version control abilities.

Issue tracking

An issue or bug tracker is a tool that stores all inquiries, bug reports, wishes and complaints you make. You can compare it to a helpdesk “trouble ticket” system. The issue tracker provides a todo list for the developer and acts as an impartial documentation of your communication with the developer. If you can’t get direct access to the issue tracker on their website, let them demonstrate the usage by playing through a typical scenario like a bug report. At least, the developer should provide you with a list of “resolved” issues for each new version of your software.

Continuous integration

This is a relatively new type of tool, but a very powerful one. It can also be named a “build server” or (less powerful) a “nightly build”. The baseline is that your project will be built by an automated process, as often as possible. In the case of continuous integration, the build happens after each commit to the source code management system (refer to the first entry of this list). Let your developer show you what happens automatically after a commit to the source code management system. Ask him about the “build time” of your project (or other projects). This is the time needed to produce a new version you can try out. If the build time is reasonably low (like a few minutes), ask for a small change to your project and wait for the resulting software.

There is a fair chance that your developer not only talks about “continuous integration”, but also “continuous delivery”. This includes words like “staging”, “build queue”, “test installation”, etc. Great! Let them explain and demonstrate their implementation of “continuous delivery”. You’ll probably be impressed and the developer had another chance to brag.

Verification (a.k.a. Testing)

This is a delicate question: “Will the source code contain automated tests?”. Our industry’s expectancy value for any kind of automated tests in a project is still dangerously near absolute zero. If you get blank stares on that question, that’s not a good sign. It doesn’t really matter too much if the answer contains the words “unit test”, “integration test” or even “acceptance test”. Most important again: Let your developer show you their implementation of automated tests in your (or a similar) project. Make sure the continuous integration server (refer to entry number three) is aware of the tests and runs them on every build. This way, everything that’s secured by tests cannot break without being noticed immediately. You probably won’t have to deal with reappearing bugs in every other version, a symptom known as “regression”.

Your developer might be really enthusiastic about testing. While every developer hour costs your precious money, this is money well spent. Think of it as an insurance against unpredictable behaviour of your software in the future. Over the course of development, you won’t notice these tests directly, as they are used internally for development. Talk to your developer about some form of reporting on the tests. Perhaps a “test coverage” report that accompanies the issue list (refer to the second entry)? Just don’t go overboard here. A low test coverage percentage is still better than no tests.

If your developer states that he is “test driven”, that’s not a psychological condition, but a modern attempt to test really thoroughly. Let him demonstrate you the advantages of this approach by playing through an implementation cycle of a small change to your project. It may foster your confidence in the insurance’s power.

Project documentation

Every software project above the trivial level contains so many details that no human brain is able to remember them all after some time. Your developer needs some place to store vital information about the project other than “in the code” and “in the issue tracker”. A popular choice to implement this requirement is providing a Wiki. You probably already know a Wiki from Wikipedia. Think about a web-based text editing tool with structuring possibilities. If you can’t access the documentation tool yourself, let your developer demonstrate it. Ask about an excerpt of your project documentation, perhaps as a PDF or HTML document. Don’t be too picky about the aesthetics, the main use case is quick and easy information retrieval. Even handwritten project documentation may pass your test, as long as it is stored in one central place.

Source code conventions

Nearly all source code is readable by a machine. But some source code is totally illegible by fellow developers or even the original author. Ask your developer about their code formatting rules. Hopefully, he can provide you with some written rules that are really applied to the code. For most programming languages, there are tools that can check the formatting against certain rules. These programs are called “code inspection tools” and fit like hand in glove with the continuous integration server (refer to the third entry). Some aspects of source code readability cannot be checked by algorithms, like naming or clarity of concepts. Good developers perform regular code reviews where fellow developers discuss the code critically and suggest improvements. The best customers explicitely ask for code reviews, even if they won’t participate in them. You will feel the difference in the produced software on the long run.

Community awareness

Software development is a rapidly advancing profession, with game-changing discoveries every other year. One single developer cannot track all the new tools, concepts and possibilities in his field. He has to rely on a community of like-minded and well-meaning experts that share their knowledge. Ask your developer about his community. What (technical) books did he read recently? What books are known by the whole development team? As a customer, you probably can’t tell right away if the books are worth their paper, but that’s not the main point of the question. Just like with tests, the amount of books read by the average programmer won’t make a very long list. If your development team is consistent enough to share a common literature ground, that’s already worth a lot.

But it’s not just books. Even books are too slow for the advancement! Ask about participation in local technical events, like user groups of the programming language of your project. What about sharing? Does the developer share his experiences and insights? The cheapest way to do that is a weblog (you’re reading one right now). Let him show you his blog. How many articles are published in a reasonable timespan, what’s the feedback? Perhaps he writes articles for a technical magazine or even a book? Now you can ask other developers for their opinion on the published work. You’ve probably found a really professional developer, congratulations.

There is more, much more

This list is in no way exhaustive in regard to what a capable developer uses in concepts, skills and tools. This is meant as the minimal set, with a lot of room for improvement. There are compilations of skills like the Clean Code Developer that go way beyond this list. Ask your developer about his personal field of interest. Hopefully, after he finished bragging and techno-babbling for some time, you’re convinced that your developer is a professional one. You shouldn’t settle for less.