Testing C++ code with OpenCV dependencies

This is a documentation of a problem I ran into when I wrote C++ tests for a simple function, inclusive workaround.

The story:

Pushing for more quality and stability we integrate google test into our existing projects or extend test coverage. One of such cases was the creation of tests to document and verify a bugfix. They called a single function and checked the fields of the returned cv::Scalar.

TEST(ScalarTest, SingleValue) {
  ...
  cv::Scalar actual = target.compute();
  ASSERT_DOUBLE_EQ(90, actual[0]);
  ASSERT_DOUBLE_EQ(0, actual[1]);
  ASSERT_DOUBLE_EQ(0, actual[2]);
  ASSERT_DOUBLE_EQ(0, actual[3]);
}

Because this was the first test using OpenCV, the CMakeLists.txt also had to be modified:

target_link_libraries(
  ...
  ${OpenCV_LIBS}
  ...
)

Unfortunately, the test didn’t run through: it ended either with a core dump or a segmentation fault. The analysis of the called function showed that it used no pointers and all variables were referenced while still in scope. What did gdb say to the segmentation fault?

(gdb) bt
#0  0x00007ffff426bd25 in raise () from /lib64/libc.so.6
#1  0x00007ffff426d1a8 in abort () from /lib64/libc.so.6
#2  0x00007ffff42a9fbb in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff42afb56 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffff54d5135 in void std::_Destroy_aux<false>::__destroy<testing::internal::String*>(testing::internal::String*, testing::internal::String*) () from /usr/lib64/libopencv_ts.so.2.4
#5  0x00007ffff54d5168 in std::vector<testing::internal::String, std::allocator<testing::internal::String> >::~vector() ()
from /usr/lib64/libopencv_ts.so.2.4
#6  0x00007ffff426ec4f in __cxa_finalize () from /lib64/libc.so.6
#7  0x00007ffff54a6a33 in ?? () from /usr/lib64/libopencv_ts.so.2.4
#8  0x00007fffffffe110 in ?? ()
#9  0x00007ffff7de9ddf in _dl_fini () from /lib64/ld-linux-x86-64.so.2
Backtrace stopped: frame did not save the PC

Apparently my test had problems at the end of the test, at the time of object destruction. So I started to eliminate every statement until the problem vanished or no statements were left. The result:

#include "gtest/gtest.h"
TEST(DemoTest, FailsBadly) {
  ASSERT_EQ(1, 0);
}

And it still crashed! So the code under test wasn’t the culprit. Another change introduced previously was the addition of OpenCV libs to the linker call. An incompatibility between OpenCV and google test? A quick search spitted out posts from users experiencing the same problems, eventually leading to the entry in OpenCVs bug tracker: http://code.opencv.org/issues/1608 or http://code.opencv.org/issues/3225. The opencv_ts library which appeared in the stack trace, exports symbols that conflict with google test version we link against. Since we didn’t need opencv_ts library, the solution was to clean up our linker dependencies:

Before:

find_package(OpenCV)

 

/usr/bin/c++ CMakeFiles/demo_tests.dir/DemoTests.cpp.o -o demo_tests -rdynamic ../gtest-1.7.0/libgtest_main.a -lopencv_calib3d -lopencv_contrib -lopencv_core -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_highgui -lopencv_imgproc -lopencv_legacy -lopencv_ml -lopencv_nonfree -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_ts -lopencv_video -lopencv_videostab ../gtest-1.7.0/libgtest.a -lpthread -lopencv_calib3d -lopencv_contrib -lopencv_core -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_highgui -lopencv_imgproc -lopencv_legacy -lopencv_ml -lopencv_nonfree -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_ts -lopencv_video -lopencv_videostab

After:


find_package(OpenCV REQUIRED core highgui)

 

/usr/bin/c++ CMakeFiles/demo_tests.dir/DemoTests.cpp.o -o demo_tests -rdynamic ../gtest-1.7.0/libgtest_main.a -lopencv_highgui -lopencv_core ../gtest-1.7.0/libgtest.a -lpthread

Lessons learned:

Know what you really want to depend on and explicitly name it. Ignorance or trust in build tools’ black magic is a recipe for blog posts.

Thoughts on Design

There is a book I am currently reading (and recommend): “The design of everyday things” by Donald A. Norman. The author describes common design errors in an easy readable way and shows or outlines the solutions for them. Despite not beeing a book about software engineering, it covers pretty well one of its greatest problems: interaction with a human. What are the points a software developer should consider when creating new or changing old features?

Natural mapping

Natural mappings are the clues that we can map to known patterns and instinctively use to interpret new unknown things. These mostly date back to prehistoric times and address the animal in us to catch our attention. To animals big things, moving things, things that are different because of color, shape or some other distinctive property are important, because they must decide based on them whether to flee or to attack. Natural mappings require near zero conscious processing power and make the user to pay attention to crucial information instantly. If you have two buttons “cancel order” and “submit order” and you want the user to click on submit, you better make the submit button big and flashy and the cancel button normal size and color it in standard grey (Have a look at the publish button and the preview button of wordpress).

Visibility

Not all users read the manual (if any exists) before they use their programs. Not all users that read the manual, can understand it or find there the steps necessary to accomplish their task. To be still able to succeed, at every step the user asks himself the following questions:

  • What is already done?
  • What is my current position in the process?
  • What is to do now?
  • How far I am away from my goal?

Consider a user who only has 15 minutes and has to fill out an order form consisting of 15 pages. A user who does not even get the total number of pages stops frustrated after very few pages, because the process seems endless. Provide him with the page count and the current page and he will be able to plan ahead and esimate the needed time. If the mandatory fields are marked, the user will concentrate on them and progress faster, incresing the probability to complete the order in time.

Feedback

Every time a user has done something, he will want to ensure that everything happened as he wanted. A mute system will inspire confusion and fear (Want to try it out? Use ed). A status message is a great signal for the user that he accomplished some of the steps on the way to his goal. Without the message “Order submitted successfully” the user won’t know whether the system accepted his input or just jumped to an another page. Additional confirmations like emails allow the user to receive status information with the additional benefit of being persistent unlike a web page in a browser.

Errors

Like any human the users input bogus data or trigger unwanted actions. In this cases the user should get appropriate feedback. When the previous steps are considered, the user will know what field is affected, why the input is not accepted and what the steps are to correct the situation. Sometimes the errors are logical and not syntactical, making them hard to impossile to detect by the system. There is no way to tell whether the user wanted to buy one or ten books. The layout of elements can be adapted to minimize the risk of accidental use: the “Close Application” button is better not to be placed near the “Save” button. When the mistake has been done, the user should be offered an edit, or at least a withdrawal option. Not all systems allow reversal of actions, and present the user a confirmation dialog with an important choice, producing unnecessary stress. There are ways to conter that problem.

Conclusion

This are simple points that can be taken into consideration when carefully designing a system. Even if they are simple, we tend to forget them because we have deadlines or understand the system on such level that we cannot even imagine what steps an inexperienced user can take and what hints he need.

Working on software as a free time activity

Why would somebody do this? Isn’t it already enough to code at work for eight hours a day, five days a week? If you ask yourself this questions, then I think you should reconsider your position.

There is a fundamental difference between work and free time. You are not constrained. You don’t have to hold a deadline. Software development is a mentally challenging task, and while some time pressure keeps you focused, a little bit more forces you to cut corners instead of considering better alternatives. If deadlines were good, they wouldn’t have “dead” in their name. In your free time you decide when you are done.

Even having considered alternatives you are not always able to implement them. There may be a corporate identity that doesn’t contain your favourite flavour of pink. There can be a module licensed under a non-commerial-only license. Or maybe your company uses an old framework missing the latest features. No such problems in your free time.

There is a theory that mastery comes from practice. By coding in your free time, you can decide whether you invest your time in deeper knowledge of some topic or in a broader horizon thus becoming a valuable employee. And sharing freshly won knowlege and experience can increase your reputation as colleague too.

The social among us even meet like-minded people at events like Java User Group or Schneide Dev Brunch. Here the amount of transported information is  much higher, since everyone has another background and focuses on another things. You can even share your mistakes and laugh with others about them.

Are there any side-effects of free time coding besides those listed before? Yes. Your personality can change. It is possible that you will gain a positive attitude and start invest your free time in your skills. Maybe you’ll even start to motivate others to do likewise.

Procrastination

Not doing things at the time they ought be done seems to be a known problem for as long as people are able to plan. They know the consequences and knowingly decide to delay the inevitable. When do you encounter such situations in your professional life? How can they be improved?

Deadline

is the trump card that can be played every time someone wants to put off a task that hinders but not completely stops the process. I’ve seen it often when talking about refactorings or automated tests – “We can do it when we have time”. When the situation does not permit changes to the schedule it is a valid and appropriate decision to delay them until later. Delaying again and again is not. To improve the situation it is necessary that everyone in the chain in the development process from the developer to the customer[1] knows the consequences of accumulating technical debt. In most cases they are just ignorant of the fact or of its long term impact and will change the course of action when “enlightened”. The next step should be the reduction of workload to catch up and improve the product in time.

Fear

of failure or fear of consequences is also often the reason someone delays important tasks. The fear can be fostered through misinterpreted quality standards. A little checklist to keep in mind:

  • Breaking the build is not bad. It is bad to let it stay broken.
  • Checkstyle is not everything. There are situations when people are “more right”. Dogmatism is unhealthy anyway.
  • Man is not a machine. Nobody can be expected to work full time without making mistakes. Even hardware fails.

To improve this situation it is necessary to revise the values and make sure that the consequences are reduced to the level where they are noticeable but not hurting or even crippling. This way any person can make decisions – even wrong ones – without anticipation of immediate punishment. Not all working environments are that toxic. Sometimes the person is just inexperienced or has low self esteem. Here teaching, leading by example and encouragement to take responsibility helps a lot.

Boredom

is like torture for conscious mind. A boring task can be endured once, but doing it every day is very unconfortable. Fortunately when something can be predicted, it can be automated. Cron jobs, continuous integration systems or small round vacuum cleaners serve as example. Even our brain automates processes by pushing them into the subconscious mind – this way we are able to speak and walk at the same time without any problems. Some bureaucratic routines however, like time tracking, cannot be automated fully and require our intervention at regular intervals. The natural behaviour to increase the intervals between the tasks or forgetting them ultimately leads to the situation comparable to technical debt where a huge amount of undone tasks grinds a system to a halt. Since most companies require some amount of bureaucracy to function, it is not possible to completely optimize it away. But it is possible to reduce the chance to forget a task. As example we are always wearing a real cowboy hut when we work in a production environment. Only when we are done, we can put it off. To make boring tasks more attractive the idea of continuous integration game can be used. In time tracking context the user would get points for booking time at the same working day and less for booking at the next day.

Conclusion
Simplified the equation is: pressure * negative feelings towards the task = danger of procrastination. To improve the situation, lower the factors!

[1] Especially in product development: Customer is not the one who buys or uses the product, but the one who pays the developer to make the product and later wants to make money with it.

A minimal set of skills for software development contractors

You aren’t sure if your developer is professional enough? Here are seven topics you can ask him about to find it out. It’s the minimal skill set a modern developer should use.

“Our company is specialized in providing professional software development for our customers”. That’s a nice statement to inspire your customers with. The only problem with it is: every contractor claims to be professional. You wouldn’t even get a project if you admitted to be “unprofessional”. But how can a customer, mostly unaware of the subtleties in the field of software development, decide if his contractor really works professionally? A lot of money currently spent on projects doomed from the beginning could be saved if the answer was that easy. But there’s a lower limit of skills that have to be present to pass the most minimal litmus test on developer professionality. This blog article gives you an overview about the things you should ask from your next software development contractor.

First a disclaimer: I’ve compiled this list of skills with the best intentions. It is definitely possible to develop software without some or even any of these skills. The development can even be performed in a very professional manner. So the absence of a skill doesn’t reveal an unprofessional contractor without fail. And on the other side, the clear presence of all skills doesn’t lead to glorious projects. The list is a rule of thumb to distinguish the “better” contractor from the “worse”. It’s a starting ground for the inexperienced customer to ask the right questions and get hopefully insightful answers.

Let’s assume you are a customer on the lookout for a suitable software development contractor, maybe a freelancer or a company. You might take this list and just ask your potential developer about every item on it. Listen to their answers and let them show you their implementation of the skill. In my opinion, the last point is the most crucial one: Don’t just talk about it, let them demonstrate their abilities. You won’t be able to differentiate the best from the most trivial implementation at first, but that’s part of the learning process. The thing is: if the developer can readily demonstrate something, chances are he really knows what he is talking about.

The minimal skills

The list is sorted by their direct impact on the overall development quality. This includes the quality perceived by you (the customer), the end user and the next developer who inherits the source code once the original developer bails out. This doesn’t mean that the topics mentioned later are “optional” in the long run.

Source code management system

This tool has many different names: source code management (SCM), revision control system (RCS) and version control system (VCS) are just a few of them. It is used to track the changes in the code over time. With this tool, the developer is able to tell you exactly which change happened when, for what version and by whom. It is even possible to undo the change later on. If your developer mentions specific tool names like Git, Subversion, Perforce or Mercurial, you are mostly settled here. Let him show you a typical sync-edit-commit cycle and try to comprehend what he’s telling you. Most developers love to brag about their sophisticated use of version control abilities.

Issue tracking

An issue or bug tracker is a tool that stores all inquiries, bug reports, wishes and complaints you make. You can compare it to a helpdesk “trouble ticket” system. The issue tracker provides a todo list for the developer and acts as an impartial documentation of your communication with the developer. If you can’t get direct access to the issue tracker on their website, let them demonstrate the usage by playing through a typical scenario like a bug report. At least, the developer should provide you with a list of “resolved” issues for each new version of your software.

Continuous integration

This is a relatively new type of tool, but a very powerful one. It can also be named a “build server” or (less powerful) a “nightly build”. The baseline is that your project will be built by an automated process, as often as possible. In the case of continuous integration, the build happens after each commit to the source code management system (refer to the first entry of this list). Let your developer show you what happens automatically after a commit to the source code management system. Ask him about the “build time” of your project (or other projects). This is the time needed to produce a new version you can try out. If the build time is reasonably low (like a few minutes), ask for a small change to your project and wait for the resulting software.

There is a fair chance that your developer not only talks about “continuous integration”, but also “continuous delivery”. This includes words like “staging”, “build queue”, “test installation”, etc. Great! Let them explain and demonstrate their implementation of “continuous delivery”. You’ll probably be impressed and the developer had another chance to brag.

Verification (a.k.a. Testing)

This is a delicate question: “Will the source code contain automated tests?”. Our industry’s expectancy value for any kind of automated tests in a project is still dangerously near absolute zero. If you get blank stares on that question, that’s not a good sign. It doesn’t really matter too much if the answer contains the words “unit test”, “integration test” or even “acceptance test”. Most important again: Let your developer show you their implementation of automated tests in your (or a similar) project. Make sure the continuous integration server (refer to entry number three) is aware of the tests and runs them on every build. This way, everything that’s secured by tests cannot break without being noticed immediately. You probably won’t have to deal with reappearing bugs in every other version, a symptom known as “regression”.

Your developer might be really enthusiastic about testing. While every developer hour costs your precious money, this is money well spent. Think of it as an insurance against unpredictable behaviour of your software in the future. Over the course of development, you won’t notice these tests directly, as they are used internally for development. Talk to your developer about some form of reporting on the tests. Perhaps a “test coverage” report that accompanies the issue list (refer to the second entry)? Just don’t go overboard here. A low test coverage percentage is still better than no tests.

If your developer states that he is “test driven”, that’s not a psychological condition, but a modern attempt to test really thoroughly. Let him demonstrate you the advantages of this approach by playing through an implementation cycle of a small change to your project. It may foster your confidence in the insurance’s power.

Project documentation

Every software project above the trivial level contains so many details that no human brain is able to remember them all after some time. Your developer needs some place to store vital information about the project other than “in the code” and “in the issue tracker”. A popular choice to implement this requirement is providing a Wiki. You probably already know a Wiki from Wikipedia. Think about a web-based text editing tool with structuring possibilities. If you can’t access the documentation tool yourself, let your developer demonstrate it. Ask about an excerpt of your project documentation, perhaps as a PDF or HTML document. Don’t be too picky about the aesthetics, the main use case is quick and easy information retrieval. Even handwritten project documentation may pass your test, as long as it is stored in one central place.

Source code conventions

Nearly all source code is readable by a machine. But some source code is totally illegible by fellow developers or even the original author. Ask your developer about their code formatting rules. Hopefully, he can provide you with some written rules that are really applied to the code. For most programming languages, there are tools that can check the formatting against certain rules. These programs are called “code inspection tools” and fit like hand in glove with the continuous integration server (refer to the third entry). Some aspects of source code readability cannot be checked by algorithms, like naming or clarity of concepts. Good developers perform regular code reviews where fellow developers discuss the code critically and suggest improvements. The best customers explicitely ask for code reviews, even if they won’t participate in them. You will feel the difference in the produced software on the long run.

Community awareness

Software development is a rapidly advancing profession, with game-changing discoveries every other year. One single developer cannot track all the new tools, concepts and possibilities in his field. He has to rely on a community of like-minded and well-meaning experts that share their knowledge. Ask your developer about his community. What (technical) books did he read recently? What books are known by the whole development team? As a customer, you probably can’t tell right away if the books are worth their paper, but that’s not the main point of the question. Just like with tests, the amount of books read by the average programmer won’t make a very long list. If your development team is consistent enough to share a common literature ground, that’s already worth a lot.

But it’s not just books. Even books are too slow for the advancement! Ask about participation in local technical events, like user groups of the programming language of your project. What about sharing? Does the developer share his experiences and insights? The cheapest way to do that is a weblog (you’re reading one right now). Let him show you his blog. How many articles are published in a reasonable timespan, what’s the feedback? Perhaps he writes articles for a technical magazine or even a book? Now you can ask other developers for their opinion on the published work. You’ve probably found a really professional developer, congratulations.

There is more, much more

This list is in no way exhaustive in regard to what a capable developer uses in concepts, skills and tools. This is meant as the minimal set, with a lot of room for improvement. There are compilations of skills like the Clean Code Developer that go way beyond this list. Ask your developer about his personal field of interest. Hopefully, after he finished bragging and techno-babbling for some time, you’re convinced that your developer is a professional one. You shouldn’t settle for less.

A small story about outsourcing

A true story about why it isn’t always cheaper to produce more cost-effective. And a story about a process that wasn’t tailored around human requirements.

Let me tell you a story about human labor and automization, cost efficiency and the result of local optimization. The story itself is true, but nearly all details are changed to protect the innocent.

An opportunity

Once, there was a company that produced sensoric equipment with a large portion of electronic circuitry. The whole device was manufactured at the company’s main factory and admired for its outstanding rigidity. Then, one day, the opportunity offered itself to outsource the assembly of the electronics to a country in the asian region. The company boss immediately recognized the business value in this change: The same parts would be produced by the company, shipped to the asian contract manufacturerer, assembled and promptly returned. Then, the company’s engineers will provide the firmware and software for the final product. By outsourcing the most generic step in the production line, the production costs could be lowered significantly.

A detail

There is one little detail that needs to be told: The sensors relied on some very specific and fragile parts only the company itself could produce. These parts were especially sensible to the atmosphere they were assembled in. A very important aspect of the production process was a special purpose machine that could assemble the parts while sustaining the necessary gas mixture and pressure. Upon closer inspection, one could say that the essence of this product’s secret ingredience weren’t the parts itself, but the specifically tailored production process.

The special purpose machine had to be transferred to the contract manufacturer in asia, otherwise, the sensors could not be assembled. This was a minor inconvenience compared to the large profits that could be realized once the outsourcing was completed.

A success

The machine was transported to the contractor, installed and tested. A special crew of workers of the contractor’s staff was trained to operate the machine properly and within the necessary conditions. After a while, the production line began its work. The first sensors assembled offshore returned home. They all worked as intended. The local engineers couldn’t tell the difference but by looking at the serial number. The company management was pleased, the profitability was increased.

A failure

Everything went well for a while. Then, the local engineers noticed a slightly higher number of faulty sensors. Not long after, the quality assurance reported decreasing performance numbers of the devices. The rigidity of the device, the unique selling point, slowly deteriorated. The company management was worried and established a task force to indentify the root cause for this change to the worse.

A mystery

The task force inspected the reported problems and couldn’t make much sense of the numbers. It wasn’t a problem of whole faulty batches (indicating incidents like transport damage), but also not of individual faulty pieces. Instead, they found that if a piece was faulty, the next few pieces from that series were also faulty. Then, there were long intervals with perfectly good pieces until another group of clearly faulty pieces occurred. Something had to go wrong during the assembly process at the contractor.

A revelation

When the task force arrived in the contractor’s factory and inspected the special purpose machine, they found that the atmosphere regulator was damaged. This automatic part of the machine takes care of the mixture and pressure of the gas in the machine during operation and keeps it in the necessary range by applying or draining specific gas. The contractor didn’t bother to replace the rather expensive part when cheap human labor is readily available. They had hired a worker to perform the atmosphere regulation manually. Some lowly paid worker had to watch the pressure numbers and provide more or less gas, just as needed. This was nearly as good as the automatic regulation and still good enough to produce quality devices.

An explanation

But, the contractor only hired one worker per shift. This worker had to go to the toilet sometimes during the work day. When he was away from the machine, it went along unregulated, soon to be misadjusted to the point of only producing junk. Once the worker returned, he would balance the numbers and bring the machine in the OK state again. This situation occurred periodically, but not too often to taint whole batches. Only during his absence, the series of faulty devices would be produced.

A conclusion

I don’t want to add much moral to this story. Perhaps one thing should be considered when recapitulating: Both the company and the contractor “optimized” their costs locally by making cost efficient decisions that turned out to be expensive in the long run. The company chose between expensive, but controlled local production and cheap outsourced assembly, arguably the most delicate step in the whole production process. The contractor chose between a high one-time investment in an automatism and the low ongoing cost of cheap human labor. Both decisions are comprehensible on their own, but lead to a situation that would never have occurred in the original setting.

A short story about priorities

When you mix your own priorities with the ones of your customers, embarrassing things might happen.

Before I can tell you the story, I have to add this disclaimer: The story is true, but it wasn’t us. To protect the identities, I’ve changed nearly all the details, hopefully without watering down the essence of the story.

The assignment

A few years ago, when developing software for mobile devices wasn’t as ordinary as it is today, a freelance developer got a call from a loyal customer. He should develop a little web application that could also be used on a smartphone. The functionality of the application was minimal: Show a neat image containing two buttons to download a song teaser in two different file formats, probably MP3 and WMA. That’s all. The whole thing was a marketing app that should be used a few weeks or months and then be mothballed. Being a marketing app, the payment was very decent.

The development

The freelancer was excited to say the least. This was his chance to enter the emerging field of mobile device software development. And because it should be a web application, he would try out all these new fancy mobile web frameworks and settle for the best. The image of the website should rotate and adjust to the device screen if the user tilted his phone. And because phone screens weren’t that well equipped with pixels as today and mobile internet was expensive, the image should be optimized for the best possible tradeoff between details and file size.

Some mobile web frameworks provided access to the device type, so there were all kinds of javascript magic tricks to be potentially applied. Well, they weren’t applied, but very well could have been. The freelancer prototyped most of them, out of curiosity.

The web application itself was completed in one session once the song files and the master image were provided by the customer. The freelancer installed everything on a staging web server and communicated back success. The customer was pleased by the quick reaction time and appointed a meeting to inspect the results.

The presentation

The customer gave the freelancer a notebook attached to a beamer to present the application. The notebook had a screen resolution many times the envisioned target platform, so the intro image was very pixellated and far from appealing. The freelancer used his smartphone to present the image on a smaller screen, but the customer only shook his head: “You should develop a web application for both phones and PCs. Most of our customers will use their PC to visit the site.” The freelancer apologized and promised to add another browser switch to present a full resolution image to PC users.

Now, the freelancer continued to present the web application on his phone. He invited the customer to tilt the device and was satisfied when the web site adjusted perfectly. The customer was pleased, too. Then, he tapped on one of the buttons. Nothing happened. The whole application consisted only of two buttons and one wasn’t working. The freelancer frantically tried to figure out what had gone wrong, when the customer tapped on the other button. No reaction from that one, too. “But I’ve uploaded both song files to the right location with the right access rights.”, the freelancer just said to himself when it dawned him: He forgot to insert the link tags in the HTML file. The buttons were just images. He never actually clicked on one of the buttons during development.

The moral of the story

Recapitulating, the freelancer was asked to develop a web application with an image and two download buttons, but he managed to cripple the image for the larger part of the anticipated users and never provide any download functionality at all. The customer’s requirements somehow got lost along the way.

This isn’t so much a story about a confused freelancer or improper requirement analysis. It’s a story about priorities. The customer expressed his priorities through the requirements. The freelancer superimposed his own priorities very early in the process (without telling anybody) and never returned to the original set until the presentation. And while it is granted to have a secondary agenda as a service provider, it should never interfere with the primary agenda – the one set by the customer.

Don’t fool yourself into thinking that this could never happen to you. It’s not always as obvious as in this story. Some, if not most of your customer’s priorities are unintentionally (or even intentionally) kept secret. They can only be traced during live exercises, which is why early prototyping and using the prototype in real scenarios is a good way to reveal them.

A review of the year 2011 at Softwareschneiderei

This is a review of the year 2011 for the Softwareschneiderei, a software development company from Karlsruhe, Germany.

The current year 2011 is coming to an end. This is the traditional time to pause and reflect on what has happened. This blogpost tries to sum up our year in software development at Softwareschneiderei. It was an interesting, entertaining and successful year for us, that’s for sure.

The official parts

Our developer blog was alive throughout the hardest times of the year, when everyone was under full project load. Every week, one of our developer shares a little posting with the world. The blog is still managed by token only. Looking at the visitor statistics, we fully appreciate your attention. The first blog post of this year looked at the remainder of a failed project. “A tale of scrap metal code” was a detailed vivisection in three parts. Over the course of the year, we wrote about bogus error messages, Groovy, Grails, GORM and some confessions about coding style and multithreading. If you have the time, spend a few minutes to browse our blog post archive for this year.

Our “official” company blog, written in german language, had no activity this year. We can certainly do better than this and take it on the list for 2012.

The company homepage, written in german language, had continuous updates and extensions this year. We coupled the “Open Source Love Day” (OSLD) with our “Homepage Comittee”, when every employee has to improve the homepage in some aspects and present the change to the “comittee”. Unfortunately, this somehow lead to fewer OSLDs this year.

The ongoing Dev Brunch sessions thinned out a bit in the second half of the year. This was a concession to the ever-growing workload. We strive to establish a tighter schedule with accompanying blog posts in the next year.

The internal parts

We were under heavy development load this year. This isn’t a bad thing, but impacts the internal communication and team building process. We tried to cope a bit by restructuring the Open Source Love Day to a “Team Day”, when the whole team meets and works on various internal or hobby projects. Some of these days were spend on Code Camps and other training events. This means less love for the open source community, but crucial together time for us.

We picked up several “new” programming languages this year. You can tell by the blog posts that we worked with Python, Ruby, Flex/ActionScript and even VisualBasic on real projects. The VisualBasic experience was a little epiphany that it’s really the developer and not the language that leads to shitty software.

One method of continuous improvement is our “creativity budget”. Basically, it’s money every developer can spend to improve his workplace. This budget wasn’t used at all this year, as the workplaces seem to be optimal. This cannot be true, so we bought brand new computers or a big RAM upgrade for everyone. And every computer has a big enough SSD now. We took our own advice seriously and invested in our productivity.

Our developer crew grew again this year. We are beginning to think about the remaining space in the new office again. But as usual, we grow slowly and deliberately. There’s nothing worse than a team of strangers.

Conclusion

The year 2011 was great! We’re looking forward to the year 2012, with our motto of christmas 2011: “cheery and spry” (the original motto is in german language “froh und munter”, I hope the translation caught the original spirit).

Have a great turn of the year, everyone. We’d love to see you again next year.

Story about bogus error messages

Most computer users know the situation where some system or service that worked for months without problems suddenly stops working. I want to tell you of a small war story which happened some weeks ago to us.

We are maintaining an own mail server with imap access for our employees. That allows for relativly easy serverside spam protection, mailing list management and archiving and so on. We use the trusted combination of postfix, cyrus and mailman for the task and everything works very reliably. Then suddenly we got the error message ssl_error_rx_record_too_long in our e-mail clients. Nothing had changed software-wise. Googleing on the internet brought up all kinds of different obscure reasons for this error but no explanation why something like that would happen out of thin air.

Fortunately, looking in cyrus’ log files quickly showed the reason: the hard drive was full! A two days before the mail system failing there was a larger upload to the server for sharing stuff with a colleague. This upload fitted almost exactly onto the free disk space and some mails later the disk was full. It was really a murphy’s law situation because some kilo bytes less free space would have made the file sharing fail with a sensible error message. But it worked and made the mail system fail suddenly without immediate connection some changes to the server.

There are some lessons to be learned here:

  • Aside from file managers most applications assume memory and disk space are unlimited. If they do hit such a limit they usually fail miserably with complete bogus errors.
  • Monitor critical resources on important systems to receive warnings ahead of time before important service fail. Tools like Nagios can help here.
  • Try to be aware of side effects of your actions. Separating services to different machines may help to reduce unexpected side effects on seemingly unrelated stuff. We used the server to run many different unrelated services.

Verbosity is not Java’s fault

One of Java’s most heard flaws (verbosity) isn’t really tied to the language it is rooted in a more deeply source: it comes from the way we use it.

Quiz: Whats one of the most heard flaws of Java compared to other languages?

Bad Performance? That’s a long overhauled myth. Slow startup? OK, this can be improved… It’s verbosity, right? Right but wrong. Yes, it is one of the most mentioned flaws but is it really inherit to the language Java? Do you really think Closures, annotations or any other new introduced language feature will significantly reduce the clutter? Don’t get me wrong here: closures are a mighty construct and I like them a lot. But the source of the problem lies elsewhere: the APIs. What?! You will tell me Java has some great libraries. These are the ones that let Java stand out! I don’t talk about the functionality of the libraries here I mean the design of the API. Let me elaborate on this.

Example 1: HTML parsing/manipulation

Say you want to parse a HTML page and remove all infoboxes and add your link to a blog box:

        DOMFragmentParser parser = new DOMFragmentParser();
        parser.setFeature("http://xml.org/sax/features/namespaces", false); 
        parser.setFeature("http://cyberneko.org/html/features/balance-tags", false);
        parser.setFeature("http://cyberneko.org/html/features/balance-tags/document-fragment", true);
        parser.setFeature("http://cyberneko.org/html/features/scanner/ignore-specified-charset", true);
        parser.setFeature("http://cyberneko.org/html/features/balance-tags/ignore-outside-content", true);
        HTMLDocument document = new HTMLDocumentImpl();
        DocumentFragment fragment = document.createDocumentFragment();
        parser.parse(new InputSource(new StringReader(html)), fragment);
        XPathFactory factory = XPathFactory.newInstance();
        XPath xpath = factory.newXPath();
        Node infobox = xpath.evaluate("//*/div[@class='infobox']", fragment, XPathConstants.NODE);
        infobox.getParentNode().removeChild(infobox);
        Node blog = xpath.evaluate("//*[@id='blog']", fragment, XPathConstants.NODE);
        NodeList children = blog.getChildNodes();
        for (int i = 0; i < children.getLength(); i++) {
            node.remove(children.item(i));
        }
        blog.appendChild(/*create Elementtree*/);

What you really want to say is:

HTMLDocument document = new HTMLDocument(url);
document.at("//*/div[@class='infobox']").remove();
document.at("//*[@id='blog']").setInnerHtml("<a href='blog_url'>Blog</a>");

Much more concise, easy to read and it communicates its purpose clearly. The functionality is the same but what you need to do is vastly different.

  The library behind the API should do the heavy lifting not the API's user.

Example 2: HTTP requests

Take this example of sending a post request to an URL:

HttpClient client = new HttpClient();
PostMethod post = new PostMethod(url);
for (Entry param : params.entrySet()) {
    post.setParameter(param.key, param.value);
}
try {
    return client.executeMethod(post);
} finally {
    post.releaseConnection();
}

and compare it with:

HttpClient client = new HttpClient();
client.post(url, params);

Yes, there are cases where you want to specify additional attributes or options but mostly you just want to send some params to an URL. This is the default functionality you want to use, so why not:

  Make the easy and most used cases easy,
    the difficult ones not impossible to achieve.

Example 3: Swing’s JTable

So what happens when you designed for one purpose but people usually use it for another one?
The following code displays a JTable filled with attachments showing their name and additional actions:
(Disclaimer: this one makes heavy use of our internal frameworks)

        JTable attachmentTable = new JTable();
        TableColumnBinder<FileAttachment> tableBinding = new TableColumnBinder<FileAttachment>();
        tableBinding.addColumnBinding(new StringColumnBinding<FileAttachment>("Attachments") {
            @Override
            public String getValueFor(FileAttachment element, int row) {
                return element.getName();
            }
        });
        tableBinding.addColumnBinding(new ActionColumnBinding<FileAttachment>("Actions") {
            @Override
            public IAction<?, ?>[] getValueFor(FileAttachment element, int row) {
                return getActionsFor(element);
            }
        });
        tableBinding.bindTo(attachmentTable, this.attachments);

Now think about you had to implement this using bare Swing. You need to create a TableModel which is unfortunately based on row and column indexes instead of elements, you need to write your own renderers and editors, not talking about the different listeners which need to map the passed indexes to the corresponding element.
JTable was designed as a spreadsheet like grid but most of the time people use it as a list of items. This change in use case needs a change in the API. Now indexes are not a good reference method for a cell, you want a list of elements and a column property. When the usage pattern changes you can write a new library or component or you can:

  Evolve your API.

Designed to be used

So why is one API design better than another? The better ones are designed to be used. They have a clearly defined purpose: to get the task done in a simple way. Just that. They don’t want to satisfy a standard or a specification. They don’t need to open up a huge new world of configuration options or preference tweaks.

Call to action

So I argue that to make Java (or your language of choice) a better language and environment we have to design better APIs. Better designed APIs help an environment more than just another new language feature. Don’t jump on the next super duper language band wagon because it has X or Y or any other cool language feature. Improve your (API) design skills! It will help you in every language/environment you use and will use. Learning new languages is good to give you new viewpoints but don’t just flee to them.