Author: vasilikvockin

Working on software as a free time activity

Why would somebody do this? Isn’t it already enough to code at work for eight hours a day, five days a week? If you ask yourself this questions, then I think you should reconsider your position.

There is a fundamental difference between work and free time. You are not constrained. You don’t have to hold a deadline. Software development is a mentally challenging task, and while some time pressure keeps you focused, a little bit more forces you to cut corners instead of considering better alternatives. If deadlines were good, they wouldn’t have “dead” in their name. In your free time you decide when you are done.

Even having considered alternatives you are not always able to implement them. There may be a corporate identity that doesn’t contain your favourite flavour of pink. There can be a module licensed under a non-commerial-only license. Or maybe your company uses an old framework missing the latest features. No such problems in your free time.

There is a theory that mastery comes from practice. By coding in your free time, you can decide whether you invest your time in deeper knowledge of some topic or in a broader horizon thus becoming a valuable employee. And sharing freshly won knowlege and experience can increase your reputation as colleague too.

The social among us even meet like-minded people at events like Java User Group or Schneide Dev Brunch. Here the amount of transported information is  much higher, since everyone has another background and focuses on another things. You can even share your mistakes and laugh with others about them.

Are there any side-effects of free time coding besides those listed before? Yes. Your personality can change. It is possible that you will gain a positive attitude and start invest your free time in your skills. Maybe you’ll even start to motivate others to do likewise.

Ugly problems, ugly solutions?

Do have workarounds to be worse than the problems?

One type of our projects is to integrate some devices into our customers infrastructure. The tasks then mostly consist of writing bridging code for third party libs of the hardware vendor. The most fun part is when the libs do not have some needed capability or feature.

The situation

In my case I was building a device driver with following requirements:

  • asynchronous execution of long running tasks.
  • ability to cancel long running tasks.
  • at any time it is asked for its current status, it has to provide it.

The device is accompanied by a DLL with a following interface(simplified):

  • doWork(), a blocking function that returns after a configurable amount of time that can range from milliseconds to hours.
  • abortWork(), is supposed to cancel the process triggered by doWork() and to make doWork() return earlier.

First impressions

I was able to fullfill two requirements pretty fast. The state ist more or less a simple getter and the doWork function was called in a separate thread. Just cancelling the execution didn’t work. More precisely it didn’t work as expected. In the time between a call to doWork() and the moment it returned, the process always used 100% of one CPU core. After that it always dropped to nearly zero. Now, what happened, when I called abortWork()? There were two things: doWork() returned, but the CPU utilization stayed the same for an indefinite amount of time. Or the call was ignored completely. Especially funny was the first case, where the API seemed to work until the process run out of cores and the system practically grinded to a halt.

The “Solution”

Banging my head against the desk didn’t help, so my first thought was to forget abortWork() and kill the thread myself. Microsoft provides a nice function called TerminateThread for that purpose. Everyone who looks at the documentation, will see that the list of side effects is quite impressive, memory leaks being the least bad ones. I couldn’t guarantee that the application would work afterwards, so I decided against it. What would be the alternative? Process shutdown. When you stop the process all blocked threads should be away. Being too soft and trying to unload the DLL is a bad idea – you have a deadlock when the DllMain waits for the worker thread to finish. My last attempt was to suicide the process!

Now I was able to abort a running task, but my app were no longer available all the time. Every attempt to get the current status between the start of a shutdown and a completed startup failed. So a semi-persistent storage containing the last status of a living application was needed. To achieve this, I created an application with the same interface as the real device driver and proxy that delegated all the requests to it, caching the status responses. That way the polling application still assumed that the last action were still running until the restarting app was fully available again.

In the end the solution consisted of two device drivers, one for caching the state and the other for doing the work. When cancelling the task was required, the latter device driver died and restarted itself again.

Final thoughts

I hope that there is a way to do this in a more elegant way and I just overlooked some facts. It is unbelievable that you can lose all control over your app by a simple call to a third party library and that the only escape is death.

Building Visual C++ Projects with CMake

In previous post my colleague showed how to create RPM packages with CMake. As a really versatile tool it is also able to create and build Visual Studio projects on Windows. This property makes it very valuable when you want to integrate your project into a CI cycle(in our case Jenkins).

Prerequisites:

To be able to compile anything following packages needed to be installed beforehand:

  •  CMake. It is helpful to put it in the PATH environment variable so that absolute paths aren’t needed.
  • Microsoft Windows SDK for Windows 7 and .NET Framework 4 (the web installer or  the ISOs).  The part “.NET Framework 4” is very important, since when the SDK for the .NET Framework 3.5 is installed you will get following parse error for your *.vcxproject files:

    error MSB4066: The attribute “Label” in element is unrecognized

    at the following position:

    <ItemGroup Label=”ProjectConfigurations”>

    Probably equally important is the bitness of the installed SDK. The x64 ISO differs only in one letter from the x86 one. Look for the X if want 64 bit.

  • .NET Framework 4, necessary to make msbuild run

It is possible that you encounter following message during your SDK setup:

A problem occurred while installing selected Windows SDK components. Installation of the “Microsoft Windows SDK for Windows 7” product has reported the following error: Please refer to Samples\Setup\HTML\ConfigDetails.htm document for further information. Please attempt to resolve the problem and then start Windows SDK setup again. If you continue to have problems with this issue, please visit the SDK team support page at http://go.microsoft.com/fwlink/?LinkId=130245. Click the View Log button to review the installation log. To exit, click Finish.

The reason behind this wordy and less informative error message were the Visual C++ Redistributables installed on the system. As suggested by Microsoft KB article removing them all helped.

Makefiles:

For CMake to build anything you need to have a CMakeLists.txt file in your project. For a tutorial on how to use CMake, look at this page. Here is a simple CMakeLists.txt to get you started:

project(MyProject)
 cmake_minimum_required(VERSION 2.6)
 set(source_files
 main.cpp
 )
 include_directories(
 ${CMAKE_CURRENT_SOURCE_DIR}
 )
 add_executable(MyProject ${source_files})

Building:

To build a project there are few steps necessary. You can enter them in your CI directy or put them in a batch file.

call "%ProgramFiles%\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /Release /x86

With this call all necessary environment variables are set. Be careful on 64 bit platforms as jenkins slave executes this call in a 32 bit context and so “%ProgramFiles%” is resolved to “ProgramFiles (x86)” where the SDK does not lie.

del CMakeCache.txt

This command is not strictly necessary, but it prevents you from working with outdated generated files when you change your configuration.

cmake -G "Visual Studio 10" .

Generates a Visual Studio 2010 Solution. Every change to the solution and the project files will be gone when you call it, so make sure you track all necessary files in the CMakeLists.txt.

cmake --build . --target ALL_BUILD --config Release

The final step. It will net you the MyProject.exe binary. The target parameter is equal to the name of the project in the solution and the config parameter is one of the solution configurations.

Final words:

The hardest and most time consuming part was the setup of prerequisites. Generic, not informative error messages are the worst you can do to a clueless customer. But… when you are done with it, you are only two small steps apart from an automatically built executable.

Procrastination

Not doing things at the time they ought be done seems to be a known problem for as long as people are able to plan. They know the consequences and knowingly decide to delay the inevitable. When do you encounter such situations in your professional life? How can they be improved?

Deadline

is the trump card that can be played every time someone wants to put off a task that hinders but not completely stops the process. I’ve seen it often when talking about refactorings or automated tests – “We can do it when we have time”. When the situation does not permit changes to the schedule it is a valid and appropriate decision to delay them until later. Delaying again and again is not. To improve the situation it is necessary that everyone in the chain in the development process from the developer to the customer[1] knows the consequences of accumulating technical debt. In most cases they are just ignorant of the fact or of its long term impact and will change the course of action when “enlightened”. The next step should be the reduction of workload to catch up and improve the product in time.

Fear

of failure or fear of consequences is also often the reason someone delays important tasks. The fear can be fostered through misinterpreted quality standards. A little checklist to keep in mind:

  • Breaking the build is not bad. It is bad to let it stay broken.
  • Checkstyle is not everything. There are situations when people are “more right”. Dogmatism is unhealthy anyway.
  • Man is not a machine. Nobody can be expected to work full time without making mistakes. Even hardware fails.

To improve this situation it is necessary to revise the values and make sure that the consequences are reduced to the level where they are noticeable but not hurting or even crippling. This way any person can make decisions – even wrong ones – without anticipation of immediate punishment. Not all working environments are that toxic. Sometimes the person is just inexperienced or has low self esteem. Here teaching, leading by example and encouragement to take responsibility helps a lot.

Boredom

is like torture for conscious mind. A boring task can be endured once, but doing it every day is very unconfortable. Fortunately when something can be predicted, it can be automated. Cron jobs, continuous integration systems or small round vacuum cleaners serve as example. Even our brain automates processes by pushing them into the subconscious mind – this way we are able to speak and walk at the same time without any problems. Some bureaucratic routines however, like time tracking, cannot be automated fully and require our intervention at regular intervals. The natural behaviour to increase the intervals between the tasks or forgetting them ultimately leads to the situation comparable to technical debt where a huge amount of undone tasks grinds a system to a halt. Since most companies require some amount of bureaucracy to function, it is not possible to completely optimize it away. But it is possible to reduce the chance to forget a task. As example we are always wearing a real cowboy hut when we work in a production environment. Only when we are done, we can put it off. To make boring tasks more attractive the idea of continuous integration game can be used. In time tracking context the user would get points for booking time at the same working day and less for booking at the next day.

Conclusion
Simplified the equation is: pressure * negative feelings towards the task = danger of procrastination. To improve the situation, lower the factors!

[1] Especially in product development: Customer is not the one who buys or uses the product, but the one who pays the developer to make the product and later wants to make money with it.

FTP integrated

When developing a feature containing unknown technology or hardware, I prefer a spike followed by integration tests. Sometimes it helps a lot.

How it all began
One of our customers employs NAS for data storage, accessing it per FTP. Some of the features like copying and moving files around were already implemented by us using Apaches FTPClient. The next feature on the list was “cleanup after x days” – deletion of files, or more important: directories. FTP, being a pretty basic protocol, does not allow for recursive deletion of directories. The only way to do it is to delete the deepest elements first,  going up one level and repeat – or in other words – implementing the recursion yourself. This was too much for our simple feature, so the decision was made to hide the complexity behind a VirtualFile, an interface already existing in our framework.

Being a novice in speaking FTP I was happy to hear that we already have acquired exactly the same type of NAS the customer has. To see how the system behaves (or not) and document it at the same time, I decided to implement the interface integration test first.

Fun
As the amount of tests and file operations started to grow, so did grow the round trip time of my test/make test pass/refactor cycle and my patience dwindled. I switched from NAS FTP-Server to a local FileZilla FTP-Server. It worked like a charm and all necessary features were implemented really fast.

The next step was to run the app using the new feature with real amount of data, real directory structure and our NAS. It failed miserably. And randomly. The app suffered from closed connections while trying to open a data connection. After some search the reason was found: FTPClient we use had active mode enabled by default. That means that to transfer data the server tried to connect to the client and the clients Firewall did not like it. After setting connection mode to passive the problem was solved.

The tests run fine, but they run slow. And they introduced a dependency on an external system. If that system broke or were disabled for any other reason, our CI would report failure without any changes in the code. Both points could be addressed by using an embedded FTP Server. We choose Apaches FTP Server. Changing the tests was easy, since the only thing to do was to setup the server before the test and to shut it down afterwards. Surprisingly some tests failed. Apaches server handled some cases differently:

  • it allowed opening output streams to directories without any exception
  • it forbid to delete current working directory
  • the name listing in the directory (NLST) returned by NAS were absolute paths to the file, Apaches server returned simple names.

After another code change the code worked correctly with all three servers.

Lessons learned
While implementing the interface I learned much about how to create and test bridging functionality:

  • Specification cannot replace tests. Searching for the FTP commands to use I looked at several websites that described the commands. None of them wrote about whether NLST returns absolute paths or only filenames. There are always holes in the spec that will be interpreted differently by vendors or the vendors do not always obey it.
  • Unit tests are great, but they are limited to your code only. When it comes to communication between system components, especially communication with foreign systems, an integration test is a must.
  • Working with a test setup that mimics production environment as close as possible is great. Without the NAS, the app would have simply failed in the best case. In the worst case it would have deleted wrong files. Neither of them make a customer happy.