Tag: rxtx

Open Source Love Day July 2010

Our Open Source Love Day for July 2010 brought love for Hudson (especially the CMake and Crap4j plugins), RXTX and JUnit.

This friday , we held our Open Source Love Day for July 2010.  We began with several internal meetings and discussion (like the Homepage Comittee meeting) and dived right in our work afterwards. Everybody had a little backlog of issues that we wanted to get done on this day. Nearly everybody succeeded (well, the author had a minor delay – read about it below). The day went by in a very fast pace, but it felt right.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

  • There are really cool new features in the latest JUnit versions and Rules are one of them. What hurt our aesthetic sense was that the field that hold the Rule instance has to be public. Checkstyle was on our side, so we tweaked JUnit to allow all kinds of visibility. You can read about the change needed here: http://github.com/KentBeck/junit/issues#issue/31. The fix is almost trivial and will hopefully be incorporated in the next versions of JUnit, so we do not publish our altered version.
  • We constantly receive requests and remarks about our cmake plugin for Hudson. This lead to a new version of the plugin fixing two issues with matrix builds and custom build types. Head over to the plugin homepage and grab the new version 1.6. The issues were in detail:
    • The plugin can be used with matrix builds now
    • Custom build types can be defined now
  • RXTX is our choice for serial port communication with Java. We fixed some issues during the last few OSLDs, with one issue left for today: When you flush your stream while using a special type of usb-to-rs232 converter, you got an exception. The corresponding issue is #102 in the RXTX issue tracker. We proposed a patch that fixes the problem.
  • Another hudson plugin is our crap4j reporter. It lacked some love for months now and finally broke when used with the latest hudson versions. Fixing the problem was a lot harder than we thought, basically because the plugin needed adjustments to recent API changes and we couldn’t figure out exactly what adjustments are necessary. You might have a look at the developer mailing list thread for this question. Finally, we got it resolved (on sunday, with a sudden stroke of insight) and a new version 0.8 is published.
  • We use an internal time tracking tool for our projects. This tool isn’t specifically open source yet, but continues to grow in terms of features and usability. The work invested in this tool helps us to continue with the OSLD, so it’s beneficial work nonetheless.
  • During the last OSLD, we had plans for a new hudson plugin and even produced a prototype. This time, we looked around the hudson plugin zoo (it’s getting a bit difficult to keep track of all of them) for inspiration and found a wonderful piece of art: The Groovy Postbuild Plugin. Using this plugin with a small groovy script served our needs exactly. No need for a full-blown plugin when you can scratch your itch with a simple script. Thanks to Serban Iordache for his great work!

What were our lessons learnt today?

  • If you need to setup a fresh workspace for an open source project, consider to prepare it over the night before, or the download delay will kill your precious work time. There is nothing more frustrating than staring at a “downloading…” progress bar while being eager to start programming.
  • Always look around what others have done before. We wanted to build a full hudson plugin from scratch when all we needed was a little groovy script placed inside another plugin. Sweet!
  • Do not hesitate to privately fix open source issues that won’t get done in time for you. Just make sure to have a management process in place to track those changes and be able to re-apply them to future versions. More important though, be able to tell exactly when NOT to re-apply them because the original project has fixed the issue.

Retrospective of the OSLD

The OSLD went smooth and was productive. We tend to work on backlogs instead of searching for random issues now, but that’s just a sign that our approach has matured and we depend on the OSLD to get work done.

Last wednesday, we held our Open Source Love Day for June 2010. This one was productive despite the heat that had us sweating the whole day long (as a sidenote: it got even warmer the days afterwards). Some features were finished and will help at least us in our projects. We still look forward for the right way to release them. Another release was even more problematic, you will read about it below.The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

Open Source Love Day June 2010

Our Open Source Love Day for June 2010 brought love for Hudson (especially the Campfire Plugin), Launch4j and RXTX. Everything went smooth and we were soaked with sweat due to massive sunshine.

Last wednesday, we held our Open Source Love Day for June 2010. This one was productive despite the heat that had us sweating the whole day long (as a sidenote: it got even warmer the days afterwards). Some features were finished and will help at least us in our projects. We still look forward for the right way to release them. Another release was even more problematic, you will read about it below.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

  • Launch4j is a java application launcher for Windows, handling all the stuff a startup script would do, too. At the last OSLD, we added the ability to restart the application in case of a crash or other unplanned exit. To utilize this feature for automatic update routines, we needed to add the additional feature of starting another command instead of the original one. If the program fills a special file with the command needed, Launch4j will execute it after the program’s exit. This patch builds onto the previous patch and we are still investigating how to publish this functionality without breaking backward compatibility. We are looking forward to release it on the next OSLD.
  • We use RXTX to perform the serial (RS232) communication on all our java projects. We worked on an issue with serial converters over the course of several OSLDs now and released the patch to the issue tracker of RXTX after a longterm stability test. See the reworked patch for issue #144. There is another issue with the flush() method that seems to affect not only virtual RS232 ports that we currently investigate. But we aren’t yet able to come up with a complete issue description or a fix, so this will be suspended until the next OSLD.
  • We have written the Campfire Hudson Plugin as part of previous OSLDs. When issues emerged, we got patches from the community here. Thank you guys! We included the changes to the code and prepared a new release, when maven failed. This is not an issue, except when it fails repeatedly and messes up the workspace and the repository. After a long time of helpless fiddling with the parameters, we decided to start over and increase the version number to 2.1 (instead of 1.2). All of a sudden, everything worked out fine. Maven is a mysterious beast.
  • The initial work for a New Hudson Plugin was made. One tradition of the OSLD has always been to scratch our own itches. While there are many useful hudson plugins, we have the immediate need for another one that doesn’t exist yet. Without going into details here (we save this for the next OSLD), we produced a proof of concept and a first iteration of the code. Stay tuned for details on the next OSLD.

What were our lessons learnt today?

  • If you don’t succeed with maven’s automatic processes, do not try to sort out things manually. You’ll just end up with a gigantic mess that won’t work either way. The best way to deal with maven failures is to revert everything and try again with different parameters.
  • The best approach to develop hudson plugins is to adapt the old “monkey see, monkey do” process. There are so many plugins already, chances are good your immediate question was already answered somewhere. Just check the found solution for accidental complexity. Sometimes, the first solution isn’t the easiest.
  • When dealing with the legacy Win32 API, combined knowledge scraping is king. We had discussions throughout the day that only consisted of little parts of recollections about knowledge that seemed long forgotten. But finally, we put the pieces together and solved the problem. It should be called teamthink, i guess.

Retrospective on the OSLD

The weather at this OSLD was way too hot to operate at normal speed. But we got some nice results and a cliffhanger for the next OSLD. We left soaked with sweat but happy that evening.

Open Source Love Day May 2010

Our Open Source Love Day for May 2010 brought love for Netty, Launch4j and RXTX. Everything went smooth again this time.

Last friday, we held our Open Source Love Day for May 2010. After the excessive yak shaving of last month’s OSLD, this one was more productive. We got one feature right nearly instantly and several others throughout the day. To keep us from feeling depressed, we allocated dedicated “research time” to get to know a project before starting to develop on it. So one point on our current list of accomplishments is only research work that should be counted as “thought investment” for further improvements.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

  • Netty is a java framework to perform network communication, transparently using different transport mechanisms. We use it in a customer project environment and it worked mostly well. But we noticed some issues that might have to do with the framework, so we decided to put it under development. But before hopping in right away, we invested some hours to really anatomize it. Our achievement isn’t noticeable yet, but might be in the future.
  • Launch4j is a java application launcher for Windows, handling all the stuff a startup script (the infamous start.bat) would do, too. One thing that’s easy to perform in a batch script is to restart the application if it went down with an error. We wanted this feature in Launch4j and we succeeded. Before releasing the changes to the project, we need some rethinking of our approach, as it is very special to our case, but if you are interested, we send you the patch on request.
  • RXTX is a project to perform serial (RS232) communication. Over the course of the last OSLDs, we worked on an issue that appears in combination with serial converters. See the OSLD posting for march 2010 for details. This issue seems to be completely fixed now. We will run some longterm stability tests with it and then release a patch to the issue tracker of RXTX, issue #144.
  • Our internal tool for business administration was greatly improved this time. The tool isn’t open sourced yet, but we are not afraid to publish it in the future. The new features will directly decrease our administrative overhead and give us more time to perform our main duty – developing software.

A sidenote regarding the yaks (see last month’s OSLD posting to understand this one)

Our monthly “Homepage Comittee” meeting went smooth this time. The setup pain of last month is now paying out and we got our homepage updated. It seems the yaks haven’t grown beards again yet.

What were our lessons learnt today?

  • Pure C code tends to produce endless code blocks. Simple refactorings like extract method/function take a long time and aren’t supported by the IDE. The sentence “there is no boolean!” left some of us impressed, too.
  • The C/C++ Eclipse workbench (in fact, a bunch of plugins for Eclipse) was a joke instead of a help. It didn’t exactly prevent us from doing our work, but that’s all it did. Launch4j came along with project files for the Bloodshed Dev-C++ IDE, which – besides its bloody name – seems to be up to its tasks. We decided to use Dev-C++ over Eclipse for development on Launch4j.
  • Hpricot is a very versatile HTML parser/manipulator. It is just pure joy to work with. (Ok, we knew that one before)

After the fruitless OSLD of april, this one was a great relief for us. The new restart feature for Launch4j worked out instantly and will solve many of our current immediate problems (and some of yours, hopefully). The solution for RXTX got us out of trouble, too. All in all, this was a successful OSLD.

Open Source Love Day April 2010

Our Open Source Love Day for April 2010 didn’t yield the desired results. Instead, we went to give our yaks a shave.

Today, we held our Open Source Love Day for April 2010. Well, to be honest, we tried to. We didn’t achieve that much to present publically and felt really depressed at the end of the day. At least, we can try to explain what went wrong. It all has to do with our own breed of yaks that needed to be shaved today. But first, lets explain the basics.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

Today, we reached out to perform the following tasks:

  • RXTX is a fine java library to access the (rather legacy) serial port to send and receive data over the RS232. On our last Open Source Love Day, we proposed a patch for a problem with disappearing emulated COM ports. This happens a lot if you use USB or ethernet converters and unplug them accidentally or because your intranet got shut down. The patch works well, but not perfectly. We noticed that the IOException we introduced gets thrown much too often. Today, we wanted to fix our fix. We only came to the conclusion that the problem might be a lot more complex than we initially thought.
  • Launch4j is another java library/tool we use to deliver production-ready software bundles. It’s a launcher that increases the user experience at the critical “first contact” for Windows users. The project is mature and useful, but out of development for nearly two years now. We want some features in it, so we thought we might build them today. But the yaks didn’t let us.
  • We have an internal tool for our business administration that desperately needs some hugs. So even if this isn’t strictly Open Source yet, we allocated some time to add a few features. But they outgrew their timebox, too.

At the end of the day, we didn’t accomplish anything ready to be released. That’s frustrating to us and has no value to anybody else. We promise to do better next time.

How the yaks got shaved

Every organization has a secret stable where yaks are raised just for the sake of raising yaks. Once a year or so, they need a shave, so you’ll find yourself shaving yaks before you know what’s happening. That’s what happened to us today.

The term “yak shaving” isn’t our invention. It describes the result of a process when in order to accomplish task A, you need to do task B first. Task B can’t be done before tasks C and D both are completed. Task D, you figure it already, is dependent on task E, which in turn… Soon and inevitably, you’ll find yourself shaving yaks because it is what needs to be done to accomplish your todo list.

This is our story: On every Open Source Love Day, we also gather to have a short “Homepage Comittee” meeting. This is our formal approval process to publish content on our website. One improvement we wanted to implement today was an automatic publishing process. It copies the content of our internal Content Management System to our webserver. Simple as that. This was the starting point of an odyssee.

To copy the content, we used wget. But wget in the version of our distribution had a bug (and missed a feature), so we needed to update wget… Soon afterwards, we found ourselves upgrading our main PostgreSQL database (which went incredibly smooth but took forever) and wrestling with certain dependencies of other parts of the system. We realized we are now officially shaving yaks when we got the error message:

You need to have ruby gems 1.3.2 installed. Your version is 1.3.1!

Remember we started with taking a snapshot of our CMS using wget. Clearly, separating our services to more machines would have helped a lot. But that would’ve taken more than one day to do. It’s on our wish list for the next Admin Day.

Let’s conclude this blog post with the insight that we need to timebox our work better when working for the OSLD. And our yaks are shaved and ready for the summertime.

About PrintStream and Exceptions

Several of our projects deal with sensor hardware of different types often connected via the good old™ serial port. That is fine most of the time because most protocols are simple and RXTX provides a nice cross-platform library for most of your serial port needs. But many new computers do not feature the old RS232 serial ports anymore or other contraints prevent the use of a plain RS232 serial port. Here come serial converters like the Advantech ADAM 4570 (serial-to ethernet) or usb-to-serial converters into play. Usually this works fine.

Now one of our customers had a test system using an unreliable converter with sensor hardware. The hardware problems uncovered a robustness issue in our software which crashed the JVM when the virtual serial port of the converter disappeared and our app tried to write to it. Despite the faulty hardware our software had to be robust because it manages many more devices other than just that one sensor over serial. Looking at the problem we discovered that the crash occurred somewhere in the native part of RXTX. So we decided to scratch our own itch (and the one of the customer) and set out to fix the issue in RXTX at a Open Source Love Day (OSLD) . So we fixed the problem and submitted the patch to the bugtracker of the RXTX project. Our sample program now worked flawlessly and threw an IOException when the serial port failed in some way.

Happy to have fixed the problem we incorporated the patch RXTX in our production software but it still crashed and no IOException appeared anywhere in the logs. After another bughunting session we spotted the subtle difference of sample and production program: the use of OutputStream insted of PrintStream. PrintStream silently swallows all exceptions which proved fatal in our use case with the unreliable stream carrier. So the final fix was essentially replacing our PrintStream code

RXTXPort port = new RXTXPort("COM6");
PrintStream p = new PrintStream(port.getOutputStream(), true, "iso8859");
p.print("command");

with using OutputStream directly:

RXTXPort port = new RXTXPort("COM6");
OutputStream o = port.getOutputStream();
o.write("command".getBytes("iso8859"));

Conclusion

Be careful when using PrintStream with unreliable stream carriers it swallows exceptions! That may shadow problems which you may want to know of. Often PrintStreams behaviour will not be a problem but in certain cases like the one depicted above it causes a lot of headaches.

Open Source Love Day March 2010

Our Open Source Love Day for March 2010 brought love for Grails, our cmake hudson plugin, RXTX and winp. Everything went smooth and was lots of fun.

Yesterday, we held our first Open Source Love Day (OSLD) for this year. The last OSLD was at December 2009. Then, we reassigned a day in January and February each to perform our relocation to the new (and much bigger) office. But now we are back to regular duty and had the time to donate some work back to the Open Source ecosystem.

The Open Source Love Day

We introduced a monthly Open Source Love Day to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

Participate at our OSLD by using the features we’ve built today:

  • Grails still has some bugs. Instead of only complaining about them, we try to fix them. There is a bug with checkboxes and nested boolean properties that bugged us in a customer project. It’s filed unter GRAILS-3299 and has a proposed patch now.
  • In previous OSLDs, we produced the cmake hudson plugin. In the corresponding blog entry, comments with bug reports began to pile up. They addressed issues with hudson master/slave setups. So we implemented a hudson master/slave test environment, using VirtualBox virtual machines to perform as slaves. This setup quickly revealed the problems that were typical enough to devote a complete blog entry about this topic soon. Fixing the problems resulted in the new cmake hudson plugin version 1.2 to be released yesterday.
  • We are using the RXTX project to perform serial (RS232) communication in several projects. We are really glad the project exists, because the “official” communications API from Sun/Oracle is nothing but a mess. With RXTX, we only had a problem with emulated COM ports. Emulated COM ports exist when you use a USB->Serial or Ethernet->Serial converter, which is what our customer chose to do. If you unplug the converter during operation, the corresponding COM port disappears. This causes RXTX to crash, bringing the JVM down, too. We wrote a test application and used it with every converter we own (and we own quite a lot of them!). Then we began tracing the RXTX source code (at C code level), altering it to “only” throw an IOException when the virtual COM port disappears. The corresponding patch will be proposed to the RXTX project soon.
  • Another API we use a lot is the tiny winp project, written by Kohsuke Kawaguchi, the creator of hudson. We kill Windows processes with it, within a project that runs on Windows 2000, Windows XP and Windows 7. The latest Windows version seemed incompatible with winp, even the 32bit edition. We didn’t find the cause for this, but developed a workaround that will be proposed to the winp project soon.

What were our lessons learnt today?

  • If you face OutOfMemoryErrors on a 64bit Java6 JVM, try to switch back to a 32bit Java5 JVM. It helped us with our Grails bugfixing (during the test phase).
  • Hudson Master/Slave support for plugins isn’t particularly hard. It’s just that you need to be aware of the topic and replace some types like java.io.File. We gathered the same experience twice with our Crap4J plugin and the cmake plugin. It’s time to tell the world about it. Stay tuned!
  • The good old error return code is an error prone coding paradigm, because all too often, users of a function/method just forget to check the returned result. This was the case with a call to WaitForSingleObject in RXTX.
  • If you don’t understand an implementation well enough to fix the cause, you might at least be able to produce a workaround. It’ll work for you and provide guidance for the original author about where the bug might hide. This is why we count our winp efforts as success, too.
  • Your project either is mavenized or it isn’t. Everything in between is half-assed.

This OSLD was a bit short, as we had some guests in the evening, but nevertheless, it was fun. Well, to be precise, it was this special software engineer’s type of fun: The whole company was remarkably quiet most of the day, with everyone working totally focussed. We scratched our own itches, enhanced our customer projects and contributed to the open source community. A very good day!

Stay tuned if you want to know more about the specifics of the hudson plugin development or the to-be-proposed patches. We will publish them here.