Tag: Open Source

Bringing your Grails app from 2.4 to 3.3

Updating to a new framework version often needs a lot of work and investigation how to fix problems that may arise. Usually there are upgrade guides that take you most of the way and make upgrading only a grind.

This also true for Grails and our upgrade experience with it. Often there are parts where you have to invest extra work and creativity. The current upgrade of our application from 2.4.5 to 3.3.8 is no exception:

The grind

The major changes and upgrade notes are part of the documentation so I will only mention them briefly:

  • Switch to the gradle build system
  • Using YAML as main configuration
  • Migration from filters to interceptors
  • New testing framework (partly optional because you can still use the old mixin framework with a plugin)
  • Package name changes
  • Former core features are now available as plugins like gsps, datasource and GORM
  • Functional tests need to use Spock+Geb or you will face weird problems and need to do extra work (we had selenium tests using selenium-server before)
  • Integration tests work differently so work needs to be done to migrate them
  • Logging using Lockback
  • Entities often need a @Entity annotation
  • Move some files to new Locations

The tricky stuff

  • A service named CounterService conflicts with spring boot autowiring so we had to rename it
  • Our TagLib tests using JUnit4 were failing with obscure errors, porting them to Spock fixed them.
  • We have so many dependencies that running the application with gradle:bootRun fails with: Createprocess error=206; the filename or extension is too long Fortunately adding grails { pathingJar = true } to build.gradle fixes the issue
  • Environment variables for gradle:bootRun are swallowed if not prefixed with “grails.”. We are using environment variables to customize running the application on the dev machines.

 

The hard parts

The most painful part was two central plugins we are using not being available anymore: shiro and searchable.

Shiro

For shiro there are some initial ports that work well for our needs, so the challenge was mostly finding the most fitting one of the forks on github. We went with the fork of Alin Pandichi and forked it ourselves to upgrade some version definitions.

Searchable becomes ElasticSearch

The real odyssee began looking for a replacement of the abandoned searchable plugin. Fortunately there is the compelling ElasticSearch-plugin which uses almost the same API as the searchable plugin:

The plugin focus on exposing Grails domain classes for the moment. It highly takes the existing Searchable Plugin as reference for its syntax and behaviour.

Unfortunately, we were unable to get it to work with our project trying many different versions, so we decided to fork and fix it for us. The main problems were:

  • Essentially, it does not work properly with hibernate as a data store because it chokes on the JavaAssist proxies hibernate often creates for domain objects.
  • An easy to fix concurrency issue
  • Not flexible enough converters

After a lot of debugging and a couple of fixes and the new feature of being able to use a spring bean as a converter we had search working smoothly and better than ever.

Wrapping it all up

The upgrade of our application to the newest incarnation of Grails was a rocky ride and took us quite some time.

On the other hand the framework got a lot better. Especially gradle is much better to manage than the previous build system.

So we are looking forward to a much better and robust development experience in the future and hope for some less revolutionary releases and easier upgrades.

CMakeBuilder Version 1.9

Introducing CMakeBuilder plugin version 1.9.

Today, I want to announce version 1.9 of the CMakeBuilder plugin for Jenkins (formerly known as Hudson). Concluding from the user feedback, there are no major missing features – at least for the moment.

So for this version, I implemented only one visible enhancement: It is now possible to use environment variables in every configuration setting. Even settings like “Preload Script” “Make Command” or “Install Command” can now be configured with the support of environment variables.

The major invisible change I did was the migration to the Jenkins development infrastructure using this very helpful guide. Moving the whole thing to git will be next.

Check it out!

Open Source Love Day October 2010

Our Open Source Love Day for October 2010 brought love for the cmake hudson plugin. Other issues were addressed but not finished. If you like to type fast and accurate, we suggest you check out typeracer.

On Friday two weeks ago, we held our Open Source Love Day for October 2010. This day was special in several ways. We strayed very far from the usual schedule for this day, there were several internal tasks that couldn’t be delayed and we introduced a “fun practice” event. But we eventually produced something valuable this day.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

The distractions

  • A regular project needed an urgent cost estimation by the whole team. This was the last opportunity because of an upcoming parental leave to have the team together for a long time.
  • Another regular project needed an urgent problem solved. This turned out to be so obscure that one of our developers had to be on-site. You can read about it in this blog entry now.
  • We received several shipments of office furniture and computer parts. They had to be checked and placed in.
  • We had a fun practice event. We discovered the online “game” typeracer and practiced our raw typing skill against each other for some time. Pro tip for beginners: don’t look at the highscores!

On this OSLD, we accomplished the following tasks:

  • A new version 1.8 of the cmakebuilder hudson plugin implements several feature requests. You can now choose to NOT clean your workspace before building and set different paths for the cmake installation for every job or node (hudson slaves). The latter option can be applied using an environment variable.

On this OSLD, we also tried the following tasks:

  • We keep an eye on Scala and its associated web framework Lift as a promising technology. One issue with Lift that bugs us is the use of “sun bastard format” properties for internationalization. We tried to teach Lift to accept UTF-8 encoded property files. After a lot of “downloading the internet” (you can always tell which project uses maven by their initial setup delay), we quickly implemented our own ResourceBundle.Control. But the Lift framework itself could not be built: “Error occurred during initialization of VM: Could not reserve enough space for object heap”. We ran out of time and will investigate in this issue on the next OSLD.
  • Grails is another web framework we use in projects. There are some bugs that really annoy us, and the OSLD is the perfect time to fix them. One of these bugs is GRAILS-6475, which we tried to reproduce with the latest code base. After writing a test case that would go green unexpectedly, we tried to provoke the error by setting up a sample project. The bug didn’t show up there, too. We left a comment in the issuetracker and ceased development.

What were our lessons learnt today?

  • You can’t tear off massive amounts of time from the OSLD and expect it to still be working. An OSLD doesn’t scale down apparently.
  • Most issues that can’t be done fail with the project’s build. The build process of a foreign project is the most crucial phase in your decision on commitment. If it fails, your participation in the project is at risk. We’ve seen many brittle, undocumented and incredible complex build processes now. And we can state one thing: It doesn’t stop with throwing maven at a project, you still have to “think the build”.

Retrospective of the OSLD

This OSLD was special in the amount of non-OSLD work done. The remaining efforts weren’t as successful as we wished. This has been an ongoing issue with our OSLD for the last months now and we are looking forward to adapt our workstyle to yield better results in the future. The distraction by typeracer was fun, though.

Open Source Love Day September 2010

Our Open Source Love Day for September 2010 brought love for the cmake hudson plugin and a brand new tool around crap4j that will be released soon.

On wednesday last week, we held our Open Source Love Day for September 2010. Our day started with the usual Homepage Comittee meeting and very soon, we were up and working. This time, our success rate wasn’t as high as we wanted, mostly because we worked on internal tools that didn’t work out quite as well as expected. But, we managed to produce something valuable this day.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

  • A new version 1.6.1 of the cmakebuilder hudson plugin was published. This version consisted of bugfixes only and right now, it still seems flawed. We are working on the issue, expect a new version 1.6.2 soon.
  • Our internal time-tracking tool got love at several issues. One issue required the use of triangle-shaping CSS, as described in this blog post from Jon Rohan. Our issue weren’t finished because Javascript code can rot into a big pile of crap really quick.
  • We managed to make a long hatched dream come true at this OSLD. As you might be aware, we are big fanboys of crap4j, a metric tool that associates test coverage with code complexity. Thus, we wrote the crap4j hudson plugin, release the CrapMap and use some internal improvements, too. The main disadvantage of crap4j is the strong dependency on a specific test coverage tool. Our goal was to use the test coverage data we already collect using Cobertura. We achieved this goal and got the whole thing working. It will be released in the next few weeks, with a detailed blog post here. Stay tuned for this new tool (it already has a name: Crapertura).

What were our lessons learnt today?

  • No matter how clever you are, Javascript outsmarts you every time it appears in superior numbers. Refactoring is the key here, but difficult and tedious to apply.
  • When you dissect a foreign API or code base, you just need to find the right grip. I cannot decribe it more precise right now, but this grip is all you need to open up the code. When playing around with the crap4j code base, as soon as we held the grip, everything else followed naturally. Perhaps “the grip” can be translated with “catching the author’s intent”. These are always magical moments.

Retrospective of the OSLD

This OSLD wasn’t as successful as we wished, partly because of missing manpower (honeymoon holiday!) and because of our inability to tame a Javascript code base. We have to work on our expertise here and we are glad that we’ve found out at an OSLD, not a time- and mission-critical customer project.

Open Source Love Day July 2010

Our Open Source Love Day for July 2010 brought love for Hudson (especially the CMake and Crap4j plugins), RXTX and JUnit.

This friday , we held our Open Source Love Day for July 2010.  We began with several internal meetings and discussion (like the Homepage Comittee meeting) and dived right in our work afterwards. Everybody had a little backlog of issues that we wanted to get done on this day. Nearly everybody succeeded (well, the author had a minor delay – read about it below). The day went by in a very fast pace, but it felt right.

The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

  • There are really cool new features in the latest JUnit versions and Rules are one of them. What hurt our aesthetic sense was that the field that hold the Rule instance has to be public. Checkstyle was on our side, so we tweaked JUnit to allow all kinds of visibility. You can read about the change needed here: http://github.com/KentBeck/junit/issues#issue/31. The fix is almost trivial and will hopefully be incorporated in the next versions of JUnit, so we do not publish our altered version.
  • We constantly receive requests and remarks about our cmake plugin for Hudson. This lead to a new version of the plugin fixing two issues with matrix builds and custom build types. Head over to the plugin homepage and grab the new version 1.6. The issues were in detail:
    • The plugin can be used with matrix builds now
    • Custom build types can be defined now
  • RXTX is our choice for serial port communication with Java. We fixed some issues during the last few OSLDs, with one issue left for today: When you flush your stream while using a special type of usb-to-rs232 converter, you got an exception. The corresponding issue is #102 in the RXTX issue tracker. We proposed a patch that fixes the problem.
  • Another hudson plugin is our crap4j reporter. It lacked some love for months now and finally broke when used with the latest hudson versions. Fixing the problem was a lot harder than we thought, basically because the plugin needed adjustments to recent API changes and we couldn’t figure out exactly what adjustments are necessary. You might have a look at the developer mailing list thread for this question. Finally, we got it resolved (on sunday, with a sudden stroke of insight) and a new version 0.8 is published.
  • We use an internal time tracking tool for our projects. This tool isn’t specifically open source yet, but continues to grow in terms of features and usability. The work invested in this tool helps us to continue with the OSLD, so it’s beneficial work nonetheless.
  • During the last OSLD, we had plans for a new hudson plugin and even produced a prototype. This time, we looked around the hudson plugin zoo (it’s getting a bit difficult to keep track of all of them) for inspiration and found a wonderful piece of art: The Groovy Postbuild Plugin. Using this plugin with a small groovy script served our needs exactly. No need for a full-blown plugin when you can scratch your itch with a simple script. Thanks to Serban Iordache for his great work!

What were our lessons learnt today?

  • If you need to setup a fresh workspace for an open source project, consider to prepare it over the night before, or the download delay will kill your precious work time. There is nothing more frustrating than staring at a “downloading…” progress bar while being eager to start programming.
  • Always look around what others have done before. We wanted to build a full hudson plugin from scratch when all we needed was a little groovy script placed inside another plugin. Sweet!
  • Do not hesitate to privately fix open source issues that won’t get done in time for you. Just make sure to have a management process in place to track those changes and be able to re-apply them to future versions. More important though, be able to tell exactly when NOT to re-apply them because the original project has fixed the issue.

Retrospective of the OSLD

The OSLD went smooth and was productive. We tend to work on backlogs instead of searching for random issues now, but that’s just a sign that our approach has matured and we depend on the OSLD to get work done.

Last wednesday, we held our Open Source Love Day for June 2010. This one was productive despite the heat that had us sweating the whole day long (as a sidenote: it got even warmer the days afterwards). Some features were finished and will help at least us in our projects. We still look forward for the right way to release them. Another release was even more problematic, you will read about it below.The Open Source Love Day

We introduced a monthly Open Source Love Day (OSLD) to show our appreciation to the Open Source software ecosystem and to donate back. We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

On this OSLD, we accomplished the following tasks:

Open Source Love Day December 2009

Our Open Source Love Day for November 2009 brought love for EGit and several hudson plugins. We got slightly frustrated over KDevelop4.

On Tuesday, we had our last regular working day for 2009. We celebrated this circumstance by having our fourth Open Source Love Day (OSLD). The day was successful, you can review our list of today’s achievements below.

We introduced a monthly Open Source Love Day to show our appreciation to the Open Source software ecosystem and to donate back We heavily rely on Open Source software for our projects. We would be honored if you find our contributions useful. Check out our first OSLD blog posting for details on the event itself.

Participate at our OSLD by using the features we’ve built today:

  • Our campfire plugin for hudson was updated to version 1.1. The new version contains the improvements Mark Woods suggested (global configuration and login recovery). Thank you, Mark!
  • The campfire plugin also switched the communication model from webpage scraping to the brand new campfire API. This should improve the stability of the plugin.
  • Some of the EGit (git plugin for eclipse) patches we sent in at the last OSLD needed some rework and polishing. You can review the details in EGit’s code review system gerrit: change 121 and change 122.
  • Our cmake hudson plugin was updated to version 1.1. The new version checks the environment (installed cmake version, etc.) before delegating the call and provides better error messages.
  • We started working on a feature of KDevelop4 that was present in KDevelop3 and is now missing: “Compile file”. The progress was slowed down by some problems. See below for details.
  • Hudson got a new major version of the IRC plugin from Christoph Kutzinksi. The plugin was in a rather desolate state before. We had used a private fork with specific additions to control our infrastructure. The plugin was on our list of OSLD patients, when Christoph merged it with the hudson instant-messaging plugin and introduced a multitude of cool new features. We beta-tested the new version and it was great. The only drawback was the complete alteration in message syntax that broke our infrastructure. So in order to scratch our own itch, we programmed a little API to parse hudson IRC plugin messages of the new 2.X version stream. Our code is published on github, have a look if you are interested and drop us a line if you found it useful.

What were our lessons learnt today?

  • If maven decides to work properly, everything is really cool.
  • Just because you use JGit/EGit on top of Eclipse, all three being platform independent, doesn’t save you from slash vs. backslash issues. EGit’s initial user experience is better on unixoid platforms than on windows systems. The patch #141 helped us beyond the showstopper of unrecognized local repositories.
  • We acquired an additional share of eclipse plugin development knowledge when polishing our EGit features.
  • Working with git and gerrit is challenging on first encounter. We are constantly learning in this area.
  • Bugzilla fails to present open issues in a manner where you can quickly pick an issue of interest. If you really want to use it for your open source project, think of a scraped website that only lists the “low hanging fruits” for newbie developers.
  • KDevelop4 has outdated documentation, the projects kdevplatform and kdevelop were moved inside the repository.
  • If you encounter a rather erratic error stating that “KDE4Workspace not found”, try excluding the debuggers/gdb subproject from your build.
  • Most of us used the waiting delays of one project (“oh, maven is downloading the internet again”) to switch over to a secondary task. So this event trains our multitasking abilities right along.

In summary, this OSLD was a fun way to end a workyear on heavy duty. We will continue to celebrate OSLDs in 2010, as it’s a fun way to peek into foreign projects, learn a lot in short time and contribute to the community.

Open Source Love Day November 2009

Our Open Source Love Day for November 2009 brought love for EGit, the Eclipse git plugin and some frustration over lacking documentation.

Today, we celebrated our third Open Source Love Day (OSLD). It was slightly degraded by a company-wide illness outbreak, but we tried to make the best out of the situation.

Open Source Love Days are our way to show our appreciation and care to the Open Source software ecosystem. Our work wouldn’t be as fun or just not possible without professional Open Source software. You can read more about our motivation and specifica in our first OSLD blog posting.

You can participate at our OSLD by using the features we’ve built today:

  • We think git is an enrichment to the SCM market, as is Eclipse to the IDE market. Improving the quality of Eclipse’s git plugin is the next logical step. The ability to diff the content of two revisions in EGit was committed today. As a bonus, the name of the committer shows up in the right manner, too. See the screenshots to get the idea:

When developing EGit, we were already using it to pull the sources. Unfortunately, the repository URL changed bigtimes since our checkout without us noticing. This got us into trouble trying to follow the contributor guide. The command line version of git isn’t that communicative yet. But after all, this is a great time to learn about the real world problems when using git. The EGit contributor guide itself is a fantastic way for a project to show initial appreciation to volunteer efforts. Thanks for caring, guys! If you are interested to review our changes yourself, fetch the patch.

  • Another part of today’s work was on the KDevelop project. We tried to fix some outstanding little features or bugs, whatever is on the list of KDevelop 4. But we spent our day fixing our development machines instead. The Ubuntu linux operating system (8.10) was way too old to get useful results and KDE needs to be up-to-date to develop KDevelop. Besides our sluggishness to keep our virtual machines on the bleeding edge, the checkout experience of KDevelop was rather sleek. What bothers us a bit is the ominous entanglement between KDevelop and KDE. It seems you can’t have one without the other and need to master both to make a stand.
  • As a third part, we wanted to contribute to the TANGO project (not the useful icon collection, but the useful control system). They migrated their main repository from CVS to SVN lately, but the migration seems unfinished still. At least, the migration effort lacks public documentation for the occasional contributor. That’s a real showstopper, because you never get beyond the very first step: setting up a working project. We won’t give up and email the project leads on this topic, but it didn’t fit into this OSLD.

What were our lessons learnt today?

  • Just having a possibility to view or download the source code doesn’t make an Open Source project. The key to success is the ability of complete strangers to hop in and perform useful work. Having terse, but accurate documentation helps a lot. The EGit contributor guide is a good example of a single document that makes the difference. If you own an open source project and want to attract occassional contributors (like us), write such a document and watch us (and others) drop you a patch. That said, we come to belief that the person that writes technical documentation for the developers is one of the most important roles on a project. Perhaps we join some projects in the future to fill that role.

To sum it up, this OSLD was limited from the beginning by developer availibility. With lacking documentation, we nearly grinded to a halt. We look forward to our next OSLD in December.

A Campfire plugin for Hudson

Our last OSLD resulted in a new hudson plugin: a build notifier for Campfire. Right now it is pretty simple and just posts the last build result in a chat room:
Campfire-Notification

You can configure your account data and the room to join in the job config page under Post Build actions:
Hudson-Campfire
But there is more to come:

  • global config
  • SSL support
  • a link back to hudson

or what is your favorite missing feature?

Our second Open Source Love Day (OSLD)

A retrospective report of our second Open Source Love Day (OSLD). We present the results of our work on hudson and git and the lessons learnt.

opensourceloveday

Today we celebrated our second Open Source Love Day (OSLD). When we say “celebrated”, we actually mean that all of us worked hard and concentrated for hours, just to have a short meeting with candy at the end of the day.

The Open Source Love Day is our way to show our appreciation to the Open Source software development ecosystem. We heavily rely on Open Source products for our customer projects, so it’s just fair to donate back. You can read more about our motivation and specifica in our first OSLD blog posting.

For this day, we adjusted the rules a bit. While Object Calisthenics are very powerful in formulating rather academic software development values in some easy-to-remember rules, they just don’t fit well with existing projects. We still kept the rules in mind, but didn’t follow them strictly. We also learnt our share from last time’s experience of jumping right into the middle of arbitrary projects without a real need to do so. Today, we scratched more of our own itches.

You can participate at our OSLD by using the feature we’ve built today:

  • Hudson gets a brand new plugin. Currently, it’s in alpha status and needs some more nurturing, but is planned to be published within the next few weeks. The proof of concept was successful today. You will read more about it on this blog soon.
  • Another of our hudson plugins, namely the cmake builder plugin, got some feature love, incorporating suggestions from plugin users. We especially thank Ole B. for his feedback. The new features are checked in and will be available with the next plugin version 0.6, scheduled to be published in a few days. You’ll read the details about the new features here.
  • We’ve produced a feature implementation for hudson, adding the ability to use environment variables for the job’s workspace path. This feature touches core hudson functionality, so we just proposed a patch and leave it up to the core hudson team to decide on its inclusion. For more details, head over to the hudson issue tracker, entry #3997.
  • And we didn’t forget about git. As we are multi-IDE users (today’s development took place using NetBeans, IDEA and Eclipse), the EGit eclipse plugin for git will soon have the ability to diff the content of two revisions. An undocumented method argument took too much time to finish the feature today. After email communication with the project owner, the feature works on our machine, but needs some polishing before being committed in the near future.

As you can see, the hudson continuous integration server received a great share of our today’s love. It’s a great tool with a great community that really deserves our contributions.

What were our lessons learnt today?

  • While implementing the variable expansion feature, the author got distracted by a similar concept and followed this red herring. Namely, instead of a hudson.util.VariableResolver, we needed to use the hudson.EnvVars class. The EnvVars are pre-filled with all global variables like JAVA_HOME, while the VariableResolver is not. This could have been avoided by looking at the actual code instead of just type names. Once you think you’ve found your type, you read code the wrong way just to sustain your assumption.
  • To implement advanced plugin features, whether for hudson or eclipse, is a matter of skill with the “monkey see, monkey do” development style. Documentation is mostly non-existent or out-dated.
  • When handling HTML and HTTP in java, some survival tricks are crucial. Stay tuned for a whole blog post on that topic.
  • We still don’t feel comfortable within the JGit source code, as we still lack advanced git feature and terminology knowledge and the project lacks documentation. Our part of the problem will decline over time, as it’s a question of tool/mindset/slang exposure.

To sum it up, this OSLD worked out much better than our first one. We had more fun and yielded better results, mostly because we adjusted our goals to better suit our working style.

What are your experiences with open source development? Drop a comment!