Tag: Continuous Integration

Using passwords with Jenkins CI server

For many of our projects the Jenkins continuous integration (CI) server is one important cornerstone. The well known “works on my machine” means nothing in our company. Only code in repositories and built, tested and packaged by our CI servers counts. In addition to building, testing, analyzing and packaging our projects we use CI jobs for deployment and supervision, too. In such jobs you often need some sort of credentials like username/password or public/private keys.

If you are using username/password they do not only appear in the job configuration but also in the console build logs. In most cases this is undesirable but luckily there is an easy way around it: using the Environment Injector Plugin.

In the plugin you can “inject passwords to the build as environment variables” for use in your commands and scripts.inject-passwords-configuration

The nice thing about this is that the passwords are not only masked in the job configuration (like above) but also in the console logs of the builds!inject-passwords-console-log

Another alternative doing mostly the same is the Credentials Binding Plugin.

There is a lot more to explore when it comes to authentication and credential management in Jenkins as you can define credentials at the global level, use public/private key pairs and ssh agents, connect to a LDAP database and much more. Just do not sit back and provide security related stuff plaintext in job configurations or your deployments scripts!

A minimal set of skills for software development contractors

You aren’t sure if your developer is professional enough? Here are seven topics you can ask him about to find it out. It’s the minimal skill set a modern developer should use.

“Our company is specialized in providing professional software development for our customers”. That’s a nice statement to inspire your customers with. The only problem with it is: every contractor claims to be professional. You wouldn’t even get a project if you admitted to be “unprofessional”. But how can a customer, mostly unaware of the subtleties in the field of software development, decide if his contractor really works professionally? A lot of money currently spent on projects doomed from the beginning could be saved if the answer was that easy. But there’s a lower limit of skills that have to be present to pass the most minimal litmus test on developer professionality. This blog article gives you an overview about the things you should ask from your next software development contractor.

First a disclaimer: I’ve compiled this list of skills with the best intentions. It is definitely possible to develop software without some or even any of these skills. The development can even be performed in a very professional manner. So the absence of a skill doesn’t reveal an unprofessional contractor without fail. And on the other side, the clear presence of all skills doesn’t lead to glorious projects. The list is a rule of thumb to distinguish the “better” contractor from the “worse”. It’s a starting ground for the inexperienced customer to ask the right questions and get hopefully insightful answers.

Let’s assume you are a customer on the lookout for a suitable software development contractor, maybe a freelancer or a company. You might take this list and just ask your potential developer about every item on it. Listen to their answers and let them show you their implementation of the skill. In my opinion, the last point is the most crucial one: Don’t just talk about it, let them demonstrate their abilities. You won’t be able to differentiate the best from the most trivial implementation at first, but that’s part of the learning process. The thing is: if the developer can readily demonstrate something, chances are he really knows what he is talking about.

The minimal skills

The list is sorted by their direct impact on the overall development quality. This includes the quality perceived by you (the customer), the end user and the next developer who inherits the source code once the original developer bails out. This doesn’t mean that the topics mentioned later are “optional” in the long run.

Source code management system

This tool has many different names: source code management (SCM), revision control system (RCS) and version control system (VCS) are just a few of them. It is used to track the changes in the code over time. With this tool, the developer is able to tell you exactly which change happened when, for what version and by whom. It is even possible to undo the change later on. If your developer mentions specific tool names like Git, Subversion, Perforce or Mercurial, you are mostly settled here. Let him show you a typical sync-edit-commit cycle and try to comprehend what he’s telling you. Most developers love to brag about their sophisticated use of version control abilities.

Issue tracking

An issue or bug tracker is a tool that stores all inquiries, bug reports, wishes and complaints you make. You can compare it to a helpdesk “trouble ticket” system. The issue tracker provides a todo list for the developer and acts as an impartial documentation of your communication with the developer. If you can’t get direct access to the issue tracker on their website, let them demonstrate the usage by playing through a typical scenario like a bug report. At least, the developer should provide you with a list of “resolved” issues for each new version of your software.

Continuous integration

This is a relatively new type of tool, but a very powerful one. It can also be named a “build server” or (less powerful) a “nightly build”. The baseline is that your project will be built by an automated process, as often as possible. In the case of continuous integration, the build happens after each commit to the source code management system (refer to the first entry of this list). Let your developer show you what happens automatically after a commit to the source code management system. Ask him about the “build time” of your project (or other projects). This is the time needed to produce a new version you can try out. If the build time is reasonably low (like a few minutes), ask for a small change to your project and wait for the resulting software.

There is a fair chance that your developer not only talks about “continuous integration”, but also “continuous delivery”. This includes words like “staging”, “build queue”, “test installation”, etc. Great! Let them explain and demonstrate their implementation of “continuous delivery”. You’ll probably be impressed and the developer had another chance to brag.

Verification (a.k.a. Testing)

This is a delicate question: “Will the source code contain automated tests?”. Our industry’s expectancy value for any kind of automated tests in a project is still dangerously near absolute zero. If you get blank stares on that question, that’s not a good sign. It doesn’t really matter too much if the answer contains the words “unit test”, “integration test” or even “acceptance test”. Most important again: Let your developer show you their implementation of automated tests in your (or a similar) project. Make sure the continuous integration server (refer to entry number three) is aware of the tests and runs them on every build. This way, everything that’s secured by tests cannot break without being noticed immediately. You probably won’t have to deal with reappearing bugs in every other version, a symptom known as “regression”.

Your developer might be really enthusiastic about testing. While every developer hour costs your precious money, this is money well spent. Think of it as an insurance against unpredictable behaviour of your software in the future. Over the course of development, you won’t notice these tests directly, as they are used internally for development. Talk to your developer about some form of reporting on the tests. Perhaps a “test coverage” report that accompanies the issue list (refer to the second entry)? Just don’t go overboard here. A low test coverage percentage is still better than no tests.

If your developer states that he is “test driven”, that’s not a psychological condition, but a modern attempt to test really thoroughly. Let him demonstrate you the advantages of this approach by playing through an implementation cycle of a small change to your project. It may foster your confidence in the insurance’s power.

Project documentation

Every software project above the trivial level contains so many details that no human brain is able to remember them all after some time. Your developer needs some place to store vital information about the project other than “in the code” and “in the issue tracker”. A popular choice to implement this requirement is providing a Wiki. You probably already know a Wiki from Wikipedia. Think about a web-based text editing tool with structuring possibilities. If you can’t access the documentation tool yourself, let your developer demonstrate it. Ask about an excerpt of your project documentation, perhaps as a PDF or HTML document. Don’t be too picky about the aesthetics, the main use case is quick and easy information retrieval. Even handwritten project documentation may pass your test, as long as it is stored in one central place.

Source code conventions

Nearly all source code is readable by a machine. But some source code is totally illegible by fellow developers or even the original author. Ask your developer about their code formatting rules. Hopefully, he can provide you with some written rules that are really applied to the code. For most programming languages, there are tools that can check the formatting against certain rules. These programs are called “code inspection tools” and fit like hand in glove with the continuous integration server (refer to the third entry). Some aspects of source code readability cannot be checked by algorithms, like naming or clarity of concepts. Good developers perform regular code reviews where fellow developers discuss the code critically and suggest improvements. The best customers explicitely ask for code reviews, even if they won’t participate in them. You will feel the difference in the produced software on the long run.

Community awareness

Software development is a rapidly advancing profession, with game-changing discoveries every other year. One single developer cannot track all the new tools, concepts and possibilities in his field. He has to rely on a community of like-minded and well-meaning experts that share their knowledge. Ask your developer about his community. What (technical) books did he read recently? What books are known by the whole development team? As a customer, you probably can’t tell right away if the books are worth their paper, but that’s not the main point of the question. Just like with tests, the amount of books read by the average programmer won’t make a very long list. If your development team is consistent enough to share a common literature ground, that’s already worth a lot.

But it’s not just books. Even books are too slow for the advancement! Ask about participation in local technical events, like user groups of the programming language of your project. What about sharing? Does the developer share his experiences and insights? The cheapest way to do that is a weblog (you’re reading one right now). Let him show you his blog. How many articles are published in a reasonable timespan, what’s the feedback? Perhaps he writes articles for a technical magazine or even a book? Now you can ask other developers for their opinion on the published work. You’ve probably found a really professional developer, congratulations.

There is more, much more

This list is in no way exhaustive in regard to what a capable developer uses in concepts, skills and tools. This is meant as the minimal set, with a lot of room for improvement. There are compilations of skills like the Clean Code Developer that go way beyond this list. Ask your developer about his personal field of interest. Hopefully, after he finished bragging and techno-babbling for some time, you’re convinced that your developer is a professional one. You shouldn’t settle for less.

Summary of the Schneide Dev Brunch at 2012-03-25

If you couldn’t attend the Schneide Dev Brunch in March 2012, here are the main topics we discussed for you to review.

This summary is a bit late and my only excuse it that the recent weeks were packed with action. But the good news is: The Schneide Dev Brunch is still alive and gaining traction with an impressive number of participants for the most recent event. The Schneide Dev Brunch is a regular brunch in that you gather together to have a late breakfast or early dinner on a sunday, only that all attendees want to talk about software development (and various other topics). If you bring a software-related topic along with your food, everyone has something to share. We were able to sit in the sun on our roofgarden and enjoy the first warm spring weekend.

We had to do introductory rounds because there were quite some new participants this time. And they brought good topics and insights with them. Let’s have a look at the topics we discussed:

Checker Framework

This isn’t your regular java framework, meant to reside alongside all the other jar files in your dependency folder. The Checker framework enhances java’s type system with “pluggable types”. You have to integrate it in your runtime, your compiler and your IDE to gain best results, but after that you’re nothing less than a superhero among regulars. Imagine pluggable types as additional layers to your class hierarchy, but in the z-axis. You’ll have multiple layers of type hierachies and can include them into your code to aid your programming tasks. A typical use case is the compiler-based null checking ability, while something like Perl’s taint mode is just around the corner.

But, as our speaker pointed out, after a while the rough edges of the framework will show up. It still is somewhat academic and lacks integration sometimes. It’s a great help until it eventually becomes a burden.

Hearing about the Checker framework left us excited to try it sometimes. At least, it’s impressive to see what you can do with a little tweaking at the compiler level.

Getting Stuck

A blog entry by Jeff Wofford inspired one of us to talk about the notion of “being stuck” in software development. Jeff Wofford himself wrote a sequel to the blog entry, differentiating four kinds of stuck. We could relate to the concept and have seen it in the wild before. The notion of “yak shaving” entered the discussion soon. In summary, we discussed the different types of being stuck and getting stuck and what we think about it. While there was no definite result, everyone could take away some insight from the debate.

Zen to Done

One topic was a review of the Zen to Done book on self-organization and productivity improvement. The methodology can be compared to “Getting Things Done“, but is easier to begin with. It defines a bunch of positive habits to try and establish in your everyday life. Once you’ve tried them all, you probably know what works best for you and what just doesn’t resonate at all. On a conceptional level, you can compare Zen to Done to the Clean Code Developer, both implementing the approach of “little steps” and continuous improvement. Very interesting and readily available for your own surveying. There even exists a german translation of the book.

Clean Code Developer mousepads

Speaking of the Clean Code Developer. We at the Softwareschneiderei just published our implementation of mousepads for the Clean Code Developer on our blog. During the Dev Brunch, we reviewed the mousepads and recognized the need for an english version. Stay tuned for them!

Book: Making software

The book “Making software” is a collection of essays from experienced developers, managers and scientists describing the habits, beliefs and fallacies of modern software development. Typical for a book from many different authors is the wide range of topics and different quality levels in terms of content, style and originality. The book gets a recommendation because there should be some interesting reads for everyone inside. One essay was particularly interesting for the reviewer: “How effective is Test-Driven Development?” by Burak Turhan and others. The article treats TDD like a medicine in a clinical trial, trying to determine the primary effects, the most effective dosage and the unwanted side effects. Great fun for every open-minded developer and the origin of a little joke: If there was a pill you could take to improve your testing, would a placebo pill work, too?

Book: Continuous Delivery

This book is the starting point of this year’s hype: “Continuous Delivery” by Jez Humble and others. Does it live up to the hype? In the opinion of our reviewer: yes, mostly. It’s a solid description of all the practices and techniques that followed continuous integration. The Clean Code Developer listed them as “Continuous Integration II” until the book appeared and gave them a name. The book is a highly recommened read for the next years. Hopefully, the practices become state-of-the-art for most projects in the near future, just like it went with CI. The book has a lot of content but doesn’t shy away from repetition, too. You should read it in one piece, because later chapters tend to refer to earlier content quite often.

Three refactorings to grace

The last topic was the beta version of an article about the difference that three easy refactorings can make on test code. The article answered the statement of a participant that he doesn’t follow the DRY principle in test code in a way. It is only available in a german version right now, but will probably be published on the blog anytime soon in a proper english translation.

Epilogue

This Dev Brunch was a lot of fun and had a lot more content than listed here. Some of us even got sunburnt by the first real sunny weather this year. We are looking forward to the next Dev Brunch at the Softwareschneiderei. And as always, we are open for guests and future regulars. Just drop us a notice and we’ll invite you over next time.

Follow-up to our Dev Brunch April 2010

A follow-up to our April 2010 Dev Brunch, summarizing the talks and providing bonus material.

Last sunday, we held our Dev Brunch for April 2010. It was the start of the open air season, introducing our new office roof garden. We brunched under a clear, sunny sky (no clouds, even no vapor trails from airplanes, as they aren’t allowed to fly because of some distant volcano ash emission) and talked about agile processes and books. A major part of the brunch was graded as “informal chatter”, just as it should be like.

The Dev Brunch

If you want to know more about the meaning of the term “Dev Brunch” or how we implement it, have a look at the follow-up posting of the brunch in October 2009. We continue to allow presence over topics. Our topics for the brunch were:

  • Agile life planning – Your software development daily routine is to incrementally build software from a backlog and release it after a fixed timebox. What might happen if you transfer this process to your private life (often called “real life”)? You’ll release a week’s life every week and have a week planning meeting on sunday, filling your week/sprint backlog with the most important items of your life’s backlog. Telling from this first hand experience of about a year, it works exceptionally well, improving the leisure time quality and making “progress” visible even at harder times. The only remaining question is who acts as the product owner.
  • Converting to Scrum – First hand experience of a team that, after years of “alarm call”-style development, successfully implemented Scrum as their primary process. The effects were at least less overtime, improved progress tracking, improved code quality (though Scrum is  only a management process!) and less pressure in the project. The customer even adjusted their wording when talking about new features: “next sprint” instead of “immediately, now”. The implemented process isn’t vanilla Scrum, but works nevertheless.
  • Defending Continuous Integration – What if, after an initial phase of excitement over the new tool, the continuous integration server really reveals flaws in your project? There seems to be the tendency to kill the messenger: Shut down the CI server and everything’s fine again. This talk was about the reasoning of both sides and some basic insights gained about machine sharing. Tip of the talk: Reserve your CI server exclusively for this task.
  • Book recommendations – A random collection of technical books read in the last half year, presented with a short summary and personal rating. Titles included were Coders At Work, Founders At Work, 97 Things Every Software Architect Should Know, 97 Things Every Project Manager Should Know, 97 Things Every Programmer Should Know, Hiring the Best Knowledge Workers, Techies & Nerds and Manage Your Project Portfolio. Yes, there are some patterns visible in the book titles. And it’s a good idea to keep some checklist of read articles for most of these books.
  • Dreyfus Model of Skill Acquisition – The Dreyfus Model differentiates five to six different stages of learning some arbitrary skill. It makes assumptions how the members of the stages work, how they process feedback information and what they need to get better. See the comment section for more information and bonus material.

As usual, the topics ranged from first-hand experiences to literature research. For additional information, check out the comment sections. Comments and resources might be in german language.

Retropection of the brunch

Holding the brunch in the bright morning sun, surrounded by rooftops and birds, really is a huge gain for the ambience factor. We even found a solution to produce fresh coffee up there. This will be a fun summer for the Dev Brunch.

A blind spot of Continuous Integration

Imagine you haven’t changed a continuously integrated project for months when suddenly a unit test breaks. Here’s why CI has a blind spot with flawed tests.

In the early days of April 2008, we updated our hudson continuous integration (CI) server to a new version. This was no unusual action, as there was a new version every day back then, bringing new features in a rapid rate. What was unusual after the upgrade was that one of the surveilled projects failed to build all of a sudden.

Sudden (test) failure without a change

The build was started manually, without a code change. The project itself was inactive back then, meaning that no changes were made for months. And suddenly, a unit test broke. The test was in the project for two whole years without ever going off. What happened?

Good unit tests

There are rules for good unit tests. A basic set are the A-TRIP rules formulated in the excellent beginner’s book “Pragmatic Unit Testing” by Andy Hunt and Dave Thomas. The failed test clearly disobeyed the “repeatable” rule (the R in A-TRIP): It didn’t result in the same result as before while the code under test didn’t change.

Write repeatable tests or your CI will be blind (partially)

The cause of the failed test was putting the clocks back because the daylight-saving time part of the year began. The unit test secured some date calculations by taking the current date and comparing it to future and past dates that got calculated. The calculation went wrong when the daylight-saving mode changes in the calculated period, which was a bug. Repeatability of the unit test was lost when “the current date” entered the code – whether on the unit test or productive code side.

Two years of blindness

How could this bug survive two years without being noticed? The project was under CI surveillance since the beginning, the unit test to detect the bug was present along with the bugged code. The answer is: We never programmed for this project around the weeks of the year when the clock is adjusted and the bug occurs. This was a coincidence influenced by the customer’s schedule. So every time CI (or we) ran the unit test, it passed. Until that day right after putting the clocks back.

How to avoid this blind spot

There are two things you can do to avoid this scenario:

  • Always inject a fixed “the current date” into your code when dealing with date calculations. Only use absolute dates in your unit tests. Time isn’t a healer for your tests, it’s a beast to be tamed.
  • Set up a nightly build for your project that runs once a day even when no changes have been made. It would have caught this bug one and a half year earlier.

To sum it up:

CI only spots bugs when they move (aka the code is changed). Nightly builds provide a (fuzzy) security layer against non-repeatable unit tests. And unit tests with flaws provide only delusory security.

Additional background information

After fully understanding the circumstances, we were curious why the customer didn’t notice the bug and asked him about it. The answer was delightful: “Our computers don’t adjust their clocks. Daylight-saving time only causes trouble.” What a wise decision!

For a good comparison of CI vs. Nightly Builds see this blog entry.