W
e actively use Hudson as our continuous integration server software. It has a nice little feature called “build history trend” that shows the duration of all archived builds. One of our major projects started out small and fast with a build duration of 01:20 minutes. One and a half year later, it reached for the 04:00 minute hurdle. It wasn’t a surprise to us, as the build has more than four times the work now and the hardware staid the same.
But a question emerged: How can we speed up our build?
Applying optimization: The basic maths
We did a quick review of our ant build scripts to ensure there’s nothing fundamentally wrong with them and then decided which road to follow first: Optimizing the build scripts or boosting the hardware? There is only one pragmatic answer to us: boost the hardware as long as it stays reasonable in price. Every optimization in the build script would need its time (which isn’t cheap) and possibly increase the script complexity (which is very expensive later on).
Optimizing the hardware
So we went on the journey to make a fast buildbox even faster. We started out with a dual core processor (2.6 GHz), a decent-but-standard harddisk and 4 GB of memory. We replaced every part on its own to see the effect. The journey includes:
Our goal is to cut our build time down by 50 percent, to a little less than 02:00 minutes. We don’t want to spend more than 500 EUR for new hardware. So now, after this introduction:
Part I: Replacing the harddisk
Our buildbox starts with a more or less normal harddisk (0.5 TB), certified for continuous usage. We could have bought just another normal harddisk of a newer generation, but that doesn’t cut it in our experience (we didn’t verify specifically, though).
Calling the carnivores
If you need to upgrade your harddisk, you can buy yourself a VelociRaptor drive and be pretty much assured that you’ll notice the difference. We had pleasant experiences with this kind of fast-spinning drives before, but this time, we wanted to go a step further and try a fast Solid State Disk (SSD). As you only need to relocate the working directories (called workspaces in hudson terminology) of your projects to the new disk, the capacity isn’t important as long as it’s greater than your project sizes. You can just plug your new disk in the buildbox and format it with a high performance file system. As our buildbox runs on Linux, relocating the workspace is just setting a symbolic link. You do not even tell hudson about it. If you happen to run on Windows, check out the “use custom workspace” setting on your job’s configuration page.
An investment of about 200 EUR and 15 minutes of installation later, we had the result: The build average before was 03:30 minutes and now 03:10 minutes. That’s not a big leap forward, as others found out, too. It’s not that the SSD was bad, it performed exceptionally well in the benchmarks, but the harddisk wasn’t the bottleneck. To further proof our assumption, we installed the fastest harddrive you can get: the RAM disk.
Only pretend to use the disk
Linux (like other unixoid systems) has the great feature of an emulated harddisk right in your memory. On Debian/Ubuntu systems, this emulated drive is mounted at /dev/shm and has a capacity of half your total physical memory. It grows dynamically, so you don’t have to worry about its initial size. But you have to check if your workspace fits into it. Our buildbox had 4 GB of RAM and 2 GB were enough to contain the hudson workspace. We configured hudson to build there (you can use symbolic links or the “custom workspace” setting as shown in the picture) and got the result: The build average went down to 02:50 minutes.
Review on the results
That’s as far as we could speed up our buildbox by just replacing the harddisk. Down from 03:30 minutes to 02:50 minutes, a reduction of 40 seconds or 20 percent. In fact, we even cheated as the buildbox doesn’t use an harddisk anymore for building. With Linux, it’s incredibly easy to utilize a RAM disk as long as you have enough RAM to loan. For Windows systems, there are several software products that can do the same. If you don’t want to loan your RAM, you can look into HyperDrives, but for a price!
So we conclude that the fastest harddisk is an emulated one and even then, its effect on the build time is limited.
Stay tuned for the next episode of our journey to a faster buildbox, when we apply a faster CPU.
Schneide Blog 
Just a comment on your methodology. Would it not be worth finding out the actual cause of your build performance issues before dropping 200Euro on a new disk that doesn’t fix your problem?
How about spending a little time profiling your build and at least seeing if your bottleneck is I/O or CPU rather than guessing at the cause and wasting time/money in the process.
Who knows, you might have been able to get to your target speed (or better) by upgrading one component, e.g. CPU, instead of the current shotgun approach of upgrading everything.
Hi Glen, thank you for your comment.
You are definitly right in saying that “analyzing the problem” is better than “throwing money on the problem” without knowing what’s going on. I’m sorry if this entry can be misunderstood.
We did an initial analysis after the decision to tackle the issue by hardware and it really indicated a CPU-related bottleneck. But as replacing the CPU substantially means buying a new box, we decided to give the “complete renewal” a try, if only to gather experience with the components’ influences and write about it. The SSD wasn’t wasted in the end, for example. It just boosts another box now.
Try to see it as an experiment with learning effects instead of money waste. Our investment in amounts of time was limited, as it’s our scarcest resource.
Really looking forward to the remaining 3 parts.
I’ve tried this myself on Windows, by moving the Visual Studio projects and source files from the 7200-RPM HDD to a RAMdrive. My build time decreased to 56% of its normal value, so I see it as a substantial improvement. (Also, creating a new MKS “sandbox” – i.e. downloading the sources from the version management server – to the RAMdrive took only 45% of the normal (HDD) time, another very visible improvement.)
So I’d say people shouldn’t dismiss the SDD/RAMdrive solution too quickly. It may depend a lot on your current system and project particularities (maybe it counts more if your project is comprised of a lot of small files instead of a few big ones).
Hi donjoe,
thank you for your feedback and experiences. I totally agree that cutting your builtime by more than the half is a big improvement. Our experiments were performed on linux with java projects (ant based build). May I ask what product you used to create the RAMdrive under Windows?
I hope this article doesn’t read like the RAMdrive isn’t worth trying. I heartfully agree that moving the disk operations into the RAM (if possible) is the first step to a faster build. Just the SSD didn’t kick it that much like we hoped it would.
Actually, I was just curious about the speed improvement, so I used a very simple free tool called Gavotte Ramdisk, after which I abandoned the whole thing because I only have 1GB of RAM (the usual sandboxes I work on will never fit in any RAMdrive I could bite out of that). Gavotte Ramdisk doesn’t take care of saving and restoring your RAMdrive on shutdown/restart and can only create a single RAMdrive at a time.
If you want a better long-term solution, you should go with the free version of VSuite Ramdisk – it has multiple drives, image saving and restoring, greater maximum drive size etc.
As for SSDs, indeed they’re still only half as fast as RAM, so they’re not bound to give you any spectacular improvements if you only plan to use them for storing project files like I did in my test. However, I suspect the same is not true if you actually install your whole OS + compiler and everything on the SSD (which is why I plan to buy one for myself in the near future, to use as a dedicated system-and-apps drive at home :).