Paul Gross's Blog

A short build time is a critical element of continuous integration. I’ve been involved in a number of build improvements on my current project (both local and on the build server), and I thought I would share some of them. Using the tricks below, we cut our build time in half (and have more to go). Obviously, every build and project are different, so many of these may not apply to other situations.

Profile the build

We profiled our build before we made any changes. We found the slowest tasks using rake—trace, which prints out task timing. Furthermore, we watched the output from the “top” command while the builds were running on the build server to reveal bottlenecks.

Turn down logging

We noticed while watching “top” that kjournald was constantly running. This indicates a lot of disk activity. Our app logs to files and syslog while running in production, but this is unnecessary during builds. We turned off nearly all logging in test mode.

Make sure the build machine is only performing builds

We discovered that our build machine had become a bit of a playground for trying new things. This is fine in moderation, but some of these projects were not cleaned up. There were a lot of unnecessary processes running on the build machine. We killed everything that was unnecessary, and increased the nice value of processes that were not critical. For example, cruisecontrol.rb runs the web interface as a separate process from the builders. We increased the nice value of the web interface so building our projects would take precedence.

Parallelizaton

If you have more than one processor (or core), and the machine is running a giant set of tests, there is a good chance that the other processor is doing very little. We discovered huge gains by running tests in parallel. Projects like deeptest and selenium grid make this easy. We went a simpler route and run our functional and acceptance tests at the same time in different processes.

Build database from dumps

Our project is on release six, so we have six versions of the database to build. We ran ddl and dml for every release to build a version six database. Each time we release, we add a new version and the build gets a little longer. We started saving build time by dumping the database schema and data from the previous release. Now, when we want to build a version six database, we restore a version five dump and then build up from there.

Trim selenium suite

Selenium is a great testing tool, but it runs slowly. Opening new browsers and clicking through the site is slow. We looked at our acceptance test suite more carefully and trimmed it. Some of the excess was duplication that was covered in another test (or easily added to another test). Other logic was better tested at the functional level and did not need an acceptance test. We reduced our suite to a few long passes through the application, rather than many smaller tests.

Better hardware

Hardware is relatively cheap compared with developer time. It is worth investing in great hardware. That said, it does not reduce the need for the above improvements. Unfortunately, in many organizations, getting new hardware can be slow. Rather than wait a month, we can work on the build and see results today. And once the new hardware arrives, it will make things even faster.

If you have other build improvement strategies, please let me know in the comments.