Reducing build time

A short build time is a critical element of continuous
integration
.
I’ve been involved in a number of build improvements on my current
project (both local and on the build server), and I thought I would
share some of them. Using the tricks below, we cut our build time in
half (and have more to go). Obviously, every build and project are
different, so many of these may not apply to other situations.

Profile the build

We profiled our build before we made any changes. We found the slowest
tasks using rake—trace, which prints out task timing. Furthermore, we
watched the output from the “top” command while the builds were running
on the build server to reveal bottlenecks.

Turn down logging

We noticed while watching “top” that kjournald was constantly running.
This indicates a lot of disk activity. Our app logs to files and syslog
while running in production, but this is unnecessary during builds. We
turned off nearly all logging in test mode.

Make sure the build machine is only performing builds

We discovered that our build machine had become a bit of a playground
for trying new things. This is fine in moderation, but some of these
projects were not cleaned up. There were a lot of unnecessary processes
running on the build machine. We killed everything that was unnecessary,
and increased the nice value
of processes that were not critical. For example,
cruisecontrol.rb runs the
web interface as a separate process from the builders. We increased the
nice value of the web interface so building our projects would take
precedence.

Parallelizaton

If you have more than one processor (or core), and the machine is
running a giant set of tests, there is a good chance that the other
processor is doing very little. We discovered huge gains by running
tests in parallel. Projects like
deeptest and selenium
grid
make this easy. We went a
simpler route and run our functional and acceptance tests at the same
time in different processes.

Build database from dumps

Our project is on release six, so we have six versions of the database
to build. We ran ddl and dml for every release to build a version six
database. Each time we release, we add a new version and the build gets
a little longer. We started saving build time by dumping the database
schema and data from the previous release. Now, when we want to build a
version six database, we restore a version five dump and then build up
from there.

Trim selenium suite

Selenium is a great testing tool, but it
runs slowly. Opening new browsers and clicking through the site is slow.
We looked at our acceptance test suite more carefully and trimmed it.
Some of the excess was duplication that was covered in another test (or
easily added to another test). Other logic was better tested at the
functional level and did not need an acceptance test. We reduced our
suite to a few long passes through the application, rather than many
smaller tests.

Better hardware

Hardware is relatively cheap compared with developer time. It is worth
investing in great hardware. That said, it does not reduce the need for
the above improvements. Unfortunately, in many organizations, getting
new hardware can be slow. Rather than wait a month, we can work on the
build and see results today. And once the new hardware arrives, it will
make things even faster.

If you have other build improvement strategies, please let me know in
the comments.

Paul Gross

Paul Gross

I'm a lead software developer in Seattle working for Braintree Payments.

Read More