Migrating from Gradle to Bazel

We recently switched a collection of Java applications and libraries (in a monorepo) from Gradle to Bazel at Braintree. Overall, the transition went well and we are much happier with Bazel than Gradle.

The problems with Gradle

We had a number of issues with Gradle that led us to seek alternate build tools:

Gradle is slow

One of our biggest issues with Gradle was the speed. Everything felt sluggish and unnecessarily slow. For example, here are some comparisons between Gradle and Bazel:

With no files changed, rerunning build:

% time ./gradlew compileJava compileTestJava
10.547 total

% time bazel build //...
0.520 total

Running all of the tests takes half the time:

% time ./gradlew test
1:58.84 total

% time bazel test //...
1:01.14 total

Changing a single file, recompiling and rerunning dependent tests also takes half the time:

% time ./gradlew test
cpu 25.802 total

% time bazel test //...
12.860 total

Gradle is just slow to figure out what has changed and what needs to be run. It’s probably due to the number of subprojects (modules) we have inside this monorepo (about 20). Bazel seems to handle this case much better, and this is before we have even optimized our build dependencies. Gradle’s support for parallel execution is still incubating, and even with it on, we didn’t notice much improvement.

Gradle’s build language is groovy

Maybe Groovy is a great language on its own merits, but it’s a difficult build language for a team that isn’t familiar with it. Most of us are familiar with build languages in the host language (e.g. Rake and Ruby), or common scripting languages like Bash, Python, etc. Groovy adds yet another thing to learn, and has its own quirks.

For example, here is how you execute a command and redirect standard output in a Gradle task:

task myTask(type: Exec) {
    commandLine 'someCommand', 'param1'

    doFirst {
        standardOutput = new FileOutputStream('output.txt')
    }
}

This is pretty different than a simple someCommand param1 > output.txt. Gradle doesn’t prevent us from writing Bash scripts, but if you want custom commands to be run as dependencies of other tasks, it can be harder to write them separately.

Bazel’s build languages (Core and Skylark) are subsets of Python, so they are more familiar to us. They also aren’t general purpose languages, which force us to write scripts in scripting languages. That means we’re using the right tool for the job, instead of Gradle for everything.

Gradle is error prone

We found Gradle to be full of gotchas. For example, we had a section in our build.gradle like this to turn on compiler warnings:

subprojects {
  compileJava {
    options.compilerArgs << "-Xlint:all"
    options.compilerArgs << "-Xlint:-processing"
    options.compilerArgs << "-Werror"
  }

  apply plugin: "java"
}

It turned out that since the apply plugin: "java" was after the compileJava section, the options were never applied. There was no error or warning; Gradle just silently ignored our code.

We also ran into a lot of frustrations with the Gradle IntelliJ IDEA plugin. It would sometimes fail to refresh after we made changes to our build files (even with auto-import turned on), and then it was hard to get it back into a good state. We’d often have to manually synchronize the project, or even bounce IntelliJ.

Another frustration we ran into was with libraries which pulled in other libraries which conflicted with each other (kafka -> slf4j-log4j12). We were able to fix this in Gradle with code like:

dependencies {
  compile("org.apache.kafka:kafka_2.11:0.8.2.1") {
    exclude module: "slf4j-log4j12"
  }
}

But this didn’t affect the IntelliJ Gradle plugin. To fix that one, we had to manually check in a file called .idea/libraries/Gradle__org_slf4j_slf4j_log4j12_1_6_1.xml:

<component name="libraryTable">
  <library name="Gradle: org.slf4j:slf4j-log4j12:1.6.1">
    <CLASSES />
    <JAVADOC />
    <SOURCES />
  </library>
</component>

This worked, but if you accidentally imported the project again, or made any other manual changes to dependencies, IntelliJ would remove this file and the error would return.

In the Bazel world, every dependency is explicitly stated. In our case, we just left out the conflicting library and only included the good one. It was much simpler.

More benefits of Bazel

Bazel brings more to the table than just fixing our issues with Gradle. Here are a few notable features:

Docker support

Bazel recently announced support for building Docker images (i.e. directly in Bazel, without Dockerfiles): Building deterministic Docker images with Bazel

With Bazel, we were able to create a macro that unified a bunch of our Docker config. Now, our apps can build Docker images with only a few lines of config (and no Dockerfile):

app_docker_image(
  java_binary = "myapp",
  main_class = "braintree.myapp.MyApplication",
)

Bazel query

Bazel supports a powerful query language. For example, if you change a library, you can query and run all of the dependent tests (API changes with extra cheese, hold the fear):

bazel test $(bazel query 'kind(test, rdeps(//..., //mylibrary))')

You can also query and graph your dependencies (Have you ever looked at your build? I mean, really looked at your build?):

bazel query 'deps(//:main)' --output graph > graph.in

dot -Tpng < graph.in > graph.png

Test tagging

Bazel lets you specify the size of your test suites (small, medium, etc) and then it will alert you if the suite time exceeds the allotted time. This can help keep test times under control.

There are also a handful of special behavior tags, such as exclusive, manual, external, and flaky: http://bazel.io/docs/bazel-user-manual.html#tags_keywords

How we switched

Once we decided to switch, we had to do a number of things to cut over.

Porting the build files

The first step was porting our existing build.gradle files over to BUILD files. We went through a few iterations on scripting this, and the latest version is on GitHub: bazel-deps.

With this tool, we were able to generate the majority of our WORKSPACE and BUILD files. We explicitly depend on less than 30 libraries (from Maven), but the transitive dependencies come out to over 200! For example, dropwizard alone brings in about 80 dependent libraries.

IntelliJ

We wrote a script to create an IntelliJ project from our Bazel BUILD files. We based it on the script that comes with Bazel (setup-intellij.sh) but modified it for our needs.

For example, our script generates an IntelliJ project with modules for each of our subprojects. This means we need to create a top level modules.xml, a subproject.iml for each project, and then add dependent projects as project level dependencies. Our script also adds Bazel generated files as dependencies to the subprojects, such as generated protobuf classes.

Aliases

We added some aliases for common Bazel commands:

alias bzb="bazel build //..."
alias bzt="bazel test //..."

Aftermath

We’ve now been on Bazel exclusively for a few weeks (after running both for a while). The cutover was a little painful while we figured out things like CI (Bazel is a memory hog by default, and our build boxes are memory constrained). We also had to migrate all of our processes, tools, READMEs, etc. over to Bazel. Inevitably, we missed a few at first which caused headaches for some of our team. Thankfully, we seem to be past all of this, and we’re all happily running Bazel now.

If you’re interested in more cool Bazel features, check out the Bazel Blog.