Lately I’ve run some experiments with remote Bazel builds of Angular and React applications. This post is a write-up of my results so far.

Remote Execution

As an alternative to building code locally, Bazel supports remote build execution.

If you have the server infrastructure for it, remote execution could mean a drastic reduction in build times from outsourcing the build to large computers in the cloud.

A natural extension of this is Remote caching. Once the code is built in the cloud, it makes sense to also cache the result to avoid future rebuilds.

Buildfarm

In my experiments I have been using Buildfarm.

One of the advantages of Buildfarm is that it offers both remote execution and caching in a single package.

It’s also relatively easy to configure Buildfarm.

The setup procedure is explained in their readme, but at a high level you start a build server with one or more remote workers. Workers can run on separate servers as long as you configure them to point to the server (worker.config).

Sharing the server between workers also enables the workers to share a single remote build cache.

In my repo I running a server and a single worker in two different docker containers. You can easily scale this up by launching instances of the worker container on multiple servers. Just remember to update worker.config to point back to the server instance.

For standalone worker containers just comment out docker networking in docker-compose-angular.yaml since it assumes a server and worker running on the same box.

This 1-to-1 setup is only for demo purposes and obviously not scalable.

I am building Buildfarm directly from master in the docker container.

Bazel

Buildfarm requires the lates version of the Bazel remote execution api. This means you have to use Bazel 0.17 or later.

In my docker containers I am installing the latest version of Bazel at the time of building.

The Angular application used for this demo is an extended version of the canonical Angular Bazel sample found here. The main difference from the original example is the addition of a large feature.

The React application is a large 8000 component application. The purpose of the high number of components is to stretch the limits of the demo’s scalability

Build Times

Angular

In the case of the Angular application, the baseline for the build is 367 seconds. This is from a local build after running Bazel clean. Obviously the build times will vary greatly based on your hardware.

Adding remote execution has the potential to reduce this drastically, but with a single computer setup, there is not much gain.

Hopefully someone will try this on a computer farm with lots of cores and report back with amazing build times :-). If you have access to fast server(s), try increasing –jobs in .bazelrc to a higher value to build more tasks in parallel.

However, even on modest hardware, there is still a huge improvement in build performance on second builds. This is because Buildfarm caches the results from past builds.

After Bazel clean the second build time is only 35 seconds due to the cache.

Next I will simulate a decent size code change by making an update to 100 source files.

Based on this update we will not be able to leverage the cache 100%.

Any changed module will have to be rebuilt, but the build time is still pretty decent - 86 seconds after Bazel clean.

React

The React application is interesting since it’s enormous. 8000 components take forever to build!

During testing I wired up 4 of my computers with each one running a separate worker. This certainly helps, but still, there is no escaping long build times given my relatively modest “server cluster”.

Fortunately second build times are much faster since build results are served out of the cache.

Sharing the Cache

Is there a way to get around the slow first builds?

A key benefit of the cache is that it can be shared across developer boxes.

Ideally only the first build on the first box should be slow.

Potentially the CI server could be used to populate the cache since it continuously builds all code revisions anyway.

I wanted to simulate this by spinning up two containers per application. After building both applications in the first container, first builds in the second container should be able to leverage the cache close to 100%.

In practice I have noticed a big difference in cache hit rates between the Angular application and the React application.

Bazel gives you the number of cache hits at the end of the build, but for Angular builds the ratio is fairly low. By contrast the React application shows a 100% cache hit ratio in the second container.

You can test this by spinning up the second container by running yarn start-react-app2.

The Angular app still benefits from the cache on second builds, but it doesn’t share the cache across computers. Meaning, you can’t directly benefit from a cache generated by your CI server. Instead each build generates its own unique cache.

I have not investigated why the cache hit ration is so low.

Both applications are written in Typescript, but the Angular application is built using a compiler with custom extensions (AoT) on top of the regular Typescript compiler. Not sure if this matters, but it’s the only difference I can think of.

Further Research

I think remote execution/caching holds a lot of promise, but I still consider this experimental. There are a number of issues and limitations to overcome before I consider this 100% reliable.

Caching seems very sensitive to differences in computer setups. Running in docker gives me 100% control over the environment.

Unfortunately I am not seeing the same cache rates when building in less controlled build environments. Also, I am not able to run builds from a Mac against a linux server. These are limiting factors I still have to research.

I have shared my code on Github if you are interested in checking it out.

The instructions for how to get started is in the readme.