In looking through the previous pages, one theme repeats over and over: managing your own code is fairly straightforward, but managing its dependencies is much more difficult. There are all sorts of dependencies: sometimes there’s a dependency on a task (such as “push the documentation before I mark a release as complete”), and sometimes there’s a dependency on an artifact (such as “I need to have the latest version of the computer vision library to build my code”). Sometimes, you have internal dependencies on another part of your codebase, and sometimes you have external dependencies on code or data owned by another team (either in your organization or a third party). But in any case, the idea of “I need that before I can have this” is something that recurs repeatedly in the design of build systems, and managing dependencies is perhaps the most fundamental job of a build system.
Dealing with Modules and Dependencies
Projects that use artifact-based build systems like Bazel are broken into a set
of modules, with modules expressing dependencies on one another via BUILD
files. Proper organization of these modules and dependencies can have a huge
effect on both the performance of the build system and how much work it takes to
maintain.
Using Fine-Grained Modules and the 1:1:1 Rule
The first question that comes up when structuring an artifact-based build is
deciding how much functionality an individual module should encompass. In Bazel,
a module is represented by a target specifying a buildable unit like a
java_library or a go_binary. At one extreme, the entire project could be
contained in a single module by putting one BUILD file at the root and
recursively globbing together all of that project’s source files. At the other
extreme, nearly every source file could be made into its own module, effectively
requiring each file to list in a BUILD file every other file it depends on.
Most projects fall somewhere between these extremes, and the choice involves a
trade-off between performance and maintainability. Using a single module for the
entire project might mean that you never need to touch the BUILD file except
when adding an external dependency, but it means that the build system must
always build the entire project all at once. This means that it won’t be able to
parallelize or distribute parts of the build, nor will it be able to cache parts
that it’s already built. One-module-per-file is the opposite: the build system
has the maximum flexibility in caching and scheduling steps of the build, but
engineers need to expend more effort maintaining lists of dependencies whenever
they change which files reference which.
Though the exact granularity varies by language (and often even within
language), Google tends to favor significantly smaller modules than one might
typically write in a task-based build system. A typical production binary at
Google often depends on tens of thousands of targets, and even a moderate-sized
team can own several hundred targets within its codebase. For languages like
Java that have a strong built-in notion of packaging, each directory usually
contains a single package, target, and BUILD file (Pants, another build system
based on Bazel, calls this the 1:1:1 rule). Languages with weaker packaging
conventions frequently define multiple targets per BUILD file.
The benefits of smaller build targets really begin to show at scale because they
lead to faster distributed builds and a less frequent need to rebuild targets.
The advantages become even more compelling after testing enters the picture, as
finer-grained targets mean that the build system can be much smarter about
running only a limited subset of tests that could be affected by any given
change. Because Google believes in the systemic benefits of using smaller
targets, we’ve made some strides in mitigating the downside by investing in
tooling to automatically manage BUILD files to avoid burdening developers.
Some of these tools, such as buildifier and buildozer, are available with
Bazel in the
buildtools directory.
Minimizing Module Visibility
Bazel and other build systems allow each target to specify a visibility — a
property that determines which other targets may depend on it. A private target
can only be referenced within its own BUILD file. A target may grant broader
visibility to the targets of an explicitly defined list of BUILD files, or, in
the case of public visibility, to every target in the workspace.
As with most programming languages, it is usually best to minimize visibility as
much as possible. Generally, teams at Google will make targets public only if
those targets represent widely used libraries available to any team at Google.
Teams that require others to coordinate with them before using their code will
maintain an allowlist of customer targets as their target’s visibility. Each
team’s internal implementation targets will be restricted to only directories
owned by the team, and most BUILD files will have only one target that isn’t
private.
Managing Dependencies
Modules need to be able to refer to one another. The downside of breaking a
codebase into fine-grained modules is that you need to manage the dependencies
among those modules (though tools can help automate this). Expressing these
dependencies usually ends up being the bulk of the content in a BUILD file.
Internal dependencies
In a large project broken into fine-grained modules, most dependencies are likely to be internal; that is, on another target defined and built in the same source repository. Internal dependencies differ from external dependencies in that they are built from source rather than downloaded as a prebuilt artifact while running the build. This also means that there’s no notion of “version” for internal dependencies—a target and all of its internal dependencies are always built at the same commit/revision in the repository. One issue that should be handled carefully with regard to internal dependencies is how to treat transitive dependencies (Figure 1). Suppose target A depends on target B, which depends on a common library target C. Should target A be able to use classes defined in target C?
Figure 1. Transitive dependencies
As far as the underlying tools are concerned, there’s no problem with this; both B and C will be linked into target A when it is built, so any symbols defined in C are known to A. Bazel allowed this for many years, but as Google grew, we began to see problems. Suppose that B was refactored such that it no longer needed to depend on C. If B’s dependency on C was then removed, A and any other target that used C via a dependency on B would break. Effectively, a target’s dependencies became part of its public contract and could never be safely changed. This meant that dependencies accumulated over time and builds at Google started to slow down.
Google eventually solved this issue by introducing a “strict transitive dependency mode” in Bazel. In this mode, Bazel detects whether a target tries to reference a symbol without depending on it directly and, if so, fails with an error and a shell command that can be used to automatically insert the dependency. Rolling this change out across Google’s entire codebase and refactoring every one of our millions of build targets to explicitly list their dependencies was a multiyear effort, but it was well worth it. Our builds are now much faster given that targets have fewer unnecessary dependencies, and engineers are empowered to remove dependencies they don’t need without worrying about breaking targets that depend on them.
As usual, enforcing strict transitive dependencies involved a trade-off. It made
build files more verbose, as frequently used libraries now need to be listed
explicitly in many places rather than pulled in incidentally, and engineers
needed to spend more effort adding dependencies to BUILD files. We’ve since
developed tools that reduce this toil by automatically detecting many missing
dependencies and adding them to a BUILD files without any developer
intervention. But even without such tools, we’ve found the trade-off to be well
worth it as the codebase scales: explicitly adding a dependency to BUILD file
is a one-time cost, but dealing with implicit transitive dependencies can cause
ongoing problems as long as the build target exists. Bazel
enforces strict transitive dependencies
on Java code by default.
External dependencies
If a dependency isn’t internal, it must be external. External dependencies are those on artifacts that are built and stored outside of the build system. The dependency is imported directly from an artifact repository (typically accessed over the internet) and used as-is rather than being built from source. One of the biggest differences between external and internal dependencies is that external dependencies have versions, and those versions exist independently of the project’s source code.
Automatic versus manual dependency management
Build systems can allow the versions of external dependencies to be managed
either manually or automatically. When managed manually, the buildfile
explicitly lists the version it wants to download from the artifact repository,
often using a semantic version string such
as 1.1.4. When managed automatically, the source file specifies a range of
acceptable versions, and the build system always downloads the latest one. For
example, Gradle allows a dependency version to be declared as “1.+” to specify
that any minor or patch version of a dependency is acceptable so long as the
major version is 1.
Automatically managed dependencies can be convenient for small projects, but they’re usually a recipe for disaster on projects of nontrivial size or that are being worked on by more than one engineer. The problem with automatically managed dependencies is that you have no control over when the version is updated. There’s no way to guarantee that external parties won’t make breaking updates (even when they claim to use semantic versioning), so a build that worked one day might be broken the next with no easy way to detect what changed or to roll it back to a working state. Even if the build doesn’t break, there can be subtle behavior or performance changes that are impossible to track down.
In contrast, because manually managed dependencies require a change in source control, they can be easily discovered and rolled back, and it’s possible to check out an older version of the repository to build with older dependencies. Bazel requires that versions of all dependencies be specified manually. At even moderate scales, the overhead of manual version management is well worth it for the stability it provides.
The One-Version Rule
Different versions of a library are usually represented by different artifacts, so in theory there’s no reason that different versions of the same external dependency couldn’t both be declared in the build system under different names. That way, each target could choose which version of the dependency it wanted to use. This causes a lot of problems in practice, so Google enforces a strict One-Version Rule for all third-party dependencies in our codebase.
The biggest problem with allowing multiple versions is the diamond dependency issue. Suppose that target A depends on target B and on v1 of an external library. If target B is later refactored to add a dependency on v2 of the same external library, target A will break because it now depends implicitly on two different versions of the same library. Effectively, it’s never safe to add a new dependency from a target to any third-party library with multiple versions, because any of that target’s users could already be depending on a different version. Following the One-Version Rule makes this conflict impossible—if a target adds a dependency on a third-party library, any existing dependencies will already be on that same version, so they can happily coexist.
Transitive external dependencies
Dealing with the transitive dependencies of an external dependency can be particularly difficult. Many artifact repositories such as Maven Central, allow artifacts to specify dependencies on particular versions of other artifacts in the repository. Build tools like Maven or Gradle often recursively download each transitive dependency by default, meaning that adding a single dependency in your project could potentially cause dozens of artifacts to be downloaded in total.
This is very convenient: when adding a dependency on a new library, it would be a big pain to have to track down each of that library’s transitive dependencies and add them all manually. But there’s also a huge downside: because different libraries can depend on different versions of the same third-party library, this strategy necessarily violates the One-Version Rule and leads to the diamond dependency problem. If your target depends on two external libraries that use different versions of the same dependency, there’s no telling which one you’ll get. This also means that updating an external dependency could cause seemingly unrelated failures throughout the codebase if the new version begins pulling in conflicting versions of some of its dependencies.
For this reason, Bazel does not automatically download transitive dependencies.
And, unfortunately, there’s no silver bullet—Bazel’s alternative is to require a
global file that lists every single one of the repository’s external
dependencies and an explicit version used for that dependency throughout the
repository. Fortunately, Bazel provides tools that are able to automatically
generate such a file containing the transitive dependencies of a set of Maven
artifacts. This tool can be run once to generate the initial WORKSPACE file
for a project, and that file can then be manually updated to adjust the versions
of each dependency.
Yet again, the choice here is one between convenience and scalability. Small projects might prefer not having to worry about managing transitive dependencies themselves and might be able to get away with using automatic transitive dependencies. This strategy becomes less and less appealing as the organization and codebase grows, and conflicts and unexpected results become more and more frequent. At larger scales, the cost of manually managing dependencies is much less than the cost of dealing with issues caused by automatic dependency management.
Caching build results using external dependencies
External dependencies are most often provided by third parties that release stable versions of libraries, perhaps without providing source code. Some organizations might also choose to make some of their own code available as artifacts, allowing other pieces of code to depend on them as third-party rather than internal dependencies. This can theoretically speed up builds if artifacts are slow to build but quick to download.
However, this also introduces a lot of overhead and complexity: someone needs to be responsible for building each of those artifacts and uploading them to the artifact repository, and clients need to ensure that they stay up to date with the latest version. Debugging also becomes much more difficult because different parts of the system will have been built from different points in the repository, and there is no longer a consistent view of the source tree.
A better way to solve the problem of artifacts taking a long time to build is to use a build system that supports remote caching, as described earlier. Such a build system saves the resulting artifacts from every build to a location that is shared across engineers, so if a developer depends on an artifact that was recently built by someone else, the build system automatically downloads it instead of building it. This provides all of the performance benefits of depending directly on artifacts while still ensuring that builds are as consistent as if they were always built from the same source. This is the strategy used internally by Google, and Bazel can be configured to use a remote cache.
Security and reliability of external dependencies
Depending on artifacts from third-party sources is inherently risky. There’s an
availability risk if the third-party source (such as an artifact repository) goes
down, because your entire build might grind to a halt if it’s unable to download
an external dependency. There’s also a security risk: if the third-party system
is compromised by an attacker, the attacker could replace the referenced
artifact with one of their own design, allowing them to inject arbitrary code
into your build. Both problems can be mitigated by mirroring any artifacts you
depend on onto servers you control and blocking your build system from accessing
third-party artifact repositories like Maven Central. The trade-off is that
these mirrors take effort and resources to maintain, so the choice of whether to
use them often depends on the scale of the project. The security issue can also
be completely prevented with little overhead by requiring the hash of each
third-party artifact to be specified in the source repository, causing the build
to fail if the artifact is tampered with. Another alternative that completely
sidesteps the issue is to vendor your project’s dependencies. When a project
vendors its dependencies, it checks them into source control alongside the
project’s source code, either as source or as binaries. This effectively means
that all of the project’s external dependencies are converted to internal
dependencies. Google uses this approach internally, checking every third-party
library referenced throughout Google into a third_party directory at the root
of Google’s source tree. However, this works at Google only because Google’s
source control system is custom built to handle an extremely large monorepo, so
vendoring might not be an option for all organizations.