How to use Composite builds as a replacement of buildSrc in Gradle
Gradle buildSrc approach has become standard for implementing custom plugins, tasks and specifying common configurations (like dependencies list and versions) but has one major flaw — it invalidates a build cache when it is changed. On the other hand, Gradle also provides an alternative composite build approach that lacks this flaw. In this article, I describe how to use composite build instead of buildSrc and the challenges to expect from migration.
My experience of Gradle configurations
Gradle build system was introduced for Android Development together with Android Studio. Android Studio has not always been used. If you have been developing applications for Android for more than 6 years, you probably remember Eclipse with Android Plugin. Today this plugin is deprecated and unsupported.
Gradle and Groovy were absolute magic at that time (actually, they still are) and I just copy-pasted everything from StackOverflow. Because applications rarely had more than one module, so keeping everything in one build.gradle file was absolutely fine.
I started using modules for external libraries whenever I wanted to fix or adopt something inside them. And I just copy-pasted all the magic from the application build.gradle to the library’s one. It worked fine, except for the fact that everything needed to be duplicated, e.g. updating dependency versions in all modules. I later discovered that you can use root build.gradle to define a common configuration of modules.
This felt much better but it still wasn’t perfect. Root build.gradle had got too big and complicated to maintain. At the time when modularization was starting to become popular and we were splitting our app into data, core, domain, presentation modules, I discovered a different approach: we could simply extract these functions into separate Gradle scripts and apply them.
This solution lacks autocompletion and you can’t even fix it. In the build.gradle approach at least you can fix missing autocomplete by using plugins { } block, but this block is unavailable in all other script files except build.gradle.
And now, years later, a lot of developers, me included, are using buildSrc to manage common configurations. After so many painful years using buildSrc in our projects is a blessing. You can use any JVM language, you have full autocomplete and IDE support, and you can even write tests: unit tests with JUnit or any other framework, integration tests which actually start a separate instance of Gradle with a test environment provided. Could it be that at last, we have found the Holy Grail of Gradle configuration?!
Drawback of buildSrc
Unfortunately, all these cool features come with one massive drawback. Any change inside buildSrc completely invalidates the build cache. It also invalidates remote build cache, in instances when you are using it. Whilst it’s not really a problem for small projects, big ones with hundreds of modules are affected badly. On the other hand, changes in Gradle script files don’t invalidate the cache but merely invalidate some tasks.
Imagine the following chain of cacheable tasks: compile (Java plugin) -> report (our custom task). compile task has a type of JavaCompile, which we receive from the built-in Java plugin. report is our custom task, which can be defined in 2 different ways: inside buildSrc and inside build.gradle. Now we are going to make any bytecode affecting change inside the report task class. With the buildSrc approach both compile and report tasks will be executed again, even if compile class, inputs and outputs have not been changed. With the build.gradle approach, only the report task will be executed again. Inputs, outputs and bytecode have not been changed for compile, so the result can be taken from cache. Gradle can’t verify that report task will produce the same outputs, that is why it is launched again and build cache is ignored. So, how can we fix it? We absolutely don’t want to go back in time and lose all the buildSrc’s fancy, cool features for the sake of maintaining build speed.
Composite builds
Generally speaking, a composite build is one that includes builds with different root projects. If you have a simple multi-module project, then all subprojects share a common configuration defined by root build.gradle. But there is a way of adding another project without affecting it and this is by using a custom configuration and building it in isolation. It can be useful to build external libraries in this way, as well as parts of your project that are completely independent from each other, but it can also be used for Gradle plugins. You can reference plugins by id from included build in your main project.
If we extract our configuration logic from the buildSrc folder, then classes from the included build aren’t threatened as part of buildSrc and Gradle don’t invalidate build-cache on every change. This is happening because configuration logic is provided to the main project as an external dependency (the same way as other plugins, e.g. Android Gradle Plugin). This means that Gradle now can correctly verify task inputs and outputs and can use build cache.
It is important to add that the following changes affect only build cache. Build cache contains serialised outputs of tasks with keys defined by inputs and classpath. When task results are taken from the cache, you will see FROM-CACHE status of the task. Incremental builds are not affected by this change and tasks will be invalidated anyway. Before launching a task, Gradle can check if task inputs and outputs have been changed since the last run. In the event that they have not, you will see UP-TO-DATE status of the task.
Migration from buildSrc to Composite build
Now, I am going to show the migration process of our Reaktive library to composite build. This project is a very good example for the following reasons:
- It has custom logic inside buildSrc
- It has external Gradle script configuration file binrary-compatibility.gradle
- It has common logic inside root build.gradle
Indeed, we have all the 3 approaches as described in the first part. Here I will show how to deal with each of them and convert them into separate plugins.
Copy
The first step is quite simple. Let’s just copy our buildSrc folder to buildSrc2 folder. If you don’t use plugins in your buildSrc folder, then now is a good time to start. Without any plugins, classes from your new module won’t be loaded into the build script classpath. Don’t remove your original buildSrc yet. If you do, you won’t be able to sync the project. To inform Gradle of the new module that is included, we add the following into settings.gradle:
The first line is added for our convenience and to remove the buildscript { repositories { } } block. By using the includeBuild function we are telling Gradle to treat the project inside buildSrc2 folder as included build. This project will be built on demand.
Plugins migration
So, how do we use it? First, let’s add our plugins definitions.
java-gradle-plugin will create corresponding properties files for plugins, so you can remove them. You can read more about java-gradle-plugin here.
If you don’t yet have any plugins and are only using buildSrc for dependencies management, you need to create an empty fake plugin and apply it to the project to make classes available in the build script.
Once you apply class-loader-plugin to the project, Deps class becomes available. And autocompletion will work in the same way as before.
Common functions migration
Inside build.gradle we had setupMultiplatformLibrary function.
This function defines some common configuration for all modules. We applied the Kotlin Multiplatform plugin and declared some essential dependencies.
To convert this function into a Gradle plugin, we need to specify a dependency on Gradle plugin and create our custom plugin, which does the same setup.
In addition, we have parameterised setup for setupAllTargetsWithDefaultSourceSets with isLinuxArm32HfpEnabled parameter. Kotlin coroutines do not support linuxArm32Hfp target, but we do. That is why we should avoid configuring this target just for the coroutines interoperability library. To do so, we could filter out project.name, but we don’t want to hardcode it. Instead, we can implement it with the extension approach like other plugins.
Unfortunately, we can’t use the opt-out system here (disableLinuxArm32Hfp()), because Kotlin Gradle plugin does not handle target removal, but only target addition. However, we can apply configuration with the help of mpp-configuration plugin and its configuration extension.
And autocompletion works as expected.
External script file migration
We use Binary Compatibility Validator Plugin which I described in the following article. Configuration for it is defined inside binrary-compatibility.gradle and applied to root build.gradle. Basically, all it does is apply plugins and configures the modules to ignore.
And we can simply convert the following build script into a plugin using the approach described previously.
Next, we can apply our new plugin inside root build.gradle.
Dependencies
With this new approach, we need dependency management both in the main project and included builds. In included builds, we use different plugins as dependencies to ensure access to related classes such as project extensions, tasks etc. For this purpose, we will create additional included builds in order to manage dependencies.
Now we can create a Deps class for all the external dependencies we have. You can check implementation here. Add a new module to settings.gradle using includeBuild(“dependencies”). Now the dependencies plugin and Deps class can be used in any project.
While implementing this approach I discovered that if I use included build as a dependency inside another included build, IDEA does not resolve its classes in the editor(although it still compiles without errors). To fix this, I manually included Deps class into buildSrc2 compilation so that it will be available wherever any plugin on buildSrc2 is applied. Quite a dirty and unstable hack, but I hope that fixes in future releases will make such misbehaviour unnecessary. Once fixed, using regular implementation(“com.badoo.reaktive.dependencies:dependencies:SNAPSHOT”) notation will be sufficient.
dependencies plugin can be used in the main project modules in the same way as described above.
Drawbacks
The only difference between using composite build instead of buildSrc is the availability of classes without a corresponding plugin.
The real difference appears when we use plugins { } block instead of apply plugin: ‘id’ or direct configuration functions call. The main advantage of using plugins block is autocompletion inside Groovy scripts. You will have access to classes related to specific plugins and extensions. But you cannot be completely sure that extensions are fully configured at the moment when you apply a plugin. For example, say you had the following setup:
Custom plugin will just print compileSdkVersion in the console in the configuration phase. In this implementation, it will be 30. Now let’s migrate it to using plugins { } block.
In this case, you will see null in the console output, because custom-plugin is applied before android configuration. There are a couple of ways to fix this:
1. Use apply plugin: ‘custom-plugin’ or static functions to set up the build. It is OK to continue using it if it is hard to migrate. The only thing you will lose is autocompletion support in Groovy scripts. Also, remember you need to apply a fake plugin to load corresponding classes.
2. Use the project.afterEvaluate { } block inside the plugin. But be careful: if you abuse it too much, your afterEvaluate blocks will start to depend on other afterEvaluate blocks and the order of their execution.
3. Try to convert your logic inside the plugin by using lazy API, tasks and other mechanisms. This requires good knowledge of Gradle API, but this way you will create reusable and independent plugins.
Conclusion
Composite builds can be used as a replacement of buildSrc to avoid Gradle cache invalidation. Migration using the proposed approach is both straightforward and painless. Included builds can use plugins from other included builds and depend on each other. To see the full power of autocompletion in your Groovy scripts you can use plugins { } block if you like. When you have no plugins, just create an empty fake plugin and apply it to load classes from the same module.
