Download to read offline
Uploaded byWix Engineering
Building a Large Java Codebase with Bazel - Natan Silnitsky
The document discusses the transition from Maven to Bazel for building a large Java codebase at Wix, emphasizing the challenges of longer build times and the use of parallelism and caching to improve efficiency. It covers solutions such as skipping tests, caching module outputs, and breaking up code into smaller modules, while also addressing semantic versioning issues. The migration process involved consolidating repositories and enhancing build execution through remote capabilities, resulting in significantly faster builds and improved developer experience.
More Related Content
Similar to Building a Large Java Codebase with Bazel - Natan Silnitsky
![Faster java ee builds with gradle [con4921]](https://cdn.slidesharecdn.com/ss_thumbnails/fasterjavaeebuildswithgradlecon4921-160927042155-thumbnail.jpg?width=640&height=640&fit=bounds)
![谷歌留痕技术教程[ 𝙩𝙤𝙥 𝟮𝟯𝟯. 𝙘 𝙤𝙢 ]](https://cdn.slidesharecdn.com/ss_thumbnails/top233-260130173900-2eb784f9-thumbnail.jpg?width=640&height=640&fit=bounds)
Building a Large Java Codebase with Bazel - Natan Silnitsky
- 1. Natan Silnitsky Backend InfraDeveloper, Wix.com Building a Large Java Codebase with Bazel natans@wix.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
- 2.
- 3. 1 h 1 2 3 4 5 1 2 3 4 1 2 3 1 2 7 0 1 0 1 Lines ofcode 2 5 0 7 Maven modules Build time EXAMPLE WIX REPO
- 4.
- 5. WIX BACKEND ~250 developers ~1100 micro-services >10MLoC, Scala ,Java, Node.js K’s of builds every day
- 6.
- 7.
- 8. #1 Skip tests! mvninstall -DskipTests gradle build -x test But you compromise on quality… more production bugs
- 9. #2 Cache moduleoutputs ● Big overhead when there are many modules Unreliable caching Better caching solution Maven Gradle
- 10. #3 Code breakup Monolith Microservices Code Gitrepos Each repo built separately Faster build time
- 11. #3 Code breakup Whatabout Inter-repo dependencies? ● Common libs (infra, sdk clients) ● Service libs (api, testkit) Code Git repos Each repo built separately Faster build time
- 12. Semantic versioning breaking change new feature bug fix 1 .3 . 1 MAJOR . MINOR . PATCH * 3rd party
- 13.
- 14.
- 15. ● #code-paths ● Integrationissues Semantic versioning…? Why not V5 V8 +Incompatible change V10 + Bug fix Supplier provides v11 with v5+fix Client change code to work with v8
- 16. ● #code-paths ● Integrationissues ● Code repetitions Semantic versioning…? Why not V5 V5 + Bug fix
- 17. ● #code-paths ● Integrationissues ● Code repetitions ● Keeps getting violated Semantic versioning…? Why not 1 . 3 . 1 breaking change new feature bug fix 1 . 3 . 0
- 18. <dependency> <groupId>com.wix.foo</groupId> <artifactId>api</artifactId> <version>1.0.0-SNAPSHOT</version> </dependency> Wix used Mavensnapshot versions instead BUILD TOOL Maven BUILD SERVER TeamCity
- 19. Maven Maven MavenMaven Maven Maven Maven Maven Maven Maven Maven Maven Maven Maven Maven Maven Maven Maven Maven Maven + Pool of agents + Distribute work + Jobs that run maven with clean cache + Dependencies TeamCity
- 20. Maven Maven Maven Maven Maven buildable buildablebuildable buildable buildable + Ensure tobuild against the latest SNAPSHOT dependency TeamCity
- 21. building blockedblocked blocked blocked + Block whiledependency is building Maven Maven Maven Maven Maven TeamCity * binary incompatibility
- 22. broken blockedblocked blocked blocked + Block whiledependency is broken Maven Maven Maven Maven Maven TeamCity
- 23.
- 24. USE BAZEL 02 Build everything fromsource Divide your code to smaller build units Heavy use of caching and parallelism
- 25. In Bazel You controlthe level of granularity to build. java_project ├── WORKSPACE └── src └── main └── java └── com ├── example │ ├── A.java │ ├── B.java │ └── C.java └── example2 ├── D.java ├── E.java └── F.java
- 26. The basic buildunit is called a target java_project ├── WORKSPACE └── src └── main └── java └── com ├── example │ ├── A.java │ ├── B.java │ ├── BUILD │ └── C.java └── example2 ├── D.java ├── E.java └── F.java Target Targets are instances of rules.
- 27. A RULE f(A,B,C)f( ) A B C sourcefiles | dependencies | tools configurations Sandbox
- 28. A RULE f(A,B,C) OUTPUTs f( ) A B C f(A,B,C) f() A B C SAME
- 29.
- 30. MUCH MORE PARALLEL Maven/ Gradle BAZEL
- 31. MUCH MORE PARALLEL Maven/ Gradle Compilation Test Run BAZEL Pending
- 32. INCREMENTAL ✔ ✔ ✔ ✔✔ ✔ ✔ ✔ ✔ ✔ BAZEL$ bazel build //...
- 33. Efficiency is attainednot by doing things faster, But by avoiding things that should not be done in the first place -Venkat Subramaniam-
- 34. GOOGLE CLOUD /OSS REMOTE EXECUTION 100s Workers -CI/local tradeoff ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ BAZEL Cache
- 35. 1 hmaven localclean IT’S WORTH IT
- 36. 1 hmaven localclean 20 minbazel local clean 8.5 minbazel remote clean IT’S WORTH IT bazel fully cached 0.1 min
- 37.
- 38. How did wemigrate? Changed code to work with newer 3rd party versions Auto Migrator Tests that failed in sandbox Consolidated ~1000 repos -> ~50 https://github.com/wix/exodus/ Splits Maven Modules to bazel targets
- 39.
- 40. How did wemigrate? Changed code to work with newer 3rd party versions Auto Migrator Tests that failed in sandbox Consolidated ~1000 repos -> ~50 E.g. Writing to non tmp destinations,Access external network endpoint *Tech debt
- 41. How did wemigrate? Changed code to work with newer 3rd party versions Auto Migrator Tests that failed in sandbox Consolidated ~1000 repos -> ~50
- 42. How did wemigrate? Changed code to work with newer 3rd party versions Auto Migrator Tests that failed in sandbox Consolidated ~1000 repos -> ~50 50 Repos work together like 1 virtual mono-repo Virtual Mono-Repo & Bazel Talk by Ittai Zeidman from Wix
- 43.
- 44. Remote Build Executionis fast and reliable
- 45. Remote Build Executionis fast and reliable Pre-submit - reduces noise
- 46. Remote Build Executionis fast and reliable Pre-submit - reduces noise Very Rich Build results API (Build events protocol)
- 47. Google Cloud buildBazel UI
- 48. Intellij support Wix isworking on improvements No Sandbox support / RBE for Mac Harder to maintain build configuration Wix built plugin features - OSS SOON
- 49. LabelDex Indexing Server ✔ ✔ ✔✔ Target Labels to JARs Google Cloud Build Results Intellij Plugin OSS SOON LabelDex
- 50. OSS SOON SmartAutoComplete +AutoDep
- 51. Automatically fix dependenciesin BUILD files by parsing errors from bazel build result log + index server. Inspired by Pants OSS SOON DepFixer Internal JARs External JARs Build result log
- 52. USE BAZEL Build everything fromsource Divide your code to smaller build units Heavy use of caching and parallelism
- 53. Faster Builds &Feedback More Frequent Cross Org Changes/Refactoring Improved local experience BAZEL IS WORTH IT
- 54. Migration Stories Migrating toBazel? 5 crucial questions... - My Blog Series on Medium We Switched from Maven to Bazel and Builds Got 10x Faster - Redfin Migrating From Maven to Bazel at Salesforce
- 55. Thank You natans@wix.com twitter@NSilnitskylinkedin/natansilnitsky github.com/natansil
- 56.
- 57. Q&A natans@wix.com twitter@NSilnitsky linkedin/natansilnitskygithub.com/natansil