Two years ago, we were given a big challenge - Transform Wix Build System, then based on Maven and Teamcity, to a new system that will support our exponentially growing scale. But, how could we move to a system so different in so many ways than the existing one? Furthermore, we were required not to break the current build system, as we migrate to the new one. Fast forward to today: Wix backend CI system is fully migrated to Bazel! The system builds in a fracture of the time - even with our largest codebases. In this talk, we will describe how we achieved this, why it took us so long, what tools we had to build on the way (and what we already have, and will, open source!), and share the principles that helped us.

1. Or Shachar,
Backend Engineer, Wix-CI
Migrating to a Bazel-based CI
system: 6 learnings
ors@wix.com linkedin/or-shachar github.com/or-shachar

2. Check emailCoffee break Ping pong
Reading news CAll your mom
<Pending mode>

3. Check emailCoffee break Ping pong
Reading news CAll your mom
Building...

4. Hi,
I'm Or.
CI
Build with Ant and Makefiles
Maven CI Pipelines
Backend Engineering
…
Backend Engineering for2017
2011
2014
2019

5. Building...
Building...
We migrated
to Bazel

6. Codebase +
build definition
Build server
feedback
Build tool
deployables

7. ~5M LOC
> 1000 repos
2000+ Maven modules
2017
Teamcity
Maven
feedback
deployables

8. ~5M LOC
> 1000 repos
2000+ Maven modules
2017
Teamcity
Maven
feedback
deployables
building / broken
blockedblocked
blocked
blocked

9. 2017
Master
stability
Push to
feedback
Push to
release
Often broken
Can take 1 hour
Can take hours

10. And so we
wait...

11. Find new machinery
that’s stable, fast, and can scale.

12. Master
stability
Push to
feedback
Push to
release
2019
Often broken
Can take 1 hour
Can take hours
2017
Master
stability
Push to
feedback
Push to
release
Mostly stable
5-10 minutes
15-25 minutes

13. 6 LEARNINGS
FROM
MIGRATING
OUR BUILD
SYSTEM TO
BAZEL.

14. #1
Use a Migration tool
Massage it, validate it.

15. Migration 101 Bazel>>Maven >>

16. .
├── foo-core [pom.xml]
│ └── src
│ ├── main
│ │ └── java
│ │ ├── [com.wix.foo.api]
│ │ │ └── FooBar.java
│ │ └── [com.wix.foo.utils]
│ │ ├── TimeUtils.java
│ │ └── NameUtils.java
│ └── test
│ └── java
│ └── [com.wix.foo.utils]
│ ├── TimeUtilsTest.java
│ └── NameUtilsTest.java
└── foo-server [pom.xml]
└── src
└── main
└── java
└── [com.wix.foo.server]
└── FooServer.java
Migration 101 Bazel>>Maven >>

17. .
├── foo-core [pom.xml]
│ └── src
│ ├── main
│ │ └── java
│ │ ├── [com.wix.foo.api]
│ │ │ └── FooBar.java
│ │ └── [com.wix.foo.utils]
│ │ ├── TimeUtils.java
│ │ └── NameUtils.java
│ └── test
│ └── java
│ └── [com.wix.foo.utils]
│ ├── TimeUtilsTest.java
│ └── NameUtilsTest.java
└── foo-server [pom.xml]
└── src
└── main
└── java
└── [com.wix.foo.server]
└── FooServer.java
Migration 101 Maven Bazel>> >>
Choose
Granularity

18. .
├── foo-core [pom.xml]
│ └── src
│ ├── main
│ │ └── java
│ │ ├── [com.wix.foo.api]
│ │ │ └── FooBar.java
│ │ └── [com.wix.foo.utils]
│ │ ├── TimeUtils.java
│ │ └── NameUtils.java
│ └── test
│ └── java
│ └── [com.wix.foo.utils]
│ ├── TimeUtilsTest.java
│ └── NameUtilsTest.java
└── foo-server [pom.xml]
└── src
└── main
└── java
└── [com.wix.foo.server]
└── FooServer.java
Migration 101 Maven Bazel>> >>
Choose
Granularity

19. .
├── foo-core [pom.xml]
│ └── src
│ ├── main
│ │ └── java
│ │ ├── [com.wix.foo.api]
│ │ │ └── FooBar.java
│ │ └── [com.wix.foo.utils]
│ │ ├── TimeUtils.java
│ │ └── NameUtils.java
│ └── test
│ └── java
│ └── [com.wix.foo.utils]
│ ├── TimeUtilsTest.java
│ └── NameUtilsTest.java
└── foo-server [pom.xml]
└── src
└── main
└── java
└── [com.wix.foo.server]
└── FooServer.java
Migration 101 Maven Bazel>> >>
Choose
Granularity

20. .
├── foo-core [pom.xml]
│ └── src
│ ├── main
│ │ └── java
│ │ ├── [com.wix.foo.api]
│ │ │ └── FooBar.java
│ │ └── [com.wix.foo.utils]
│ │ ├── TimeUtils.java
│ │ └── NameUtils.java
│ └── test
│ └── java
│ └── [com.wix.foo.utils]
│ ├── TimeUtilsTest.java
│ └── NameUtilsTest.java
└── foo-server [pom.xml]
└── src
└── main
└── java
└── [com.wix.foo.server]
└── FooServer.java
Migration 101 Maven Bazel>> >>
Choose
Granularity
1:1:1 idiom - one target per
one directory, representing a
single package

21. .
├── foo-core [pom.xml]
│ └── src
│ ├── main
│ │ └── java
│ │ ├── [com.wix.foo.api]
│ │ │ └── FooBar.java
│ │ └── [com.wix.foo.utils]
│ │ ├── TimeUtils.java
│ │ └── NameUtils.java
│ └── test
│ └── java
│ └── [com.wix.foo.utils]
│ ├── TimeUtilsTest.java
│ └── NameUtilsTest.java
└── foo-server [pom.xml]
└── src
└── main
└── java
└── [com.wix.foo.server]
└── FooServer.java
Migration 101 Maven Bazel>> >>
Choose
Granularity
>>
Convert to
Bazel rules
java_library
java_library
java_test
java_image

22. .
├── foo-core [pom.xml]
│ └── src
│ ├── main
│ │ └── java
│ │ ├── [com.wix.foo.api]
│ │ │ └── FooBar.java
│ │ └── [com.wix.foo.utils]
│ │ ├── TimeUtils.java
│ │ └── NameUtils.java
│ └── test
│ └── java
│ └── [com.wix.foo.utils]
│ ├── TimeUtilsTest.java
│ └── NameUtilsTest.java
└── foo-server [pom.xml]
└── src
└── main
└── java
└── [com.wix.foo.server]
└── FooServer.java
java_library
java_library
java_test
java_binary
Migration 101 Maven Bazel>> >>
Choose
Granularity
>>
Convert to
Bazel rules
>>
internal
deps
java_library
java_library
java_test
java_image

23. .
├── foo-core [pom.xml]
│ └── src
│ ├── main
│ │ └── java
│ │ ├── [com.wix.foo.api]
│ │ │ └── FooBar.java
│ │ └── [com.wix.foo.utils]
│ │ ├── TimeUtils.java
│ │ └── NameUtils.java
│ └── test
│ └── java
│ └── [com.wix.foo.utils]
│ ├── TimeUtilsTest.java
│ └── NameUtilsTest.java
└── foo-server [pom.xml]
└── src
└── main
└── java
└── [com.wix.foo.server]
└── FooServer.java
java_library
java_library
java_test
java_binary
@junit
@jetty
@guava20
Migration 101 Maven Bazel>> >>
Choose
Granularity
>>
Convert to
Bazel rules
>>
external
deps
java_library
java_library
java_test
java_image
@guava28

24. .
├── foo-core [pom.xml]
│ └── src
│ ├── main
│ │ └── java
│ │ ├── [com.wix.foo.api]
│ │ │ └── FooBar.java
│ │ └── [com.wix.foo.utils]
│ │ ├── TimeUtils.java
│ │ └── NameUtils.java
│ └── test
│ └── java
│ └── [com.wix.foo.utils]
│ ├── TimeUtilsTest.java
│ └── NameUtilsTest.java
└── foo-server [pom.xml]
└── src
└── main
└── java
└── [com.wix.foo.server]
└── FooServer.java
java_library
java_library
java_test
java_binary
@junit
@jetty
Migration 101 Maven Bazel>> >>
Choose
Granularity
>>
Convert to
Bazel rules
>>
external
deps
java_library
java_library
java_test
java_image
@guava28

25. . [WORKSPACE.bazel]
├── foo-core [pom.xml]
│ └── src
│ ├── main
│ │ └── java
│ │ ├── [com.wix.foo.api] [BUILD.bazel]
│ │ │ └── FooBar.java
│ │ └── [com.wix.foo.utils] [BUILD.bazel]
│ │ ├── TimeUtils.java
│ │ └── NameUtils.java
│ └── test
│ └── java
│ └── [com.wix.foo.utils] [BUILD.bazel]
│ ├── TimeUtilsTest.java
│ └── NameUtilsTest.java
└── foo-server [pom.xml]
└── src
└── main
└── java
└── [com.wix.foo.server] [BUILD.bazel]
└── FooServer.java
Migration 101 Maven Bazel>> >>
Choose
Granularity
>>
Convert to
Bazel rules
>>
Add
deps
>>
Write
Bazel files

27. Code graph >> Targets graph
Third party
definitions
>> >>
✓
wix/exodus
>>

28. Automatic tooling
is not enough
>> >>
!
bazel build
bazel test
migration
trustees

29. ✖ Missing compile/runtime deps
✖ Third party alignment
✖ Trying to read from ./src/test/resources
✖ Tests that try to download during test runtime
✖ More...
TECHNICAL DEBT == MORE WORK
exodus/troubleshooting

30. >> >> >>
?!
>>
bazel build
bazel test
Did I lose
anyting?

31. >> <<
Test reports Test reports
compare!
>> >> >>
?!
>>

32. >> >> >>
?!
>> >>
✓
>>
Exodus
Overrides

33. 1 Use a migration tool

34. #2
Measure your system
but measure right.

35. 2
3
4
5
1
2
3
4
1
2
3
1
2
7
0
1
0
1
WIX FRAMEWORK (2017)
Lines of code
2
5
0 7
Maven
modules
3
5
2 3
Bazel
Targets
2
5

36. 89% reduction
Correct Build Times in Minutes
* 8-cores machine

37. 89% reduction
Not Good
Enough
Correct Build Times in Minutes
* 8-cores machine

38. Push-to-deployable times per commit Stability rate
"Super-Artifact" CI Quality Board

39. Push-to-deployable times per commit Stability rate
"Super-Artifact" CI Quality Board

40. Push-to-deployable times per commit Stability rate
"Super-Artifact" CI Quality Board

41. 1 Use a migration tool
2 Measure the system

42. #3
Local dev experience
don’t forget your users!

43. CI & Console runs 🤝 Build Tool

44. ✖ Dependency management
✖ Code completion
✖ Refactoring
✖ MacOS cache issues
Developers and IDEs 🤝 Build Tool

45. Local-dev velocity matters!
Bad CILocal dev
CIBad Local dev

46. ✓ Bazel training
✓ IDE issues
✓ Develop solutions!
Think local development!
CIOkay Local dev
Wix Bazel Local dev team:
○Tooling alignment
○Dev Cache
○Wix IntelliJ Bazel Plugin

47. 1 Use a migration tool
2 Measure the system
3 Think local dev experience

48. #4
Introduce a new CI pipline
but keep the old one too.

49. Code push
Trigger
Artifact
Deployment
backoffice
inform
GA/Testbed deployment
CI Pipeline
Production
Server

50. Code push
Trigger
Artifact
Deployment
backoffice
Inform
GA/Testbed Deployment
CI Pipeline
Production
Server
Replace
This part

51. ✓ Remote cache and remote execution
✓ PR prechecks (no push to master)
✓ Automatic discovery of deployables
✓ Cloud based build server - no build queues
New Bazel CI Pipeline

52. Inform
Code push
Trigger
Artifact
Deployment
backoffice
GA/Testbed Deployment
CI Pipeline
Production
Server
Trigger
Bazel
Artifact

53. Bazel
Maven
✓
Two CI pipelines, side by side
Self-service switch
Bazel
Maven✓

54. 1 Use a migration tool
2 Measure the system
3 Think local dev experience
4 Run parallel CI pipelines

55. #5
Optimize for Speed
e.g., remote execution cache

56. How to make faster bazel builds?

57. ✓ Remote cache - saves ~90% of execution
✓ Remote execution - improves throughput
5-8 minutes -
fully cached build

58. ✓ Remote cache - saves ~90% of execution
✓ Remote execution - improves throughput
✓ Repository cache - Saves downloading http binaries
✓ Docker cache - Saves downloading docker images

59. ✓ Remote cache - saves ~90% of execution
✓ Remote execution - improves throughput
✓ Repository cache - Saves downloading http binaries
✓ Docker cache - Saves downloading docker images
✓ Bazel flags - builds without the bytes, multithread digest...

60. When we started
we compiled Bazel from sources
Bazel keeps evolving
Each release
brought a lot of value, but also
required us to adjust
Tap to important issues, influence the priorities
Become a Bazel activist!

61. 1 Use a migration tool
2 Measure the system
3 Think local dev experience
4 Run parallel CI pipelines
5 Optimize for speed

62. #6
Migrating poly repo is hard
but possible!

63. Bazel works well with mono repo
All code in one place
Single bazel execution
✓ Single CI job

64. It's too
much for
me!
I'm not
ready yet
Mono-repo
on Git
You
change
too much!
Mono RepoMercurial
Poly-Repo
on Git
✓

65. from 1000+ repos
...to just about 50.

66. Goal: Virtual mono repository
java_library(
name = "core",
srcs = [":sources"],
deps = [
"//src/main/java/com/wix/local",
"@framework//src/main/java/com/wix/external",
],
)
git_repository(
name = "framework",
url = "git@github.com:wix/framework.git",
commit = "<auto-updated>",
)

67. Virtual mono-repo:
Virtual Mono-Repo Service holds all
repos of wix with their latest HEAD
version
Bazel wrapper generates a local bzl file
with `git_repository` definition of all
repositories
Each WORKSPACE file loads the
definition from the generated bzl file.
framework(031d53ce983d26a861d053c760190de669aa458d)
users(1202d41c7dc0739deddde3176c7de34ed9b978b7)
events(ab6d1d07477e3742fbc9f9c1e81e0ec402650b1b)
external_repos.bzl
git_repository(
name = "framework",
Commit = "031d53ce983d2
)
...
WORKSPACE.bazel
1
2
3

68. ~50 migrations

69. External Dependencies
Guava:25
Commons:7
Junit:3.2
wix-framework:1.0-SNAPSHOT
wix-utils:1.0.SNAPSHOT
...

70. External Dependencies
Guava:25
Commons:7
Junit:3.2
wix-framework:1.0-SNAPSHOT
wix-utils:1.0.SNAPSHOT
... java_library(
name = "core",
srcs = [":sources"],
deps = [
"//src/main/java/com/wix/local",
"@com_google_guava",
"@com_wix_framework",
],
)

71. Migration in "Isolation Mode"
✓ Faster move to Bazel CI
✓ Faster feedback
✓ Faster deployables
✖ Keep Maven builds green
✖ Reproducibility hit
✖ Stale SNAPSHOT locally

72. Isolation Mode
Social Mode
java_library(
name = "core",
srcs = [":sources"],
deps = [
"//src/main/java/com/wix/local",
"@com_google_guava",
"@com_wix_framework",
],
)
scala_library(
name = "core",
srcs = [":sources"],
deps = [
"//src/main/java/com/wix/local",
"@com_google_guava",
"@framework//src/java/com/wixpress/lib",
],
)

73. ✖ Complex
✖ Non-revertible
java_library(
name = "core",
srcs = [":sources"],
deps = [
"//src/main/java/com/wix/local",
"@com_google_guava",
"@com_wix_framework",
],
)
java_library(
name = "core",
srcs = [":sources"],
deps = [
"//src/main/java/com/wix/local",
"@com_google_guava",
"@framework//src/java/com/wixpress/lib",
],
)

74. java_library(
name = "core",
srcs = [":sources"],
deps = [
"//src/main/java/com/wix/local",
"@com_google_guava",
"@com_wix_framework",
],
)
external_binary(
name = "com_wix_framework",
artifact = "com.wix.framework:1.0-SNAPSHOT"
)
external_source(
name = "com_wix_framework",
exports = [
"@framework//src/java/com/wixpress/lib"
]
)
workspace_rule

75. external_binary external_source
@com_wix_framework
Current repo in
social mode?
Requested
module in
social mode?
no
no yes
yes

76. Social mode
✓ Global Third party alignment
✓ Missing runtime deps
✓ Testonly targets
✓ ...

77. 1 Use a migration tool
2 Measure the system
3 Think local dev experience
4 Run parallel CI pipelines
5 Optimize for speed
6 Migrate poly-repo

78. Where are we now?

79. Safe haven
CLOSED
✓ No more SNAPSHOT consumption
✓ All deployable from bazel
✓ All developers use bazel only

80. 1 Use a migration tool
2 Measure the system
3 Think local dev experience
4 Run parallel CI pipelines
5 Optimize for speed
6 Migrate poly-repo

81. Thank You
ors@wix.com
6 LEARNINGS FROM MIGRATING
OUR BUILD SYSTEM TO BAZEL

82. Q&A
ors@wix.com