For each PR bitbucket generates {pullRequestId}/merge branch, which simulates state of target branch after merge. To ensure a green trunk we have to run all related builds on this branch. After merge of one PR results of all other open PRs will be rollbacked and you need to restart all related builds because of updated upstream. Accordingly, we get a large number of rebuilds, which is a big problem in a monorepo. We solved this problem by developing a bb plugin based on independency graph.

4. Maria Guseva
The solution of merge hell in
monorepo

5. Yandex
5

6. Code storage
6
ARCADIA
Frontend project 1
Frontend project N
…
Mobile project 1
Mobile project N
…
Backend project 1
Backend project N
…

7. Yandex monorepo statistics*
7
› 1.8M files
› 6M commits
› 2TB of data
› 3k commits in trunk per work day
* without mobile projects and some others

8. Popular repos vs Yandex monorepo
8

9. Repository
9https://www.fourtheorem.com/blog/monorepo

10. Monorepo. Pros
10
› Common infrastructure
› Common code
› Atomic commits to multiple projects
› Code reuse
› Easier global refactoring
› No exotic languages and libraries

11. Monorepo. Cons
11
› Common infrastructure development
› Resources on adding and updating libraries
› Constant work on a code “health”

12. Trunk-based development
12https://trunkbaseddevelopment.com/

13. 13
Green trunk – only tested updates
can be merged into trunk

14. Merge branch
14
M M1 M2

15. Bitbucket merge hell
15
For each PR bitbucket generates {pullRequestId}/merge branch
PR_N
PR_1
PR_2
…
PR_1 Merged
PR_N
PR_2
…
Mail
MailMaps
Maps
N - 1
N + (N -1) + (N - 2) + … + 1 = о(N2)

16. Yandex mobile monorepo statistics
16
› ~ 30 different services
› ~ 300 open PRs
› ~ 50 merged PRs per work day

17. TeamCity builds duration

18. 18

19. Possible solutions
19
Run builds only on user branch changes

20. Possible solutions
20
Merge-pin custom branch
N = 2

21. 21
Build manager plugin – CI tools integration plugin
based on “independency” graph
The solution

22. Build manager plugin
22
Bitbucket
Pull request
event listener
Merge check
Conveyor
Build result
cache

23. Сonveyor
23
Сonveyor
Config list
Changes list
Filtration
Build result
cache
TeamCity
Commit status
publisher

24. Commit status publisher
24
Sends information about builds
/rest/build-status/1.0/commits/{commitId}
{
“state”:”SUCCESSFUL”,
“key”:”Infra_Test_Changing_Build”,
“name”:”TestChanging::Build”,
“url”:”https://teamcity.ru/viewLog.html?buildId=123”,
“description”:”Tests passed: 5”
}

25. Build manager plugin root config
25
- maps/android/maps-sdk/ci.yaml
- infra/bitbucket-plugins/ci.yaml
…
- mail/android/mail-app/ci.yaml

26. Build manager plugin project config
26
name: infra-bitbucket-plugins
tcBuilds:
- name: TestChanging::Build
id: Infra_Test_Changing_Build
optional: true
includedPaths:
- path: infra/bitbucket-plugins/*
includedTargetBranches:
- branch: trunk
excludedPaths:
- path: "*.md"
excludedTargetBranches:
- branch: feature/*
› Possibility for staring
builds from different CI
tools
› Optional builds doesn’t
prevent merge

27. Example
27
name: mail
tcBuilds:
- name: Mail::Build
id: Mail_Build
includedPaths:
- path: mail/*
includedTargetBranches:
- branch: trunk
excludedPaths:
- path: "*.md"
name: maps
tcBuilds:
- name: Maps::Build
id: Maps_Build
includedPaths:
- path: maps/*
trunk
mail/Main.java
common/Main.java
mail/readme.md
maps/readme.md
Mail::Build
Maps::Build
Mail::Build
mail/test/MainTest.java

28. Features
28
› Skip checks
› Rerun build

29. Performance
29

30. Performance
30
3000 * 20 (mins) / 60 (mins in hour) / 24 (hours in day) = 40+ days

31. Merge optimization
31
PR1
PR2
PR3
PR4
PR5
Project1 Project2 Project3
Merged
Merged
Merged
Recalculate {prId}/merge
Run builds
Merged
Recalculate {prId}/merge
Compare diffs

32. Summary
32
› Green trunk in monorepo is possible
› Invest in infrastructure – make developers happy
› Don’t be scary to develop new tools

33. Thanks
Maria Guseva
maria-guseva@yandex-team.ru
@gusevamaria