Downloaded 17 times
Uploaded byAysylu Greenberg
Building A Distributed Build System at Google Scale (StrangeLoop 2016)
The document discusses the challenges and solutions involved in building a distributed build system at Google scale, addressing key elements such as project dependencies, continuous integration, and artifact management. It highlights the scale of operations, including the number of developers, codebase size, and build processes, while explaining the importance of working within a single repository for efficient collaboration and simplified dependency management. Additionally, the document outlines the architecture of the 'buildrabbit' system, emphasizing its resilience and ability to handle migration with zero downtime.
More Related Content
Similar to Building A Distributed Build System at Google Scale (StrangeLoop 2016)
Building A Distributed Build System at Google Scale (StrangeLoop 2016)
- 1. Building a Distributed BuildSystem at Scale Aysylu Greenberg September 17, 2016
- 2.
- 3. Building Distributed Build System atGoogle Scale
- 4. Building Distributed Build System atGoogle Scale
- 5. What does abuild system do? Photo by smemon / CC BY
- 6. Build Scenario Project withdependencies Makefile main.o : main.c defs.h cc -c main.c main.c : curl –O http://remote/main.c
- 7. Build Scenario Project withdependencies Find dependencies Makefile main.o : main.c defs.h cc -c main.c main.c: curl –O http://remote/main.c
- 8. Build Scenario Project withdependencies Find dependencies Build project with the dependencies Download build artifacts
- 9. Test Scenario Project withdependencies Find dependencies Build project with the dependencies Download build artifacts Run the test Get the results of the test
- 10. Building Distributed Build System atGoogle Scale
- 11. Building Distributed Build System atGoogle Scale
- 12. Scale • Engineers: >30,000developers in 40+ offices
- 13. Scale • Engineers: >30,000developers in 40+ offices • Commits: 15K by humans + 30K by robots/day
- 14. Scale • Engineers: >30,000developers in 40+ offices • Commits: 15K by humans + 30K by robots/day • Source code: 2 billion LOC
- 15. Scale • Engineers: >30,000developers in 40+ offices • Commits: 15K by humans + 30K by robots/day • Source code: 2 billion LOC • Builds and tests: 5M per day through BuildRabbit
- 16. Scale • Engineers: >30,000developers in 40+ offices • Commits: 15K by humans + 30K by robots/day • Source code: 2 billion LOC • Builds and tests: 5M per day through BuildRabbit • Petabytes of output artifacts
- 17. Scale • Engineers: >30,000developers in 40+ offices • Commits: 15K by humans + 30K by robots/day • Source code: 2 billion LOC • Builds and tests: 5M per day through BuildRabbit • Petabytes of output artifacts • 1 repository
- 18. Working in OneRepository Photo by flyingblogspot / CC BY SA
- 19. Repository of Artifactsvs Building from Source Photo by flyingblogspot / CC BY SA
- 20. Working in OneRepository • Linear revision history
- 21. Working in OneRepository • Linear revision history • Extensive code sharing & reuse • Everything can be cross-referenced • Large-scale refactoring for codebase modernization • Easier collaboration across teams
- 22. Working in OneRepository • Linear revision history • Extensive code sharing & reuse • Simplified dependency management • No confusion about authoritative version of the file • No forking of shared libraries • Components for library releases = Git subtree or Git subcomponents to separate release from WIP versions
- 23. Working in OneRepository • Linear revision history • Extensive code sharing & reuse • Simplified dependency management • Predictable, repeatable builds from source
- 24. Working in OneRepository • Linear revision history • Extensive code sharing & reuse • Simplified dependency management • Predictable, repeatable builds from source • Optimizations to avoid compiling same artifacts
- 25. Working in OneRepository • Linear revision history • Extensive code sharing & reuse • Simplified dependency management • Predictable, repeatable builds from source • Optimizations to avoid compiling same artifacts • Decouple each team's processes as much as possible
- 26. Building Distributed Build System atGoogle Scale
- 27. Building Distributed Build System atGoogle Scale
- 28. Build System Story Photoby imanka / CC BY
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36. BuildRabbit: Distributed BuildSystem IN THE CLOUD
- 37.
- 38.
- 39. Building Distributed Build System atGoogle Scale
- 40. Building Distributed Build System atGoogle Scale
- 41. From Push toPull: Client / Server Client for User BuildRabbit Scheduler BuildRabbit Worker
- 42. From Push toPull: Build Service User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info event RPC stream
- 43. From Push toPull: Build Service User Persistent Queue BuildRabbit Worker Build Artifacts Build Progress Info event RPC stream
- 44.
- 45.
- 46.
- 47.
- 48. Migration with ZeroDowntime Photo by moonjazz / CC BY SA
- 49.
- 50.
- 51. FOCUS ON MIXEDMODE MIGRATE BACKENDS FIRST
- 52. TARGET LAUNCH-FRIENDLY CLIENTS FOCUSON MIXED MODE MIGRATE BACKENDS FIRST
- 53.
- 54. DECOUPLE LAUNCH OFSERVICES TARGET LAUNCH-FRIENDLY CLIENTS FOCUS ON MIXED MODE MIGRATE BACKENDS FIRST
- 55.
- 56. MAXIMUM VISIBILITY INTOSYSTEM STATE DECOUPLE LAUNCH OF SERVICES TARGET LAUNCH-FRIENDLY CLIENTS FOCUS ON MIXED MODE MIGRATE BACKENDS FIRST
- 57. PRACTICE THE LAUNCH MAXIMUMVISIBILITY INTO SYSTEM STATE DECOUPLE LAUNCH OF SERVICES TARGET LAUNCH-FRIENDLY CLIENTS FOCUS ON MIXED MODE MIGRATE BACKENDS FIRST
- 58. PRACTICE THE LAUNCH,A LOT MAXIMUM VISIBILITY INTO SYSTEM STATE DECOUPLE LAUNCH OF SERVICES TARGET LAUNCH-FRIENDLY CLIENTS FOCUS ON MIXED MODE MIGRATE BACKENDS FIRST
- 59. HAVE A SOLIDROLLBACK PLAN PRACTICE THE LAUNCH, A LOT MAXIMUM VISIBILITY INTO SYSTEM STATE DECOUPLE LAUNCH OF SERVICES TARGET LAUNCH-FRIENDLY CLIENTS FOCUS ON MIXED MODE
- 60. Distributed System DesignPatterns Photo by oslointhesummertime / CC BY
- 61. Distributed vs CentralizedControl Plane Photo by oslointhesummertime / CC BY
- 62. Distributed Design Patterns:Distributed Control Design implementation is scaled from 1 machine Bigger overhead during evolution
- 63. Distributed Design Patterns:Centralized Control Distributed design from the start Bigger overhead during design phase
- 64. Building a Distributed BuildSystem at Scale Aysylu Greenberg @aysylu22
Editor's Notes
- #9 Package and deploy if necessary
- #15 1/2 of the codebase changes daily
- #19 Rachel Potvin’s talk @Scale
- #21 1 source of truth, Atomic changes
- #22 Increased visibility, flexible team boundaries
- #23 No painful cross-repository merging of copied code
- #42 utilization + routing, client has to have retry logic, 1 connection to get progress info and outputs, if connection broken, retry
- #43 API built to last, using old facade at first; Handle retries for the user; Smarter scheduling + incrementality + utilization
- #44 API built to last, using old facade at first; Handle retries for the user; Smarter scheduling + incrementality + utilization
- #47 2nd: runs the same version; CENTRALIZED control over build system components