SlideShare a Scribd company logo
Google Confidential and Proprietary
Continuous Integration at
Google Scale
By John Micco
Developer Infrastructure
Google Confidential and Proprietary
● >30,000 developers in 40+ offices
● 13,000+ projects under active development
● 30k submissions per day (1 every 3 seconds)
● Single monolithic code tree with mixed language code
● Development on one branch - submissions at head
● All builds from source
● 30+ sustained code changes per minute with 90+ peaks
● 50% of code changes monthly
● 150+ million test cases / day, > 150 years of test / day
● Supports continuous deployment for all Google teams!
Speed and Scale
Google Confidential and Proprietary
Overview
1. Continuous Integration Goals
2. Continuous Integration at Google
3. Future of testing
Google Confidential and Proprietary
● Provide real-time information to build monitors
○ Identify failures fast
○ Identify culprit Changes
○ Handle flaky tests
● Provide frequent green builds for cutting releases
○ Identify recent green builds
○ Show results of all testing together
○ Allow release tooling to choose a green build
○ Handle flaky tests
"green build" = all tests contained in that build are passing at a given
Change.
Continuous Integration
Google Confidential and Proprietary
● Develop Safely
○Sync to last green changelist
○Identify whether change will break the build before submit
○Submit with confidence
○Handle flaky tests
Continuous Integration (cont)
Google Confidential and Proprietary
Standard Continuous Build System
● Triggers builds in continuous cycle
● Cycle time = longest build + test cycle
● Tests many changes together
● Which change broke the build?
Google Confidential and Proprietary
● Triggers tests on every change
● Uses fine-grained dependencies
● Change 2 broke test 1
Google Continuous Build System
Google Confidential and Proprietary
Continuous Integration Display
Google Confidential and Proprietary
● Identifies failures sooner
● Identifies culprit change precisely
○ Avoids divide-and-conquer and tribal knowledge
● Lower compute costs using fine grained dependencies
● Keeps the build green by reducing time to fix breaks
● Accepted enthusiastically by product teams
● Enables teams to ship with fast iteration times
○ Supports submit-to-production times of less than 36
hours for some projects
Benefits
Google Confidential and Proprietary
● Requires enormous investment in compute resources (it
helps to be at Google) grows in proportion to:
○ Submission rate
○ Average build + test time
○ Variants (debug, opt, valgrind, etc.)
○ Increasing dependencies on core libraries
○ Branches
● Requires updating dependencies on each change
○ Takes time to update - delays start of testing
Costs
Google Confidential and Proprietary
● Makes testing available before submit
● Uses fine-grained dependencies
○ Recalculate any dependency changes
● Uses same pool of compute resources at high priority
● Avoids breaking the build
● Captures contents of a change and tests in isolation
○ Tests against head
○ Identifies problems with missing files
● Integrates with
○ submission tool - submit iff testing is green
○ Code Review Tool - results are posted to the review thread
Developing Safely - presubmit
Google Confidential and Proprietary
Example Presubmit Display
Google Confidential and Proprietary
Practical Matters - Test Growth
● Sources of growth in test execution time
○ More developers = increased submission rate
○ More tests
○ Longer running tests
○ Tests consuming more resources (threading)
● Examine the growth trends
○ Predict compute needs
○ Look for any build system features required
Build / Test Compute Resources
Jan
2011
Jan
2012
Jan
2013
Jul 2012
Jul 2011
Google Confidential and Proprietary
● Problems
○ Quadratic execution time growth w/ 2 factors
■ Submit rate - grows linearly
■ Test pool size - grows linearly
○ Ultimately cannot run every affected test @ every change
○ Low latency results still top requirement
● Solution: Just in time scheduling (JIT)
Test Growth
Continuous Integration:
● Run every test affected at every changelist.
In Production:
● Build and run tests concurrently on Google’s distributed
build and test backend.
JIT
as often as possible
JIT Scheduling
Schedule tests to run only when
system has capacity.
Produce project-wide results at
periodic changelists.
Milestone Property
A changelist C is a milestone iff ...
● All tests affected at C are run
● All tests affected since the previous
milestone are run.
● All these tests are run at their greatest
affecting changelist <= C.
Milestone Property
A changelist C is a milestone iff ...
● All tests affected at C are run
● All tests affected since the previous
milestone are run.
● All these tests are run at their greatest
affecting changelist <= C.
Exactly the work
necessary to deliver a
conclusive project status
Confidential + Proprietary
Change Lists
AffectedTestTargetset
Cut milestone
at this CL
21
Confidential + Proprietary
Change Lists
AffectedTestTargetset
22
Confidential + Proprietary
Change Lists
AffectedTestTargetset
23
Confidential + Proprietary
Change Lists
AffectedTestTargetset
24
Confidential + Proprietary
Change Lists
AffectedTestTargetset
25
Google Confidential and Proprietary
JIT scheduling results
● JIT scheduler changed compute growth from quadratic to linear!
● Without it, compute demand would have already consumed all of
Google's capacity
○ We literally either would have had to stop testing this way or stop
running user searches
● Enabled Google to keep providing fast feedback to developers with
reasonable compute costs
Google Confidential and Proprietary
Cuprit Finding - Transition to Fail
A
Time
Targets
Changelists
1 2 3
Passed
Affected, but not run (yet)
Milestone
Non-milestone
4
Failed
Schedule these
Google Confidential and Proprietary
Cuprit Finding - Transition to Fail
A
Time
Targets
Changelists
1 2 3
Passed
Affected, but not run (yet)
Milestone
Non-milestone
4
Failed
A: Change 3 broke test A.
Google Confidential and Proprietary
Future Direction
● Atif Memon is working with us (on sabbatical) to analyze our data
● He is finding that the dependency distance between a target and
the triggering source file is highly correlated with the probability of
introducing a non-flaky failure. We are working with him to publish
his findings.
Google Confidential and Proprietary
Q & A
For more information:
● http://google-engtools.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html
● http://www.youtube.com/watch?v=b52aXZ2yi08
● http://www.infoq.com/presentations/Development-at-Google
● http://google-engtools.blogspot.com/
● http://misko.hevery.com/2008/11/11/clean-code-talks-dependency-injection/
● https://www.youtube.com/watch?v=KH2_sB1A6lA&feature=youtube_gdata_player
Q & A

More Related Content

What's hot

The Rust Programming Language
The Rust Programming LanguageThe Rust Programming Language
The Rust Programming Language
Mario Alexandro Santini
 
Configuration as Code in Bamboo
Configuration as Code in BambooConfiguration as Code in Bamboo
Configuration as Code in Bamboo
Atlassian
 
1. Overview of Distributed Systems
1. Overview of Distributed Systems1. Overview of Distributed Systems
1. Overview of Distributed Systems
Daminda Herath
 
Significance of AI in Testing
Significance of AI in TestingSignificance of AI in Testing
Significance of AI in Testing
Agile Testing Alliance
 
ISTQB - Software development life cycle
ISTQB - Software development life cycleISTQB - Software development life cycle
ISTQB - Software development life cycle
HoangThiHien1
 
소프트웨어 아키텍처 평가(Atam)
소프트웨어 아키텍처 평가(Atam)소프트웨어 아키텍처 평가(Atam)
소프트웨어 아키텍처 평가(Atam)
영기 김
 
Software Engineering - chp4- design patterns
Software Engineering - chp4- design patternsSoftware Engineering - chp4- design patterns
Software Engineering - chp4- design patterns
Lilia Sfaxi
 
Stlc ppt
Stlc pptStlc ppt
Stlc ppt
Bhavik Modi
 
Software testing ppt
Software testing pptSoftware testing ppt
Software testing ppt
Heritage Institute Of Tech,India
 
Black & White Box testing
Black & White Box testingBlack & White Box testing
Microservices pattern language (microxchg microxchg2016)
Microservices pattern language (microxchg microxchg2016)Microservices pattern language (microxchg microxchg2016)
Microservices pattern language (microxchg microxchg2016)
Chris Richardson
 
Software Testing Fundamentals
Software Testing FundamentalsSoftware Testing Fundamentals
Software Testing Fundamentals
Chankey Pathak
 
Architecture business cycle
Architecture business cycleArchitecture business cycle
Architecture business cycle
Himanshu
 
Slides: Why You Need End-to-End Data Quality to Build Trust in Kafka
Slides: Why You Need End-to-End Data Quality to Build Trust in KafkaSlides: Why You Need End-to-End Data Quality to Build Trust in Kafka
Slides: Why You Need End-to-End Data Quality to Build Trust in Kafka
DATAVERSITY
 
Testing Metrics
Testing MetricsTesting Metrics
Testing Metrics
PM Venkatesha Babu
 
Real Life Clean Architecture
Real Life Clean ArchitectureReal Life Clean Architecture
Real Life Clean Architecture
Mattia Battiston
 
architectural design
 architectural design architectural design
architectural design
Preeti Mishra
 
AI & Machine Learning Pipelines with Knative
AI & Machine Learning Pipelines with KnativeAI & Machine Learning Pipelines with Knative
AI & Machine Learning Pipelines with Knative
Animesh Singh
 
Serverless Summit - Quiz
Serverless Summit - QuizServerless Summit - Quiz
Serverless Summit - Quiz
CodeOps Technologies LLP
 
Non functional performance requirements v2.2
Non functional performance requirements v2.2Non functional performance requirements v2.2
Non functional performance requirements v2.2
Ian McDonald
 

What's hot (20)

The Rust Programming Language
The Rust Programming LanguageThe Rust Programming Language
The Rust Programming Language
 
Configuration as Code in Bamboo
Configuration as Code in BambooConfiguration as Code in Bamboo
Configuration as Code in Bamboo
 
1. Overview of Distributed Systems
1. Overview of Distributed Systems1. Overview of Distributed Systems
1. Overview of Distributed Systems
 
Significance of AI in Testing
Significance of AI in TestingSignificance of AI in Testing
Significance of AI in Testing
 
ISTQB - Software development life cycle
ISTQB - Software development life cycleISTQB - Software development life cycle
ISTQB - Software development life cycle
 
소프트웨어 아키텍처 평가(Atam)
소프트웨어 아키텍처 평가(Atam)소프트웨어 아키텍처 평가(Atam)
소프트웨어 아키텍처 평가(Atam)
 
Software Engineering - chp4- design patterns
Software Engineering - chp4- design patternsSoftware Engineering - chp4- design patterns
Software Engineering - chp4- design patterns
 
Stlc ppt
Stlc pptStlc ppt
Stlc ppt
 
Software testing ppt
Software testing pptSoftware testing ppt
Software testing ppt
 
Black & White Box testing
Black & White Box testingBlack & White Box testing
Black & White Box testing
 
Microservices pattern language (microxchg microxchg2016)
Microservices pattern language (microxchg microxchg2016)Microservices pattern language (microxchg microxchg2016)
Microservices pattern language (microxchg microxchg2016)
 
Software Testing Fundamentals
Software Testing FundamentalsSoftware Testing Fundamentals
Software Testing Fundamentals
 
Architecture business cycle
Architecture business cycleArchitecture business cycle
Architecture business cycle
 
Slides: Why You Need End-to-End Data Quality to Build Trust in Kafka
Slides: Why You Need End-to-End Data Quality to Build Trust in KafkaSlides: Why You Need End-to-End Data Quality to Build Trust in Kafka
Slides: Why You Need End-to-End Data Quality to Build Trust in Kafka
 
Testing Metrics
Testing MetricsTesting Metrics
Testing Metrics
 
Real Life Clean Architecture
Real Life Clean ArchitectureReal Life Clean Architecture
Real Life Clean Architecture
 
architectural design
 architectural design architectural design
architectural design
 
AI & Machine Learning Pipelines with Knative
AI & Machine Learning Pipelines with KnativeAI & Machine Learning Pipelines with Knative
AI & Machine Learning Pipelines with Knative
 
Serverless Summit - Quiz
Serverless Summit - QuizServerless Summit - Quiz
Serverless Summit - Quiz
 
Non functional performance requirements v2.2
Non functional performance requirements v2.2Non functional performance requirements v2.2
Non functional performance requirements v2.2
 

Viewers also liked

La Hora del Código (2016)
La Hora del Código (2016)La Hora del Código (2016)
La Hora del Código (2016)
Alberto Pacheco
 
Continuous integration by Rémy Virin
Continuous integration by Rémy VirinContinuous integration by Rémy Virin
Continuous integration by Rémy Virin
CocoaHeads France
 
Continuous integration at scale
Continuous integration at scaleContinuous integration at scale
Continuous integration at scale
Vivek Singh
 
GdG DevFestMed 2016 - 06/11/2016
GdG DevFestMed 2016 - 06/11/2016GdG DevFestMed 2016 - 06/11/2016
GdG DevFestMed 2016 - 06/11/2016
Gaetano Paternò
 
Inverting The Testing Pyramid
Inverting The Testing PyramidInverting The Testing Pyramid
Inverting The Testing Pyramid
Naresh Jain
 
Achieving DevOps using Open Source Tools in the Enterprise
Achieving DevOps using Open Source Tools in the EnterpriseAchieving DevOps using Open Source Tools in the Enterprise
Achieving DevOps using Open Source Tools in the Enterprise
CollabNet
 
Continuous Delivery Sounds Great but it Won't Work Here
Continuous Delivery Sounds Great but it Won't Work HereContinuous Delivery Sounds Great but it Won't Work Here
Continuous Delivery Sounds Great but it Won't Work Here
Jez Humble
 
How to Implement Domain Driven Design in Real Life SDLC
How to Implement Domain Driven Design  in Real Life SDLCHow to Implement Domain Driven Design  in Real Life SDLC
How to Implement Domain Driven Design in Real Life SDLC
Abdul Karim
 
How Google Works
How Google WorksHow Google Works
How Google Works
Eric Schmidt
 
How to Make Awesome SlideShares: Tips & Tricks
How to Make Awesome SlideShares: Tips & TricksHow to Make Awesome SlideShares: Tips & Tricks
How to Make Awesome SlideShares: Tips & Tricks
SlideShare
 
Getting Started With SlideShare
Getting Started With SlideShareGetting Started With SlideShare
Getting Started With SlideShare
SlideShare
 

Viewers also liked (11)

La Hora del Código (2016)
La Hora del Código (2016)La Hora del Código (2016)
La Hora del Código (2016)
 
Continuous integration by Rémy Virin
Continuous integration by Rémy VirinContinuous integration by Rémy Virin
Continuous integration by Rémy Virin
 
Continuous integration at scale
Continuous integration at scaleContinuous integration at scale
Continuous integration at scale
 
GdG DevFestMed 2016 - 06/11/2016
GdG DevFestMed 2016 - 06/11/2016GdG DevFestMed 2016 - 06/11/2016
GdG DevFestMed 2016 - 06/11/2016
 
Inverting The Testing Pyramid
Inverting The Testing PyramidInverting The Testing Pyramid
Inverting The Testing Pyramid
 
Achieving DevOps using Open Source Tools in the Enterprise
Achieving DevOps using Open Source Tools in the EnterpriseAchieving DevOps using Open Source Tools in the Enterprise
Achieving DevOps using Open Source Tools in the Enterprise
 
Continuous Delivery Sounds Great but it Won't Work Here
Continuous Delivery Sounds Great but it Won't Work HereContinuous Delivery Sounds Great but it Won't Work Here
Continuous Delivery Sounds Great but it Won't Work Here
 
How to Implement Domain Driven Design in Real Life SDLC
How to Implement Domain Driven Design  in Real Life SDLCHow to Implement Domain Driven Design  in Real Life SDLC
How to Implement Domain Driven Design in Real Life SDLC
 
How Google Works
How Google WorksHow Google Works
How Google Works
 
How to Make Awesome SlideShares: Tips & Tricks
How to Make Awesome SlideShares: Tips & TricksHow to Make Awesome SlideShares: Tips & Tricks
How to Make Awesome SlideShares: Tips & Tricks
 
Getting Started With SlideShare
Getting Started With SlideShareGetting Started With SlideShare
Getting Started With SlideShare
 

Similar to 2016 04-25 continuous integration at google scale

Continuous Infrastructure First
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure First
Kris Buytaert
 
Testing in a continuous delivery environment
Testing in a continuous delivery environmentTesting in a continuous delivery environment
Testing in a continuous delivery environment
Stefan Verhoeff
 
Continues delivery - Introduction
Continues delivery - IntroductionContinues delivery - Introduction
Continues delivery - Introduction
Erez Attar
 
Release & Iterate Faster: Stop Manual Testing
Release & Iterate Faster: Stop Manual TestingRelease & Iterate Faster: Stop Manual Testing
Release & Iterate Faster: Stop Manual Testing
Drew Hannay
 
Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...
Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...
Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...
Weaveworks
 
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
Vietnam Open Infrastructure User Group
 
Gerrit Code Review
Gerrit Code ReviewGerrit Code Review
Gerrit Code Review
Johannes Barop
 
GitOps , done Right
GitOps , done RightGitOps , done Right
GitOps , done Right
Kris Buytaert
 
Expedia 3x3 presentation
Expedia 3x3 presentationExpedia 3x3 presentation
Expedia 3x3 presentation
Drew Hannay
 
Building Sustainable Software: An Introduction to Software Engineering
Building Sustainable Software: An Introduction to Software EngineeringBuilding Sustainable Software: An Introduction to Software Engineering
Building Sustainable Software: An Introduction to Software Engineering
Muhammad Shehata
 
Release Engineering
Release EngineeringRelease Engineering
Release Engineering
GDG Odessa
 
Cloud Native CI/CD with Spring Cloud Pipelines
Cloud Native CI/CD with Spring Cloud PipelinesCloud Native CI/CD with Spring Cloud Pipelines
Cloud Native CI/CD with Spring Cloud Pipelines
Lars Rosenquist
 
Cloud Native CI/CD with Spring Cloud Pipelines
Cloud Native CI/CD with Spring Cloud PipelinesCloud Native CI/CD with Spring Cloud Pipelines
Cloud Native CI/CD with Spring Cloud Pipelines
Lars Rosenquist
 
Agile lifecycle handbook by bhawani nandan prasad
Agile lifecycle handbook by bhawani nandan prasadAgile lifecycle handbook by bhawani nandan prasad
Agile lifecycle handbook by bhawani nandan prasad
Bhawani N Prasad
 
Software management for tech startups
Software management for tech startupsSoftware management for tech startups
Software management for tech startups
Mahmoud Atef Abdelsamie
 
Agile Development Lifecycle and Best Practices with Denodo
Agile Development Lifecycle and Best Practices with DenodoAgile Development Lifecycle and Best Practices with Denodo
Agile Development Lifecycle and Best Practices with Denodo
Denodo
 
Agile process with a fixed cost
Agile process with a fixed costAgile process with a fixed cost
Agile process with a fixed cost
Ralph Johnson
 
Working Agile with Scrum and TFS 2013
Working Agile with Scrum and TFS 2013Working Agile with Scrum and TFS 2013
Working Agile with Scrum and TFS 2013
Moataz Nabil
 
Bots on guard of sdlc
Bots on guard of sdlcBots on guard of sdlc
Bots on guard of sdlc
Alexey Tokar
 
3 Keys to Performance Testing at the Speed of Agile
3 Keys to Performance Testing at the Speed of Agile3 Keys to Performance Testing at the Speed of Agile
3 Keys to Performance Testing at the Speed of Agile
Neotys
 

Similar to 2016 04-25 continuous integration at google scale (20)

Continuous Infrastructure First
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure First
 
Testing in a continuous delivery environment
Testing in a continuous delivery environmentTesting in a continuous delivery environment
Testing in a continuous delivery environment
 
Continues delivery - Introduction
Continues delivery - IntroductionContinues delivery - Introduction
Continues delivery - Introduction
 
Release & Iterate Faster: Stop Manual Testing
Release & Iterate Faster: Stop Manual TestingRelease & Iterate Faster: Stop Manual Testing
Release & Iterate Faster: Stop Manual Testing
 
Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...
Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...
Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...
 
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
Room 2 - 4 - Juncheng Anthony Lin - Redhat - A Practical Approach to Traditio...
 
Gerrit Code Review
Gerrit Code ReviewGerrit Code Review
Gerrit Code Review
 
GitOps , done Right
GitOps , done RightGitOps , done Right
GitOps , done Right
 
Expedia 3x3 presentation
Expedia 3x3 presentationExpedia 3x3 presentation
Expedia 3x3 presentation
 
Building Sustainable Software: An Introduction to Software Engineering
Building Sustainable Software: An Introduction to Software EngineeringBuilding Sustainable Software: An Introduction to Software Engineering
Building Sustainable Software: An Introduction to Software Engineering
 
Release Engineering
Release EngineeringRelease Engineering
Release Engineering
 
Cloud Native CI/CD with Spring Cloud Pipelines
Cloud Native CI/CD with Spring Cloud PipelinesCloud Native CI/CD with Spring Cloud Pipelines
Cloud Native CI/CD with Spring Cloud Pipelines
 
Cloud Native CI/CD with Spring Cloud Pipelines
Cloud Native CI/CD with Spring Cloud PipelinesCloud Native CI/CD with Spring Cloud Pipelines
Cloud Native CI/CD with Spring Cloud Pipelines
 
Agile lifecycle handbook by bhawani nandan prasad
Agile lifecycle handbook by bhawani nandan prasadAgile lifecycle handbook by bhawani nandan prasad
Agile lifecycle handbook by bhawani nandan prasad
 
Software management for tech startups
Software management for tech startupsSoftware management for tech startups
Software management for tech startups
 
Agile Development Lifecycle and Best Practices with Denodo
Agile Development Lifecycle and Best Practices with DenodoAgile Development Lifecycle and Best Practices with Denodo
Agile Development Lifecycle and Best Practices with Denodo
 
Agile process with a fixed cost
Agile process with a fixed costAgile process with a fixed cost
Agile process with a fixed cost
 
Working Agile with Scrum and TFS 2013
Working Agile with Scrum and TFS 2013Working Agile with Scrum and TFS 2013
Working Agile with Scrum and TFS 2013
 
Bots on guard of sdlc
Bots on guard of sdlcBots on guard of sdlc
Bots on guard of sdlc
 
3 Keys to Performance Testing at the Speed of Agile
3 Keys to Performance Testing at the Speed of Agile3 Keys to Performance Testing at the Speed of Agile
3 Keys to Performance Testing at the Speed of Agile
 

Recently uploaded

Refactoring legacy systems using events commands and bubble contexts
Refactoring legacy systems using events commands and bubble contextsRefactoring legacy systems using events commands and bubble contexts
Refactoring legacy systems using events commands and bubble contexts
Michał Kurzeja
 
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdfThe Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
kalichargn70th171
 
Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.
KrishnaveniMohan1
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
gapen1
 
How GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdfHow GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdf
Zycus
 
ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.
Maitrey Patel
 
Computer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdfComputer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdf
chandangoswami40933
 
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
widenerjobeyrl638
 
What’s New in VictoriaLogs - Q2 2024 Update
What’s New in VictoriaLogs - Q2 2024 UpdateWhat’s New in VictoriaLogs - Q2 2024 Update
What’s New in VictoriaLogs - Q2 2024 Update
VictoriaMetrics
 
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data PlatformAlluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio, Inc.
 
Stork Product Overview: An AI-Powered Autonomous Delivery Fleet
Stork Product Overview: An AI-Powered Autonomous Delivery FleetStork Product Overview: An AI-Powered Autonomous Delivery Fleet
Stork Product Overview: An AI-Powered Autonomous Delivery Fleet
Vince Scalabrino
 
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
Yara Milbes
 
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom KittEnhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Peter Caitens
 
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdfBaha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid
 
TheFutureIsDynamic-BoxLang-CFCamp2024.pdf
TheFutureIsDynamic-BoxLang-CFCamp2024.pdfTheFutureIsDynamic-BoxLang-CFCamp2024.pdf
TheFutureIsDynamic-BoxLang-CFCamp2024.pdf
Ortus Solutions, Corp
 
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in NashikUpturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
campbellclarkson
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
ervikas4
 
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptxOperational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
sandeepmenon62
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
kgyxske
 

Recently uploaded (20)

Refactoring legacy systems using events commands and bubble contexts
Refactoring legacy systems using events commands and bubble contextsRefactoring legacy systems using events commands and bubble contexts
Refactoring legacy systems using events commands and bubble contexts
 
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdfThe Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
 
Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
 
How GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdfHow GenAI Can Improve Supplier Performance Management.pdf
How GenAI Can Improve Supplier Performance Management.pdf
 
ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.ACE - Team 24 Wrapup event at ahmedabad.
ACE - Team 24 Wrapup event at ahmedabad.
 
Computer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdfComputer Science & Engineering VI Sem- New Syllabus.pdf
Computer Science & Engineering VI Sem- New Syllabus.pdf
 
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
美洲杯赔率投注网【​网址​🎉3977·EE​🎉】
 
What’s New in VictoriaLogs - Q2 2024 Update
What’s New in VictoriaLogs - Q2 2024 UpdateWhat’s New in VictoriaLogs - Q2 2024 Update
What’s New in VictoriaLogs - Q2 2024 Update
 
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data PlatformAlluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
 
Stork Product Overview: An AI-Powered Autonomous Delivery Fleet
Stork Product Overview: An AI-Powered Autonomous Delivery FleetStork Product Overview: An AI-Powered Autonomous Delivery Fleet
Stork Product Overview: An AI-Powered Autonomous Delivery Fleet
 
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
 
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom KittEnhanced Screen Flows UI/UX using SLDS with Tom Kitt
Enhanced Screen Flows UI/UX using SLDS with Tom Kitt
 
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdfBaha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
Baha Majid WCA4Z IBM Z Customer Council Boston June 2024.pdf
 
TheFutureIsDynamic-BoxLang-CFCamp2024.pdf
TheFutureIsDynamic-BoxLang-CFCamp2024.pdfTheFutureIsDynamic-BoxLang-CFCamp2024.pdf
TheFutureIsDynamic-BoxLang-CFCamp2024.pdf
 
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in NashikUpturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in Nashik
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
 
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptxOperational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
 

2016 04-25 continuous integration at google scale

  • 1. Google Confidential and Proprietary Continuous Integration at Google Scale By John Micco Developer Infrastructure
  • 2. Google Confidential and Proprietary ● >30,000 developers in 40+ offices ● 13,000+ projects under active development ● 30k submissions per day (1 every 3 seconds) ● Single monolithic code tree with mixed language code ● Development on one branch - submissions at head ● All builds from source ● 30+ sustained code changes per minute with 90+ peaks ● 50% of code changes monthly ● 150+ million test cases / day, > 150 years of test / day ● Supports continuous deployment for all Google teams! Speed and Scale
  • 3. Google Confidential and Proprietary Overview 1. Continuous Integration Goals 2. Continuous Integration at Google 3. Future of testing
  • 4. Google Confidential and Proprietary ● Provide real-time information to build monitors ○ Identify failures fast ○ Identify culprit Changes ○ Handle flaky tests ● Provide frequent green builds for cutting releases ○ Identify recent green builds ○ Show results of all testing together ○ Allow release tooling to choose a green build ○ Handle flaky tests "green build" = all tests contained in that build are passing at a given Change. Continuous Integration
  • 5. Google Confidential and Proprietary ● Develop Safely ○Sync to last green changelist ○Identify whether change will break the build before submit ○Submit with confidence ○Handle flaky tests Continuous Integration (cont)
  • 6. Google Confidential and Proprietary Standard Continuous Build System ● Triggers builds in continuous cycle ● Cycle time = longest build + test cycle ● Tests many changes together ● Which change broke the build?
  • 7. Google Confidential and Proprietary ● Triggers tests on every change ● Uses fine-grained dependencies ● Change 2 broke test 1 Google Continuous Build System
  • 8. Google Confidential and Proprietary Continuous Integration Display
  • 9. Google Confidential and Proprietary ● Identifies failures sooner ● Identifies culprit change precisely ○ Avoids divide-and-conquer and tribal knowledge ● Lower compute costs using fine grained dependencies ● Keeps the build green by reducing time to fix breaks ● Accepted enthusiastically by product teams ● Enables teams to ship with fast iteration times ○ Supports submit-to-production times of less than 36 hours for some projects Benefits
  • 10. Google Confidential and Proprietary ● Requires enormous investment in compute resources (it helps to be at Google) grows in proportion to: ○ Submission rate ○ Average build + test time ○ Variants (debug, opt, valgrind, etc.) ○ Increasing dependencies on core libraries ○ Branches ● Requires updating dependencies on each change ○ Takes time to update - delays start of testing Costs
  • 11. Google Confidential and Proprietary ● Makes testing available before submit ● Uses fine-grained dependencies ○ Recalculate any dependency changes ● Uses same pool of compute resources at high priority ● Avoids breaking the build ● Captures contents of a change and tests in isolation ○ Tests against head ○ Identifies problems with missing files ● Integrates with ○ submission tool - submit iff testing is green ○ Code Review Tool - results are posted to the review thread Developing Safely - presubmit
  • 12. Google Confidential and Proprietary Example Presubmit Display
  • 13. Google Confidential and Proprietary Practical Matters - Test Growth ● Sources of growth in test execution time ○ More developers = increased submission rate ○ More tests ○ Longer running tests ○ Tests consuming more resources (threading) ● Examine the growth trends ○ Predict compute needs ○ Look for any build system features required
  • 14. Build / Test Compute Resources Jan 2011 Jan 2012 Jan 2013 Jul 2012 Jul 2011
  • 15.
  • 16. Google Confidential and Proprietary ● Problems ○ Quadratic execution time growth w/ 2 factors ■ Submit rate - grows linearly ■ Test pool size - grows linearly ○ Ultimately cannot run every affected test @ every change ○ Low latency results still top requirement ● Solution: Just in time scheduling (JIT) Test Growth
  • 17. Continuous Integration: ● Run every test affected at every changelist. In Production: ● Build and run tests concurrently on Google’s distributed build and test backend. JIT as often as possible
  • 18. JIT Scheduling Schedule tests to run only when system has capacity. Produce project-wide results at periodic changelists.
  • 19. Milestone Property A changelist C is a milestone iff ... ● All tests affected at C are run ● All tests affected since the previous milestone are run. ● All these tests are run at their greatest affecting changelist <= C.
  • 20. Milestone Property A changelist C is a milestone iff ... ● All tests affected at C are run ● All tests affected since the previous milestone are run. ● All these tests are run at their greatest affecting changelist <= C. Exactly the work necessary to deliver a conclusive project status
  • 21. Confidential + Proprietary Change Lists AffectedTestTargetset Cut milestone at this CL 21
  • 22. Confidential + Proprietary Change Lists AffectedTestTargetset 22
  • 23. Confidential + Proprietary Change Lists AffectedTestTargetset 23
  • 24. Confidential + Proprietary Change Lists AffectedTestTargetset 24
  • 25. Confidential + Proprietary Change Lists AffectedTestTargetset 25
  • 26. Google Confidential and Proprietary JIT scheduling results ● JIT scheduler changed compute growth from quadratic to linear! ● Without it, compute demand would have already consumed all of Google's capacity ○ We literally either would have had to stop testing this way or stop running user searches ● Enabled Google to keep providing fast feedback to developers with reasonable compute costs
  • 27. Google Confidential and Proprietary Cuprit Finding - Transition to Fail A Time Targets Changelists 1 2 3 Passed Affected, but not run (yet) Milestone Non-milestone 4 Failed Schedule these
  • 28. Google Confidential and Proprietary Cuprit Finding - Transition to Fail A Time Targets Changelists 1 2 3 Passed Affected, but not run (yet) Milestone Non-milestone 4 Failed A: Change 3 broke test A.
  • 29. Google Confidential and Proprietary Future Direction ● Atif Memon is working with us (on sabbatical) to analyze our data ● He is finding that the dependency distance between a target and the triggering source file is highly correlated with the probability of introducing a non-flaky failure. We are working with him to publish his findings.
  • 30. Google Confidential and Proprietary Q & A For more information: ● http://google-engtools.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html ● http://www.youtube.com/watch?v=b52aXZ2yi08 ● http://www.infoq.com/presentations/Development-at-Google ● http://google-engtools.blogspot.com/ ● http://misko.hevery.com/2008/11/11/clean-code-talks-dependency-injection/ ● https://www.youtube.com/watch?v=KH2_sB1A6lA&feature=youtube_gdata_player Q & A