SlideShare a Scribd company logo
1 of 24
Continuous Integration as a
Development Team’s Way of Life
Melissa Benua (@queenofcode)
Senior Software Engineer
PlayFab, Inc
Better Software West 2016
Continuous Integration + Continuous Delivery
Continuous Integration (1999):
“The practice of merging all
developer working copies to a
shared mainline several times a
day.”
Continuous Delivery (2010):
“Every code change can be
deployed to production.”
Develop
BuildTest
Deploy
Principles of Go-Fast
• Many small changes are
better than fewer big changes
• Every single change gets the
full set of tests
• Automate everything
• Code in the master branch
can go live at any point in time
• Code reviews are necessary,
but also automated
Why Go Fast?
• Test automation leads to
faster development speed
• Faster development speed
leads to faster turnaround
times
• Faster turnaround times lead
to more features
• More features lead to $$$
CI + CD Strategy
Branch Strategy Deployment Strategy
Dev Build
• Build Against Staging
• Deploy to Staging
• Run Integration Tests
Live Build
• Build Against Production
• Deploy to Production Staging
• Run Integration Tests
Deploy
• Swap Production Staging into Production
• Monitor
Pipeline Overview
Self-hosting parts of the integration pipeline can be cheap and easy!
Git
•Source
control
Phabricator
•Track
•Peer Review
Jenkins
•Build
•Test
Cloud Compute
•Release
•Monitor
Source Control
Don’t lose your work!
Git!
• Why git?
• Simple to start with
• Plugins for every need
• Forking allows great freedom
• Choice of hosted or self hosted
• Distributed reliability and safety
• Easy partial roll backs
• Ubiquitous
MASTER
The ‘GitHub’ Flow
C2
C10
C1 C3 C4
C11
Create branch
‘myfeature’
Merge ‘myfeature’
into master
Code Review + Tracking
Do good work!
Phabricator!
• http://phabricator.org/
• Runs on Linux
• Written in PHP
• Spun out of Facebook
• Moderate plugin system
• Can handle most languages
• Use as many or as few
Phabricator services as you like
• Supports: issue tracking, scrum
boards, source auditing, code
review, more!
Code Review Holy Grail
Submit
•Dev submits code to own git branch
•Dev types ‘arc diff’ to start Phabricator code review
Build
•Phabricator starts build stage of review
•Git branch is built + unit tested on Jenkins, results report back to diff
Deploy
•Phabricator starts integration stage of review
•New environment deployed in cloud by Jenkins
•Integration tests run against new environment, results report back to diff
Code Reviews with Phabricator
• Differential: code review tool
• Harbormaster: build management
tool
• Manifest: issue tracking tool
• Setup Process:
• https://github.com/uber/
phabricator-jenkins-plugin
• Harbormaster sets up rules of how
to connect to Jenkins
• Herald sets up what jobs run
against what code
• Jenkins runs jobs and posts back
test + coverage results
Code Coverage with Phabricator
• Accepts coverage as part of Jenkins
test results postback
• Uses simple custom format:
• N => Not executable. This is a
comment or whitespace which should
be ignored when computing test
coverage.
• C => Covered. This line has test
coverage.
• U => Uncovered. This line is executable
but has no test coverage.
• X => Unreachable. If your coverage
analysis can detect unreachable code,
you can report it here.
• ‘myclass.cs’ => ‘NNCNNUNXUC’
Cloud Build + Test
Check your work!
Jenkins!
• http://jenkins-ci.org/
• Installs on Windows or Linux
• Written in Java
• Extensive plugin system
• Can build most languages
• Jobs can be chained together
and communicate with each
other
• Uses webhooks for cross-
service communication
Build and Test with Jenkins
Develop: Diff Build
• Compile change against mainline
• Execute unit tests
Build: Continuous Integration
• Compile change as a part of mainline submit
• Execute unit tests
Deploy: Continuous Deployment
• Start staging environment
• Deploy staging environment
• Execute integration tests
Jenkins in the Wild
Deploy + Monitor
See your work live!
Cloud Deployment
• Each service packaged and deployed
by Jenkins:
• Staging: All builds update staging
environment services and all tests are
run
• Production: Builds are cherry pick
deployed
– Deploy to Production Staging
– Run tests against new staging
environment
– Roll staged traffic over to new
environment
– Roll back immediately on failures
• All deployments managed via Jenkins
• Packaging includes config changes
• Common Pitfalls:
– In place updates
– Swap all traffic at once
– No roll back mechanism
Cloud Monitoring
• How to know you’re down
• Use counters
• Count what makes sense
• Know your service KPIs
• Don’t just count, track deviation
• Page when it’s wrong, before it’s
bad
• Log
• Don’t rely on being able to debug
on the server
Not sure if error spike due
to new code
Or terrible users
Cloud Monitoring
– Deviation from minute to minute can
tell you a lot at high volumes
– Allows finding what would otherwise
have been lost in the noise
– Fine grained tracking is most useful.
Per API + per error code, for
example
– Track enough data to be able to
match deviations with deployments
– Logs are important, but often aren’t
enough to know something is wrong
unless it’s broken
• Logs tell you about A request,
counters tell you about ALL requests
Statistics!
• > 1000000 lines of code
• ~1500 automated tests
• ~60% automated code
coverage
• ~200k lines of
churn/month
• ~5 production
deployments/week
• ~10 cloud services
Questions?
Melissa Benua
@queenofcode
mbenua@gmail.com
https://www.linkedin.com/in/mbenua
http://www.slideshare.net/MelissaBenua

More Related Content

What's hot

TFS Build - Continuous Integration
TFS Build - Continuous IntegrationTFS Build - Continuous Integration
TFS Build - Continuous Integration
Esteban Garcia
 
Continuous deployment steve povilaitis
Continuous deployment   steve povilaitisContinuous deployment   steve povilaitis
Continuous deployment steve povilaitis
Steve Povilaitis
 
Using Continuous Integration To Ensure Project Health New
Using Continuous Integration To Ensure Project Health NewUsing Continuous Integration To Ensure Project Health New
Using Continuous Integration To Ensure Project Health New
bartlowe
 

What's hot (20)

Continuous Integration and Builds
Continuous Integration and BuildsContinuous Integration and Builds
Continuous Integration and Builds
 
CD
CDCD
CD
 
TFS Build - Continuous Integration
TFS Build - Continuous IntegrationTFS Build - Continuous Integration
TFS Build - Continuous Integration
 
CI with TFS
CI with TFSCI with TFS
CI with TFS
 
What is Continuous Integration and Continuous Delivery
What is Continuous Integration and Continuous Delivery What is Continuous Integration and Continuous Delivery
What is Continuous Integration and Continuous Delivery
 
Jenkins Test Automation with codeBeamer ALM
Jenkins Test Automation with codeBeamer ALMJenkins Test Automation with codeBeamer ALM
Jenkins Test Automation with codeBeamer ALM
 
Continuous deployment steve povilaitis
Continuous deployment   steve povilaitisContinuous deployment   steve povilaitis
Continuous deployment steve povilaitis
 
Using Continuous Integration To Ensure Project Health New
Using Continuous Integration To Ensure Project Health NewUsing Continuous Integration To Ensure Project Health New
Using Continuous Integration To Ensure Project Health New
 
Learn Key Insights from The State of Web Application Testing Research Report
Learn Key Insights from The State of Web Application Testing Research ReportLearn Key Insights from The State of Web Application Testing Research Report
Learn Key Insights from The State of Web Application Testing Research Report
 
Continuous Integration, Deploy, Test From Beginning To End 2014
Continuous Integration, Deploy, Test From Beginning To End 2014Continuous Integration, Deploy, Test From Beginning To End 2014
Continuous Integration, Deploy, Test From Beginning To End 2014
 
STARWest: Use Jenkins For Continuous 
Load Testing And Mobile Test Automation
STARWest: Use Jenkins For Continuous 
Load Testing And Mobile Test AutomationSTARWest: Use Jenkins For Continuous 
Load Testing And Mobile Test Automation
STARWest: Use Jenkins For Continuous 
Load Testing And Mobile Test Automation
 
Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...
Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...
Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...
 
CloudBees Continuous Integration and Test with Appvance PerformanceCloud
CloudBees Continuous Integration and Test with Appvance PerformanceCloudCloudBees Continuous Integration and Test with Appvance PerformanceCloud
CloudBees Continuous Integration and Test with Appvance PerformanceCloud
 
Sencha Roadshow 2017: What's New in Sencha Test
Sencha Roadshow 2017: What's New in Sencha TestSencha Roadshow 2017: What's New in Sencha Test
Sencha Roadshow 2017: What's New in Sencha Test
 
Deeper Root Cause Analysis to App Performance Bottlenecks with Appvance APM I...
Deeper Root Cause Analysis to App Performance Bottlenecks with Appvance APM I...Deeper Root Cause Analysis to App Performance Bottlenecks with Appvance APM I...
Deeper Root Cause Analysis to App Performance Bottlenecks with Appvance APM I...
 
Simple Continous Integration
Simple Continous IntegrationSimple Continous Integration
Simple Continous Integration
 
Continous integration
Continous integrationContinous integration
Continous integration
 
Continous Integration: A Case Study
Continous Integration: A Case StudyContinous Integration: A Case Study
Continous Integration: A Case Study
 
Running JMeter Tests In Appvance PerformanceCloud
Running JMeter Tests In Appvance PerformanceCloudRunning JMeter Tests In Appvance PerformanceCloud
Running JMeter Tests In Appvance PerformanceCloud
 
Keeping Your CI/CD Pipeline as Fast as It Needs to Be
Keeping Your CI/CD Pipeline as Fast as It Needs to BeKeeping Your CI/CD Pipeline as Fast as It Needs to Be
Keeping Your CI/CD Pipeline as Fast as It Needs to Be
 

Viewers also liked

Viewers also liked (20)

Commander's Intent: Managing Through Uncertainty
Commander's Intent: Managing Through UncertaintyCommander's Intent: Managing Through Uncertainty
Commander's Intent: Managing Through Uncertainty
 
PlayFab Advanced Cloud Script
PlayFab Advanced Cloud ScriptPlayFab Advanced Cloud Script
PlayFab Advanced Cloud Script
 
Say hello to the new PlayFab!
Say hello to the new PlayFab!Say hello to the new PlayFab!
Say hello to the new PlayFab!
 
No Free Lunch: Transactions in Online Games
No Free Lunch: Transactions in Online GamesNo Free Lunch: Transactions in Online Games
No Free Lunch: Transactions in Online Games
 
Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
Behind the Scenes: Deploying a Low-Latency Multiplayer Game GloballyBehind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
 
Building the pipeline for FUN - Game Development
 Building the pipeline for FUN - Game Development Building the pipeline for FUN - Game Development
Building the pipeline for FUN - Game Development
 
The Future is Operations: Why Mobile Games Need Backends
The Future is Operations: Why Mobile Games Need BackendsThe Future is Operations: Why Mobile Games Need Backends
The Future is Operations: Why Mobile Games Need Backends
 
iPhone Battery Saving Tips & Tricks
iPhone Battery Saving Tips & TricksiPhone Battery Saving Tips & Tricks
iPhone Battery Saving Tips & Tricks
 
Connecting with Customers & Clients via Social Media
Connecting with Customers & Clients via Social MediaConnecting with Customers & Clients via Social Media
Connecting with Customers & Clients via Social Media
 
88 Inspiring Career Quotes Every Professionals Should Know
88 Inspiring Career Quotes Every Professionals Should Know88 Inspiring Career Quotes Every Professionals Should Know
88 Inspiring Career Quotes Every Professionals Should Know
 
Propel With Purpose: Power of Your Network
Propel With Purpose: Power of Your NetworkPropel With Purpose: Power of Your Network
Propel With Purpose: Power of Your Network
 
The 5 biggest mistakes every job seeker makes
The 5 biggest mistakes every job seeker makesThe 5 biggest mistakes every job seeker makes
The 5 biggest mistakes every job seeker makes
 
What Is Twitter: A guide for NFPs
What Is Twitter: A guide for NFPsWhat Is Twitter: A guide for NFPs
What Is Twitter: A guide for NFPs
 
Propel With Purpose: Go Culture Shopping
Propel With Purpose: Go Culture ShoppingPropel With Purpose: Go Culture Shopping
Propel With Purpose: Go Culture Shopping
 
Propel With Purpose: Power of Swans
Propel With Purpose: Power of SwansPropel With Purpose: Power of Swans
Propel With Purpose: Power of Swans
 
biipmi.com - Guide to Creating Your First Course Listing
biipmi.com - Guide to Creating Your First Course Listingbiipmi.com - Guide to Creating Your First Course Listing
biipmi.com - Guide to Creating Your First Course Listing
 
The Hacker's guide to fundraising
The Hacker's guide to fundraisingThe Hacker's guide to fundraising
The Hacker's guide to fundraising
 
How To Create A Social Media Strategy
How To Create A Social Media StrategyHow To Create A Social Media Strategy
How To Create A Social Media Strategy
 
biipmi.com - Guide to Creating Your First Job Posting
biipmi.com - Guide to Creating Your First Job Postingbiipmi.com - Guide to Creating Your First Job Posting
biipmi.com - Guide to Creating Your First Job Posting
 
Technology Policy for Hong Kong
Technology Policy for Hong KongTechnology Policy for Hong Kong
Technology Policy for Hong Kong
 

Similar to Continuous Integration as a Way of Life

Similar to Continuous Integration as a Way of Life (20)

Continuous Integration as a Development Team’s Way of Life
Continuous Integration as a Development Team’s Way of LifeContinuous Integration as a Development Team’s Way of Life
Continuous Integration as a Development Team’s Way of Life
 
Jenkins_1679702972.pdf
Jenkins_1679702972.pdfJenkins_1679702972.pdf
Jenkins_1679702972.pdf
 
jenkins.pdf
jenkins.pdfjenkins.pdf
jenkins.pdf
 
Continuous Deployment of your Application @JUGtoberfest
Continuous Deployment of your Application @JUGtoberfestContinuous Deployment of your Application @JUGtoberfest
Continuous Deployment of your Application @JUGtoberfest
 
SQL Server DevOps Jumpstart
SQL Server DevOps JumpstartSQL Server DevOps Jumpstart
SQL Server DevOps Jumpstart
 
Continuous Deployment To The Cloud @DevoxxPL 2017
Continuous Deployment To The Cloud @DevoxxPL 2017 Continuous Deployment To The Cloud @DevoxxPL 2017
Continuous Deployment To The Cloud @DevoxxPL 2017
 
Continuous Deployment of your Application @SpringOne
Continuous Deployment of your Application @SpringOneContinuous Deployment of your Application @SpringOne
Continuous Deployment of your Application @SpringOne
 
Continuous Integration
Continuous IntegrationContinuous Integration
Continuous Integration
 
MyHeritage - QA Automations in a Continuous Deployment environment
MyHeritage -  QA Automations in a Continuous Deployment environmentMyHeritage -  QA Automations in a Continuous Deployment environment
MyHeritage - QA Automations in a Continuous Deployment environment
 
Continuous Delivery for Open Source Java projects
Continuous Delivery for Open Source Java projectsContinuous Delivery for Open Source Java projects
Continuous Delivery for Open Source Java projects
 
DevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CD
DevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CDDevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CD
DevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CD
 
Continuous Integration for Salesforce1 Platform
Continuous Integration for Salesforce1 PlatformContinuous Integration for Salesforce1 Platform
Continuous Integration for Salesforce1 Platform
 
Test-Driven-Development for Networking: Making CI Work for You by Colin McNam...
Test-Driven-Development for Networking: Making CI Work for You by Colin McNam...Test-Driven-Development for Networking: Making CI Work for You by Colin McNam...
Test-Driven-Development for Networking: Making CI Work for You by Colin McNam...
 
Production Ready WordPress #WPLDN
Production Ready WordPress #WPLDNProduction Ready WordPress #WPLDN
Production Ready WordPress #WPLDN
 
Getting to Walk with DevOps
Getting to Walk with DevOpsGetting to Walk with DevOps
Getting to Walk with DevOps
 
Continous integration and delivery for single page applications
Continous integration and delivery for single page applicationsContinous integration and delivery for single page applications
Continous integration and delivery for single page applications
 
Devops architecture
Devops architectureDevops architecture
Devops architecture
 
Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013
 
Why NXTware Remote for Jenkins
Why NXTware Remote for JenkinsWhy NXTware Remote for Jenkins
Why NXTware Remote for Jenkins
 
Continuous Deployment to the cloud
Continuous Deployment to the cloudContinuous Deployment to the cloud
Continuous Deployment to the cloud
 

Recently uploaded

Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Lisi Hocke
 

Recently uploaded (20)

Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmux
 
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?
 
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Concepts
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...
Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...
Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
 
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
 
微信号购买
微信号购买微信号购买
微信号购买
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
 
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
 
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
 

Continuous Integration as a Way of Life

  • 1. Continuous Integration as a Development Team’s Way of Life Melissa Benua (@queenofcode) Senior Software Engineer PlayFab, Inc Better Software West 2016
  • 2. Continuous Integration + Continuous Delivery Continuous Integration (1999): “The practice of merging all developer working copies to a shared mainline several times a day.” Continuous Delivery (2010): “Every code change can be deployed to production.” Develop BuildTest Deploy
  • 3. Principles of Go-Fast • Many small changes are better than fewer big changes • Every single change gets the full set of tests • Automate everything • Code in the master branch can go live at any point in time • Code reviews are necessary, but also automated
  • 4. Why Go Fast? • Test automation leads to faster development speed • Faster development speed leads to faster turnaround times • Faster turnaround times lead to more features • More features lead to $$$
  • 5. CI + CD Strategy Branch Strategy Deployment Strategy Dev Build • Build Against Staging • Deploy to Staging • Run Integration Tests Live Build • Build Against Production • Deploy to Production Staging • Run Integration Tests Deploy • Swap Production Staging into Production • Monitor
  • 6. Pipeline Overview Self-hosting parts of the integration pipeline can be cheap and easy! Git •Source control Phabricator •Track •Peer Review Jenkins •Build •Test Cloud Compute •Release •Monitor
  • 8. Git! • Why git? • Simple to start with • Plugins for every need • Forking allows great freedom • Choice of hosted or self hosted • Distributed reliability and safety • Easy partial roll backs • Ubiquitous
  • 9. MASTER The ‘GitHub’ Flow C2 C10 C1 C3 C4 C11 Create branch ‘myfeature’ Merge ‘myfeature’ into master
  • 10. Code Review + Tracking Do good work!
  • 11. Phabricator! • http://phabricator.org/ • Runs on Linux • Written in PHP • Spun out of Facebook • Moderate plugin system • Can handle most languages • Use as many or as few Phabricator services as you like • Supports: issue tracking, scrum boards, source auditing, code review, more!
  • 12. Code Review Holy Grail Submit •Dev submits code to own git branch •Dev types ‘arc diff’ to start Phabricator code review Build •Phabricator starts build stage of review •Git branch is built + unit tested on Jenkins, results report back to diff Deploy •Phabricator starts integration stage of review •New environment deployed in cloud by Jenkins •Integration tests run against new environment, results report back to diff
  • 13. Code Reviews with Phabricator • Differential: code review tool • Harbormaster: build management tool • Manifest: issue tracking tool • Setup Process: • https://github.com/uber/ phabricator-jenkins-plugin • Harbormaster sets up rules of how to connect to Jenkins • Herald sets up what jobs run against what code • Jenkins runs jobs and posts back test + coverage results
  • 14. Code Coverage with Phabricator • Accepts coverage as part of Jenkins test results postback • Uses simple custom format: • N => Not executable. This is a comment or whitespace which should be ignored when computing test coverage. • C => Covered. This line has test coverage. • U => Uncovered. This line is executable but has no test coverage. • X => Unreachable. If your coverage analysis can detect unreachable code, you can report it here. • ‘myclass.cs’ => ‘NNCNNUNXUC’
  • 15. Cloud Build + Test Check your work!
  • 16. Jenkins! • http://jenkins-ci.org/ • Installs on Windows or Linux • Written in Java • Extensive plugin system • Can build most languages • Jobs can be chained together and communicate with each other • Uses webhooks for cross- service communication
  • 17. Build and Test with Jenkins Develop: Diff Build • Compile change against mainline • Execute unit tests Build: Continuous Integration • Compile change as a part of mainline submit • Execute unit tests Deploy: Continuous Deployment • Start staging environment • Deploy staging environment • Execute integration tests
  • 19. Deploy + Monitor See your work live!
  • 20. Cloud Deployment • Each service packaged and deployed by Jenkins: • Staging: All builds update staging environment services and all tests are run • Production: Builds are cherry pick deployed – Deploy to Production Staging – Run tests against new staging environment – Roll staged traffic over to new environment – Roll back immediately on failures • All deployments managed via Jenkins • Packaging includes config changes • Common Pitfalls: – In place updates – Swap all traffic at once – No roll back mechanism
  • 21. Cloud Monitoring • How to know you’re down • Use counters • Count what makes sense • Know your service KPIs • Don’t just count, track deviation • Page when it’s wrong, before it’s bad • Log • Don’t rely on being able to debug on the server Not sure if error spike due to new code Or terrible users
  • 22. Cloud Monitoring – Deviation from minute to minute can tell you a lot at high volumes – Allows finding what would otherwise have been lost in the noise – Fine grained tracking is most useful. Per API + per error code, for example – Track enough data to be able to match deviations with deployments – Logs are important, but often aren’t enough to know something is wrong unless it’s broken • Logs tell you about A request, counters tell you about ALL requests
  • 23. Statistics! • > 1000000 lines of code • ~1500 automated tests • ~60% automated code coverage • ~200k lines of churn/month • ~5 production deployments/week • ~10 cloud services