Uploaded bySalesforce Engineering
A Story of Continuous Integration
This document tells the story of how a class project evolved into a company with thousands of engineers and millions of lines of code using continuous integration practices over 15 years. It describes how the project started with manual code sharing and testing, then adopted version control and regular integration. It details how the company scaled up practices like automated builds, testing, and failure tracking to support growth from 2 people to thousands of engineers and over 200,000 tests while maintaining quality. The key takeaways are shortening feedback loops through practices like continuous integration, adopting principles like versioning everything and automating as much as possible, and having scalable solutions at each stage of growth.
More Related Content
![[Webinar] Test First, Fail Fast - Simplifying the Tester's Transition to DevOps](https://cdn.slidesharecdn.com/ss_thumbnails/testfirstfailfastfinal-170323164615-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to A Story of Continuous Integration
More from Salesforce Engineering
A Story of Continuous Integration
- 1. A Story ofContinuous Integration jcharlety@salesforce.com cmourot@salesforce.com
- 2. Your speakers C. Mourot 20142005 JBCharlety 20132004
- 3. We want totell you a success story This story is a fiction. Any resemblance to actual events is purely coincidental. 6 months 12 months 18 months 3 years 5 years 10 years time
- 4. Once upon atime… • Class Project • 2 people • USB keys / shared drive • < 100 tests • because the teachers wanted some tests • coded the last day • manual execution
- 5. At the beginning… Howeffectively share the code • Use a Version Control System (VCS) How often run the tests? • Let’s nominate a “build master”! Let’s meet your uncle to get some love money ($) x100 x3 I’m not able to start the server! Who has got the last version of the code? !
- 6. Sharing the code mainlinea1a3 a1 a a2 a3 integration commit, checkin build ok build: compile+test file sync, update codeline
- 7. At Salesforce more than1000 engineers one codeline (almost) 150+ scrum teams more than 15 locations
- 8. Demo to abusiness angel ($$) 6 months later… x500 I worked all night to fix your $@!#% bugs! x10 x1 It has not been compiling for days on my machine! Why is my feature not in the demo? I checked in yesterday! mainline Do we have tests?
- 9. 6 months later…the day after What happened? Integration chaos Let’s empower the build master! mainline asap!
- 10. 12 months later… x1000 x15 Ineed to fix a bug asap for a customer x10 Damned! But I don’t want to ship all those new features mainline
- 11. Let’s branch! Versioning thecode mainline v1.0 backport?
- 12. Be careful whenchoosing your branching pattern! Versioning the code http://www.bradapp.com/acme/branching/branch-structs.html
- 13.
- 14. v4 shipped inproduction 18 months later… x2000 Customer Z is very angry with the last version: it’s buggy! x20 x10 Every morning, I fix the build on my machine mainline I spend all my time fixing the build Everybody is committing a lot of code, the day before the demo Build ok! Do we have tests?
- 15. 18 months later…the day after Automatic build • Build server • Run after each checkin • Compile + Tests + Alert Everyone is responsible for the build • Don’t break the build! • No more buildmaster Scope/Time/Quality mainline
- 16. At Salesforce Quality :the triangle Time and Quality are always fixed • Milestone/Release dates never change • Feature & Release Signoff requirements are quantifiable and deterministic Scope is the only thing that you can change • Team cuts features to meet FF/RF signoff requirements Scope Time Quality Planned Scope Actual Scope
- 17. 3 years later… x5000 x35 Buildlast more than 2 hours ! x100 mainline 3 days to get the build results! Are you kidding me? What? I thought we had continuous integration! The build is broken all the time!
- 18. • How tospeed up the build? • Buy a faster server • Buy more servers • Break in different chunks • Build stage • Test types • Choice • After each commit, compile only • Run the basic tests twice a day • Run the extended each night Let’s scale
- 19. At Salesforce As ofQ2 2013 (20%) UI Tests Functional Tests (70%) Unit Tests (5%) Type Of Test Number of Tests Unit Test 12 000+ Basic Functional 20 000+ Full Functional 140 000+ Extended Functional 25 000+ Basic UI 500+ Full UI 40 000+ Extended UI 10 000+
- 20. • How tonot break the build? • Unbreakable build • How to speed up test execution? • Add more servers to run in // • Who broke the tests? • Nominate a buildmaster to create bugs and assign them to the suspects Just bought another company in the US 5 years later… x20k x110 I’m always fixing the build because the other team broke it! x300 When is the night? The nightly build lasts 3 days More than 2000 tests are failing!
- 21. • automatic compilationof changes before check-in • basic tests run for each checkin • extended tests run on a regular basis Precheckin The Unbreakable Build - UNBREAKABLE - mainline
- 22. • Who brokethe tests? • Who is responsible to fix the tests? • How to handle flappy tests? It’s time to automate the tests failures process! Growing, growing, growing! 10 years later… x200k x1000 My team delivers new features, doesn’t fix test! More than 8000 tests are failing! x300 Tests are ok on my machine, they are failing on the servers
- 23. At Salesforce TFMS: Automationof the tests failures process Basics • Auto-create a bug for new test failures • Auto-close Test Failure bug • Re-open unresolved Test Failure • Detect duplicate Test Failure bug records Advanced functionality • Suspect identification • Lock the line • Flapper detection LucasfilmLtd.
- 24. At Salesforce Suspect ChangeIdentification Full TF TF TF TF CL1 CL2 CL5 CL4 CL3
- 25. At Salesforce Code Check-in& Build Pipeline Code Review Submit to Pre- Checkin Commit to perforce Repo Build Run Tests Report Test Results File Bugs for Test Failures PRECHECKIN AUTOBUILD TFMS
- 26. At Salesforce Let’s recap Code • Multi-million line code base • Several major branches • Hundreds of checkin per day Tests • More than 200,000 tests • 3000 hours serialized runtime • Number of Full Test Runs Processed – 1700/day Continuous Integration • Number of remote instances (VMs) – 1300 • Prod-like test environments Bonus: a portfolio of patents related to testing • See google patents: salesforce testing
- 27. 15 years later,what’s next? • From Continuous Integration to Continuous Delivery • Release more often, release smaller chunks • No more release freeze? • DevOps movement • cloud architecture • 0 downtime Ship features in production
- 28. 15 years later,what’s next? • distributed testing • testing for UI - devices and browsers Huge challenges
- 29. Key takeaways Shorten thefeedback loops • Compilation • Tests • Customers Key principles of continuous integration • Version Control everything • If it hurts, do it more frequently • Automate everything A solution for each scale
- 30.
- 31.