Continuous integration (CI) is a buzzword in software development today. We know it means “run lots of builds,” but having a continuous integration pipeline opens up opportunities well beyond making sure your team's code compiles. What if this pipeline could improve everything from the quality of code reviews to how often and safely you deploy to production and how you monitor your product in the wild? What if CI could provide insights into how automated tests are performing and how to improve them? Melissa Benua describes how to set up a basic CI infrastructure and then transform it into a way of life for development and test teams. Using free or nearly free tools, Melissa walks through a practical approach to making sure your code works—all the time and at every stage of the release train. Come away with practical advice for creating builds and running automation on the fly without spending hundreds of hours or thousands of dollars.

1. Continuous Integration as a
Development Team’s Way of Life
Melissa Benua (@queenofcode)
Senior Software Engineer
PlayFab, Inc
Better Software West 2016

2. Continuous Integration + Continuous Delivery
Continuous Integration (1999):
“The practice of merging all
developer working copies to a
shared mainline several times a
day.”
Continuous Delivery (2010):
“Every code change can be
deployed to production.”
Develop
BuildTest
Deploy

3. Principles of Go-Fast
• Many small changes are
better than fewer big changes
• Every single change gets the
full set of tests
• Automate everything
• Code in the master branch
can go live at any point in time
• Code reviews are necessary,
but also automated

4. Why Go Fast?
• Test automation leads to
faster development speed
• Faster development speed
leads to faster turnaround
times
• Faster turnaround times lead
to more features
• More features lead to $$$

5. CI + CD Strategy
Branch Strategy Deployment Strategy
Dev Build
• Build Against Staging
• Deploy to Staging
• Run Integration Tests
Live Build
• Build Against Production
• Deploy to Production Staging
• Run Integration Tests
Deploy
• Swap Production Staging into Production
• Monitor

6. Pipeline Overview
Self-hosting parts of the integration pipeline can be cheap and easy!
Git
•Source
control
Phabricator
•Track
•Peer Review
Jenkins
•Build
•Test
Cloud Compute
•Release
•Monitor

7. Source Control
Don’t lose your work!

8. Git!
• Why git?
• Simple to start with
• Plugins for every need
• Forking allows great freedom
• Choice of hosted or self hosted
• Distributed reliability and safety
• Easy partial roll backs
• Ubiquitous

9. MASTER
The ‘GitHub’ Flow
C2
C10
C1 C3 C4
C11
Create branch
‘myfeature’
Merge ‘myfeature’
into master

10. Code Review + Tracking
Do good work!

11. Phabricator!
• http://phabricator.org/
• Runs on Linux
• Written in PHP
• Spun out of Facebook
• Moderate plugin system
• Can handle most languages
• Use as many or as few
Phabricator services as you like
• Supports: issue tracking, scrum
boards, source auditing, code
review, more!

12. Code Review Holy Grail
Submit
•Dev submits code to own git branch
•Dev types ‘arc diff’ to start Phabricator code review
Build
•Phabricator starts build stage of review
•Git branch is built + unit tested on Jenkins, results report back to diff
Deploy
•Phabricator starts integration stage of review
•New environment deployed in cloud by Jenkins
•Integration tests run against new environment, results report back to diff

13. Code Reviews with Phabricator
• Differential: code review tool
• Harbormaster: build management
tool
• Manifest: issue tracking tool
• Setup Process:
• https://github.com/uber/
phabricator-jenkins-plugin
• Harbormaster sets up rules of how
to connect to Jenkins
• Herald sets up what jobs run
against what code
• Jenkins runs jobs and posts back
test + coverage results

14. Code Coverage with Phabricator
• Accepts coverage as part of Jenkins
test results postback
• Uses simple custom format:
• N => Not executable. This is a
comment or whitespace which should
be ignored when computing test
coverage.
• C => Covered. This line has test
coverage.
• U => Uncovered. This line is executable
but has no test coverage.
• X => Unreachable. If your coverage
analysis can detect unreachable code,
you can report it here.
• ‘myclass.cs’ => ‘NNCNNUNXUC’

15. Cloud Build + Test
Check your work!

16. Jenkins!
• http://jenkins-ci.org/
• Installs on Windows or Linux
• Written in Java
• Extensive plugin system
• Can build most languages
• Jobs can be chained together
and communicate with each
other
• Uses webhooks for cross-
service communication

17. Build and Test with Jenkins
Develop: Diff Build
• Compile change against mainline
• Execute unit tests
Build: Continuous Integration
• Compile change as a part of mainline submit
• Execute unit tests
Deploy: Continuous Deployment
• Start staging environment
• Deploy staging environment
• Execute integration tests

18. Jenkins in the Wild

19. Deploy + Monitor
See your work live!

20. Cloud Deployment
• Each service packaged and deployed
by Jenkins:
• Staging: All builds update staging
environment services and all tests are
run
• Production: Builds are cherry pick
deployed
– Deploy to Production Staging
– Run tests against new staging
environment
– Roll staged traffic over to new
environment
– Roll back immediately on failures
• All deployments managed via Jenkins
• Packaging includes config changes
• Common Pitfalls:
– In place updates
– Swap all traffic at once
– No roll back mechanism

21. Cloud Monitoring
• How to know you’re down
• Use counters
• Count what makes sense
• Know your service KPIs
• Don’t just count, track deviation
• Page when it’s wrong, before it’s
bad
• Log
• Don’t rely on being able to debug
on the server
Not sure if error spike due
to new code
Or terrible users

22. Cloud Monitoring
– Deviation from minute to minute can
tell you a lot at high volumes
– Allows finding what would otherwise
have been lost in the noise
– Fine grained tracking is most useful.
Per API + per error code, for
example
– Track enough data to be able to
match deviations with deployments
– Logs are important, but often aren’t
enough to know something is wrong
unless it’s broken
• Logs tell you about A request,
counters tell you about ALL requests

23. Statistics!
• > 1000000 lines of code
• ~1500 automated tests
• ~60% automated code
coverage
• ~200k lines of
churn/month
• ~5 production
deployments/week
• ~10 cloud services

24. Questions?
Melissa Benua
@queenofcode
mbenua@gmail.com
https://www.linkedin.com/in/mbenua
http://www.slideshare.net/MelissaBenua