Continuous integration is the new buzzword in software development because it opens up opportunities well beyond making sure all your team's code compiles cleanly. What if this pipeline could improve everything from the quality of code reviews, to how you monitor your product “in the wild,” and when your automated tests are executed? What if it could provide insight into how well those tests are performing? Melissa Benua explores how to setup a basic integration pipeline and transform that pipeline into a way of life for development teams. Using free or nearly-free tools, Melissa walks through a practical approach for employing continuous integration processes to make sure your code works—all the time and at every stage of release. Come away with practical advice for creating builds and running automation on the fly without spending hundreds of hours or thousands of dollars.

1. T14
Test
Techniques
5/5/16
13:30
Continuous
Integration:
A
New
Way
of
Life
Presented
by:
Melissa
Benua
PlayFab,
Inc.
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.stareast.techwell.com/

2. Melissa
Benua
PlayFab,
Inc.
Senior
backend
engineer
for
PlayFab
Melissa
Benua
has
worked
in
nearly
every
software
development
role: engineer,
test,
DevOps,
and
program
management
in
her
career.
Melissa
has
created
and
run
high-availability,
high-quality
services
at
Boeing
and
Microsoft
on
Bing,
Cortana,
and
Xbox
One.
She
discovered
her
love
of
massively-scaled
systems
while
working
on
the
Bing
backend,
honing
the
art
of
keeping
highly-available
complex
systems
up
while
undergoing
massive
code
churn.
Whether
working
with
classic
or
cutting-edge
testing
models,
Melissa
isn't
afraid
to
mix
traditional
approaches
with
bold
new
ideas
to
make
her
products
better,
faster,
and
more
reliable.
She
is
passionate
about
about
sharing
best
practices
with
colleagues
and
the
tech-world.

3. CONTINUOUS INTEGRATION AS A
DEVELOPER'S WAY OF LIFE
Melissa Benua
Senior Software Engineer
PlayFab, Inc
STAREAST 2016

4. Continuous Integration + Continuous Delivery
Continuous Integration (1999):
“The practice of merging all
developer working copies to a
shared mainline several times a
day.”
Continuous Delivery (2010):
“Every code change can be
deployed to production.”
Develop
BuildTest
Deploy

5. Principles of Go-Fast
Many small changes are better
than fewer big changes
Every single change gets the full
set of tests
Automate everything
Code in the master branch can
go live at any point in time
Code reviews are necessary, but
also automated

6. Why Go Fast?
Test automation leads to faster
development speed
Faster development speed leads
to faster turnaround times
Faster turnaround times lead to
more features
More features lead to $$$

7. CI + CD Strategy
Branch Strategy Deployment Strategy
Dev Build
• Build Against Staging
• Deploy to Staging
• Run Integration Tests
Live Build
• Build Against Production
• Deploy to Production Staging
• Run Integration Tests
Deploy
• Swap Production Staging into Production
• Monitor

8. Pipeline Overview
Self-hosting parts of the integration pipeline can be
cheap and easy!
Git
• Source
control
Phabricator
• Track
• Peer Review
Jenkins
• Build
• Test
Cloud
Compute
• Release
• Monitor

9. SOURCE CONTROL
Don’t lose your work!

10. Git!
Why git?
• Simple to start with
• Plugins for every need
• Forking allows great freedom
• Choice of hosted or self
hosted
• Distributed reliability and
safety
• Easy partial roll backs
• Ubiquitous

11. MASTER
The ‘GitHub’ Flow
C2
C10
C1 C3 C4
C11
Create branch
‘myfeature’
Merge ‘myfeature’
into master

12. CODE REVIEW + TRACKING
Do good work!

13. Phabricator!
• http://phabricator.org/
Runs on Linux
Written in PHP
Spun out of Facebook
Moderate plugin system
Can handle most languages
Use as many or as few Phabricator
services as you like
Supports: issue tracking, scrum
boards, source auditing, code
review, more!

14. Code Reviews with Phabricator
Differential: code review tool
Harbormaster: build management
tool
Manifest: issue tracking tool
Setup Process:
• https://github.com/uber/
phabricator-jenkins-plugin
• Harbormaster posts to Jenkins
every branch submit
• Jenkins runs build + test
• Jenkins posts back test results +
coverage results

15. Code Coverage with Phabricator
Accepts coverage as part of Jenkins test
results postback
Uses simple custom format:
• N Not executable. This is a comment
or whitespace which should be
ignored when computing test
coverage.
• C Covered. This line has test
coverage.
• U Uncovered. This line is executable
but has no test coverage.
• X Unreachable. If your coverage
analysis can detect unreachable code,
you can report it here.
‘myclass.cs’ => ‘NNCNNUNXUC’

16. CLOUD BUILD + TEST
Check your work!

17. Jenkins!
• http://jenkins-ci.org/
• Installs on Windows or
Linux
• Written in Java
• Extensive plugin system
• Can build most languages
• Jobs can be chained
together and communicate
with each other
• Uses webhooks for cross-
service communication

18. Build and Test with Jenkins
Develop: Diff Build
•Compile change against mainline
•Execute unit tests
Build: Continuous Integration
•Compile change as a part of mainline submit
•Execute unit tests
Deploy: Continuous Deployment
•Start staging environment
•Deploy staging environment
•Execute integration tests

19. Jenkins in the Wild

20. DEPLOY + MONITOR
See your work live!

21. Cloud Deployment
Each service packaged and deployed by
Jenkins:
• Staging: All builds update staging
environment services and all tests are
run
• Production: Builds are cherry pick
deployed
• Deploy to Production Staging
• Run tests against new staging environment
• Roll staged traffic over to new environment
• Roll back immediately on failures
All deployments managed via Jenkins
Packaging includes config changes
Common Pitfalls:
• In place updates
• Swap all traffic at once
• No roll back mechanism

22. Cloud Monitoring
How to know you’re down
• Use counters
• Count what makes sense
• Know your service KPIs
• Don’t just count, track deviation
• Page when it’s wrong, before it’s
bad
• Log
Don’t rely on being able to debug on
the server
Not sure if error spike due to
new code
Or terrible users

23. Cloud Monitoring
Deviation from minute to minute can
tell you a lot at high volumes
• Allows finding what would
otherwise have been lost in the
noise
• Fine grained tracking is most
useful. Per API + per error code, for
example
• Track enough data to be able to
match deviations with deployments
• Logs are important, but often aren’t
enough to know something is
wrong unless it’s broken
• Logs tell you about A request, counters
tell you about ALL requests

24. Integration Tests as Monitoring
Author one set of HTTP-level tests
• Same as how clients connect
• Self-contained and self-initializing
• Repeatable and reliable
Deploy tests both within build env and
within monitoring cloud
Collect data from tests into one central
location
Present data for use by devops and
customers
http://www.slideshare.net/MelissaBenua

25. Questions?
Melissa Benua
@queenofcode
mbenua@gmail.com
https://www.linkedin.com/in/mbenua
http://www.slideshare.net/MelissaBenua