Agile Saturday Lecture - 16-Feb-2014

1. © All rights reserved
Want to use slides from
this presentation?
Ask for permission:
ethanram@gmail.com
Making the Agile Leap
to Continuous Deployment
Ethan Ram / 15 Feb 2014
V1.1

2. Ethan Ram - Bio
• Chief architect of Videobet
• Leading Videobet’s R&D and OPs
teams Agility efforts
• As VP R&D of GameGround
transformed the R&D to run in
Continuous Deployment
environment
• Hacker in soul and a Windows
Internals hobbyist
2/15/2014
© All rights reserved
Find me here:
• ethnram@gmail.com
• LinkedIn
• The Null Terminator
blog
2

3. Videobet
• A Playtech division
• Innovative Gaming
Platform Provider
– Established 9 years ago
– 150 employees in Tallinn
and Kiev
– Operate in 9 countries
– 60% of UK market
• R&D
– Front-end teams (C++)
– Server teams (Java)
– Game development teams
2/15/2014
© All rights reserved
3

4. Agenda
#1
inciple
Pr
anifesto
Agile M
y
to satisf
ty i s
a nd
st priori
ur highe r through early le
O
e
ab
e custom elivery of valu
th
us d
ontinuo
c
e.
softwar
2/15/2014
•
•
•
•
•
•
•
© All rights reserved
Why Continuous Deployment
Setting the goals
Deployment technics
Development technics
Making the leap
Some unexpected results
Q&A
4

5. Goal: Continuous Deployment
Wix “From Jan ‘13 to Jun ’13 we did 1500
At SlideShare, instead of big launches, we
Deployments, 470 A/B Tests,
We have been doing continuous deployment release code all the time... it reduces risk.
200 Feature Toggles”
at Plex since our inception in 1995. We Since we are putting out small changes at a
Yoav Avrahami
formalized the process with our own “deploy”time, if it breaks we know what it is and can
button in 2003, and hit the 25k deployment
roll back.
mark less than two years thereafter. We
Rashmi
currently average between 50-100
deployments per day to our massive online
Flickr "10+ Deploys Per Day"
ERP system.
WordPress.com - we’ve averaged about 16 Conover
Steve of the exciting things… about Digg is
Jerry Foster
One
product releases a day, every day for theOne of the exciting things… about Digg is
last
continuous deployment --when developers fix
continuous deployment when developers fix
four and a half years… Everyone in our
athat or add new feature, there's no need
bug
company has access to a deploy button abug or add aanew feature, there's no need
to wait for scheduled release… the
to
releases the latest checked in code to about wait for aascheduled release… the
turnaround time for aachange drops
turnaround time for change drops
400 production servers in our web tier in less
dramatically.
dramatically.
than 30 seconds (across 3 data centers).
Andrew Bayer
Andrew Bayer
Toni Schneider
2/15/2014
© All rights reserved
5

6. Major Release
2/15/2014
© All rights reserved
6

7. No. I Can’t Do That 
• “We’re starting this campaign next week and
I need the product to do X”
• “We need to stop showing this feature to
non-US customers”
• “I think this idea would work. When can we
start A/B testing live?”
• “I need to know if users are clicking on this
link and if they see this error message”
• “Forget about the other features. We have to
go live by the end of the month!”
2/15/2014
© All rights reserved
7

8. 2/15/2014
Continuous Deployment:
Yes We Can!
© All rights reserved
8

9. How to Streamline Deployment?
• Speed-up Dev-ops
• Lower risk of deploying something broken
• Streamline feature implementation
2/15/2014
© All rights reserved
???
9

10. Monitoring & Rollback
• Must haves – real-time
– Monitor that features are working,
not just servers
– Monitor the business results
– Visualize and put it on the big
screen
– One-click rollback
• Rollback: the ultimate risk
mitigation
– Prepare to fail!
• Will the rollback work?
– Rollback drills
2/15/2014
© All rights reserved
10

11. Deployment & Servers Setup
Go from – 8 hrs downtime on upgrade
• Physical machines
• IP based network topology
• Layered servers: web-apps,
backend, cache, db
• Feature servers
• Each team has a different
deployment setup and scripts
• Complex database schema upgrades
To – 8 mins deploy, no
downtime
• Virtual machines
• DNS based network topology
• All servers on each VM
– DB too!!!
• Same VM setup on all
environments
– With different config
– With different data-set
• One deployment script run on
dev machine >> production
– Target tag and environment name
as input
I T’S EASY!
2/15/2014
• Versioned mini schema
migrations
© All rights reserved
11

12. DB Versioning With Flyway DB
Flyway DB
makes
you Agile!
• Mini schema migrations kept with source
code
– Applying migrations added between v1 to v2 is
easy
– Roll-back of migrations supported built-in
• Each developer has her own DB
• Merging team-branch migrations is even
simpler than merging code
• Integrative with Maven/Ant/scripts and Java
APIs
– Server can upgrade/lower DB schema by itself
• Data for unit-testing can be added and
removed on the fly
2/15/2014
© All rights reserved
12

13. Staging on Production
• Get real users to use new version ASAP
– Call it Beta, Early Availability, Version Trial, Testbed
• Deploy new version to part of the servers
and direct it traffic of part of the user-base
–
–
–
–
–
Deploy to a subset of servers
Run manual tests & automated tests
Direct some traffic to new version
Monitor! Measure!
Deploy to the rest after some time
• Not only for consumer-based internet
products!
2/15/2014
© All rights reserved
13

14. Quality Assurance
• Integration Test Automation
– Focus first on covering main data flows
– More main features, less details
• Part of the Continuous Integration
– Build > Unit-test > Integration Tests
– All tests must pass to allow build promotion
…
not so
easy…
• Run the same tests on all environments
– Developers’ & production environments too
• Don’t give-up on manual testing
• ‘Test frenzy’ as version signs-off
– One hour session, structured, lead by QA
– ALL employees join. The secretary too.
2/15/2014
© All rights reserved
14

15. R&D Kanban Board: CD Workflow
BACKLOG
Backlog
SELECTED
Selected
• Bugs
• Improvements.
• Daily
prioritization
2/15/2014
IN
In PROGRESS
Progress
•Feature owner • Coding
• Unit testing
assigned
•Task
• Integration
testing
Breakdown
•User stories
•Design/gfx
•Test plan
RESOLVED
Resolved
VERIFIED
Verified
• Need
verification
• Drop build to
QA
• Automated
integration
testing
• Manual testing
TESTBED
Testbed
• Verified
• Ready to be
installed on
TestBed
• Optional step:
install on
TestBed
• Regression
Testing
• Enforce WIP limits
• One owner for each column
• Feature owner responsible for feature end-to-end…
• QA plan and test completion (no need for bugs!)
• Unit testing and integration testing
• Use DBA/HTML/FLASH and QA resources as needed
• Test it on testbed/production
© All rights reserved
DONE
Done
• Running on
production
• Done / Not
relevant
16

16. Developing on Trunk
2/15/2014
© All rights reserved
17

17. Use feature toggle flag
2/15/2014
© All rights reserved
18

18. Developing on the Trunk
• Use feature toggle flags
• Each dev has a full system - update, compile, run it all, test-it
• Write backwards compatible code
– Assume some servers/services do not have the new feature/code
yet
– DB fields may be missing
– Mark older APIs with [deprecated/obsolete]
• Dynamic Configuration
– Central config, dynamically loaded
– Plan for no app/system restarts
Thor n
Fr ee?!?
• Schema changes are added to code as DB versioning scripts
• Larger features that require a branch are rare!
• Education: Do not break the trunk!
2/15/2014
© All rights reserved
19

19. Product: From Revolution to Evolution
V2.2
V2.3 / R403
R417
2/15/2014
© All rights reserved
20

20. Leaping into CD
• Initiative must come from high mgmt.
– It’s a large investment… and risky
money
t ime
• It’s not all or nothing >>>
– First milestone: weekly release
– Set a deadline and commit: “we start
releasing once a week right after the next
major release in September”
• It’s a project to deliver
– Multi-function task-force: devs, QA,
deployment
– Full time!
– Plan a lot – detailed task list etc.
• Training to all teams: process, tools
• Decide on a CD weekly schedule
– Tuesday afternoon is best time for a release
• Make results visible early!
– And celebrate success!
2/15/2014
© All rights reserved
21

21. The first CD release felt a bit like this…
..Just Go For It!
2/15/2014
© All rights reserved
22

22. Some Unexpected Results
• … the sky did not fall on us 
• Excellent process transparency
–
–
–
–
R&D stopped fighting with Product Mgmt.
Team leaders stopped blaming each other
Devs thought it’s much more fun
Everyone got more involved
• We managed to attract much better engineers
• R&D managers thought its awesome!
2/15/2014
© All rights reserved
23

23. Want to use slides from this presentation?
Ask for permission: ethanram@gmail.com
Q&A
Thanks for contributing!
Yoav Avrahami @ Wix
Mike Mcgarr @ Excella
Thanks!
2/15/2014
© All rights reserved
24