Continuous delivery may be a natural for greenfield workloads, but how do you take an existing seven-year-old SaaS application and move from multi-month to weekly release cycles? Find out how our team — developers, QA, and ops — worked together to change our process and along the way changed their own ideas of what was possible.

1. CONTINUOUS DELIVERY:
HOW RIGHTSCALE RELEASES
WEEKLY

2. Speakers
• Tim Miller
• VP of Engineering
• Daniel Onorato
• Director of QA
• Robert Sulway
• Director of Development

3. Agenda
• Defining Continuous Delivery
• Motivations and Strategies
• Defining the Weekly Release
• Keys to Success
• Automating Is a “Must”
• The Dashboard
• (Surprising?) Benefits of Shorter Release Cycles

4. Your Roadmap to Application Agility
Code Build Integrate Test Release Deploy Operate
Agile Development
Continuous Integration
Continuous Delivery
Continuous Deployment
DevOps

5. POLLING QUESTIONS

6. Broker Cloud Services with RightScale
Self-Service Cloud Analytics
RightScale
Cloud Management
Design
Virtualized
Environments
Public
Clouds
IaaS+/PaaS
Services
Private
Clouds
Bare
Metal
Automate
Multi-Cloud Orchestration & Governance
OperateDeploy Report Optimize
5

7. • Complex code base, lives in over 200 repositories
• Some code going back 7 years
• Running on ~700 servers in Production
• 7-8 week release cycles
• Developers, QA & Ops all set in their ways, process gets layered on top of
process
Landscape
6

8. • Get planned features to market more quickly
• Respond to urgent requests
• Reduce WIP
• Master Branch Instability – not getting tested during
regression
• Increase quality, decrease escalations
• Details of features “fresh in minds” on release
• Mindset: My feature is going into production soon
• Think about deployment!
Motivations

9. • Spoiler Alert: These became benefits
• The biggest challenge was getting past “We can’t..”
• …because it would lower quality bar, QA always chasing the next
release
• ...roll out deployments in parallel, need to roll out serially
• …because it will take too much time/people for release process
The Challenges

10. • Go Big
• Teams would need to innovate
• Required to shed the old and do things differently
• Add in some constraints to make things more interesting
• Can’t lower the quality bar in any way
• Can’t add human resources to achieve the goal
• Can’t reduce velocity
• If you are going to do it weekly, can’t have downtime during releases
Strategies
9

11. • Process
• Relentless adherence to our “contracts”
• Contract for the definition of “done” for features
• Contract for the “ready to ship” weekly features collection
• Dashboards (Jira/Travis) to see daily status of “ready to ship”
• Environments
• Branching strategy
• Automation to stand up functional test environments for given features
• Staging environment to test the weekly features collection as a whole
Keys to Success

12. Keys to Success...continued
1111
• Automation
• Write once, run anywhere system level test automation (feature test,
stage integration, production)
• Matrix driven automation test execution for release deployment
validation in production with dashboard
• Automation to facilitate rolling production deployment
• Organization
• Embedded QA Engineering on scrum teams
• Tightly integrated Dev/Ops team

13. Contracts We Live By

14. What it is:
• A vehicle to release a collection of features on a weekly basis when
they are “done”
What it is not:
• A mandate to plan, design, implement, test and deploy a given feature
in a week’s time
Defining the Weekly Release

15. • For Libraries:
• Unit test coverage in (ci) for new feature, either measured or manually verified
to exist by QA
• For Apps:
• Unit test coverage running in (ci) for new feature(either measured or manually
verified to exist by QA)
• Functional test coverage in (ci) for new feature (either measured or manually
verified to exist by QA)
• Automated system test coverage run on a mini-rs and ready for merge into
master test collateral repo
• Note: we will require either the unit, functional or system test. The QA Eng/Devs will
determine required coverage at the time a feature is worked.
• Adequate end user documentation has been created.
• Product owner sign-off for feature
Definition of “done”

16. • For both Libraries and Apps:
• Downtime window is not required
• Compatible with a rolling system restart
• Documented migration instructions exist (if necessary)
• Documented deployment instructions exist (if necessary)
• Any required ServerTemplate updates must be coded and integrated on
a mini-rs
Definition of “done”

17. A weekly features collection is deemed “ready to ship” when:
• Verification the new feature(s) unit test(ci), functional test(ci) and/or
system level tests (staging) are passing on staging branch
• Verification of a full nightly regression run on staging on staging branch
with no impact on existing features and pass/fail trends
• All regression issues are resolved
Definition of “ready to ship”

18. Branching Strategy

19. Weekly View

20. Automation is a Must

21. Managing Many Environments

22. The RightScale DevOps Process
Mini-RightScale
Dev Environment
Staging
Environment
Production
Environment
Deployment
Automation
Deployment
Automation
Pull
code &
test
Deployment
Automation
Operations Automation
Pull
code &
test
Pull
code &
release
Monitor
alarm
react
Develop
Templates

23. • Write once, run anywhere test collateral
• Automation frameworks and collateral support running a given test in
each of our possible test targets without updating test collateral once
written. (mini-rs, staging, production).
• $ bundle exec rake SPEC=ui/functional/server_array_functional.rb
SETTINGS=mini-rs_93,nightly_settings SHARD=93
• $ bundle exec rake SPEC=ui/functional/server_array_functional.rb
SETTINGS=us_3,nightly_settings SHARD=3
• Automated feature tests add to collection of regression tests once
authored
• Tying test frameworks into matrix based invocation system.
Streamlining QA Automation

24. • Gemify (Ruby) test collateral repos to allow sharing among
various automation frameworks.
• Can speed up test execution in many cases
• Speeds up test suite implementation
• Example UI test using API test resources for setup
Sharing Test Collateral Repos

25. require 'rester'
require 'right_api_15’
describe "Verifies CRUD operations on the new ELB browser:", :crud => true do
before(:all) do
Session.login(:account_name => :test_account)
url = config(:LOCATION)
email = config(:EMAIL)
password = config(:PASSWORD)
RightApi15.default_login(login_info)
end
context "Setup" do
it "creates Network required for this test" do
@clouds.each do |cloud|
@api_created['network'] = NetworkManager::Network.create(:cloud => cloud, :cidr_block => '13.0.0.0/16’)
end
…
it "creates a load balancer using network/subnet with no name" do
@clouds.each do |cloud|
cloud_href = RightApi15::MetaData.get_cloud_href(:cloud_name => cloud)
network = @api_created['network'].select {|n| n.cloud_href == cloud_href}.first
subnet = @api_created['subnet'].select {|sn| sn.href.include?(cloud_href)}.first
security_group = @api_created['security_group'].select {|sg| sg.cloud_href == cloud_href}.first
LoadBalancer.create(:cloud => cloud,
:load_balancer_name => "#{@name}-api",
:VPC => { :vpc_option => network.resource_uid,
:subnets => subnet.resource_uid,

26. Matrix Based Automation Test Invocation

27. The Dashboard

28. Our “Ready to Ship” Dashboard

29. Regression and Feature Bug Status

30. Automated Test Trends

31. Regression Test Trends

32. CI Dashboard

33. Have Your Cake…

34. Benefits of shorter release cycles
3333
• Master code branch very stable at all times
• Fewer regression bugs found in integration cycle
• Automation test coverage guaranteed to grow and exist (definition of
done)
• Forced us to keep pushing on automation and tying systems together to
lessen the human resource tax associated with each release (work in
progress)
• Forced us to engineer feature updates/migrations such that they can be
performed in a rolling fashion with no system maintain window requiring
down time.
• Releases happen in the middle of the our normal work day as opposed to
Friday nights / Weekends. Everyone much happier with this.
• Features are deployed very close in time to there development. Details
still fresh in our minds

35. 34
Next Steps and Q&A
E-book:
Continuous Integration and Delivery in the Cloud: How We
Do It at RightScale: How RightScale Does It
www.rightscale.com/ci-cd-ebook