Software Quality in the Devops World 
The Impact of Continuous Delivery on Testing and 
Monitoring (and vice versa) 
© Copyright 2014 Pivotal. © Copyright 2014 Pivotal. AAllll rriigghhttss rreesseerrvveedd.. 
1 
Cornelia Davis (@cdavisafc) 
Director, Platform Engineering, Cloud Foundry
“Software is Eating the World” 
© Copyright 2014 Pivotal. All rights reserved. 
2
Software is Changing Industries 
© Copyright 2014 Pivotal. All rights reserved. 
3 
$3.5B valuation 
Financial Services 
$10B valuation 
Travel & Hospitality 
$17B valuation 
Transportation 
$3.2B Acquisition by Google 
Home Automation 
$20B valuation 
Entertainment 
$26B valuation 
Tesla--Automotive
You are either building a software business… 
! 
! 
Or losing to someone who is. 
© Copyright 2014 Pivotal. All rights reserved. 4
Amazon, a book store in Seattle, 
deploys code every 11 seconds 
© Copyright 2014 Pivotal. All rights reserved. 5
What is Continuous Delivery? 
© Copyright 2014 Pivotal. All rights reserved. 
6 
$ 
Business 
Development 
QA 
Operations 
Customer
What is Continuous Delivery? 
© Copyright 2014 Pivotal. All rights reserved. 
7 
$
Continuous Delivery - How? 
© Copyright 2014 Pivotal. All rights reserved. 8
Warner Music: Software Factories 
Warner Software Factory Platform 
• New applications and major updates 
- Before: 6 months, team of 10 developers 
- After: 6 weeks, same team 
- Speed/Agility: 400% faster on new platform 
- HR Hard Savings: $1.1M per application update 
delivered 
© Copyright 2014 Pivotal. All rights reserved. 9
Developers are Responsible for Quality 
• Test Driven Development 
• Integration Tests 
! 
! 
! 
! 
! 
• Automation 
• Unit Tests 
! 
Foo 
Foo Bar 
© Copyright 2014 Pivotal. All rights reserved. 10
“Operations is the secret sauce” 
© Copyright 2014 Pivotal. All rights reserved. 
11 
http://radar.oreilly.com/2007/10/operations-is-a-competitive-ad.html
Agile 
• Show screen of 
tracker 
© Copyright 2014 Pivotal. All rights reserved. 12
Design for Failure 
• “Sufficiently Paranoid” 
• Circuit breaker 
• http://martinfowler.com/bliki/CircuitBreaker.html 
• https://github.com/Netflix/Hystrix 
• SpringCloud: 
http://projects.spring.io/spring-cloud 
© Copyright 2014 Pivotal. All rights reserved. 13
http://projects.spring.io/spring-cloud 
© Copyright 2014 Pivotal. All rights reserved. 14
Agile Methodologies Meet Agile (3rd) Platform 
DEVELOPMENT 
Other dev apps 
App1 
John’s dev ver 
© Copyright 2014 Pivotal. All rights reserved. 
15 
DEVELOPMENT 
Other dev apps 
App1 
Mary’s dev ver 
Agile Development 
QA 
Other QA apps 
App1 
QA version 
QA 
PRODUCTION 
Other Prod apps 
App1 
Production version 
Production 
No code or configuration changes!
DEMO 
© Copyright 2014 Pivotal. All rights reserved. 16
Build Pipelines 
© Copyright 2014 Pivotal. All rights reserved. 
17
Pipelines - Example 
© Copyright 2014 Pivotal. All rights reserved. 
18 
CF Runtime 
CF Services 
Prod 
Services: Rel; Runtime: Rel 
Svc: CR a Rel; RT: CR a Rel 
A1 
Services: Mast; Runtime: Mast 
Svc: Mast a CR; RT: Mast a CR 
Dijon Tabasco 
Services: Dev; Runtime: CR Services: CR; Runtime: Dev 
Services: Dev a Mast Runtime: Dev a Mast
Distributed, Loosely-coupled Systems… 
… are very hard to reason about 
Foo Bar 
Baz 
© Copyright 2014 Pivotal. All rights reserved. 19
And Now You Go Into Prod 
Your software will be used by humans… 
• Smoke tests in prod 
• App architecture 
• Monitoring 
• Blue/Green 
Deployments 
© Copyright 2014 Pivotal. All rights reserved. 20
Twelve Factors 
• One Codebase/Many Deploys 
• Explicit Isolated 
Dependencies 
• Config via Environment 
• Attached Backing Services 
• Separate Build/Release/Run 
• Stateless Processes 
• Export Services via Port 
Bindings 
• Scale Out via Processes 
• Disposable Instances 
• Dev/Prod Parity 
• Logs == Event Streams 
• Admin Tasks == Processes 
http://12factor.net 
© Copyright 2014 Pivotal. All rights reserved. 21
DEMO 
© Copyright 2014 Pivotal. All rights reserved. 22
Blue/Green Deployments 
© Copyright 2014 Pivotal. All rights reserved. 
23
Blue/Green Deployments 
© Copyright 2014 Pivotal. All rights reserved. 
24
Blue/Green Deployments 
© Copyright 2014 Pivotal. All rights reserved. 
25
Blue/Green Deployments 
© Copyright 2014 Pivotal. All rights reserved. 
26
Blue/Green Deployments 
© Copyright 2014 Pivotal. All rights reserved. 
27
DEMO 
© Copyright 2014 Pivotal. All rights reserved. 28
Summary 
• Developers are responsible for quality 
• Tests 
• Techniques for all stages of the application lifecycle 
• Design for failure 
• Self-healing 
• Zero-downtime upgrades 
© Copyright 2014 Pivotal. All rights reserved. 29
Thank You! 
@cdavisafc 
© Copyright 2014 Pivotal. All rights reserved. 30
A NEW PLATFORM FOR A NEW ERA

Software Quality in the Devops World: The Impact of Continuous Delivery on Testing and Monitoring (and vice versa)

  • 1.
    Software Quality inthe Devops World The Impact of Continuous Delivery on Testing and Monitoring (and vice versa) © Copyright 2014 Pivotal. © Copyright 2014 Pivotal. AAllll rriigghhttss rreesseerrvveedd.. 1 Cornelia Davis (@cdavisafc) Director, Platform Engineering, Cloud Foundry
  • 2.
    “Software is Eatingthe World” © Copyright 2014 Pivotal. All rights reserved. 2
  • 3.
    Software is ChangingIndustries © Copyright 2014 Pivotal. All rights reserved. 3 $3.5B valuation Financial Services $10B valuation Travel & Hospitality $17B valuation Transportation $3.2B Acquisition by Google Home Automation $20B valuation Entertainment $26B valuation Tesla--Automotive
  • 4.
    You are eitherbuilding a software business… ! ! Or losing to someone who is. © Copyright 2014 Pivotal. All rights reserved. 4
  • 5.
    Amazon, a bookstore in Seattle, deploys code every 11 seconds © Copyright 2014 Pivotal. All rights reserved. 5
  • 6.
    What is ContinuousDelivery? © Copyright 2014 Pivotal. All rights reserved. 6 $ Business Development QA Operations Customer
  • 7.
    What is ContinuousDelivery? © Copyright 2014 Pivotal. All rights reserved. 7 $
  • 8.
    Continuous Delivery -How? © Copyright 2014 Pivotal. All rights reserved. 8
  • 9.
    Warner Music: SoftwareFactories Warner Software Factory Platform • New applications and major updates - Before: 6 months, team of 10 developers - After: 6 weeks, same team - Speed/Agility: 400% faster on new platform - HR Hard Savings: $1.1M per application update delivered © Copyright 2014 Pivotal. All rights reserved. 9
  • 10.
    Developers are Responsiblefor Quality • Test Driven Development • Integration Tests ! ! ! ! ! • Automation • Unit Tests ! Foo Foo Bar © Copyright 2014 Pivotal. All rights reserved. 10
  • 11.
    “Operations is thesecret sauce” © Copyright 2014 Pivotal. All rights reserved. 11 http://radar.oreilly.com/2007/10/operations-is-a-competitive-ad.html
  • 12.
    Agile • Showscreen of tracker © Copyright 2014 Pivotal. All rights reserved. 12
  • 13.
    Design for Failure • “Sufficiently Paranoid” • Circuit breaker • http://martinfowler.com/bliki/CircuitBreaker.html • https://github.com/Netflix/Hystrix • SpringCloud: http://projects.spring.io/spring-cloud © Copyright 2014 Pivotal. All rights reserved. 13
  • 14.
    http://projects.spring.io/spring-cloud © Copyright2014 Pivotal. All rights reserved. 14
  • 15.
    Agile Methodologies MeetAgile (3rd) Platform DEVELOPMENT Other dev apps App1 John’s dev ver © Copyright 2014 Pivotal. All rights reserved. 15 DEVELOPMENT Other dev apps App1 Mary’s dev ver Agile Development QA Other QA apps App1 QA version QA PRODUCTION Other Prod apps App1 Production version Production No code or configuration changes!
  • 16.
    DEMO © Copyright2014 Pivotal. All rights reserved. 16
  • 17.
    Build Pipelines ©Copyright 2014 Pivotal. All rights reserved. 17
  • 18.
    Pipelines - Example © Copyright 2014 Pivotal. All rights reserved. 18 CF Runtime CF Services Prod Services: Rel; Runtime: Rel Svc: CR a Rel; RT: CR a Rel A1 Services: Mast; Runtime: Mast Svc: Mast a CR; RT: Mast a CR Dijon Tabasco Services: Dev; Runtime: CR Services: CR; Runtime: Dev Services: Dev a Mast Runtime: Dev a Mast
  • 19.
    Distributed, Loosely-coupled Systems… … are very hard to reason about Foo Bar Baz © Copyright 2014 Pivotal. All rights reserved. 19
  • 20.
    And Now YouGo Into Prod Your software will be used by humans… • Smoke tests in prod • App architecture • Monitoring • Blue/Green Deployments © Copyright 2014 Pivotal. All rights reserved. 20
  • 21.
    Twelve Factors •One Codebase/Many Deploys • Explicit Isolated Dependencies • Config via Environment • Attached Backing Services • Separate Build/Release/Run • Stateless Processes • Export Services via Port Bindings • Scale Out via Processes • Disposable Instances • Dev/Prod Parity • Logs == Event Streams • Admin Tasks == Processes http://12factor.net © Copyright 2014 Pivotal. All rights reserved. 21
  • 22.
    DEMO © Copyright2014 Pivotal. All rights reserved. 22
  • 23.
    Blue/Green Deployments ©Copyright 2014 Pivotal. All rights reserved. 23
  • 24.
    Blue/Green Deployments ©Copyright 2014 Pivotal. All rights reserved. 24
  • 25.
    Blue/Green Deployments ©Copyright 2014 Pivotal. All rights reserved. 25
  • 26.
    Blue/Green Deployments ©Copyright 2014 Pivotal. All rights reserved. 26
  • 27.
    Blue/Green Deployments ©Copyright 2014 Pivotal. All rights reserved. 27
  • 28.
    DEMO © Copyright2014 Pivotal. All rights reserved. 28
  • 29.
    Summary • Developersare responsible for quality • Tests • Techniques for all stages of the application lifecycle • Design for failure • Self-healing • Zero-downtime upgrades © Copyright 2014 Pivotal. All rights reserved. 29
  • 30.
    Thank You! @cdavisafc © Copyright 2014 Pivotal. All rights reserved. 30
  • 31.
    A NEW PLATFORMFOR A NEW ERA