This document discusses the evolution of the DevOps Quality Management Office (QMO). It outlines the vision of continuous business-driven testing to reduce the time between development and operations. Key aspects of the DevOps-driven testing approach include continuous integration and delivery, lean techniques, standardization, test optimization, and establishing a hybrid test organization. The document also compares traditional vs DevOps testing approaches and provides examples of DevOps testing success levers. It proposes that the QMO can advise on developing a DevOps strategy and roadmap to improve throughput, availability, and time to market.

1. Evolution of the DevOps
Quality Management Office
May 2015

2. 2Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
How do we position testing ? DevOps Quality Management Office(QMO)
Vision: Continuous business driven testing to shrink development to operations time at continuous quality
DevOps driven Testing SLA’s
Shift left and right with Continuous Test
Four C’s – Continuous Integrate, deliver, deploy & monitor
Lean techniques & Agile methods
Standardization
Prepackaged workbenches
Hybrid test organization
QE,TDD,BDD
Test & Services virtualization
Environment provisioning & Test Data management
Test optimization
OATS & Risk based testing
Development Test Operations

3. 3Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
What are our levers ?
Continuous quality with continuous delivery
>99%
Application
availability
>30%
Throughput
improvement
98%
Prevention of
high severity defects
>30%
Time-to-
Market
Success Levers
AdvancedIntermediateBasic
70+%
50%
40+%
70+%
100%
100%
60%
60+%
50%
30%
40%
40%
70%
50%
30%
70%
Lifecycle Automation
Continuous Test
Lean and Agile adoption
Test virtualization
Test optimization
Standardization
Continuous monitoring
QE leverage
Functional, non functional, Data, UAT
Build, integrate and deliver
Kanban, Scrum, Safe, XP, SaFe
Services and Environment automation
Risk based testing and OA leverage
Model based work benches
Availability & sentiment analysis
TDD, BDD, Unit automation

4. 4Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
What is the difference in traditional versus DevOps testing?
Process
Skills and People
Lean techniques and Agile
Continuous Test – Build, Integrate, Deliver, Deploy
Time to Market – Reduce time between development and operations
Standardization – Model-based testing
Optimization – Risk-based testing, OATS
Environment specialists
Services automation specialist
QE
Emerging Technology Adoption
Test virtualization – Services & Environment
Sentimental analysis and Continuous monitoring
Cloud adoption
DevOps Traditional Testing

5. 5Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
What Speed is Right for an Test Organization?
It depends*…..
And it is a journey of continuous improvement…
*By Technology Fast Faster Fastest
 Back-end apps & legacy
 Front-end, client server, ERP
 Web and mobile apps
Methodology
 Waterfall and basic agile
 Enterprise Agile
Test methods
 Black box, Build and Integrate
 Continuous test, white box, service
virtualization
 Continuous deploy, environment set up
automated
Env virtualization, cloud technology
Release frequency

6. 6Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
How do we decide? QMO advisory at a strategic level
DevOps Strategy assessment to determine objectives (Throughput, Availability, Time to market)
Step
1
DevOps QMO assessment to identify bottlenecks in testing mapping to delivery bottlenecks
Step
2
Leverage Kanban, VoC, Value Stream Mapping, DSM, Risk based technique
Step
3
Identify no more than 5 test themes (Examples include test environment, automation, unit testing, service
virtualization)
Step
4
Build roadmaps accompanied by a business case (Roadmaps include organization, process and tools
transformation)
Step
5

7. 7Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
With Agile test transformation as a strong lever
 What is it? Benchmarks 17 key process areas in Agile testing.
 How to use? Use to create agile test transformation roadmap. Process and automation tool kit reduces
implementation time to the tune of 30%
 When to use? For quick assessments during transformation roadmap creation. Periodic assessments for
ongoing improvement in delivery.
Current State Future State
Aware of Following Excelling

8. 8Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
And opportunities to Continuously test
Continuous
Integration Sprint
Product
Backlog
Priorities Spring
Backlog
Product
(Particularly usable)
Acceptance
Tests
Deployment
(production)
To Do… To Do…
Continuous
Delivery
Acceptance
Tests
Deployment
(production)
Sprint
Product
Backlog
Priorities Spring
Backlog
To Do…
Product
(Particularly usable)
Continuous
Deployment
Acceptance
Tests
Deployment
(production)
Sprint
Product
Backlog
Priorities Spring
Backlog
Product
(Particularly usable)

9. 9Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
Building on our automation test types – Test Left and Right
Requirements Architecture & Design Development Testing Release
ShiftLeft
Test Driven Development
Automation: 80 – 100%
Unit Tests
Automation: 80 – 100%
Behavior driven development
Automation: 50 – 70%
Service Virtualization
Automation 40 – 60%
Non functional tests
Automation: 80-100%
Performance Tests
Automation; 100%Regression test
Automation 60 – 80%
Test Management/Defect
Management
Automation: 70 – 100%
Sprint Acceptance Tests
Automation 40 – 50%
Exploratory tests
Automation: 30 – 40%
Functional Tests
Automation: 40 – 50%
ShiftRight
UAT Automation
40 – 60%
Continuous Monitoring
Auto: 40 – 60%
Continuous Integration
Auto: 80 – 100%
Continuous Delivery
Auto:10-60%
Environment virtualization
Auto: 40 – 60%

10. 10Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
With tools to continuously deliver
Assess Agile
QBP
Implement
Agile Toolkit
Manage QMO
Tools SLA’s
People
Perforce
TFS Code
SVN/GIT
Eclipse
ClearCase
Subversion
VStudio
IntelliJ
Docker
Nexus
Puppet
Chef
Third Party & Open
Source
Accurev
Mercurial
Hudsonl
Jenkins
TFS Build
Bamboo
El. Cloud
TestNG
NUnit
MsTest
Cobertura
JaCoCo
SAP SCM
Selenium
NCover
Ant Helion
OpenStack
Amazon
AnsibleMaven
vCenter
JUnit
Netbeans Salt
..
ProductionStagingQADev
HPAgileManager
Continuous Assessment
Continuous Delivery
Continuous Testing
Continuous Integration
HP Integrations to 3rd party & Open Source
HP ALM, Mobile Center, LR/Storm Runner, UFT, Fortify, NV, SV
HP CODAR
HP AppPulse, Real User Monitoring, SiteScope, Diagnostics
HP CSA
HP DevOps

11. 11Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
Sample Use Case
HP Agile
Manager
HP Storm
Runner Load
HP Performance /HP
LoadRunner
HP
ALM
HP Unified Functional
testing
Continuous Testing, Delivery and Assessment
HP CODAR HP BSMHP AppPulse MobileHP Cloud Service
Automation
HP AppPulse
Designer
Orchestrator
Provisioning engine
OOTB integrations
App Deployer
Configuration models
1 2 3

12. 12Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
Measuring DevOps driven QMO SLA’s
DevOps SLA’s QMO SLA’s Testing KPI’s
 Increase in number of
releases
 Application availability
>99%
 Reduce Mean time to
recover
 Drive operational
efficiencies
 Reduction in test cycle time
• Application Quality
• Speed and quality
 Operational efficiencies
 Reduce cycle time
 Zero critical defects in
production
 Phase containment –Shift
left
 Reduce testing during
mean time to recover
 Reduction in testing costs
release on release

13. 13Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
With a clear vision of success
Test Execution Productivity
Velocity
Defect leakage
Severity 1 and 2 defects
Features delivered per release
Effort Variance
Schedule Variance
Number of Releases / Year
Reduction in Defects and Support Calls
Release Predictability Measure
Commitment Reliability
Sprint /
Iteration
Release
Engagement
Level
Metrics
Business
KPI
Sample Dashboards
Effort Manual VS Automation Number of releases YoY Defect Leakage to Sprint Number of support ticket YoY
-1064
462
2026.4
-2,000
-1,000
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
1 2 3 4 5
Effort
manual
Effort
Automation
Difference
0
2
4
6
8
10
12
14
1 2 3 4 5
No of releases
0
10
20
30
40
50
60
70
80
1 2 3 4 5
Defects
0
200
400
600
800
1000
1200
1400
1600
1 2 3 4 5
No of Tickets

14. 14Copyright © 2015 Capgemini. All Rights Reserved
Evolution of the DevOps Quality Management Office | May 2015
Putting it together – Sample roadmap
Deployments in Months Deployments in Weeks
 Automated functional testing
 Automated non-functional testing: performance, security
 Automated test lifecycle management
 Agile for programs
 TDD and BDD
 Service virtualization
 Continuous delivery
 Enterprise Agile
 Overall: Much higher degree of automation
 Continuous deployment
 Full environment virtualization
 Automation of UAT and exploratory testing
FAST
FASTER
FASTEST

15. The information contained in this presentation is proprietary and confidential.
Copyright © 2015 Capgemini. All rights reserved.
Rightshore® is a trademark belonging to Capgemini.
www.capgemini.com
About Capgemini
With more than 145,000 people in over 40 countries, Capgemini is one of the
world's foremost providers of consulting, technology and outsourcing services. The
Group reported 2014 global revenues of EUR 10.573 billion.
Together with its clients, Capgemini creates and delivers business and technology
solutions that fit their needs and drive the results they want. A deeply multicultural
organization, Capgemini has developed its own way of working, the Collaborative
Business Experience™, and draws on Rightshore®, its worldwide delivery model.
Learn more about us at www.capgemini.com.