CWIN17 New-York / Drive continuous delivery with continous testing

1
Drive Continuous Delivery With
Continuous Testing
Jeff Bradley
Regional CTO Continuous Testing
CA Technologies
September 2017

© 2017 CA. Confidential. All rights reserved.
Eliminating the barriers between
ideas and outcomes?
How Good are You at …

Your Software Factory Must be an Accelerator.
But is it an Inhibitor?
ProductConcept
EFFICIENT, HIGH-QUALITY,
PRODUCTIVE PIPELINE
MORE CUSTOMER
VALUE
BUSINESS
ALIGNMENT

© 2017 CA. Confidential. All rights reserved. 4
THE
MODERN
SOFTWARE
FACTORY
The capabilities you need,
from mobile to mainframe,
to deliver the experiences
customers want.
Agile DevOps Security
MAXIMIZE APPLICATION PERFORMANCE
MAKE SECURITY A COMPETITIVE ADVANTAGE
CREATE AN AGILE BUSINESS
BUILD BETTER APPS, FASTER

Continuous Testing
How do you build better apps, faster?

Why Many Companies Think They’re Agile…
They moved some Dev projects from waterfall to agile
They’re having daily standups
They have a scrum master
Product owner is part of the team
They are all talking and walking agile…
And are talking about Continuous Delivery
BUT…

Testing is STILL a Bottleneck… Even in DevOps Shops
http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf
…of delays were occurring at
the Test/QA stage of the
cycle.
“Where are the main hold-ups in the software production process?”
32% 16% 22%
63%
21% 23% 30%
63%A 2017 survey of self-
proclaimed DevOps
practitioners found that …
7

Why Quality is So Hard to Achieve?
Clear evidence that you are unlikely to enjoy
the full benefits of continuous delivery unless you
modernize your testing tools and
practices.
§ 75% highlight continuous testing as critical or
important
§ 76% see Continuous Testing as critical or
highly desirable to ‘shift-left’ testing activity from
downstream back into development
Traditional testing tools
are often too complex
and expensive to use
Adoption of
continuous testing is
much easier with
the right tools
Continuous testing
requires a major
shift in mindset
and culture
agree or strongly agree
63%
79%
72%
“Continuous Testing as a Digital Business Enabler”, 2017
“
8

“Continuous Testing as a Digital Business Enabler”, 2017
Tangible Evidence
of the incremental value continuous testing represents for “Leaders”
more likely to have succeeded in
left-shifting testing activity to
requirements phase
more confident in quality of
output
more confident in speed
of delivery
more likely to experience
rapid revenue growth
more likely to deliver 10x
faster iterations
more likely to reduce defects
by 50%+
2.5x 1.5x 1.9x
2.6x 2.4x
3.9x
SPEED
QUALITY
ROI
9

Challenges to Achieving Continuous Delivery & Testing
Requirements
Development
QA / Testing
Operations
Release
70% of all
testing is still
manual4
63% of testers admit they
can’t test across all the different
devices and OS versions5
50% of time
spent looking for
test data6
79% of teams face prohibitive
restrictions, time limits or access
fees on needed 3rd party services3
X X X
64% of total defect cost originate in the requirements analysis and design phase1.
50% of developers time is spent
fixing and finding defects2
?
80% of teams experience delays in development and QA
due to unavailable dependencies3
?
57% are dissatisfied with the time it takes to deploy new features7
!
!
Greater than 50% production outages from
configuration management
!
Ave. cost of downtime per minute = $4,700
53% percent of visits are abandoned if a
mobile site takes more than three seconds to load
IDEA
Security 50% more time spent on security
defects in lower-performing teams8
30% of teams only security scan
once per year9
70xrequired manual pen
test scan cost vs. automation10
X X

What is Continuous Testing?
First need to understand what it’s not!
It’s not… Test Automation
It’s not… Continuous Integration
It’s not… Unit Testing
It’s not… done only in the Test environment
It’s not… a QA-led activity
11

What is Continuous Testing?
The practice of testing
across every activity in the SDLC to uncover and fix
unexpected behaviors as soon as they are injected

Key Elements of Continuous Testing
Automatically generate
test cases from
requirements
1
2
Simulate test
environment
3
Access test data on-
demand & ensure PII
compliance
4
Start multi-layer /
backend “requests
& response” testing
prior to UI
5
Democratize
performance testing
Integrate open source
testing tools6
7
Ensure comprehensive,
cloud-based API testing
8
Built-in automated
application security
testing
9
Promote artifacts when tests
pass by orchestrating &
automating the pipeline
10
Harness application insight
across the SDLC to
improve user experience

Most Customers Cannot Do
Continuous Testing Today,
But They Aspire To Get There….

If This is Your Current SDLC
You are waterfall
15
Dev Test
X WEEKS X WEEKS X WEEKS
Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations

If This is Your Current State
16
Sprint 1 Sprint 2 Sprint 3
X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS
Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations
You are water-SCRUM-fall

If This is Your Current State
17
Sprint 1 Sprint 2 Sprint 3 Sprint N
Sprint 2 Sprint N
Sprint 3 Sprint N
User
Stories
Dev and
Manual
Testing
Automated
Regression
Testing
X WEEKS X WEEKS X WEEKS X WEEKS
Sprint 1
Sprint 1 Sprint 2
Release
X WEEKS X WEEKS
Sprint 3
You are not “really” agile

What Testing at the Speed of Agile Should Look Like!
18
No Sprint-lag effect
Release in end of Sprint
In-Sprint Test Automation
In-Sprint everything to achieve “Continuous Testing”
Sprint 1 Sprint 2 Sprint 3 Sprint N
Sprint 2 Sprint N
Sprint 3 Sprint N
User
Stories
Dev and
Exploratory
Manual
Testing
Test Automated
Scripting &
Exectution
X WEEKS X WEEKS X WEEKS X WEEKS
Sprint 1
Sprint 1 Sprint 2
Release
Sprint 3
ReleaseReleaseRelease
HIGHLIGHTS

TESTING
REALITY
Dev Tools
TOOLSET
TEST “Ice
Cream Cone”
UI Tools
Dev Tools
TOOLSET TEST
PYRAMID
API
Unit
UI
API Tools
UI Tools
TESTING GOAL
API Tools
API
Unit
UI
94%
3%
3%
5%
45%
50%
Non-functional: static, performance,
security, etc…
*Actual Customer Example of Their Testing Strategy

Why are Traditional Testing Activities a Bottleneck?
Enable
Interfaces
(real or virtual)
Create Detailed
Test Cases
Execute
Test Cases
Identify Interfaces
(internal and 3P)
Automate Tests
(usually Regression)
Create Defects Fix Defects
Define an Idea or
Customer Needs
Write
Requirements
Create
Requirements
Traceability
Matrix
Read
Requirements
Manual
Processes
Find/Create
Test Data
Testing hasn’t changed in the past 30 years
Identify Test Data
Requirements

But Continuous Testing Changes the Game…
Enable
Interfaces
(real or virtual)
Create Detailed
Test Cases
Execute
Test Cases
Identify Interfaces
(internal and 3P)
Automate Tests
(usually Regression)
Create Defects Fix Defects
Define an Idea or
Customer Needs
MODEL
Requirements
Create
Requirements
Traceability
Matrix
Read
Requirements
Find/Create
Test Data
Identify Test Data
Requirements
NOW ALL AUTOMATED

Simple View of Continuous Testing
REQUIREMENTS
MODELING
TEST
EXECUTION
MONITORING
Continuous Integration Continuous Testing Continuous
Delivery
Automated
Deployment
WhatHow
• Flowchart Diagram
TEST
DESIGN
DEVELOPMENT

MONITORING
Delivery
Continuous
Deployment
WhatHow
• Test-Driven Development
• Acceptance Test-Driven Development
REQUIREMENTS
MODELING
TEST
EXECUTIONTEST
DESIGN
DEVELOPMENT
• Test Automation Scripts
• Test Data

MONITORING
Delivery
Continuous
Deployment
WhatHow
• Test Environments
• Test Automation Engines
• Test Frameworks
REQUIREMENTS
MODELING TEST
EXECUTION
TEST
DESIGN
DEVELOPMENT

MONITORING
Delivery
Continuous
Deployment
WhatHow
• Analyze
• Learn
• Update Model
REQUIREMENTS
MODELING
TEST
EXECUTION
TEST
DESIGN
DEVELOPMENT

4 Pillars of Continuous Testing
Can the application
code flow across
environments without
manual intervention?
Did developers create
the correct features?
Are users
perceiving value in the
application delivered?
Did developers create
the application code
correctly?
Pipeline
Automation
Code
Quality
Customer
Experience
Application
Quality
Code Correct? Automated Flow?
User Value? Features Correct?

Building On
Top Of The
Pillars
DEV
Validate Build
Promotion
Criteria
Model
Requirements
(Story Level)
Assign Definition
of Done for
Backlog Items
Sprint
Planning
Gold QA
Build
TESTProvision QA
Environment
and Deploy
Build
Validate Build
Configuration
Validate
Build
Promotion
Criteria
Gold QA Build
Received
Auto
Generate
Virtual
Services
Definition and
Test Data
Definition
Auto
Generate
Test Cases
(manual and
automated)
Enhance
Modeled
Requirements
(Feature Level)
Gold Stage
Build
STAGE
Start Full
Application
Monitoring
Validate
Build
Promotion
Criteria
Gold Stage Build
Received Auto Generate
Virtual Services
Definition
and Test Data
Definition
Auto Generate
Test Cases
(manual and
automated)
Enhance Modeled
Requirements
(Epic Level)
Gold
Production
Build
PROD
Validate
Approvals
for
Production
Promotion
Gold Prod
Build
Received
Auto Generate
Prod
Environment
Configuration
(for A/B and
Canary)
Auto
Generate
Deployment
Paths
(A/B and
Canary)
Model
A/B Tests
and
Canary
Builds
(Pick
User Base)
CX
Hypothesis
Back to
Sprint
Backlog
CONTINUOUS TESTING

Translate
Requirements Into
Visual Models
DEV
Validate All
Backlog Items
Have a Definition
of Done
Validate All Backlog
Items Are Prioritized
Regression Test
Promotion Criteria
Validation Process
Test Code Build
Process
Test Trigger for
BVTs
Create, maintain
and Run BVTs
Regression Test
Environment
Provisioning
Process
Test Automated
Code Scan
Triggers
Test Virtual
Service and
Data
Provisioning
Process
Test-Driven
Development
Development Environment
Validate Build
Promotion Criteria
Model
Requirements
(Story Level)
Assign Definition
of Done for
Backlog Items
Sprint
Planning
Create Virtual
Services and
Automated
Synthetic Data
Generation
Automated
Code Scans
(Quality/Security),
Code Commit
Prioritize
Backlog
(Ideas and
Hypothesis)
Gold QA
Build
App and API
Code Development
and Automated
Unit Tests
Update
Configuration
and Provision
Environment
Create and
Deploy Build.
Run Automated
BVT - Build
Verification
Tests
Continuous
Testing Tasks
Continuous
Testing Tasks
DEV
Test Automated
Code Scan
Triggers
Test Automated
Code Scan Triggers

Regression Test
Promotion Criteria
Validation Process
Add Layers Into Visual Models
(Automation,
Test Data, Virtual Services)
Regression Test
Application
Monitoring
Process
Configuration
Regression
Test Processes
for:
§ Environment
Provisioning
§ Build
Deployment
Regression Test
Build
Configuration
Test Environment
Validate Build
Promotion Criteria
Gold QA Build
Received
Auto Generate
Virtual Services
Definition and
Test Data
Definition
Auto Generate
Test Cases
(manual and
automated)
Validate Build
Configuration
Provision QA
Environment
and Deploy
Build
Provision
Virtual Services
and Test Data
Start Full
Application
Monitoring
Enhance
Modeled
Requirements
(Feature Level)
Gold Stage
Build
Intelligent Test
Automation:
§ Capture and
Analyze Test
Result Data
§ Correlate
Results to Past
Executions
§ Formulate
Hypothesis for
Improvement
Automated
Test
Execution TEST
Continuous
Testing Tasks
Regression Test
Provisioning Processes for:
§ Virtual Services
§ Test Data
TEST
Regression Test
Provisioning Processes
for:
§Virtual Services
§Test Data

Translate Requirements
Into Visual Models
Regression Test
Promotion Criteria
Validation Process
Regression
Test
Application
Monitoring
Process
Configuration
Regression Test
Processes for:
§ Environment
Provisioning
§ Build
Deployment
Regression
Test Build
Configuratio
n
Staging Environment
Validate Build
Promotion Criteria
Gold Stage Build
Received
Auto Generate
Virtual Services
Definition and Test
Data Definition
Auto Generate
Test Cases
(manual and
automated)
Validate Build
Configuration
Provision
Stage
Environment
and Deploy
Build
Provision
Virtual Services
and Test Data
(external only)
Start Full
Application
Monitoring
Enhance
Modeled
Requirements
(Epic Level)
Gold Prod
Build
Intelligent Test
Automation:
§ Capture and
Analyze Test
Result Data
§ Correlate
Results to Past
Executions
§ Formulate
Hypothesis for
Improvement
Automated
Test
Execution
(Functional/
Performance)
STAGE
Continuous
Testing Tasks
Regression Test
Provisioning Processes for:
§ Virtual Services
§ Test Data
STAGE
Regression Test
Provisioning Processes
for:
§Virtual Services
§Test Data

Regression
Test
Application
Monitoring
Process
Configuration
Regression
Test Build
Deployment
Process to
Production
Regression
Test Processes
for:
§ Environment
Provisioning
Production Environment
Validate Approvals for
Production Promotion
Gold Prod Build
Received
Auto Generate
Prod Environment
Configuration (for
A/B and Canary)
Auto Generate
Deployment Paths
(A/B and Canary) Validate Build
Configuration
Update Prod
Environment
Configuration
(A/B and
Canary
Builds
as needed)
Deploy
Build to Production
(automatically)
Start Full
Application
Monitoring
(IT and CX)
Model
A/B Tests and
Canary Builds
(Pick User Base)
CX Hypothesis
Back to Sprint
Backlog
§ Test the Data
Collection
and Analysis
Process
§ Test the New
Parameters
Automatically
Added to the
Hypothesis
Formulation
Cognitive
Process
Analyze
CX Metrics and
Formulate
Initial
Hypothesis
Translate A/B Test
Requirements and Canary Build
User Base Into Visual Models
Test Process to
Obtain Go Live
Approvals
PROD
Continuous
Testing Tasks
Regression
Test Build
Configuration
PROD
Regression
Test Build
Configuration

End-to-End Integrated Continuous Testing from CA
Automatically generate
test automation scripts
from requirements
1
2
Simulate test
environment
3
Access test data on-
demand & ensure PII
compliance
4
Start multi-layer /
backend “requests
& response” testing
prior to UI
5
Democratize
performance testing
Integrate open source
testing tools6
7
Ensure comprehensive,
cloud-based API testing
8
Built-in automated
application security
testing
9
Promote artifacts when tests
pass by orchestrating &
automating the pipeline
10
Harness application insight
across the SDLC to
improve user experience
Continuous
Delivery
Director

CA’s Track Record Helping our Customers
IDEA
Requirements
Development
QA / Testing
Operations
Release
Reduce time needed
to generate synthetic
test data by
10X
Faster in delivering test
cases & automation1
50%
Time reduction to
replicate defects3
50%
reduction in app
dev/test cycle2
23%
time to market
improvement3
90%
availability for
testing platforms3
1 x 500
throughput
increase in testing4
35%
cost reduction with 3rd
party access fees3
Decrease downtime
from release failures
by
Reduce application
deployment time almost
94%
Increase weekly
deployments by
300%
Up to 50%greater
app performance 5
Early load testing to prep
for an actual
25 Million Users6
3 Month
Improvement in new automation
And to full in-sprint automation
50%
80%
Security
1 in 3
Fortune 100 firms
choose Veracode
3 Trillion
Lines of code scanned
32M
Security flaws
fixed7
12,000
Developer remediation
coaching sessions

THANK YOU

GM Financial has operations in more than 20
countries, providing auto finance solutions to
approximately 16,000 dealers worldwide
Connect and automate the entire app
pipeline from development, to test, to
production:
CA Service Virtualization
CA Test Data Manager
CA Continuous Delivery Director
CA Agile Requirements Designer
CA Release Automation
SOLUTION:
CHALLENGE:
Cycle time too long to support
exponential expansion of funded
loans and comprehensive audit and
compliance requirements
• 5x faster loan funding
• 98% reduction in
deployment time from 1-2 hours
manually to 2-3 minutes
• 80% increase in test
coverage
• 90% time savings in
regression testing from 93 to 9
hours
RESULTS:
“We anticipate growth in
the next year from 1,000
automated deployments a
month to 4,000– 5,000 a
month.”
Matt McComas
AVP of Critical App Infrastructure,
GM Financial
Continuous Testing

Mission: Establish an end-to-end best
practices testing solution supported by
complimentary tools that integrates,
automates, and stabilizes the application
delivery pipeline
• CA Release Automation
• CA Service Virtualization
• CA Test Data Manager
• CA Continuous Delivery Director
SOLUTION:
CHALLENGE:
• Limited test coverage
• Manual functional testing
• Unstable test environments
• No test data management
• No automated release process
• $500,000 in cost savings
• 38,000 hours saved in manual testing in just 6 months
• 85% increase in test coverage
• 90% time savings in regression testing from 40 hours to 1
Continuous Testing & Delivery
113 116
56
123
80 87
150
223
251 238
15 7 6 5
0
50
100
150
200
250
300
COUNT OF SEVERITY 1, 2, & 3 INCIDENTS
37

• CA Agile Requirements Designer
• CA Test Data Manager
• CA Application Test
• CA Release Automation
SOLUTION:
CHALLENGE:
Struggling to keep pace with the
demands of the business. Needed to
deliver applications and updates more
quickly, of higher quality and with
fewer defects.
Continuous Testing & Delivery
“CA Technologies offered a compelling solution with
its integrated tools that provide seamless, end-
to-end test management…. CA Technologies
solutions were able to manage all the technologies
used across our applications, which other solutions
were unable to do.”
“With CA Technologies solutions,
we can identify & rectify
defects more quickly and easily
before they are released into
production. This will help us
improve the speed and quality of
software delivery.”
38

CWIN17 New-York / Drive continuous delivery with continous testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to CWIN17 New-York / Drive continuous delivery with continous testing

Similar to CWIN17 New-York / Drive continuous delivery with continous testing (20)

More from Capgemini

More from Capgemini (20)

Recently uploaded

Recently uploaded (17)

CWIN17 New-York / Drive continuous delivery with continous testing