The document discusses the importance of continuous testing in a DevOps environment, highlighting that its primary goal is to assess business risk coverage while integrating seamlessly into the software delivery pipeline. Wayne Ariola emphasizes the shift from traditional testing to continuous testing, which allows for actionable feedback at each stage of the delivery process and mitigates risks associated with software releases. Key testing practices such as risk-based prioritization and session-based exploratory testing are essential for enhancing automation and ensuring quality in rapidly changing development cycles.

1.
W5
DevOps
&
Testing
10/4/17
11:30
Traditional
Testing:
The
Silent
Killer
of
DevOps
Presented
by:
Wayne
Ariola
Tricentis
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Wayne
Ariola
Tricentis
Known
for
his
efforts
to
promote
continuous
testing,
Wayne
Ariola
has
contributed
to
the
design
of
many
innovative
technologies
and
received
several
patents
for
his
inventions.
A
recognized
leader
on
topics
such
as
service
virtualization,
SOA
and
API
quality,
software
quality
governance,
and
application
security,
Wayne
is
a
frequent
contributor
to
industry
publications
and
author
of
Continuous
Testing
for
IT
Leaders.
He
has
been
a
contributor
to
software
testing
for
fifteen
years
and
in
the
software
industry
for
more
than
twenty.

3. © 2016 by
© 2016 by .© 2016 by .
Wayne Ariola
Traditional Testing
The Silent Killer of DevOps
© 2016 by
Key Points for Today
h$ps://www.tricen1s.com/what-­‐is-­‐con1nuous-­‐tes1ng/
• Primary
goal
is
assessing
business
risk
coverage
• Establishes
a
safety
net
that
helps
the
team
protect
the
user
• Always
on,
stable
test
environment
• Seamlessly
integrated
into
the
soDware
delivery
pipeline
and
DevOps
toolchain
• Delivers
ac1onable
feedback
appropriate
for
each
stage
of
the
delivery
pipeline

4. © 2016 by 3
© 2016 by
Testing at the Speed of DevOps
What has Changed?1
2
3
Agenda
The three differences with Continuous Testing

5. © 2016 by 5
Until today, how would you rate the success
of test automation…
a success story
so so
a failure
?
© 2016 by
Rate the Success of Test Automation…
0
5
10
15
20
25
30
35
40
45
50
Q3-­‐2016
Q4-­‐2016
Q1-­‐2017
Q2-­‐2017
Success
So
So
Failure

6. © 2016 by
Test automation has not kept its promises.
© 2016 by
80%
Manual Testing
20%
Automated Testing
UI
API
Today
The reliance on
manual testing is the
top technical challenge
in app development.
World Quality Report, 2015/16
Test automation
requires developers.

7. © 2016 by
We didn’t do anything wrong,
but somehow, we lost.
‹‹ Stephen Elop, CEO Nokia, 2013 ››
© 2016 by
What has Changed?
§ People
§ Process
§ Technology
§ Tools

8. © 2016 by
Offshore Labor
38 kUSD/yr
60%
Traditional
Onshore Labor
100 kUSD/yr
Save cost
through
labor arbitrage
Satisfied?
People
© 2016 by
Process
Test as a time-boxed event
Fixed
Fixed
Plan
Develop
Build
Test
Release
Plan
Develop
Build
Test
Release
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Test is Continuous

9. © 2016 by
Technology and Disruption
• Security
• Safety
Cri1cal
• Financial
• FDA
• Automo1ve
• Aerospace
Compliance
• DevOps
• Agile
• Lean
• Bi-­‐Modal
Ini1a1ves
• Microservices
• Containers
• Cloud
• IoT
Technology
• Produc1on
Outage
• Applica1on
Slow
• User
Experience
• Connec1vity
Failure
• SaaS
• Mobile
• APIs
• IoT
Switching
Cost
© 2016 by
Top Tier Vendors
IBM Punts Development to HCL
http://www.theregister.co.uk/2016/10/03/
ibm_hcl_software_outsourcing_deal/
HPE Strategy Clear As Mud
http://it-tna.com/2016/09/28/hpe-strategy-clear-as-mud/
Tricen@s Delivering Con@nuous
Tes@ng at the Speed of DevOps
Challengers Leaders
VisionariesNiche Players
SmartBear Micro Focus
Test Plant
HPEIBM
ranorex
Microsoft
Worksoft
COMPLETENESS OF VISION
ABILITYTOEXECUTE
Tricentis

10. © 2016 by
Continuous Testing
Analysts Align
“Continuous Testing is
required for extreme
automation”
voke
“Continuous Delivery
Needs Continuous
Testing”
Forrester
“DevOps initiatives will
cause enterprises to
implement continuous
testing”
Gartner
”Continuous Testing has
obvious application in
agile environments.”
Bloor
© 2016 by
The Perfect Storm – Digital Disruption
Does
the
release
candidate
have
an
acceptable
level
of
risk?
Mitigate Business Risk
Con1nuous
Tes1ng
• Model
Based
Test
Automa1on
• Risk
Based
Tes1ng
• Exploratory
Tes1ng
• Mobile
Tes1ng
• Security
• Safety
Cri1cal
• FDA
• Automo1ve
• Aerospace
Compliance
• DevOps
• Agile
• Lean
• Bi-­‐Modal
Ini1a1ves
• Microservices
• Containers
• Cloud
• IoT
Technology
• Produc1on
Outage
• Applica1on
Slow
• User
Experience
• Connec1vity
Failure
• SaaS
• Mobile
• APIs
• IoT
Switching
Cost
• BI/DW
Tes1ng
• API
Tes1ng
• Test
Data
Management
• Service
Virtualiza1on

11. © 2016 by
© 2016 by .© 2016 by .
3 Key Differences
Continuous Testing v. Test Automation
© 2016 by
Continuous testing is the process of executing automated tests as
part of the software delivery pipeline in order to obtain feedback on the
business risks associated with a software release candidate as rapidly
as possible.
What is Continuous Testing

12. © 2016 by
Continuous Testing – What’s Different?
What’s
Different?
RISK
A
feedback
system
for
be$er
trade-­‐off
decisions
between
release
scope,
1me
and
quality
BREADTH
The
boundaries
of
an
applica1on
are
eroding.
Advanced
automa1on
is
required
for
end-­‐2-­‐end
tes1ng
TIME
Its
speed
because
1me-­‐to-­‐market
ma$ers
not
because
development
can
release
faster
© 2016 by
Survey on Risk
0%
5%
10%
15%
20%
25%
30%
35%
40%
Consultant Individual contributor Manager Senior manager
How Well Do Tests Assess Business Risks?
Very poorly Poorly Average Well Very well
2017
Survey
on
the
Future
Role
of
the
Tester
h$ps://www.surveymonkey.com/r/SWB7F3Q

13. © 2016 by
Ever-Growing Challenge
Technical Complexity
& Rate of Change
Years
Months
Months
Weeks
Weeks
Days
Days
Hours
Hours
Minutes
Delivery
Cycle Time
Perceived Disruption
Today
Introduce Continuous
Testing to meet demand
© 2016 by 22
18%(official) overall
test automation
degree
8%(official) enterprise
test automation
degree
Really ?
X
?

14. © 2016 by
80%
Manual Testing
20%
Automated Testing
UI
API
Today
The reliance on
manual testing is the
top technical challenge
in app development.
World Quality Report, 2015/16
Test automation
requires developers.
© 2016 by
Continuous Testing requires Automation
24
80%
Manual Testing
20%
Automated Testing
UI
API
Today Future
Exploratory Tests
Automated UI Tests
API Tests
Test Driven Service Virtualization
+85%
- key enabler for high automation rates -

15. © 2016 by
© 2016 by .© 2016 by .
Key Testing Practices for
Continuous Testing
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization
Test Case Design
UI- and API based test automation
Active Test Data Management
Test Driven Service Virtualization
CT Integration (CI, CD)
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!

16. © 2016 by
Introduce Session Based Exploratory Testing
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
© 2016 by 28
Is Exploratory Testing executed as part of
your testing process?
No
Performed
ad-­‐hoc
Yes,
executed
consistently
No,
but
under
considera1on

17. © 2016 by
Session Based Exploratory Testing
All testing starts with a Risk Based Prioritization
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization
Optimize your risk coverage: Test Case Design
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!

18. © 2016 by
Session Based Exploratory Testing
Risk Based Prioritization
Test Case Design
UI- and API based test automation
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization
Test Case Design
UI- and API based test automation
Prevent false+: Active Test Data Management
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!

19. © 2016 by
Session Based Exploratory Testing
Risk Based Prioritization
Test Case Design
UI- and API based test automation
Active Test Data Management
Provide stable infrastructure: Test Driven Service
Virtualization
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
© 2016 by
Session Based Exploratory Testing
Risk Based Prioritization
Test Case Design
UI- and API based test automation
Active Test Data Management
Test Driven Service Virtualization
CT Integration (CI, CD)
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!

20. © 2016 by
Session Based Exploratory Testing
Risk Based Prioritization
Test Case Design
UI- and API based test automation
Active Test Data Management
Test Driven Service Virtualization
CT Integration (CI, CD)
Automate
!
Explore
!
Manage
!
Integrate
!
Optimize
!
© 2016 by
Key Points of Continuous Testing
h$ps://www.tricen1s.com/what-­‐is-­‐con1nuous-­‐tes1ng/
• Primary
goal
is
assessing
business
risk
coverage
• Establishes
a
safety
net
that
helps
the
team
protect
the
user
• Always
on,
stable
test
environment
• Seamlessly
integrated
into
the
soDware
delivery
pipeline
and
DevOps
toolchain
• Delivers
ac1onable
feedback
appropriate
for
each
stage
of
the
delivery
pipeline

21. © 2016 by
Q&A
§ Continuous Testing vs Test Automation Paper
https://www.tricentis.com/resource-assets/continuous-testing-vs-test-
automation/
§ What is Continuous Testing (w/ 14 Key Points)
https://www.tricentis.com/what-is-continuous-testing/
§ Survey – The Role of "Tester" in the New Digital Age
https://www.surveymonkey.com/r/SWB7F3Q