As agile development practices mature and DevOps principles begin to infiltrate our corporate cultures, organizations realize the distinct opportunity to accelerate software delivery.

1. Paraso&
Proprietary
and
Confiden1al
1
5/25/16
Evolving from
Automated to Continuous Testing
for Agile and DevOps
Rix.Groenboom@Parasoft.com
Solution Architect

2. Paraso&
Proprietary
and
Confiden1al
2
Paraso&
Proprietary
and
Confiden1al
2
To
start
with…
§ Created:
BC
196
§ Found:
1799
§ London:
1801
§ Bri1sh
Museum:
1802
§ Transla1on:
1822
§ Egyp1an
hieroglyphs
§ Demo1c
script
§ Ancient
Greek

3. Paraso&
Proprietary
and
Confiden1al
3
Paraso&
Proprietary
and
Confiden1al
3
Anno
2016,
you
cannot
keep
things
for
yourself…

4. Paraso&
Proprietary
and
Confiden1al
4
Paraso&
Proprietary
and
Confiden1al
4
Impact
of
Faulty
So&ware
-­‐5.70%
-­‐1.9B

5. Paraso&
Proprietary
and
Confiden1al
5
Paraso&
Proprietary
and
Confiden1al
5
Anno
2016:
So&ware
Failures
=
Headlines
-$2.55 Bn
-4.08%
The day of the announcement
companies lost an average
of shareholder value
So&ware
failures
make
headline
news—
eroding
customer
confidence,
shareholder
value
and
brand
equity

6. Paraso&
Proprietary
and
Confiden1al
6
Paraso&
Proprietary
and
Confiden1al
6
Escala1ng
Cost
of
Failure:
Public
companies

7. Paraso&
Proprietary
and
Confiden1al
7
Paraso&
Proprietary
and
Confiden1al
7
Re-­‐Inven1ng
so&ware
development
The
evolu1on
of
the
so#ware
development
lifecycle
(SDLC)
over
the
past
3
years
is
forcing
the
transforma1on
of
so&ware
tes1ng
Transforma)on
Business
Impact
• Business
risk
associated
with
so1ware
failure
at
an
all
4me
high
• Public
companies
lose
an
average
of
$2.5
Bn
in
market
cap
on
the
day
of
the
announcement
• “Digital
Industrial
Revolu4on”
makes
all
a
so1ware
company
• Trend
toward
Dev/Test
insourcing
as
compe44ve
advantage
is
via
so1ware
• Strong
adop4on
of
“agile”
development
prac4ces
• Forces
more
technical
tes4ng,
early.
Agile
is
forcing
“shi1-­‐le1”
• DevOps
is
applying
lean
prac4ces
to
SDLC
• DevOps
is
exposing
20
years
of
process
barriers
protected
by
silos
and
culture
• Cloud
has
disrupted
tradi4onal
SDLC
and
is
driving
down
costs
• Switching
costs
associated
with
applica4ons
at
an
all
4me
low

8. Paraso&
Proprietary
and
Confiden1al
8
Paraso&
Proprietary
and
Confiden1al
8
How
to
manage
the
Disrup1on
?
• Security
• Safety
Cri1cal
• FDA
• Automo1ve
• Aerospace
Compliance
• DevOps
• Agile
• Lean
• Bi-­‐Modal
Ini1a1ves
• Microservices
• Containers
• Cloud
• IoT
Technology
• Produc1on
Outage
• Applica1on
Slow
• User
Experience
• Connec1vity
Failure
Does
the
release
candidate
have
an
acceptable
level
of
risk?
Mitigate Business Risk

9. Paraso&
Proprietary
and
Confiden1al
9
Automa1on
expands
through
whole
SDLC
Con1nuous
Delivery
Con1nuous
Integra1on
Delivery
Team
Version
Control
Build
&
Unit
Test
Automated
Acceptance
Tests
User
Acceptance
Tests
Release
Check In Trigger
Trigger Trigger
TriggerTrigger
Approval Approval
Check In
Check In
Feedback
Feedback
Con1nuous
Deployment
Produc1on
Business

10. Paraso&
Proprietary
and
Confiden1al
10
Paraso&
Proprietary
and
Confiden1al
10
Fixed
Fixed
Impact
of
Test
in
the
Evolving
SDLC
(1)
Plan
Develop
Build
Test
Release
Test
as
a
1me-­‐boxed
event
is
the
prevailing
method
“Are
we
done
tes1ng?”

11. Paraso&
Proprietary
and
Confiden1al
11
Paraso&
Proprietary
and
Confiden1al
11
Impact
of
Test
in
the
Evolving
SDLC
(2)
Plan
Develop
Build
Test
Release
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
“Does
the
release
candidate
have
an
acceptable
level
of
risk?”

12. Paraso&
Proprietary
and
Confiden1al
12
Paraso&
Proprietary
and
Confiden1al
12
Impact
of
Test
in
the
Evolving
SDLC
(3)
Speeding
up
the
conveyor
belt
does
not
yield
be>er
results…
…
you
need
Quality
@
Speed
The
new
so&ware
paradigm
requires
the
business
to
define
and
establish
automated
quality
gates
that
con1nuously
mi1gate
the
risk
of
business
failure

13. Paraso&
Proprietary
and
Confiden1al
13
Paraso&
Proprietary
and
Confiden1al
13
How
to
achieve
this
Quality
@
Speed
?
Take
the
top-­‐down
view:
§ Business
risks
drive
quality
ac1vi1es
§ Team
collaborates
on
the
defini1on
of
risk
§ Baseline
and
threshold
criteria
measured
and
monitored
§ Results
of
quality
ac1vi1es
are
visible
and
translated
for
all
levels
of
business
This
approach
will
allow
you
to
go
from
“automated”
to
“con1nuous”…

14. Paraso&
Proprietary
and
Confiden1al
14
Paraso&
Proprietary
and
Confiden1al
14
From
Automated
to
Con1nuous
(1)

15. Paraso&
Proprietary
and
Confiden1al
15
Paraso&
Proprietary
and
Confiden1al
15
From
Automated
to
Con1nuous
(2)
The
steps
required
for
moving
from
“Automated”
to
“Con1nuous”
§ Business
expecta1ons
clearly
defined
§ Business
risks
iden1fied
§ Per
applica1on,
team,
release
§ Defects
automa1cally
priori1zed
versus
business
drivers
§ No
defect
le&
behind
§ Mi1gate
business
risks
§ Tes1ng
in
‘complete’
test
environments
con1nuously
§ Leveraging
simula1on
–
Service
Virtualiza1on
§ Protec1ng
the
real-­‐user
experience
business
process
§ Feedback
loop
for
defect
preven1on
§ Learn
by
capturing
in
paoerns
§ Defect
preven1on
prac1ces
enhanced

16. Paraso&
Proprietary
and
Confiden1al
16
Paraso&
Proprietary
and
Confiden1al
16
Con1nuous
Tes1ng:
Quality
Product
&
Process
Automated
Tes)ng
provides
the
“sensors”
(detec)on
processes)
§ Fixing
defects
is
the
expected
outcome
§ Iden1fies
opportuni1es
for
defect
preven1on
§ Prevents
a
release
candidate
from
going
to
the
next
stage
of
the
release
cycle
Con)nuous
Tes)ng
addresses
the
broader
challenge
§ Fix
the
product
§ Op1mize
the
process
(improve
the
effec1veness
of
the
“sensors”)
Con1nuously
refine
the
process
in
light
of
business
risks
as
well
as
so&ware
quality
Con)nuous
Tes)ng
addresses
the
broader
challenge
§ Fix
the
product
§ Op1mize
the
process
(improve
the
effec1veness
of
the
“sensors”)
§ Con1nuously
refine
the
process
in
light
of
business
risks
as
well
as
so&ware
quality

17. Paraso&
Proprietary
and
Confiden1al
17
Paraso&
Proprietary
and
Confiden1al
17
Con1nuous
Tes1ng
Con)nuous
Tes)ng

18. Paraso&
Proprietary
and
Confiden1al
18
Paraso&
Proprietary
and
Confiden1al
18
Transforming
the
Iden1fica1on
of
Risk
From
To
Causal
Observa1ons
Probabilis1c
Defect
Documenta1on
Most
Likely
Causes
Only
Structured
Data
Structured
and
Unstructured
Filters
Business
Policies
Ad
Hoc
Automated
Tool
Dependent
Tool
Independent
Business
Objectives

19. Paraso&
Proprietary
and
Confiden1al
19
Paraso&
Proprietary
and
Confiden1al
19
Con1nuous
Tes1ng
mi1gates
business
risk
Why
Con)nuous
Tes)ng?
Systema1c,
centralized
decision
making
to
factor
business
risk
into
SDLC
Safety
net
to
allow
developers
to
bring
innova1ons
to
market
faster
A
feedback
system
for
beoer
trade-­‐off
decisions
between
release
scope,
1me
and
quality

20. Paraso&
Proprietary
and
Confiden1al
20
Paraso&
Proprietary
and
Confiden1al
20
Con1nuous
Tes1ng
builds
on
automa1on
Con)nuous
Tes)ng
Not
more
automa1on,
beoer
automa1on
Business
and
development
leaders
agree
upon
and
automate
the
mi1ga1on
of
risks
throughout
the
SDLC
Both
business
and
technical
managers
make
beoer
trade-­‐off
decisions
between
release
scope,
1me
and
quality
Organiza1ons
evolve
from
just
measuring
product
quality
to
measuring
the
process
effec1veness

21. Paraso&
Proprietary
and
Confiden1al
21
Paraso&
Proprietary
and
Confiden1al
21
Con1nuous
Tes1ng
monitors
the
process
Con)nuous
Tes)ng
Con)nuous
Deployment
Con)nuous
Delivery
Con)nuous
Release
Con)nuous
Integra)on
§ Con1nuous
Tes1ng
focuses
on
a
comprehensive
view
of
business
risk
§ Con1nuous
Tes1ng
provides
a
workflow
for
priori1zing
fixes
and
quality
improvements,
and
preven1ng
recurrence,
according
to
business
risk
and
business
priori1es
Con)nuous
Tes)ng
enables
con)nuous
improvement

22. Paraso&
Proprietary
and
Confiden1al
22
Paraso&
Proprietary
and
Confiden1al
22
Feature
Release
Business
Stakeholder
Developer
Customer
Support
Con1nuous
Tes1ng
in
ac1on:
Use
Quality
Gates
SoKware
Feature
Con1nuous
measurements
mean
con1nuous
refinement
of
the
process
Real-­‐1me
feedback
from
Objec1ve
assessment
and
go/no
go
Defects
are
eliminated
at
the
point
that
they
are
easiest
to
fix
Business
Stakeholder
Developer
Customer
Support
Quality
gates:
Organiza1ons
can
automa1cally
promote
so&ware
through
quality
gates
when
business
expecta1ons
have
been
met

23. Paraso&
Proprietary
and
Confiden1al
23
Paraso&
Proprietary
and
Confiden1al
23
Summary:
The
elements
of
Con1nuous
Tes1ng
Con)nuous
Tes)ng
re-­‐posi)ons
the
ques)on
from
“are
you
done
tes)ng?”
to
“is
the
level
of
risk
understood
and
accepted?”
Con)nuous
Tes)ng
Risk
Assessment
Policy
Analysis
Requirements
Traceability
Test
Environment
Access
Test
Op)miza)on
Advanced
Analysis
Ensure
access
to
complete
test
environments
Automate
defect
preven1on
and
Policy
measurement
Expand
test
coverage
and
measure
test
effec1veness
Define
ac1onable
prac1ces
Connect
func1onal
with
non-­‐func1onal
requirements
Process
improvement
opportuni1es

24. Paraso&
Proprietary
and
Confiden1al
24
Paraso&
Proprietary
and
Confiden1al
24
How
we
can
help
?
Does
the
release
candidate
have
an
acceptable
level
of
risk?
Mitigate Business Risk
Con1nuous
Tes1ng
• Test
Configura1on
and
Simula1on
• Test
Environment
Management
• Defect
Detec1on
• Change
Control
Development
Tes1ng
• Risk
Defini1on
• Process
Control
• Defect
Preven1on
• Visibility
§ Methodology:
§ Research:
§ SW
to
support
development
&
con1nuous
tes1ng:

25. Paraso&
Proprietary
and
Confiden1al
25
Paraso&
Proprietary
and
Confiden1al
25
Let
us
help
you
to
decipher
your
SDLC…

26. Paraso&
Proprietary
and
Confiden1al
26
Paraso&
Proprietary
and
Confiden1al
26
Paraso&
Con1nuous
Tes1ng
Paraso&
develops
automated
so&ware
quality
solu1ons
that
prevent
and
detect
risks
associated
with
applica1on
failure
Emphasizes
business
expecta1ons
to
mi1gate
business
risks
Enables
development
tes1ng
Promotes
early
detec1on
and
remedia1on
Facilitates
integra1on
tes1ng
Provides
ubiquitous
test
environment
access
with
Service
Virtualiza1on
Assists
in
end-­‐to-­‐end
system
tes1ng
The
ParasoK
Con)nuous
Tes)ng