This document features a presentation by Wayne Ariola from Parasoft discussing the significant costs of software defects and how they can impact businesses, including market capitalization and customer confidence. It emphasizes the need for a shift from traditional time-boxed testing to continuous testing and the adoption of service virtualization to improve testing efficiency and reduce dependencies. Additionally, the presentation covers strategic insights into managing quality in the software development lifecycle (SDLC) and the necessity for integrating testing early in the development process.

1. T13
Test Management
5/7/2015 1:30:00 PM
What Do Defects Really Cost?
Much More Than You Think
Presented by:
Wayne Ariola
Parasoft
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Wayne Ariola
Parasoft
Chief strategy officer Wayne Ariola leads the development and execution of Parasoft’s long-
term strategy. Wayne has contributed to the design of core Parasoft technologies and has been
awarded several patents for his inventions. He is a recognized leader on service virtualization,
API quality, quality policy governance, and application security. Wayne brings more than twenty
years of strategic consulting experience in the technology and software development
industries. Prior to joining Parasoft, he helped co-found a start-up and consulted in the strategic
change practice at PricewaterhouseCoopers.

3. What Do Defects Really Cost?
Much More Than You Think
What Do Defects Really Cost?
Much More Than You Think
Parasoft Proprietary and Confidential 1
2015-04-30
Much More Than You Think
Wayne Ariola
Chief Strategy Officer - Parasoft
Much More Than You Think
Wayne Ariola
Chief Strategy Officer - Parasoft

4. Session AbstractSession Abstract
Software quality has traditionally been a “time-boxed”
exercise and we can’t possibly expect that accelerating the
SDLC via DevOps will yield better results from a quality
perspective. Automating the build-test-release-deploy
process will undeniably get the application into the hands of
users faster. However, if you haven’t also re-engineered the
Parasoft Proprietary and Confidential 2Parasoft Proprietary and Confidential 2
users faster. However, if you haven’t also re-engineered the
processes by which the application is built and tested, you’re
quite likely flooding your users with more defects at a more
rapid rate.

5. Re-Invention…Of TimeRe-Invention…Of Time
Parasoft Proprietary and Confidential 3Parasoft Proprietary and Confidential 3
http://www.scienceworldreport.com/articles/22225/20150210/cool-optical-lattice-
clocks-1000-times-more-precise-cesium-atomic.htm

6. Software Drives InnovationSoftware Drives Innovation
Parasoft Proprietary and Confidential 4Parasoft Proprietary and Confidential 4
The Cost of
Quality
associated with
software has
shifted
dramatically
Software has
shifted from
process enabler to
business
differentiator
SDLC Speed will
be the difference
between a first
mover and a
follower
Switching costs
associated with
software are
dramatically lower

7. Software Failures = HeadlinesSoftware Failures = Headlines
Financial Airlines/Aero Government Media Internet/Tel
Parasoft Proprietary and Confidential 5
Automotive Technology Retail

8. Software Failures = Headlines 2014Software Failures = Headlines 2014
The day of the
announcement companies
lost an average
Software failures make headline news—
eroding customer confidence, shareholder value and brand equity
Parasoft Proprietary and Confidential 6Parasoft Proprietary and Confidential 6
-$2.3 Bn
-3.75%
lost an average
of shareholder value

9. Escalating Cost of Failure: PublicEscalating Cost of Failure: Public
-1.50%
-1.00%
-0.50%
0.00%
2012 2013 2014 2015 (est.)
Impact on Market Capitalization
Parasoft Proprietary and Confidential 7Parasoft Proprietary and Confidential 7
-3.12%
-3.37%
-3.75%
-4.18%y = -0.0036x - 0.0272
-4.50%
-4.00%
-3.50%
-3.00%
-2.50%
-2.00%

10. Software Failures = HeadlinesSoftware Failures = Headlines
5,500,000
3,490,000
4,000,000
5,000,000
6,000,000
Market does not forget— news about failure increase and
average of 167%
Parasoft Proprietary and Confidential 8Parasoft Proprietary and Confidential 8
821,000
1,080,000
451,000
1,400,000
896,000
1,550,000
650,000
3,490,000
2,240,000
0
1,000,000
2,000,000
3,000,000
4,000,000
Cloud Services Airline Bank Software Retail
1st Event 2nd Event

11. The Cost of Software Failure - SonyThe Cost of Software Failure - Sony
MarketValue
-22% -33% -30%-19% -11% -12%
Cumulative Loss = $18B
Parasoft Proprietary and Confidential 9Parasoft Proprietary and Confidential 9
http://www.reuters.com/article/2011/04/26/us-sony-stoldendata-idUSTRE73P6WB20110426
http://www.digitaltrends.com/gaming/sony-fined-almost-400000-for-2011-playstation-security-breach/
MarketValue
Event 15 Days 30 Days

12. Impact of Test in the Evolving SDLCImpact of Test in the Evolving SDLC
Plan Develop Build Test Release
Test as a time-boxed event is the prevailing method
Parasoft Proprietary and Confidential 10Parasoft Proprietary and Confidential 10
Fixed Fixed

13. Testing is a BottleneckTesting is a Bottleneck
Business Views
Fully Automated Provisioning
Data Analytics
Organic SystemsSoftware Quality Processes
Parasoft Proprietary and Confidential 11Parasoft Proprietary and Confidential 11
Basic Tasks
Skilled Tasks
Process Oriented
End-to-End Tasks
Data Analytics
Automation is Incremental
Efficiency

14. Impact of Test in the Evolving SDLCImpact of Test in the Evolving SDLC
Plan Develop
Build
ReleaseBuild
Build
Build
Build
Build
Build
Build
Build
Build
BuildParasoft Proprietary and Confidential 12Parasoft Proprietary and Confidential 12
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
More than automation – “continuous” tests must assess business risk

15. From Automated to Continuous TestingFrom Automated to Continuous Testing
Parasoft Proprietary and Confidential 13Parasoft Proprietary and Confidential 13

16. Getting from “Automated” to “Continuous”Getting from “Automated” to “Continuous”
We have to ask a new question
No Longer: “Are we done testing?”
Parasoft Proprietary and Confidential 14Parasoft Proprietary and Confidential 14
“Does the release candidate have an
acceptable level of risk?”

17. Getting from “Automated” to “Continuous”Getting from “Automated” to “Continuous”
There is a big leap moving from “Automated” to “Continuous”
Business expectations clearly defined
Business risks identified
Per application, team, release
Defects automatically prioritized versus business drivers
No defect left behind
Parasoft Proprietary and Confidential 15Parasoft Proprietary and Confidential 15
No defect left behind
Mitigate business risks
Clear ownership and workflow for remediation
Collaborative risk reduction
Collapse the remediation cycle time
Feedback loop for defect prevention
Pattern definition
Defect prevention practices enhanced

18. Transforming the Identification of RiskTransforming the Identification of Risk
From To
Business
Objectives
Parasoft Proprietary and Confidential 16Parasoft Proprietary and Confidential 16
From To
Causal Observations Probabilistic
Defect Documentation Most Likely Causes
Only Structured Data Structured and Unstructured
Filters Business Policies
Ad Hoc Automated
Tool Dependent Tool Independent

19. Next Generation TestingNext Generation Testing
Parasoft Proprietary and Confidential 17Parasoft Proprietary and Confidential 17

20. Process IntelligenceProcess Intelligence
Process Intelligence aggregates “siloed” information throughout the SDLC
and identifies business risks early and automatically
Correlate disparate data across:
Dependent teams
Dependent components
Infrastructure systems
Parasoft Proprietary and Confidential 18Parasoft Proprietary and Confidential 18
Infrastructure systems
Testing practices and tools
Apply advanced statistical analysis
From pre-build patterns
From constructed
With Open domain APIs
Advantages
Defect remediation
Exception-based notifications

21. The Next Generation Software Quality ‘System’The Next Generation Software Quality ‘System’
Continuous Assessment of RiskContinuous Verification of Requirements
Development Operations
Development
(E1)
Production
Test
(E2)
Integration
(E3)
Pre-Production
(E4 – En)
Release Candidate
Parasoft Proprietary and Confidential 19Parasoft Proprietary and Confidential 19
Test Environment Management - Provisioning and Elastic Cloud Scale
Test Optimization
Environment Simulation (SV)
Auto Test Construction – Synthetic
APM – Real Time Data
Development Tests

22. “Shift-Left”“Shift-Left”
Parasoft Proprietary and Confidential 20Parasoft Proprietary and Confidential 20

23. “Shift-Left”“Shift-Left”
Parasoft Proprietary and Confidential 21Parasoft Proprietary and Confidential 21

24. “Shift-Left”“Shift-Left”
Parasoft Proprietary and Confidential 22Parasoft Proprietary and Confidential 22

25. 30
Environment Access?Environment Access?
The average number of
dependent application
associated with the
System Under Test (SUT)
Parasoft Proprietary and Confidential 23Parasoft Proprietary and Confidential 23
30
BUT, Dev/QA only
have “trustworthy”
access to SIXSIXSIXSIX
of the applications
302928272625
242322212019
181716151413
121110987
654321

26. Test Environment ConfigurationTest Environment Configuration
The average percentage
of time spent
configuring the test
environment.
30%
Parasoft Proprietary and Confidential 24Parasoft Proprietary and Confidential 24
50%
The average percentage
of the test plan able
to complete once
configured.

27. Survey Results…Survey Results…
When testing with a
dependentdependentdependentdependent application
only need access to a
small percentage
of the applications
20%
Parasoft Proprietary and Confidential 25Parasoft Proprietary and Confidential 25
of the applications
functionality

28. Service Virtualization delivers a
simulated dev / test environment
allowing an organization to test
Service VirtualizationService Virtualization
Parasoft Proprietary and Confidential 26Parasoft Proprietary and Confidential 26
allowing an organization to test
anytime or anywhere

29. So, What are you Virtualizing?So, What are you Virtualizing?
AUT
Parasoft Proprietary and Confidential 27Parasoft Proprietary and Confidential 27
• Mainframe
• SOA / Web services
• Java
• .NET languages
• XML
• WSDL
• WADL
• UDDI
• WSIL
• SOAP
• PoX (Plain XML)
• REST
• JSON
• BPEL
• Web Applications
• Mobile Interfaces
• AJAX
• JSP
• JavaScript
• HTML
• CSS
• WS-*
• Standards
• MTOM(XOP) / MIME / DIME
• OAuth
• TCP/IP
• webMethods Broker
• webMethods IS
• HL7
• FTP
• HTTP 1.0
• HTTP 1.1
• HTTPS
• JMS
• IBM WebSphere MQ
• Sonic MQ
• RMI
• EJB
• SMTP
• Tibco Rendezvous
• .NET WCF
• ISO 8583
• EDI
• SAP
• custom

30. Stubs Vs Service VirtualizationStubs Vs Service Virtualization
Stubs
Inside-out approach that disassociates a test case
with a dependent systems
Brittle
Limited reuse
Service Virtualization provides a complete environment for developing and
testing versus complex, dependent systems
Parasoft Proprietary and Confidential 28Parasoft Proprietary and Confidential 28
Limited reuse
Static response
Service Virtualization
Creates an environment in which to run “rich” test scenarios
Virtualized assets represent real system behavior
All virtualized-assets are reusable
Virtualized assets are programmable and extensible

31. Complexity is a Barrier to InnovationComplexity is a Barrier to Innovation
Scheduled Access
Parasoft Proprietary and Confidential 29Parasoft Proprietary and Confidential 29
3rd Party System
Evolving Component
Mainframe

32. Access to Dev / Test SystemsAccess to Dev / Test Systems
Configuration
Complex
ERP
Mainframe
External
Database
Cloud
App
Parasoft Proprietary and Confidential 30Parasoft Proprietary and Confidential 30
Test Access
Simple Difficult
Configuration
Internal
Service
Internal
Application 3rd Party
Service
External
Application
Internal
Database
App
Message
Queue
ESB
Web
Server

33. Service Virtualization & DependenciesService Virtualization & Dependencies
Service Virtualization
Configuration
Access
Dependent applications difficult
Scheduling conflicts
High access fees
Geo-political boundaries
3rd party or partner applications
Configuration
Complex
Parasoft Proprietary and Confidential 31Parasoft Proprietary and Confidential 31
Virtual Lab
Configuration
Access
Configuration
Complex to configure
No control
Limited variability
Consumes test time
DifficultSimple
Easy

34. The Total Cost of System AccessThe Total Cost of System Access
Service Virtualization dramatically reduces the cost of delivering
a dev / test environment
Hardware Savings
Staged Application Instance
Cost
Only manage the functionality
you need
Only manage data you need
Low-Risk “disposable instance”
that can be accessed at anytime
Dramatically reduce test
Parasoft Proprietary and Confidential 32Parasoft Proprietary and Confidential 32
Access EfficiencyStaged w/ Virtual Test Lab
System Behavior
Dramatically reduce test
environment setup and
management
Service Virtualization

35. Service Virtualization ExamplesService Virtualization Examples
Mobile Application development and extension
Agile/Parallel development limited by system dependencies
Parasoft Virtualize is used throughout the SDLC to speed access to
and reduce the costs of managing development and test environments
Parasoft Proprietary and Confidential 33Parasoft Proprietary and Confidential 33
Capacity Constrained staged environments
Limited access to mainframes, ERPs, or 3rd party systems
Test data management for complex transactions

36. The Test Environment ChallengeThe Test Environment Challenge
Test environment access is outside the control of development and test leaving
gaps in the process
IT Operations
Parallel development
delays… Need simple,
realistic access to
dependent components…
Too much time
Scheduling
Dependent
Applications
Staged Assets
Parasoft Proprietary and Confidential 34Parasoft Proprietary and Confidential 34
Too much time
waiting for access…Need
reliable test data
Need a realistic test
environment easy to
maintain
Configuration
Access Limits
3rd Party Assets Virtual
Environments
Hyper Visor
App App App

37. The Test Environment ChallengeThe Test Environment Challenge
Test environment access is outside the control of development and test leaving
gaps in the process
IT Operations
Scheduling
Dependent
Applications
Staged Assets
Parasoft Proprietary and Confidential 35Parasoft Proprietary and Confidential 35
Configuration
Access Limits
3rd Party Assets Virtual
Environments
Hyper Visor
App App App
Create, Manage, Provision

38. Value of Service VirtualizationValue of Service Virtualization
Increase Access to “Trustworthy” Test Environments
Provide a complete test environment
Provide access 24/7
Flexibility provides a platform for extending test scenarios, coverage
Reduce Operational Expenditures
Service Virtualization drives better software quality
while reducing CapEx and OpEx
Parasoft Proprietary and Confidential 36Parasoft Proprietary and Confidential 36
Reduce Operational Expenditures
Significantly reduce test environment configuration time
Simplify test environment access
Remove the bottleneck of test environment setup
Reduce Capital Expenditures
Software assets can be run anywhere
Significantly reduce bandwidth demand
Reduce dependency on staged environments

39. Parasoft Proprietary and Confidential 37Parasoft Proprietary and Confidential 37

40. Parasoft Service VirtualizationParasoft Service Virtualization
Questions?
Parasoft Proprietary and Confidential 38Parasoft Proprietary and Confidential 38
Wayne Ariola
Chief Strategy Officer
Parasoft Corporation
Wayne.ariola@Parasoft.com

41. Virtual Lab ManagementVirtual Lab Management
Strengths Weaknesses
“Low” Risk
Reduce CAPEX
Reduce Server Sprawl
Manage Environment
Must Have Access
Permissions to Control
Configuration
Permutations
Parasoft Proprietary and Confidential 39Parasoft Proprietary and Confidential 39
Manage Environment
Complexity
Uniformity for Global
Teams
Operational Efficiency
Permutations
Test Data
Performance
Hardware Capacity
Constraints

42. The Value/ROI Drivers – HumanThe Value/ROI Drivers – Human
Virtual Lab Service Virtualization
Reduce OPEX Eliminate scheduling delays
associated with staged test labs
Eliminate ‘some’ configuration
delays
Eliminates configuration time by
delivering more granular ‘service’
versus system
Eliminates scheduling delays
Understand
Environment
Complexity
Gives a good view of inter-
relationship of applications
within geo/political control
Gives a complete view of all
system dependencies
Parasoft Proprietary and Confidential 40Parasoft Proprietary and Confidential 40
Complexity
Operational
Efficiency
Reduces demand on test lab
resources
Reduces delays of scheduling,
setup and re-configuration
Eliminates delays associated with
System configuration
Data management
Performance configuration
Setup and tear down

43. The Value/ROI Drivers – PhysicalThe Value/ROI Drivers – Physical
Virtual Lab Service Virtualization
Reduce CAPEX
and server
sprawl
Versus physical test lab will
significantly reduce CAPEX
Can reduce incremental CAPEX
due to physical test capacity
constraints
Can reduce demand on
“virtualized” infrastructure
resources
Eliminate Can eliminate physical test lab
BUT be careful what you want to
Eliminates the need for
incremental increases in physical
Parasoft Proprietary and Confidential 41Parasoft Proprietary and Confidential 41
Eliminate
physical test
labs
(Careful!)
BUT be careful what you want to
virtualize due to compliance and
data privacy concerns
incremental increases in physical
test labs or virtual lab capacity

44. The Value/ROI Drivers – Project/ProcessThe Value/ROI Drivers – Project/Process
Virtual Lab Service Virtualization
Time to Market
(Careful!)
Increases time to market by
providing access to “production-
like” environments within
geo/political control
Increases time to market by
providing more complete access
Provides a net new environment
earlier in the process
Access anytime
or anywhere
Good access to completed and
built applications within control
Access to both complete and
incomplete components
Manage and Good view of applications Much broader view of total
Parasoft Proprietary and Confidential 42Parasoft Proprietary and Confidential 42
Manage and
understand
Environment
Good view of applications
managed within resource pool
Much broader view of total
system interdependencies
Understanding of 3rd party
impacts
Defect
Reproduction
Captures great information for
applications under control
Extends information by including
components outside of control
Better Testing Increased code coverage
Increase test efficiency
Increased test coverage
Increased # and quality of tests
Increase test efficiency
Increased test coverage
Increased # and quality of tests

45. What’s the View from Your Seat?What’s the View from Your Seat?
Strategy
Risk Profile
Governance
Business
Portfolio
Application
Parasoft Proprietary and Confidential 43Parasoft Proprietary and Confidential 43
Actions
Project
Process
Risk Profile Application
Release
Iteration
Code

46. From Automated to ContinuousFrom Automated to Continuous
Continuous testing accelerates the SDLC by
managing quality expectations and actionable tasks
Requirements
Defined
Policy Management
Development
Defect Prevention
Development Testing Integration Testing System Testing
Defect Remediation Tasks
Parasoft Proprietary and Confidential 44Parasoft Proprietary and Confidential 44
Development
Static Analysis
Unit/Component
Peer Review
Automated Tests
API/Service Tests
Smoke Test
Security Tests
Automated Tests
Functional Tests
Scenario Tests
Performance Tests
CI
Build
Go
Release Path
No Go
Business
Decision
Service Virtualization – Test Environment Access