The demand to accelerate software delivery and for teams to continuously test and release high quality software sooner has never been greater. However, whether your release strategy is based on schedule or quality, the entire delivery process hits the wall when agility stops at testing. When software/services that are part of the delivered system or required environments are unavailable for testing, the entire team suffers. Al Wagner explains how to remove these testing interruptions, decrease project risk, and release higher quality software sooner. Using a real-life example, Al shows you how service virtualization can be applied across the lifecycle to shift integration, functional, and performance testing to the left. Gain an understanding of how service virtualization can be incorporated into your automated build and deployment process, making continuous testing a reality for your organization. Learn what service virtualization can do for you and your stakeholders. The ROI is worth it!

1. W8
Test Techniques
5/7/2014 1:45:00 PM
Continuous Testing through
Service Virtualization
Presented by:
Allan Wagner
IBM
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Allan Wagner
IBM
IBM-certified IT Specialist Allan (Al) Wagner is currently a technical manager and evangelist
with IBM, driving thought leadership, strategic initiatives, and tangible solutions with a specific
focus on quality management, test automation, and service virtualization. In his fifteen years of
practical IT field experience, Al has assisted, supported, mentored, and enabled teams with
their IT application infrastructure, development, implementation, and operations challenges. A
frequent conference speaker on software quality principles and techniques, Al also has
authored or coauthored the IBM Service Virtualization for Dummies book and numerous papers
on quality management and test automation.

3. 1
Continuous Testing
through
Service Virtualization
@Al_Wagner
awagner@ca.ibm.com
Technical Manager
IBM
Session W8
Test Techniques
Wednesday, May 7th, 1:45pm
1
Business stakeholder expectations
Better – Cheaper - Faster
In the past
• Organizations looked to find a
balance between speed, cost &
risk.
• Development teams were not as
agile as they are today; testing
never seems to have enough
time.
• Software quality suffered.
In the past
• Organizations are asking teams to
release higher quality software
sooner.
• Development teams are becoming
more and more agile; testing still
never seems to have enough
time.
• Software quality continues to
suffer.

4. 2
2
Business stakeholder expectations
Better – Cheaper - Faster
In the past
• Organizations looked to find a
balance between speed, cost &
risk.
• Development teams were not as
agile as they are today; testing
never seems to have enough
time.
• Software quality suffered.
In the past
• Organizations are asking teams to
release higher quality software
sooner.
• Development teams are becoming
more and more agile; testing still
never seems to have enough
time.
• Software quality continues to
suffer.
What are organizations “actually“
doing to deliver higher quality
software to market faster?
3
Adding more Testers is not a sustainable model
Labor costs continue to increase
Testers
Testers
Testers
Development
Team
Testers

5. 3
4
Adopting technology only increases testing complexity
Test environment setup is harder and more expensive than ever before!
external
users
internal
users
Application
Layer
Mainframe
ERP
Systems
Custom
Applications
Databases
Enterprise Service Bus
Functionality Usability
Performance/Scalability
Reliability Consumability
Before, testers
designed tests
to measure
quality based
on a number
of attributes.
5
hybrid
Network Latency
Security/Privacy
Availability
Cost
Data Integrity
Adopting technology only increases testing complexity
Test environment setup is harder and more expensive than ever before!
external
users
internal
users
private
Business Partner
Services
public
External 3rd Party
Services
Application
Layer
Mainframe
ERP
Systems
Custom
Applications
Databases
Enterprise Service Bus
Functionality Usability
Performance/Scalability
Reliability Consumability
Before, testers
designed tests
to measure
quality based
on a number
of attributes.
Now, there are
many new
things testers
also need to
consider.

6. 4
6
Faster and more frequent builds are not enough
The transformation to Agile isn’t benefitting Testers
Tester
Setup test environment
Developer
Write & deliver
code
Nightly build(s)
Compile, unit test, publish
Developer
Developer
Developer
Developer
Developer
“N” nightly builds are piling up
7
Faster and more frequent builds are not enough
The transformation to Agile isn’t benefitting Testers
Tester
Setup test environment
Developer
Write & deliver
code
Nightly build(s)
Compile, unit test, publish
Developer
Developer
Developer
Developer
Developer
“N” days to install & configure
“N” nightly builds are piling up
Quality
GateBVT
Multiple
Test Environments
FVT
Perf Test
Security Test
SIT
UAT
I just spent 2 weeks
setting this up &
the build is no good?
When agility stops at
testing, the entire
organization hits the wall

7. 5
8
Bottom line – current approaches are not helping!
Today’s testing challenges still remain
Development VelocityTesting Labor Test Lab Complexity
Outsourcing labor is no longer
a sustainable model as global
wages are increasing
Product and application
complexity are driving
test lab cost increases
Test teams can no longer
keep up with agile
development
Heterogeneous Environments
Public Cloud
Private Cloud
Data Warehouse Mainframe
Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
Web/
Internet
Routing
Service
Third-party
Services Portals
Content
Providers
EJB
Shared
ServicesArchives
Business
Partners
Messaging
Services
Traditional
Testing
Other costs
Testing costs
9
What Testers really need!
Validating today’s complex applications
Production-like test environments which are:
 realistic & complete
 available on demand
 inexpensive to setup
Heterogeneous
Environments
Public
Cloud
Private
Cloud
Data Warehouse Mainframe
Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under TestRouting
Service
Third-party
Services
Portals
Content
Providers
EJB
Shared
Services
Archives
Business
Partners
Messaging
Services
Other costs
Testing costs
Automated tests which are:
 able to run unattended validating application
integration, functionality, security & performance
 capable of validating business flows across
multiple channels
 durable, maintainable & resilient to change
Orchestration from build to production
deployment which is:
 reliable
 repeatable
 traceable
Better
Faster
Cheaper

8. 6
10
Accelerate
software delivery –
for faster time to value
Balance speed, cost,
quality and risk –
for increased capacity
to innovate
Reduce time to
customer feedback –
for improved customer
experience
Continuous
Monitoring
Collaborative
Development
Continuous
Release and
Deployment
Continuous
Customer
Feedback
and
Optimization
Continuous
Business
Planning
Monitor
and Optimize
Develop
and Test
Release
and Deploy
Plan
and Measure
DevOps –
Continuous
innovation,
feedback and
improvements
Enables clients to seize market opportunities
and reduce time to customer feedback
Increasing adoption of DevOps
Extending Lean and Agile practices across the entire software value chain
Continuous
Testing
11
All the world’s a “test” stage,
and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers

9. 7
12
Continuous Deployment and Release
All the world’s a “test” stage,
and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers
Collaborative
Development
The goal:
create the
build package
Continuous
Monitoring
The goal:
production
deployment
Continuous Testing
simultaneously across multiple test environments
Everything between dev &
prod is a test stage
Quality
Gate
Security
FVT
Performance
SIT
UAT
Quality
Gate
BVT
Dev UT
At least 1/3 of is testing!
13
All the world’s a “test” stage,
and all Testers merely players; They have their exits and their entrances; [play on Shakespeare]
The Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers
Collaborative
Development
The goal:
create the
build package
Continuous
Monitoring
The goal:
production
deployment
Continuous Testing
simultaneously across multiple test environments
Everything between dev &
prod is a test stage
Quality
Gate
Security
FVT
Performance
SIT
UAT
Quality
Gate
BVT
Test
scripts
Test
data
Test
configurations
Test
reports
Version
Control
Orchestration & Automation
Dev UT
At least 1/3 of is testing!

10. 8
14
Reality check
Which answer would you pick?
What percentage of your time are you spending setting up, maintaining, and
re-cycling test environments?
1. 0% to 20%
2. 21% to 30%
3. 31% to 40%
4. 41% to 50%
5. More than 50%
11%
18%
14%
43%
14%
Responses from GRUC Webcast – October 22nd, 2013
15
Solving the Test Environment dilemma
Making production-like test environments a reality
 Traditional test environments are both underutilized and insufficient
– Environments are expensive: hardware, software, and labor
– Tester and developer time wasted waiting for appropriate environment
 Agile test environments require four enabling capabilities
Dynamic Infrastructure
Test Data
Management
Deployment
Automation
Test Environments
DevOps Foundation
Service
Virtualization

11. 9
16
Reality check
Which answer would you pick?
How does your organization deal with testing bottlenecks due to unavailable or
missing software and services?
1. Manually write simulation stubs to emulate missing functionality.
2. Use commercial service virtualization solution.
3. Defer testing until later when the code is available to test.
4. De-scope testing of that functionality.
5. None of the above.
9%
9%
56%
6%
19%
Responses from GRUC Webcast – October 22nd, 2013
17
What is Service Virtualization?
Making the unavailable available for testing
 Virtual components simulate the behavior
of a service or application during testing
– System dependencies are a key challenge in
setting up test environments
– Unavailable/inaccessible services slow down
the testing process
– Costly 3rd party access fees can significantly
impact testing availability
 Virtual components run on commodity
hardware, private cloud, public cloud
 Each developer and tester can easily have
their own test environment
 Developers and testers can continue to use
current testing procedures and tools
 Streamline creation of production-like
environments and negative testing
Heterogeneous Environments
Public Cloud
Private Cloud
Data Warehouse Mainframe
Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under TestRouting
Service
Third-party
Services Portals
Content
Providers
EJB
Shared
Services
Archives
Business
Partners
Messaging
Services
Databases Mainframe
applications
App Under Test
Third-party
Services
Packaged apps, messaging services, etc.
Virtual Components

12. 10
18
Virtual components simulate
the “real world” behavior
Category Detail
Simple Hard-coded response returned for given input
Non-deterministic One-of-n hard-coded responses
Data driven
Input and/or output data specified in external
data source (Excel, file, database)
Data model, stateful
Input and/or output data kept in data model with
complex relationships. Supports CRUD and
other stateful behavior
Behavioral
Extends model-driven to provide pre-packaged
functionality, e.g. shopping basket, real-time
data feed, trading exchange, order matching
What are virtual components?
Simulating application functionality, behavior, & performance
19
Test Environment
Volume Constraints
Volume Constraints
2hr. access window
Maintenance Constraints
Database / JDBC
Server/ Web Services
Mainframe / MQ/CICS
Smart Device / SD-IO
MQ/CICS
Web Services
Smart Devices
JDBC
Virtual Environment
Capture
& Model
System Under Test
Listen / Capture
Store
Test Case
Repository
Creating virtual components
Listen and capture

13. 11
20
Build virtual components
from the design specs
HelloService
Turn specifications into virtual components
Creating virtual components
Author from scratch using design specifications
21
Continuous Testing improves software quality
Actual Service/App
Virtual Service/App
 Service Virtualization is an enabler for
continuous testing
 Services, applications, systems are
introduced into the continuous integration
cycle in a prioritized, controlled fashion.
Controlled integration helps isolate defects
for faster resolution.
Service virtualization enables
continuous integration testing
Using virtual components
Enables testers to continuously test what they want, when they want, how they want

14. 12
22
Avoiding the big bang
Control integration
Scenario: Testers may often wait or defer testing certain areas of
functionality until all of the components are available and deployed
as a complete application. This approach results in major defects
being discovered late in the process and could put the release date
at risk.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database
ERP
System
Custom
Application
23
Avoiding the big bang
Control integration
Scenario: Testers may often wait or defer testing certain areas of
functionality until all of the components are available and deployed
as a complete application. This approach results in major defects
being discovered late in the process and could put the release date
at risk.
Service virtualization puts
control in the hands of the
tester.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
 Simulate the behavior and
performance of the
dependent components.
Mainframe Database
ERP
System
Custom
Application
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service

15. 13
24
Avoiding the big bang
Control integration
Scenario: Testers may often wait or defer testing certain areas of
functionality until all of the components are available and deployed
as a complete application. This approach results in major defects
being discovered late in the process and could put the release date
at risk.
Service virtualization puts
control in the hands of the
tester.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
 Simulate the behavior and
performance of the
dependent components.
 Begin end to end testing
earlier and continuously.
Mainframe Database
ERP
System
Custom
Application
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
Virtual
Service
25
Avoiding the big bang
Control integration
Scenario: Testers may often wait or defer testing certain areas of
functionality until all of the components are available and deployed
as a complete application. This approach results in major defects
being discovered late in the process and could put the release date
at risk.
Service virtualization puts
control in the hands of the
tester.
Testers
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
 Simulate the behavior and
performance of the
dependent components.
 Begin end to end testing
earlier and continuously.
 Introduce the real
implementations as they
become available.
Mainframe Database
ERP
System
Custom
Application
Virtual
Service

16. 14
26
Stand up test environments faster
Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component
adding integration with an existing 3rd party service to implement new
required functionality. To validate this minor change, the time to stand
up a complete test lab combined with the 3rd party service access
fees adds substantial project cost – cost which is unnecessary.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
App Layer
new
27
Stand up test environments faster
Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component
adding integration with an existing 3rd party service to implement new
required functionality. To validate this minor change, the time to stand
up a complete test lab combined with the 3rd party service access
fees adds substantial project cost – cost which is unnecessary.
Service virtualization lowers the
cost of testing and eliminates
delays.
 Create and deploy a virtual
component to simulate the 3rd
party service functionality and
performance without the fees.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
Virtual
Service
App Layer
new

17. 15
28
Stand up test environments faster
Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component
adding integration with an existing 3rd party service to implement new
required functionality. To validate this minor change, the time to stand
up a complete test lab combined with the 3rd party service access
fees adds substantial project cost – cost which is unnecessary.
Service virtualization lowers the
cost of testing and eliminates
delays.
 Create and deploy a virtual
component to simulate the 3rd
party service functionality and
performance without the fees.
 Quickly stand up test labs using
virtual services to simulate
dependencies which have not
changed.
Testers
Business Partner
Services
3rd Party
Services
Virtual
Service
Virtual
Service
App Layer
Virtual
Service
new
29
Stand up test environments faster
Deploy what has changed, virtualize the rest
Scenario: The team is making changes to an application component
adding integration with an existing 3rd party service to implement new
required functionality. To validate this minor change, the time to stand
up a complete test lab combined with the 3rd party service access
fees adds substantial project cost – cost which is unnecessary.
Service virtualization lowers the
cost of testing and eliminates
delays.
 Create and deploy a virtual
component to simulate the 3rd
party service functionality and
performance without the fees.
 Quickly stand up test labs using
virtual services to simulate
dependencies which have not
changed.
 Test applications end to end with
confidence but at a lower cost and
without delay.
Testers
Business Partner
Services
3rd Party
Services
Virtual
Service
Virtual
Service
App Layer
Virtual
Service
new

18. 16
30
Author automated scripts earlier
Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as
possible, creation of automated test scripts is often delayed until
everything is available in a test lab – typically late in the development
cycle or perhaps not until after the actual release of the application
software.
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database
ERP
System
Custom
Application
Testers
31
Author automated scripts earlier
Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as
possible, creation of automated test scripts is often delayed until
everything is available in a test lab – typically late in the development
cycle or perhaps not until after the actual release of the application
software.
Service virtualization makes the
unavailable available for testing
purposes.
 Create virtual components from
development specifications. Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database
ERP
System
Custom
Application
JSON RESTWSDL
Virtual
Service
Virtual
Service
Virtual
Service
Testers

19. 17
32
Author automated scripts earlier
Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as
possible, creation of automated test scripts is often delayed until
everything is available in a test lab – typically late in the development
cycle or perhaps not until after the actual release of the application
software.
Service virtualization makes the
unavailable available for testing
purposes.
 Create virtual components from
development specifications.
 Record the functionality and
behavior of the application using
virtual services.
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database
ERP
System
Custom
Application
Virtual
Service
Virtual
Service
Virtual
Service
Testers
rec
101101100101101100
33
Author automated scripts earlier
Increasing Tester productivity – author automated scripts earlier, execute tests sooner
Scenario: While teams may want to automate testing as soon as
possible, creation of automated test scripts is often delayed until
everything is available in a test lab – typically late in the development
cycle or perhaps not until after the actual release of the application
software.
Service virtualization makes the
unavailable available for testing
purposes.
 Create virtual components from
development specifications.
 Record the functionality and
behavior of the application using
virtual services.
 Execute automated tests at any
layer in the application using virtual
services to simulate functionality
under development.
Application
Layer
Enterprise Service Bus
Business Partner
Services
3rd Party
Services
Mainframe Database
ERP
System
Custom
Application
WSDL
Virtual
Service
Virtual
Service
Virtual
Service
Testers
101101100 101101100

20. 18
34
Blocking defects - they won’t slow me down
Reducing testing down time
Scenario: As defects are discovered, some may actually block the
testing effort from proceeding. Testers may be forced to wait for the
fix before being able to continue and development priorities may
result in the fix being delayed. This creates a testing bottleneck!
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
App Layer
new
35
Blocking defects - they won’t slow me down
Reducing testing down time
Scenario: As defects are discovered, some may actually block the
testing effort from proceeding. Testers may be forced to wait for the
fix before being able to continue and development priorities may
result in the fix being delayed. This creates a testing bottleneck!
Service virtualization eliminates
bottlenecks and increases
productivity.
 Deploy a virtual service to
simulate the required
functionality and behavior
replacing the defective
component.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
App Layer
newVirtual
Service

21. 19
36
Blocking defects - they won’t slow me down
Reducing testing down time
Scenario: As defects are discovered, some may actually block the
testing effort from proceeding. Testers may be forced to wait for the
fix before being able to continue and development priorities may
result in the fix being delayed. This creates a testing bottleneck!
Service virtualization eliminates
bottlenecks and increases
productivity.
 Deploy a virtual service to
simulate the required
functionality and behavior
replacing the defective
component.
 Continue testing without delay.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
App Layer
newVirtual
Service
37
Blocking defects - they won’t slow me down
Reducing testing down time
Scenario: As defects are discovered, some may actually block the
testing effort from proceeding. Testers may be forced to wait for the
fix before being able to continue and development priorities may
result in the fix being delayed. This creates a testing bottleneck!
Service virtualization eliminates
bottlenecks and increases
productivity.
 Deploy a virtual service to
simulate the required
functionality and behavior
replacing the defective
component.
 Continue testing without delay.
 Replace the virtual service with
the updated source code without
having to redeploy your entire
application.
Testers
Business Partner
Services
3rd Party
Services
Enterprise Service Bus
Mainframe Database
ERP
System
Custom
Application
App Layer
newfixed

22. 20
38
The ROI of Service Virtualization
The results are real!
3 Year Benefits:
• Increased project delivery capacity 100% (40 projects to 80 projects)
• Avoided hiring three additional FTE's, saving almost $600k.
• Reduced incidents found in production from 2.5 to .3 per project, saving $3 million
• Identified defects earlier and freed the team up to focus on new capabilities, saving over $560k
• Reduced consulting and third party testing fees with better test automation, saving $408k
• Eliminated the need for additional hardware and labor to set up test environments, saving $700k
• Generated new revenue for the business – estimated at tens of millions!
A major European bank implemented service virtualization and automated integration
testing as part of an Agile transformation to enable continuous testing. As a result of
IBM capabilities, their system integration test (SIT) process was reduced from 3
weeks down to four hours!
The Total Economic Impact™ of
IBM Rational Service Virtualization
and Test Automation Solutions, a
July 2013 commissioned study
conducted by Forrester Consulting
on behalf of IBM
39
Orchestration
Continuous Testing
InfoSphere Optim
Test Data Management
Rational Test
Virtualization
Server
Rational Test
Workbench
• Capture service
behavior and model
virtual services
• Test services and
applications long
before their user
interfaces becomes
available and do
integration testing
• Performance test,
regression test,
functional test &
integration test
• Virtualize
heterogeneous
hardware, software,
and services to
provide 24x7 testing
capabilities
• Reduce the cost of
traditional testing
environments
• Build Virtual Services
to simulate over 70+
protocols
Rational
Performance
Test Server
• Reuse Rational Test
Workbench scripts to
drive performance
testing
• Combine virtual
services and the real
implementations to
performance test
earlier
• Probe for
identification of
system bottlenecks
• Create referentially
intact, “right-sized”
test databases
• Automate test result
comparisons to
identify hidden errors
• Protect confidential
data used in test,
training &
development
• Refresh test
environments faster
shortening iterative
testing cycles and
accelerating time to
market
IBM’s continuous testing solution
Rational Quality
Manager
• Track quality using a
comprehensive and
customizable test
plan
• Link test cases with
requirements,
defects, development
work items and builds
• Author and execute
automated or manual
test scripts
• Provide up-to-date
project quality status
using customizable
dashboards
Test Data
Management
Service
Virtualization
Test
Automation
with integration to
Collaborative
Development
& Test
Completing the Continuous Testing story
Service Virtualization is the enabler but it is only part of a Continuous Testing solution

23. 21
40
Orchestrate and automate the deployment of applications, middleware
configuration, and database changes into development, test and
production environments.
Databases Internal
Messages
Third-party
Services
virtual components
Simultaneously
test across
multiple test
stages
Dev QA
Test Data
Management
Deploy
Provision
Test
Combining continuous testing withOrchestration from build to production
Reliable, repeatable, & traceable deployment automation
Deployment
Automationintegrated with
Test Environments
Dynamic Infrastructure
 Test using real world
network conditions
Service
Virtualization
Test
Automation
 Deploy and refresh
right size secure data
 Deploy what is ready,
virtualize the rest
 Continuously test in
production-like env.
Network
Virtualization
41
Continuing on your Service Virtualization journey
Increasing awareness and accelerating learning
Get your copy today!
Service Virtualization
For Dummies
ibm.co/ServiceVirtualizationForDummies
Visit
Service Virtualization
on ibm.com
ibm.co/servicevirtualization

24. 22
42
Continuous testing as our planet gets smarter
If you think testing is hard now, just wait – but don’t wait too long
intelligent interconnectedinstrumented
We are already on the road to the Internet of Things
Sooner than later, someone is going to ask you to test it!
43

25. 23
44