Tech Vision: Next-Generation Performance Testing With BlazeMeter, Service Virtualization and Test Data Management

World®
’16
Next-Generation Performance Testing
with BlazeMeter, Service Virtualization
and Test Data Management
Sudhrity Mondal - Sr. Principal Consultant - CA Technologies
DO5X56S
DEVOPS

2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type of
warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

Abstract
Application Performance has always been an important component of application quality, but
the growth in online channels of interaction means that for many organizations, performance is
now critical. Poor performance can impact customer experience, revenues, profitability, brand,
and in some cases, the viability of organizations.
Current approaches for ensuring the performance of an application prior to implementation into
production can be antiquated, expensive and time consuming. This session outlines next
generation performance test approach that can be adopted by organizations using CA
BlazeMeter, Service Virtualization and Test Data Management solutions. Adopting this approach
can offer a wide range of benefits including efficiency gains in application delivery, cost savings,
improved agility, better performing applications and improve customer experience.
Sudhrity
Mondal
CA Technologies
Sr. Principal Consultant

Agenda
APPLICATION PERFORMANCE TEST
THE HURDLES TO PERFORMANCE TESTING
NEXT-GEN PERFORMANCE TESTING
Q & A
1
2
3
4

Your Apps Are the FACE of Your Business
Does their performance drive LOYALTY or FRUSTRATION?

Why Performance Means Everything
Overview
68% 25%
Of customer install base at risk of being lost
if businesses fail to deliver a positive
application experience*
Of users will demand an app load time of
6 seconds or less*
* Software: The New BattleGround for Brand Loyalty, 2015, Zogby Analytics

Why Performance Test
§ Identifies problems early on before they become costly to resolve
§ Produces better quality, more scalable code
§ Prevents revenue and credibility loss due to poor Web site
performance
§ Capacity planning for future expansion or events
§ To ensure that the system meets performance expectations under
given levels of load
§ Expose bugs that do not surface in cursory testing

Application Performance Test
§ Process of determining the speed or effectiveness of an application
and the resources used to run the application.
§ Evaluates the response time, throughput and utilization of system
to execute its required functions
– in comparison with different versions of the same application, or
– competitive product

Metrics/KPI
• Customer Experience
• Application responsiveness
• Server utilization
• Network
Types of Test
• Stress Testing
• Load Testing
• Performance Testing
• Soak Testing
• Benchmark Testing
• Baseline Testing
Factors
• Throughput
• Response Time
• Arrival Rate
• Benchmarking
• Error Rate/Availability
Application Performance Test

Performance Test – Application Scenarios
§ What aspect of application going to test?
– Breakdown into separate scenarios or tests?
– Actions to be performed?
§ Example – Book a flight:
– Log into application
– Search airline flights
– Book flight
– Pay for flight with credit card
– Logout of application
§ Scenario mix
Script % of
users
Total
Search & Book flight 20 200
Search for flights without booking 60 600
Change an existing reservation 10 100
Ticket Agent running a report 10 100
Total 100 1000

Performance Test – Results Summary
§ Top 5 Slowest Pages
– Slowest pages encountered during load test
§ Top 5 Slowest Transactions
– Do not meet the performance criteria defined at test inception
§ Top 5 Slowest Query Statements
– Slowest Query which is typically 80% of performance problems

Performance Test – Core activities
§ Plan and design tests based on acceptance criteria
§ Prepare and/or Record test plan(s)
§ Identify and prepare test environment(s) (prod/pre-prod)
§ Provision Test Data
§ Execute tests
§ Analyze results
§ Perform tuning/Change configuration, retest

Performance Test – Environments
§ Production
§ Pre-Production/Staging
§ Component level performance tests during CI/CD in SDLC
§ Application/Environment Dependencies and Constraints
Data
Constraints
System
Constraints
Cost
Constraints Scalability

Performance Test Data
§ Test data and where is it used
§ Creation of test data
§ Unique/non-unique data
§ Application owner role
§ Test Data privileges
§ PII data masking
§ Parameterization of values
Test
Scenario
Id Login Id Name Password Group Pay-Period
Approver 1 45678329 Smith,
John
123456 8532 03/01/06-
03/31/06
Approver 2 74563422 Hansen,
Joe
123456 2345 03/01/06-
03/31/06
Approver 3 74569422 Strong,
Rodney
123456 5345 03/01/06-
03/31/06
Data
Security
Test Data
Coverage Automation Data
Volumes

Limited test resources and
lack of Automation
of all testing is
still manual70%
Source: Bloor Research, 2014
Test Data
for setup & staging
of time spent
looking for50%
Source: Bloor Research, 2014
The Hurdles to Performance Testing
Build of various dev & testing
Environments
of critical dependencies
are unavailable when
dev and test teams
need them
56%
Source: voke Market Snapshot TM Report: Service
Virtualization – January 2015
Performance Testing Still the Bottleneck

Agile Aligned Performance Testing Supported with Advanced Technology
Next-Gen Performance Testing
TEST ENVIRONMENT
SIMULATION
Simulate missing systems,
APIs to save up to 640 Hrs
PERFORMANCE TEST
AUTOMATION
Shift-Left, Integrate with CI
100% Test Coverage
TEST DATA
MANAGEMENT
Reduce time spent
waiting on data by 95%
ORCHESTRATED PERFORMANCE TESTING
Streamlined set-up of complete and fully
automated performance test execution

DEMOCRATIZING
PERFORMANCE TESTING IN THE
CONTINUOUS DELIVERY ERA
CA BlazeMeter is a next-generation, SaaS and On-Premise, continuous application performance
testing solution designed to help DevOps and Agile teams begin web, mobile, microservices, and
API testing early in the SDLC and in production.
Pure SaaS,
No Setup
100% Open-
Source
Compatibility
(“Load and Go”)
Public Cloud,
Virtual Private
Cloud and/or
Behind The
Firewall
Massively
Scalable
– 2 Million+
virtual users
Performance
Test
Orchestration
using Taurus

Why Worry About Test Data?
Compliance/security
Customer satisfaction/quality
Productivity/agility/efficiency
Skyrocketing costs
Server Error in ‘/contractor’ Application.
Runtime Error

Test Data Management Approaches
COPY PRODUCTION DATA SUBSET MASK
For most companies, this is TDM:
§ Slow and error prone
§ 20%–30% coverage
§ No negative or future test scenarios
§ No data for new development
Data is masked and inserted into different
data stores for dev/test
§ Refresh timing varies due to work required
§ Data not available in parallel or on demand
§ Time wasted waiting for or looking for data
§ High infrastructure costs

Substitution Variables
Combinable Functions
CA Test Data Manager
Data Model
Generation
Bulking Scripts
Production Data / Files
Test
Data
Warehouse
Perf Test Environments
1 2
4 5
Secure Data Subsets
XML
Files
XLS
SQL
Files
CSV Files
API
HTML
Files
FD
TXT
Files
NoSQL
3
6
TDM Approaches – Synthetic Test Data Generation

Data Bulking
Performance Data
Delivered
Format Bulking
Requirements
Functions
Structure
1 million
records
Data Generation
Generate a smaller
number of records first
(example: 10,000)
Use Bulking/Multiplier function to multiply
the records to a much larger number
(example: 1,000,000)
Synthetic Data Generation for Performance Testing

The Hurdle to Performance?
Constraints/Bottlenecks for Performance Testing
Incomplete
Developmen
t
of teams experience delays in
development and QA due to
unavailable dependencies
56%
Over
80%
79%
Source: voke Market Snapshot TM Report: Service Virtualization – January
2015
of teams face prohibitive
restrictions, time limits or
access fees on needed 3rd
party services
of critical dependencies are
unavailable when dev and
test teams need them
Data
Constraints
System
Constraints
Cost
Constraints
Scalability
Constraints
http://bit.ly/SV-ROI

23 © 2016 CA. ALL RIGHTS RESERVED.
Un-Constrained Performance Testing
ESB
Mainframe
System of
Record
ERP SaaS/API
Availability Sync Capacity Cost
CA Service
Virtualization
Order Management
CA BlazeMeter
Service
Virtualization
the Wind tunnel
of Software
BlazeMeter
Run hundreds of Load
Tests in Parallel up to 2
million Virtual Users
from 25+ Geo locations
Test Data
Manager
Create high-quality,
compliant, ‘fit for
purpose’ test data
CA BlazeMeter
• Drive Performance Test On-
Premise or in the Cloud and
orchestrate
CA Test Data Managemer
• Generate Performance Test Data
CA Service Virtualization
• Simulate Missing Environments

One Challenge: Data Synchronization
Application
iSeries/AS400 z/OS
DB2, IMS,
VSAM
Oracle, MSSQL, DB2
Informix, Teradata,..
Flat Files –
CSV, Excel, XML,..
Test Execution
Production Data
SYSTEM UNDER TEST
SYSTEMS OF RECORD
What about the data
in your Virtual
Services?
Subsets/Masks/Generates
Test Data
CA SV
CA Service Virtualization
simulates missing
Systems/Interfaces

Data Synchronization between Virtual and Real systems
Application
iSeries/AS400 z/OS
DB2, IMS,
VSAM
Oracle, MSSQL, DB2
Flat Files –
CSV, Excel, XML,..
Test Execution
Production Data
SYSTEM UNDER TEST
TEST FAILURE+ =
DATA NOT
SYNCED
CA TDM Interface
SYSTEMS OF RECORD If Test Data isn’t aligned, test cases will
FAIL because of data sync issues
CA SV

The Solution
Application
iSeries/AS400 z/OS
DB2, IMS,
VSAM
Oracle, MSSQL, DB2
Flat Files –
CSV, Excel, XML,..
Test Execution
Production Data
SYSTEM UNDER TEST
TEST SUCCESS!+ =
DATA
SYNCED
SYSTEMS OF RECORD
CA TDM Integration bridges the Data
alignment gap and allows for
creation/update of virtual services with
synchronized data
CA SV
CA TDM Interface

Solution Benefits
BETTER BUSINESS
OUTCOMES
READY FOR
SEASONAL PEAKS
INCREASE
SECURITY
IMPROVED
QUALITY
REDUCED COSTS
§ Applications perform
as expected
§ Improved Customer
Experience
§ Prevent Revenue Loss
§ Applications
tested for peak
loads
§ Identify & resolve
bottlenecks before
critical business
events
§ Capacity Planning
§ Match the right data
to the right test
§ Continuous
Performance
Testing as part of
CI/CD
§ Achieve 100%
performance test
coverage
§ Eliminate the risk of
internal/external
breaches and fines
§ Ensure PII is
protected
§ Simulate systems
that are outside
scope with realistic
latency
§ Eliminate Wait
times for
environment
availability

Questions?

Must See Demos
Modernize
App Delivery
Integrated CD
Theater 5 - DOV501P
Deliver Test
Data Faster
Test Data Manager
Theater 5 - DOV511P
Deliver
Better Apps
Service Virtualization
Theater 5 - DOV507P
Orchestrate
Your Release
Release Automation
Theater 5 - DOV513P

Thank you.
Stay connected at communities.ca.com

Tech Vision: Next-Generation Performance Testing With BlazeMeter, Service Virtualization and Test Data Management

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Tech Vision: Next-Generation Performance Testing With BlazeMeter, Service Virtualization and Test Data Management

Similar to Tech Vision: Next-Generation Performance Testing With BlazeMeter, Service Virtualization and Test Data Management (20)

More from CA Technologies

More from CA Technologies (20)

Recently uploaded

Recently uploaded (20)

Tech Vision: Next-Generation Performance Testing With BlazeMeter, Service Virtualization and Test Data Management