The audacity of quality requirement-non functional testing- Aware in BugDay Bangkok 2012

The Audacity of Quality Requirements: Non Functional Testing
Testing as Managed Services

“Think  Communicate  Implement”
By - Sudeepta Guchhait (Deep)

“Quality is the link to Success”


INTENT
 How to DEFINE business problem?
 How to DERIVE high quality non functional test requirements?
 How to VERIFY high quality non functional test requirements?
 How to TEST non functional test requirements?
 How to DISCOVER the bottlenecks?
 Key BEST Practices? – Take Away


How to DEFINE business problem, due to
lower quality Non Functional test
requirements?

 Voice of Customer
 Proposition – Testing Upfront


Voice of Customer


Proposition – Test Upfront
Defect Removal When to Test
Requirement (15%): Performance Modeling, Think time
% Original Defects

modeling

100%
100% 15% 20% Test design (20%):
• Identification of critical business transactions, defining the test
20% SLA, User load distribution, Transactional volume and peak
80%
35% usage
60%
10% % Infrastructure (20%):
Reduction – Investment needs to be made to procure the production like
40% NFT test environment (Physical as well Logical design)
– Appropriate test tool procurement
20%
Testing (35%):
0%
Original Req Infra & Testing Prod Defects – Test execution to be done for performance test, load test,
Design
Defects Test lab Stress, Scalability (Capacity) and Availability (Reliability).

Qualitative Consistency — on time, on budget delivery to business, increased client satisfaction, Decreased
Benefits implementation risk.

Quantitative
Benefits Identify system & application bottlenecks to fine tune those.

√ Improved software quality with varied data sets
√ Fully tested instance in line with the expectation of business users


Challenges of Performance testing

Creation of tests
 Determine what to simulate?
 Determine which performance factors are critical?
 Analyze desired test results
 Fine tune the bottlenecks

Infrastructure
 Build a realistic test environment (Infrastructure usage)
 Select the appropriate test tool

Usability of tests
 Software build should be functionally stable
 Tests are resource-intensive


How to DERIVE high quality Non Functional
test requirements?

 Non Functional Objective
 Practical Conversation
 Performance [ M / T / E ]


Non Functional Objectives
Performance Testing

Application Response Time Capacity Planning
How long does it take to Complete At what point does degradation in
a task? Performance occur?

Reliability Bottleneck Identification
How stable is the system under a What is the cause of degradation
heavy load? In Performance?

Acceptance Product Evaluation
Is the system stable enough to go What is the best server configuration
into production? for 100-500-1000 users?


Practical conversation on Performance of Application
 “I think we need Performance Testing, but what is it exactly?”
 “I know you want it to be fast, but how fast?”
 “I think around 300 users will use the system, they will do all kinds of activity,
so can we determine performance ?
 “What you will do with production data in performance?”
 “Post Test charts looks nice, let me know the performance is good or bad”
 “Does that mean we’re done? Can we release?”


Identification of Performance Transactions
80/20 rule
- 20% of the critical transaction will be performed by 80% of real business
users.

- high visibility activities
• Creating initial user profile
• Updating payment information
- high importance activities
• Withdrawals
• Transfer funds
• Paying bill on-line
- performance intensive activities
• Importing the monthly report from “the other system”
• Requesting payment history
• Heavy weight transactions (Stock trade, IPO subscription etc.)
- Identify the “heavy hitters” even though they may be used less often


Performance Modeling (Validation)
“Performance validation is the process by which software is tested with the
intent of determining if the software meets pre-existing performance
requirements. This process aims to evaluate compliance.”

Primarily used for…

 Determining SLA compliance
 Determine user behavior load
 Determine the volume usage


Performance Testing
“Performance testing is the process by which software is tested to determine the
current system performance. This process aims to gather information about
current performance, but places no value judgments on the findings.”


 Determining capacity of existing systems
 Creating benchmarks for future systems
 Evaluating degradation with various loads and/or configurations


Performance Engineering
“Performance engineering is the process by which software is tested and tuned
with the intent of realizing the required performance. This process aims to
optimize the most important application performance trait, user experience.”


 Fine application bottlenecks
 Extending the capacity of old systems.
 Fixing problems of the application that are not meeting
requirements/SLAs.


How to VERIFY high quality Non Functional
test requirements?

 Defect Root Cause
 Issue in Requirement & Mitigation


Defect Root Cause Identification

 Identify Key Scenarios: Identify scenarios where performance is important
and scenarios that pose the most risk to performance objectives
 Identify Workload: Identify how many users and how many concurrent users
system needs to support
 Identify Performance Objectives: Identify performance objectives for each of
the key scenarios.
 Baseline is the process of running a set of tests to capture performance
metric data for the purpose of evaluating the effectiveness of subsequent
performance improving changes to the system or application.
 Benchmarking is the process of comparing your system performance against
an industry standard that is endorsed by some other organization.


Defect Root Cause Identification - Example
API Processing Monitoring
Step 1: Funds Transfer Menu (App1 –App2) Network Monitoring Server Monitoring

App1 App2

F1 5 A Fn
A 3
2 4 6
1 P P 7
12 I 10 I 8
11 9

12 times * number of sec
API Processing Monitoring
Server Monitoring

Point-to-Point – Component Level View
1. Captures the internal processing time between steps (low-level system processing)
2. No ability to measure this with testing tool alone
3. Requires monitors to be installed to collect internal processing time at each step


Requirement Design

Issue 1: Illustrative Example

Step 1: Determine user load distribution across products/functions/modules

High-level products/functions/
modules specified in the
documents

Deposit
Withdrawal
ESANDA Payment
BA / SME SolArc/SME Cheque
EOD- Subscription


Issue in Requirement & Mitigation
Real-time
user
simulation
Information given in document Distribution missing in document may not
Total User Base 1444 be realistic
Business Process % of Users
No. of Deposit ??
Details Time
User
Withdrawal ??
8:30 am –
Concurrency 1441 Payment ??
10:30 Am
Cheque ??
12:00 pm –
Peak load 1441
03:30 pm EOD Subscription ??

User load distribution is
discussed and agreed during
the requirement Workshops.


Issue in Requirement & Mitigation
Step 2: Decompose high-level scenarios (products/functions/modules) into low-level scenarios or test
steps and assign response time for each step.

Internal Workshop

High-Level Scenario(s)

Withdrawal

Deposit Scenario
Scenario 1
2 Transactio
Withdrawal n
ESANDA Payments Step - 1 Step - 1 Response
Cheque time for
BA / SME SolArc/SME Test Team Step - 2 Step - 2
steps
EOD
Subscription would be
discussed
& agreed
Step - N Step - 28 in the
Workshops to decompose the Step – 30 requireme
high-level scenarios into the low- nt
level test cases/test steps. workshop.


Model Real Users / Think Time
Why must they be accurately modeled?
- Results from inaccurately modeled tests are nearly always inaccurate, and often
lead to incorrect decisions.
- The only way to predict actual user experience (end-to-end response time) is
to execute tests using realistic User Community Model (s).
- Extrapolating expected performance based on incomplete models doesn’t work.

You know what, end users will move at different speed in your Application
And…
It’s your job to design how to model the think time and execute script in varying speed

"The one thing that matters the most is not how your application behaves under
theoretical or simulated conditions, but how well it works when you plug it into the
wall and let everyone come hit your box from all across the world“ - Serdar
Yegulalp


How to TEST Non Functional
Requirements?

 Methodology
 Approach
 Execution & Monitoring
 Diagnose & Tune
 Analysis


Test Methodology


Test Approach

Performance Test Approach
Plan Design Execution

Understand Project Test Environment Script Development Test Data Create Performance
Goals and Timelines Setup & Validation Preparation Test Scenarios

Understand Customer Estimate Load Create Performance Analyze Initial Benchmark
Environment Generators Test Cases Production Data Execution

Understand Setup Controller & Generate Automated Tuning Scenarios
Identify Required Data
Application Load Generators Test Scripts (Repetitive)

Identify Business Customize Automated Generate Test Report
Setup Monitors, Logs Generate Test Data
Critical Processes Test Scripts & Log Defects

Identify Load Module Verification of Test Analyze & Identify
& Volumetrics Data Bottlenecks

Identify Performance Tuning Application
Test Scenarios

Determine Acceptable
Test Conditions
Performance Testing Final Benchmark
Execution
Team
Understand Tuning
Methodology
Collaborative Final Report

Create Performance Support
Test Plan


Execution & Monitoring

PERFORMANCE MONITORS

Automation Tool Automation Tool Internet/ Web Server Application Database
Controller Virtual Users WAN Server

 Replaces real users with thousands of virtual users
 Generates consistent, measurable, and repeatable load,
managed from a single point of control
 Efficiently isolates performance bottlenecks
across all tiers/layers with automated reporting
and analyses


Post Test Analysis
• Analyse most difficult and important data
• Check performance criteria is met or not
• Plan for next level of performance improvements
• Provide input to level phase of application design and development, tuning


Diagnose & Tune

Diagnose Tuning


How to DISCOVER Non Functional
Bottlenecks?

 What are bottlenecks?
 Analysis
 Example


What are the Bottlenecks?
A term used to describe a limiting resource in the system under test. Bottlenecks
are typically identified as the cause of slow or unacceptable performance. They
directly affect both the performance and scalability of the system under test.

Some of the common areas bottleneck can appear in any system are
 CPU burst
 Memory Leakage
 Disk threshold
 Network utilization
 Operating System limitations

Bottlenecks exist within systems for a number of reasons. These include:
 Inadequate server hardware to support the projected system load
 Inadequate network capacity, both internal and external
 Poor system and architecture design decisions
 Databases that are incorrectly implemented or tuned
 Developers not being mindful of performance considerations during development


Bottleneck Analysis
 Data collection to be done during test execution for bottleneck analysis
 Profilers – Record the time spent in different parts of a program. Profilers like SQL
profiler can be used for recording the time spent on SQL queries, stored procedures, etc.

 Traces – Record occurrence of various specified events. These traces can be on the
client side ( server response, communication between server and client, done by the load
controller) or at the server side (e.g. event viewer on the servers). Calls to web and app
servers can be accounted for by looking into the trace logs.

 Counters – To record the health of the servers during test execution. Load generating
tools or utilities like perfmon for windows, vmstat, iostat for unix, netstat for network
monitoring can be used.


Examples

The CPU utilization of the application server reached
unacceptable levels shortly before the response times
increased
Monitoring the CPU queue length resulted in the chart in
Figure, which showed a direct correlation between the
queue length and the poor performance.


Insufficient Database Calls

• Slow down in many transactions even under light load
Symptom • Running out of DB connections

Cause • Inefficient Database Calls

– Is Database fully tuned?
– Is connection pool in use and optimized?
– Are too many calls being made?
Diagnosis – Are stored procedures in use?
– Profile all SQL calls
– Isolate long running queries with JDBC instrumentation


Key Best Practices? - Take Away


Best Practices & Take Away
Performance Requirements Gathering is the key
 Workload to be driven by current and future business volume patterns Impact analysis of
one operation on the other

Data population
 Sufficient data volumes lead to correctly optimistic results 50% - 70% - 100% of Prod :
Good, Better, Best

Gap Between Test and Production Environments
 Performance great on test envi, but could bomb on production

Usage of WAN Emulator
 Obtain real-user perception for end to end performance (simulating WAN &Network)

Run Duration
 Test Run Duration to cover steady state
 Transient state readings alone, will give a wrong picture of performance

Test Early
 Performance-target-driven development
 Fine tune the database, application server
 Packaged solutions : Benchmark vanilla product for typical workloads


The audacity of quality requirement-non functional testing- Aware in BugDay Bangkok 2012

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to The audacity of quality requirement-non functional testing- Aware in BugDay Bangkok 2012

Similar to The audacity of quality requirement-non functional testing- Aware in BugDay Bangkok 2012 (20)

More from Prathan Dansakulcharoenkit

More from Prathan Dansakulcharoenkit (20)

Recently uploaded

Recently uploaded (20)

The audacity of quality requirement-non functional testing- Aware in BugDay Bangkok 2012