The audacity of quality requirement-non functional testing- Aware in BugDay Bangkok 2012

The Audacity of Quality Requirements: Non Functional Testing Testing as Managed Services “Think  Communicate  Implement” By - Sudeepta Guchhait (Deep) “Quality is the link to Success” Copyright © 2012 Aware Corporation Ltd.

INTENT How to DEFINE business problem? How to DERIVE high quality non functional test requirements? How to VERIFY high quality non functional test requirements? How to TEST non functional test requirements? How to DISCOVER the bottlenecks? Key BEST Practices? – Take Away Copyright © 2012 Aware Corporation Ltd.

Proposition – Test Upfront Defect Removal When to Test Requirement (15%): Performance Modeling, Think time % Original Defects modeling 100% 100% 15% 20% Test design (20%): • Identification of critical business transactions, defining the test 20% SLA, User load distribution, Transactional volume and peak 80% 35% usage 60% 10% % Infrastructure (20%): Reduction – Investment needs to be made to procure the production like 40% NFT test environment (Physical as well Logical design) – Appropriate test tool procurement 20% Testing (35%): 0% Original Req Infra & Testing Prod Defects – Test execution to be done for performance test, load test, Design Defects Test lab Stress, Scalability (Capacity) and Availability (Reliability). Qualitative Consistency — on time, on budget delivery to business, increased client satisfaction, Decreased Benefits implementation risk. Quantitative Benefits Identify system & application bottlenecks to fine tune those. √ Improved software quality with varied data sets √ Fully tested instance in line with the expectation of business users Copyright © 2012 Aware Corporation Ltd.

Challenges of Performance testing Creation of tests  Determine what to simulate?  Determine which performance factors are critical?  Analyze desired test results  Fine tune the bottlenecks Infrastructure  Build a realistic test environment (Infrastructure usage)  Select the appropriate test tool Usability of tests  Software build should be functionally stable  Tests are resource-intensive Copyright © 2012 Aware Corporation Ltd.

Non Functional Objectives Performance Testing Application Response Time Capacity Planning How long does it take to Complete At what point does degradation in a task? Performance occur? Reliability Bottleneck Identification How stable is the system under a What is the cause of degradation heavy load? In Performance? Acceptance Product Evaluation Is the system stable enough to go What is the best server configuration into production? for 100-500-1000 users? Copyright © 2012 Aware Corporation Ltd.

Practical conversation on Performance of Application  “I think we need Performance Testing, but what is it exactly?”  “I know you want it to be fast, but how fast?”  “I think around 300 users will use the system, they will do all kinds of activity, so can we determine performance ?  “What you will do with production data in performance?”  “Post Test charts looks nice, let me know the performance is good or bad”  “Does that mean we’re done? Can we release?” Copyright © 2012 Aware Corporation Ltd.

Identification of Performance Transactions 80/20 rule - 20% of the critical transaction will be performed by 80% of real business users. - high visibility activities • Creating initial user profile • Updating payment information - high importance activities • Withdrawals • Transfer funds • Paying bill on-line - performance intensive activities • Importing the monthly report from “the other system” • Requesting payment history • Heavy weight transactions (Stock trade, IPO subscription etc.) - Identify the “heavy hitters” even though they may be used less often Copyright © 2012 Aware Corporation Ltd.

Performance Modeling (Validation) “Performance validation is the process by which software is tested with the intent of determining if the software meets pre-existing performance requirements. This process aims to evaluate compliance.” Primarily used for…  Determining SLA compliance  Determine user behavior load  Determine the volume usage Copyright © 2012 Aware Corporation Ltd.

Performance Testing “Performance testing is the process by which software is tested to determine the current system performance. This process aims to gather information about current performance, but places no value judgments on the findings.” Primarily used for…  Determining capacity of existing systems  Creating benchmarks for future systems  Evaluating degradation with various loads and/or configurations Copyright © 2012 Aware Corporation Ltd.

Performance Engineering “Performance engineering is the process by which software is tested and tuned with the intent of realizing the required performance. This process aims to optimize the most important application performance trait, user experience.” Primarily used for…  Fine application bottlenecks  Extending the capacity of old systems.  Fixing problems of the application that are not meeting requirements/SLAs. Copyright © 2012 Aware Corporation Ltd.

Defect Root Cause Identification  Identify Key Scenarios: Identify scenarios where performance is important and scenarios that pose the most risk to performance objectives  Identify Workload: Identify how many users and how many concurrent users system needs to support  Identify Performance Objectives: Identify performance objectives for each of the key scenarios.  Baseline is the process of running a set of tests to capture performance metric data for the purpose of evaluating the effectiveness of subsequent performance improving changes to the system or application.  Benchmarking is the process of comparing your system performance against an industry standard that is endorsed by some other organization. Copyright © 2012 Aware Corporation Ltd.

Defect Root Cause Identification - Example API Processing Monitoring Step 1: Funds Transfer Menu (App1 –App2) Network Monitoring Server Monitoring App1 App2 F1 5 A Fn A 3 2 4 6 1 P P 7 12 I 10 I 8 11 9 12 times * number of sec API Processing MonitoringServer Monitoring Point-to-Point – Component Level View 1. Captures the internal processing time between steps (low-level system processing) 2. No ability to measure this with testing tool alone 3. Requires monitors to be installed to collect internal processing time at each step Copyright © 2012 Aware Corporation Ltd.

Requirement Design Issue 1: Illustrative Example Step 1: Determine user load distribution across products/functions/modules High-level products/functions/ modules specified in the documents Deposit Withdrawal ESANDA Payment BA / SME SolArc/SME Cheque EOD- Subscription Copyright © 2012 Aware Corporation Ltd.

Issue in Requirement & Mitigation Real-time user simulation Information given in document Distribution missing in document may notTotal User Base 1444 be realistic Business Process % of Users No. of Deposit ?? Details Time User Withdrawal ?? 8:30 am – Concurrency 1441 Payment ?? 10:30 Am Cheque ?? 12:00 pm – Peak load 1441 03:30 pm EOD Subscription ?? User load distribution is discussed and agreed during the requirement Workshops. Copyright © 2012 Aware Corporation Ltd.

Issue in Requirement & MitigationStep 2: Decompose high-level scenarios (products/functions/modules) into low-level scenarios or teststeps and assign response time for each step. Internal Workshop High-Level Scenario(s) Withdrawal Deposit Scenario Scenario 1 2 Transactio Withdrawal n ESANDA Payments Step - 1 Step - 1 Response Cheque time forBA / SME SolArc/SME Test Team Step - 2 Step - 2 steps EOD Subscription would be discussed & agreed Step - N Step - 28 in the Workshops to decompose the Step – 30 requireme high-level scenarios into the low- nt level test cases/test steps. workshop. Copyright © 2012 Aware Corporation Ltd.

Model Real Users / Think TimeWhy must they be accurately modeled?- Results from inaccurately modeled tests are nearly always inaccurate, and often lead to incorrect decisions.- The only way to predict actual user experience (end-to-end response time) is to execute tests using realistic User Community Model (s).- Extrapolating expected performance based on incomplete models doesn’t work.You know what, end users will move at different speed in your ApplicationAnd…It’s your job to design how to model the think time and execute script in varying speed"The one thing that matters the most is not how your application behaves under theoretical or simulated conditions, but how well it works when you plug it into the wall and let everyone come hit your box from all across the world“ - Serdar Yegulalp Copyright © 2012 Aware Corporation Ltd.

Test Approach Performance Test Approach Plan Design Execution Understand Project Test Environment Script Development Test Data Create Performance Goals and Timelines Setup & Validation Preparation Test Scenarios Understand Customer Estimate Load Create Performance Analyze Initial Benchmark Environment Generators Test Cases Production Data Execution Understand Setup Controller & Generate Automated Tuning Scenarios Identify Required Data Application Load Generators Test Scripts (Repetitive) Identify Business Customize Automated Generate Test Report Setup Monitors, Logs Generate Test Data Critical Processes Test Scripts & Log Defects Identify Load Module Verification of Test Analyze & Identify & Volumetrics Data Bottlenecks Identify Performance Tuning Application Test Scenarios Determine Acceptable Test Conditions Performance Testing Final Benchmark Execution Team Understand Tuning Methodology Collaborative Final Report Create Performance Support Test Plan Copyright © 2012 Aware Corporation Ltd.

Execution & Monitoring PERFORMANCE MONITORS Automation Tool Automation Tool Internet/ Web Server Application Database Controller Virtual Users WAN Server  Replaces real users with thousands of virtual users  Generates consistent, measurable, and repeatable load, managed from a single point of control  Efficiently isolates performance bottlenecks across all tiers/layers with automated reporting and analyses Copyright © 2012 Aware Corporation Ltd.

Post Test Analysis • Analyse most difficult and important data • Check performance criteria is met or not • Plan for next level of performance improvements • Provide input to level phase of application design and development, tuning Copyright © 2012 Aware Corporation Ltd.

What are the Bottlenecks? A term used to describe a limiting resource in the system under test. Bottlenecks are typically identified as the cause of slow or unacceptable performance. They directly affect both the performance and scalability of the system under test. Some of the common areas bottleneck can appear in any system are  CPU burst  Memory Leakage  Disk threshold  Network utilization  Operating System limitations Bottlenecks exist within systems for a number of reasons. These include:  Inadequate server hardware to support the projected system load  Inadequate network capacity, both internal and external  Poor system and architecture design decisions  Databases that are incorrectly implemented or tuned  Developers not being mindful of performance considerations during development Copyright © 2012 Aware Corporation Ltd.

Bottleneck Analysis Data collection to be done during test execution for bottleneck analysis  Profilers – Record the time spent in different parts of a program. Profilers like SQL profiler can be used for recording the time spent on SQL queries, stored procedures, etc.  Traces – Record occurrence of various specified events. These traces can be on the client side ( server response, communication between server and client, done by the load controller) or at the server side (e.g. event viewer on the servers). Calls to web and app servers can be accounted for by looking into the trace logs.  Counters – To record the health of the servers during test execution. Load generating tools or utilities like perfmon for windows, vmstat, iostat for unix, netstat for network monitoring can be used. Copyright © 2012 Aware Corporation Ltd.

ExamplesThe CPU utilization of the application server reachedunacceptable levels shortly before the response timesincreased Monitoring the CPU queue length resulted in the chart in Figure, which showed a direct correlation between the queue length and the poor performance. Copyright © 2012 Aware Corporation Ltd.

Insufficient Database Calls • Slow down in many transactions even under light load Symptom • Running out of DB connections Cause • Inefficient Database Calls – Is Database fully tuned? – Is connection pool in use and optimized? – Are too many calls being made? Diagnosis – Are stored procedures in use? – Profile all SQL calls – Isolate long running queries with JDBC instrumentation Copyright © 2012 Aware Corporation Ltd.

Best Practices & Take Away Performance Requirements Gathering is the key  Workload to be driven by current and future business volume patterns Impact analysis of one operation on the other Data population  Sufficient data volumes lead to correctly optimistic results 50% - 70% - 100% of Prod : Good, Better, Best Gap Between Test and Production Environments  Performance great on test envi, but could bomb on production Usage of WAN Emulator  Obtain real-user perception for end to end performance (simulating WAN &Network) Run Duration  Test Run Duration to cover steady state  Transient state readings alone, will give a wrong picture of performance Test Early  Performance-target-driven development  Fine tune the database, application server  Packaged solutions : Benchmark vanilla product for typical workloads Copyright © 2012 Aware Corporation Ltd.

The audacity of quality requirement-non functional testing- Aware in BugDay Bangkok 2012

by Prathan D. on May 24, 2012

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 404

Statistics

The audacity of quality requirement-non functional testing- Aware in BugDay Bangkok 2012 Presentation Transcript