2. Partners Testing Products
Pre-Sales, Sales, & Post Sales Support
Requirement Management Solution Development and Debugging Tools
Optimal Trace DevPartner Studio
Raven Flow DevPartner Java
Bounds checker
Functional Testing Tools Security checker
TestPartner
QA Director
Silk Test IT Service Management Products
Vantage
Performance Testing Tools Vantage for Network Performance
QALoad Vantage for Java/.NET Performance
Web Load Vantage for Server Performance
Silk Performer
2
3. IonIdea -Testing Services Offerings
Ion - Virtual Testing Team
Functional Testing Automation Services
Performance Testing Services
Software Testing Automation Consulting
Scripted Functional Testing Services
Software Testing outsourced
Training – Test Automation
3
4. Delivering High Performing Applications
What’s Needed: Load Testing with Infrastructure Monitoring
Virtual Users DBMS
Web Servers Application Servers
Mainframe
Combining load testing with infrastructure monitoring:
• Provides realistic emulation of transaction and traffic usage patterns to:
• Ensure servers will scale to meet current and future
needs
• Correlate important performance relationships
between all components of the application
4
infrastructure
5. Delivering High Performing Applications
What’s Needed: End-to-End Analysis of All Application Components
Virtual Users
DBMS
Web Servers Application Servers
Method JDBC-ODBC
Method
Method Mainframe
Method Web Service Method
URL JSP-ASP
Method Method Web Service Method MQ
JSP-ASP
End-to-end performance analysis:
• Provides the depth of analysis necessary to pinpoint performance
problems
5
7. Why Load Testing is Not Enough
NOT ENOUGH INFORMATION RESULT…
• Only delivers general response • Missed delivery dates
time, throughput or server metrics • Poor-quality applications
• Does not identify where bottlenecks are, • High-end resources
across environment or inside application involved in resolving
• Doesn’t get to the root cause of the problems and waiting until
problem the end of testing
• Leads to finger-pointing • Costly/unnecessary
infrastructure changes
NOT TIMELY to fix problems
• You have to wait until after
load testing to understand
whether you have a problem
7
8. What Sets Performance Triage apart?
PREDICTION: TROUBLESHOOTING:
• Predict performance • Deeper analysis during
under varying conditions load test
• Identify impact of network • Pinpoint Java and .NET
on application from performance problems
multiple locations • Perform fewer
• Pinpoint bottlenecks application retests
across application tiers
• Fix code prior to
conducting load testing
8
10. Performance Triage – Two mixes
Re-active
Leverage profiling to deal with performance issues before
load testing
Helps to make best use of limited testing timeframes
Pro-active
Well structured approach to performance testing to prevent
the ‘hair on fire’ scenarios
10
12. PT in Action - Case Study
Introduction
Setting the scene
• Online banking arm of large corporate finance house
• Urgent requirement to validate existing infrastructure capacity
and to investigate capacity to handle further growth
• Limited time to execute
12
13. PT in Action - Case Study
Performance Goals
• Response Time SLA
• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• All transaction response times less than 20 seconds
• Based on broadband bandwidth
• Concurrency SLA
• Support 600 concurrent end users
• Server Utilization SLA
• Server utilization < 50% measured as CPU, Memory and Disk I/O
13
14. PT in Action – Case Study
Load testing alone won’t identify hidden problems
Bank Data Centre
External Users
Web Servers
WAN Insufficient Capacity
Sensitivity
App Servers
Slow Methods
Internal Users
DB Servers
Contention Issues Bad SQL
14
15. PT in Action - Case Study
Identify and profile key transactions
• Enrollment WAN
Sensitivity Bad SQL
• Signon
• View account summary
• View account details
• Intra FI transfer
• View check images Profile
15
16. PT in Action – Case Study
Profile – Predicting WAN sensitivity
Increase in response time of 11 seconds when
connecting over T1 link with 50ms latency
16
17. PT in Action – Case Study
Profile – Bad SQL performance
SQL call taking in excess of 13 seconds
to complete
17
18. PT in Action – Case Study
Profiling identifies hidden problems
Bank Data Centre
External Users
Web Servers
Insufficient Capacity
WAN Sensitivity
App Servers
Internal Users Slow Methods
DB Servers
Contention Issues
Bad SQL
18
19. PT in Action – Case Study
Load Testing
Insufficient Capacity
Performance goals
Contention Issues
• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• All transaction response times less than 20 seconds Slow Methods
• Support 600 concurrent end users
• Server utilization < 50% measured as CPU, Memory and Disk I/O
Load Test
19
20. PT in Action – Case Study
Load Testing – Transaction performance FAIL!!
Response Time
Transaction performance exceeds response time SLA
Concurrent users
Concurrent SLA: 600 users
Response time SLA: < 20 seconds
Elapsed time
Transaction response time
20
21. PT in Action – Case Study
Load Testing – Server performance FAIL!!
Insufficient Capacity
Web server CPU utilization breaches SLA
Concurrent users
Concurrent SLA: 600 users
Server CPU SLA: < 50 %
Web server CPU %
Elapsed time
21
22. PT in Action – Case Study
Load Testing – Application server FAIL!!
Slow Method
Analysis inside the JVM and CLR
Slow performing method
impacting response time
and web server CPU
22
23. PT in Action – Case Study
Load Testing – Application server FAIL!!
Memory Leak
Analysis inside the JVM and CLR
Memory not being
released
23
24. PT in Action – Case Study
Load Testing – Transaction performance SUCCESS!!
Transaction performance remains below response time SLA
Concurrent users
Concurrent SLA: 600 users
Response time SLA: < 20 seconds
Elapsed time
Transaction response time
24
25. PT in Action – Case Study
Load Testing – Server performance SUCCESS!!
Web server CPU utilization remains below SLA
Concurrent users
Concurrent SLA: 600 users
Server CPU SLA: < 50 %
Web server CPU %
Elapsed time
25
26. PT in Action – Case Study
Profiling + Load Testing = All problems identified
Bank Data Centre
External Users
Web Servers
Insufficient Capacity
WAN Sensitivity
App Servers
Internal Users Slow Methods
DB Servers
Contention Issues
Bad SQL
26
27. PT in Action – Case Study
Deliverables
• Response Time SLA PASS!!
• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• All transaction response times less than 20 seconds
• Based on broadband bandwidth
• Concurrency SLA
PASS!!
• Support 600 concurrent end users
• Server Utilization SLA
• Server utilization < 50% measured as CPU, Memory and Disk I/O
PASS!!
27
28. PT – ‘Pro-active’ remix
PT – Becoming pro-active
Avoiding the ‘Hair on Fire’ scenario
28
29. PT ‘Pro-active remix’ – Mission Statement
• Performance considerations should be factored
into the application lifecycle at the earliest
opportunity
29
31. PT ‘Pro-active remix’ – Requirements capture
Performance goals
• Response Time SLA
• Login / Signon less than 10 seconds
• All other page response times less than 8 seconds
• Transaction response time less than 20 seconds
• Based on broadband bandwidth
• Application Concurrency SLA
• 600 concurrent end users
• Server Resource Utilization SLA
• Less than 50% measured as CPU, Memory, Network Utilization and Disk I/O
31
32. PT ‘Pro-active remix’ – Requirements capture
Test environment
Compare production to test system and identify
differences
32
34. PT ‘Pro-active remix’ – Requirements capture
Transaction detail + Data
View Account Summary
• Home Page <- Target URL
• Logon Page <- Login credentials
• Verification Page
• Challenge Page
• Retrieving default account summary
• Retrieving maximum amount of account history (90 days max)
• Logoff
34
35. PT ‘Pro-active remix’ – Requirements capture
Identify data requirements
• A minimum of 100 sets of unique end user login credentials
• Test database to be a recent cut of live database
• Each consumer banking logon account should contain:
• 2-3 different accounts. (If multiple hosts are accessed during a logon,
additional considerations may be warranted).
• Several items of history in one of the accounts – generally a direct
draw/checking account. Best results are obtained when this transaction
history contains 20 or more items per month distributed over 90 days.
35
36. PT ‘Pro-active remix’ – Requirements capture
Virtual user allocation
Allocation of virtual users to transactions
36
37. PT ‘Pro-active remix’ – Requirements capture
Summary
Critical requirements identified
Performance goals ( SLA’s )
Validation of performance test environment
Data requirements identified
Key transactions identified
Virtual user allocation
37
