The document outlines a webinar on planning and executing effective performance testing, featuring presenters from Soasta and Mentora. It highlights Soasta's achievements in cloud-based testing solutions and emphasizes the importance of various roles in performance testing, including business analysts, developers, systems engineers, and data analysts. Key points include skills needed for each role and the significance of collecting, interpreting, and reporting performance data for actionable results.

1. 1© 2013 SOASTA. All rights reserved.Webinar
Present
How to plan, execute &
deliver actionable results
that matter!
with

2. 2© 2013 SOASTA. All rights reserved.
Utilizing diverse skills for effective, realistic performance tests
TODAY’S PRESENTERS
Dan Downing: Managing Principal, Mentora - @dandowning_ma
Brad Johnson: VP Product Marketing, SOASTA- @bradjohnsonsv
Ed Salazar: Sr. Performance Engineer, SOASTA
Agenda:
• Poll question
• Understanding and Illustrating the “four hats”
• CloudTest demonstration
Questions:
Submit in chat box during event

3. 3© 2013 SOASTA. All rights reserved.
o First End-to-End Quality as a Service Platform
• 1st Cloud-Based Load Testing Solution
• 1st and Largest Global Test Cloud (17 Countries, 54 Locations, 800K Cloud Servers)
• 1st Continuous Mobile Test Automation Solution
• 1st “real-time” Real User Monitoring (RUM) Solution for web and mobile apps
o Over 400 Global Corporate Customers
• 20,000 Mobile Developers and Testers use SOASTA Cloud Services
• Over 2,500 Mobile and Web Apps have been Tested with SOASTA.
o Award Winning & Patented Technology
• Industry Leader: Gartner Magic Quadrant & IDC Cloud Testing
• Wall Street Journal Top 50 Hottest Companies three years running
o Global Offices
• San Francisco, New York, London, Mumbai, Shanghai & Tokyo

4. 4© 2013 SOASTA. All rights reserved.
• Thought leadership in performance testing since 2001
• Acquired by Forsythe in 2012
• Specialize in performance testing of large-scale ecommerce, mobile
and enterprise ERP systems
• SOASTA delivery partner
We bring the tools, infrastructure and
subject matter expertise needed for
any project.

5. 5© 2013 SOASTA. All rights reserved.
The Four Hats of Performance Testing
o Business Analyst: for effective test planning
o Developer: for creating realistic, maintainable scripts
o Systems Engineer: to identify and configure resource monitors
o Data Analyst: to interpret and report results

6. 6© 2013 SOASTA. All rights reserved.
Business Analyst
o Understand business drivers and derive test objectives in
dialogue with the business and technical sponsors
o Select, prioritize and quantify the use cases to be tested
with the business SMEs
o Define the approach, Key Performance Indicators and
"success"
o Recruit the team: "It takes a small village to execute a
successful performance test"
o List the key activities, owners and schedule
SKILLS: Translate business risk to performance
requirements, project management

7. 7© 2013 SOASTA. All rights reserved.
Scalability Throughput
Capacity Workload Achieved
Business Analyst

8. 8© 2013 SOASTA. All rights reserved.
Business Analyst

9. 9© 2013 SOASTA. All rights reserved.
Business Analyst
Is this a
good SLA?
Is 12
seconds
a
problem
?
Did we test for
Chrome 30?
At what point do users
really leave?
Did we
test for
Canada?
Did we
know this
page was
so
popular?

10. 10© 2013 SOASTA. All rights reserved.
Development Standards
o Project directory
• Use cases, scripts, script data, load scenarios, results, analyses/reports
o Naming
• Scripts: e.g., UC1_Browse, UC2_Search, UC3_AddToCart
• Dynamic value parameters: p_searchPhrase, p_productId
• Transaction timers: Common ones -- home, login, logout and script-specific steps --
UC1_01_enterPhrase, UC1_02_search, UC1_03_prodDetail
o Script data -- Group related data values into a file
• environment.dat (env, url, port); login.dat (loginId, pswd)
o Script modularity
• Initialization, end: one-time steps
• business logic: logically grouped, iterative navigation and action steps
SKILLS: Structure a team development project
Developer

11. 11© 2013 SOASTA. All rights reserved.
(no)
Developer

12. 12© 2013 SOASTA. All rights reserved.
o Create a logical diagram of the system-under-test and the load
generating environment
o Define the component configuration, operating system, software
o Choose monitoring tools
o Select and configure monitoring points
o Execute test
SKILLS: Overlay applications onto hardware, Unix and Windows
performance monitor configuration
Systems Engineer

13. 13© 2013 SOASTA. All rights reserved.
Systems Engineer
System Under Test

14. 14© 2013 SOASTA. All rights reserved.
Systems Engineer
System Under Test

15. 15© 2013 SOASTA. All rights reserved.
Systems Engineer
System Under Test

16. 16© 2013 SOASTA. All rights reserved.
Systems Engineer
System Under Test

17. 17© 2013 SOASTA. All rights reserved.
Systems Engineer
System Under Test
Load injection
Monitoring

18. 18© 2013 SOASTA. All rights reserved.
Systems Engineer
CPU, IO, JVM heap,
DB connection Pool
CPU, IO, memory,
queued requests
CPU, IOPS, DB
locking contention
Bandwidth throughput
Load injection
Monitoring

19. 19© 2013 SOASTA. All rights reserved.
Customers
Systems Engineer
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e

20. 20© 2013 SOASTA. All rights reserved.
“Conductor”
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e

21. 21© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e

22. 22© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e

23. 23© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
“Conductor”
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)

24. 24© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Chicago (Rackspace)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)

25. 25© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Chicago (Rackspace)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)
SOASTA
Global
Test Cloud

26. 26© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Chicago (Rackspace)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
SOASTA
Global
Test Cloud
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
“Conductor”

27. 27© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Chicago (Rackspace)
SOASTA
Global
Test Cloud
Access
for all
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
“Conductor”

28. 28© 2013 SOASTA. All rights reserved.
o Collect: Response times, errors, resources, anecdotes
o Aggregate: average, max, 95th percentile, end-to-end, at
varying granularities
o Visualize: Response times, resources, bandwidth “over load”
o Interpret: Make observations, create and test hypotheses,
support with data, draw conclusions
o Assess: Compare to acceptable results, make
recommendations
o Report: Executive summary, supporting detail; assemble
stakeholders and do read-outs
SKILLS: Pattern recognition, Excel pivots, visualization
tools, basic statistics
Data Analyst

29. 29© 2013 SOASTA. All rights reserved.
Data Analyst

30. 30© 2013 SOASTA. All rights reserved.
Great performance testing needs to master four sets of
complementary skills – recruit your team accordingly!
o Without solid business requirements much hard work may yield little
value
o Poorly designed scripts make results analysis harder and maintenance
difficult
o Testing without monitoring is like flying an airplane without instruments
o Well interpreted, actionable results delivers the business value – this is
where you earn your stripes

31. 31© 2013 SOASTA. All rights reserved.
Thanks
Contact SOASTA:
info@soasta.com
866.344.8766
Follow us:
twitter.com/cloudtest
facebook.com/cloudtest
Knowledge Center
White Papers
Webinar Recordings
Case Studies
Additional Resources
CloudLink Community
Support
Tutorials
Video
Contact Mentora:
info@mentora.com
866.636.8672
Follow us:
twitter.com/MentoraGroup
Blog.Mentora.com
www.mentora.com
For whitepapers, presentations and
resources visit
Mentora.com/Resources
www.SOASTA.com
www.soasta.com/FREE
Get Our Free Products