0
1© 2013 SOASTA. All rights reserved.Webinar
Present
How to plan, execute &
deliver actionable results
that matter!
with
2© 2013 SOASTA. All rights reserved.
Utilizing diverse skills for effective, realistic performance tests
TODAY’S PRESENTER...
3© 2013 SOASTA. All rights reserved.
o First End-to-End Quality as a Service Platform
• 1st Cloud-Based Load Testing Solut...
4© 2013 SOASTA. All rights reserved.
• Thought leadership in performance testing since 2001
• Acquired by Forsythe in 2012...
5© 2013 SOASTA. All rights reserved.
The Four Hats of Performance Testing
o Business Analyst: for effective test planning
...
6© 2013 SOASTA. All rights reserved.
Business Analyst
o Understand business drivers and derive test objectives in
dialogue...
7© 2013 SOASTA. All rights reserved.
Scalability Throughput
Capacity Workload Achieved
Business Analyst
8© 2013 SOASTA. All rights reserved.
Business Analyst
9© 2013 SOASTA. All rights reserved.
Business Analyst
Is this a
good SLA?
Is 12
seconds
a
problem
?
Did we test for
Chrome...
10© 2013 SOASTA. All rights reserved.
Development Standards
o Project directory
• Use cases, scripts, script data, load sc...
11© 2013 SOASTA. All rights reserved.
(no)
Developer
12© 2013 SOASTA. All rights reserved.
o Create a logical diagram of the system-under-test and the load
generating environm...
13© 2013 SOASTA. All rights reserved.
Systems Engineer
System Under Test
14© 2013 SOASTA. All rights reserved.
Systems Engineer
System Under Test
15© 2013 SOASTA. All rights reserved.
Systems Engineer
System Under Test
16© 2013 SOASTA. All rights reserved.
Systems Engineer
System Under Test
17© 2013 SOASTA. All rights reserved.
Systems Engineer
System Under Test
Load injection
Monitoring
18© 2013 SOASTA. All rights reserved.
Systems Engineer
CPU, IO, JVM heap,
DB connection Pool
CPU, IO, memory,
queued reque...
19© 2013 SOASTA. All rights reserved.
Customers
Systems Engineer
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Se...
20© 2013 SOASTA. All rights reserved.
“Conductor”
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
SUT/AU...
21© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
Gen...
22© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
Gen...
23© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
SUT/AUT
Web
Server...
24© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
Gen...
25© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
Gen...
26© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
Gen...
27© 2013 SOASTA. All rights reserved.
Analytics
CloudTest
Databas
e
Main
Systems Engineer
Perf Tester 1
Analytics
Load
Gen...
28© 2013 SOASTA. All rights reserved.
o Collect: Response times, errors, resources, anecdotes
o Aggregate: average, max, 9...
29© 2013 SOASTA. All rights reserved.
Data Analyst
30© 2013 SOASTA. All rights reserved.
Great performance testing needs to master four sets of
complementary skills – recrui...
31© 2013 SOASTA. All rights reserved.
Thanks
Contact SOASTA:
info@soasta.com
866.344.8766
Follow us:
twitter.com/cloudtest...
Upcoming SlideShare
Loading in...5
×

The Four Hats of Load and Performance Testing with special guest Mentora

1,045

Published on

Performance testing may be the most critical function to assuring business success and continuity under unexpected application stress conditions. Professionals in this domain develop several key skills to model realistic workloads, develop robust scripts, monitor complex environments, and deliver actionable results.

In this webinar hear how good teams effectively utilize the skills associated with the four hats of performance testing:

- Business Analyst, for effective test planning
- Developer, for creating maintainable scripts
- Systems Engineer, to identify and configure resource monitors
- Data Analyst, to interpret and report results

Dan Downing, Managing Principal at Mentora, is a veteran performance tester, teacher, author, and presenter, with 30 years of enterprise testing expertise. Join Dan and fellow test industry veteran, Brad Johnson, SOASTA’s VP of Product, as they explore these four key areas where skills and expert tools must intersect to deliver speed and quality in today’s fast moving companies.

About the presenters:

Dan Downing, Managing Principal, Application Testing, Mentora

Dan leads the Enterprise Application Performance Testing practice and serves as the principal consultant for quality assessments and large enterprise projects. He has 30 years technical and leadership experience as programmer, sales engineer, product manager, senior manager, and has led enterprise load testing projects for a variety of industries. Dan is widely regarded as a subject matter expert in load testing and created the 5-Steps of Load Testing methodology taught at Mercury Interactive. He is a frequent presenter at software quality conferences such as STAR, STPCon, and Workshop on Performance and Reliability for which he is one of the organizers.

Brad Johnson, VP Product, SOASTA

Brad Johnson has been supporting testers since the turn of the last century as head of monitoring and test products at Compuware, Mercury Interactive and Borland. He joined the new school of testing in 2009 when he signed on with SOASTA to deliver cloud testing on the CloudTest platform to a skeptical and established software testing market. Now, with the experience of tens-of-thousands of tests and hundreds of companies embracing the cloud, and using the same for mobile test automation, he’s helping expand the horizons of testers everywhere.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,045
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
20
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "The Four Hats of Load and Performance Testing with special guest Mentora"

  1. 1. 1© 2013 SOASTA. All rights reserved.Webinar Present How to plan, execute & deliver actionable results that matter! with
  2. 2. 2© 2013 SOASTA. All rights reserved. Utilizing diverse skills for effective, realistic performance tests TODAY’S PRESENTERS Dan Downing: Managing Principal, Mentora - @dandowning_ma Brad Johnson: VP Product Marketing, SOASTA- @bradjohnsonsv Ed Salazar: Sr. Performance Engineer, SOASTA Agenda: • Poll question • Understanding and Illustrating the “four hats” • CloudTest demonstration Questions: Submit in chat box during event
  3. 3. 3© 2013 SOASTA. All rights reserved. o First End-to-End Quality as a Service Platform • 1st Cloud-Based Load Testing Solution • 1st and Largest Global Test Cloud (17 Countries, 54 Locations, 800K Cloud Servers) • 1st Continuous Mobile Test Automation Solution • 1st “real-time” Real User Monitoring (RUM) Solution for web and mobile apps o Over 400 Global Corporate Customers • 20,000 Mobile Developers and Testers use SOASTA Cloud Services • Over 2,500 Mobile and Web Apps have been Tested with SOASTA. o Award Winning & Patented Technology • Industry Leader: Gartner Magic Quadrant & IDC Cloud Testing • Wall Street Journal Top 50 Hottest Companies three years running o Global Offices • San Francisco, New York, London, Mumbai, Shanghai & Tokyo
  4. 4. 4© 2013 SOASTA. All rights reserved. • Thought leadership in performance testing since 2001 • Acquired by Forsythe in 2012 • Specialize in performance testing of large-scale ecommerce, mobile and enterprise ERP systems • SOASTA delivery partner We bring the tools, infrastructure and subject matter expertise needed for any project.
  5. 5. 5© 2013 SOASTA. All rights reserved. The Four Hats of Performance Testing o Business Analyst: for effective test planning o Developer: for creating realistic, maintainable scripts o Systems Engineer: to identify and configure resource monitors o Data Analyst: to interpret and report results
  6. 6. 6© 2013 SOASTA. All rights reserved. Business Analyst o Understand business drivers and derive test objectives in dialogue with the business and technical sponsors o Select, prioritize and quantify the use cases to be tested with the business SMEs o Define the approach, Key Performance Indicators and "success" o Recruit the team: "It takes a small village to execute a successful performance test" o List the key activities, owners and schedule SKILLS: Translate business risk to performance requirements, project management
  7. 7. 7© 2013 SOASTA. All rights reserved. Scalability Throughput Capacity Workload Achieved Business Analyst
  8. 8. 8© 2013 SOASTA. All rights reserved. Business Analyst
  9. 9. 9© 2013 SOASTA. All rights reserved. Business Analyst Is this a good SLA? Is 12 seconds a problem ? Did we test for Chrome 30? At what point do users really leave? Did we test for Canada? Did we know this page was so popular?
  10. 10. 10© 2013 SOASTA. All rights reserved. Development Standards o Project directory • Use cases, scripts, script data, load scenarios, results, analyses/reports o Naming • Scripts: e.g., UC1_Browse, UC2_Search, UC3_AddToCart • Dynamic value parameters: p_searchPhrase, p_productId • Transaction timers: Common ones -- home, login, logout and script-specific steps -- UC1_01_enterPhrase, UC1_02_search, UC1_03_prodDetail o Script data -- Group related data values into a file • environment.dat (env, url, port); login.dat (loginId, pswd) o Script modularity • Initialization, end: one-time steps • business logic: logically grouped, iterative navigation and action steps SKILLS: Structure a team development project Developer
  11. 11. 11© 2013 SOASTA. All rights reserved. (no) Developer
  12. 12. 12© 2013 SOASTA. All rights reserved. o Create a logical diagram of the system-under-test and the load generating environment o Define the component configuration, operating system, software o Choose monitoring tools o Select and configure monitoring points o Execute test SKILLS: Overlay applications onto hardware, Unix and Windows performance monitor configuration Systems Engineer
  13. 13. 13© 2013 SOASTA. All rights reserved. Systems Engineer System Under Test
  14. 14. 14© 2013 SOASTA. All rights reserved. Systems Engineer System Under Test
  15. 15. 15© 2013 SOASTA. All rights reserved. Systems Engineer System Under Test
  16. 16. 16© 2013 SOASTA. All rights reserved. Systems Engineer System Under Test
  17. 17. 17© 2013 SOASTA. All rights reserved. Systems Engineer System Under Test Load injection Monitoring
  18. 18. 18© 2013 SOASTA. All rights reserved. Systems Engineer CPU, IO, JVM heap, DB connection Pool CPU, IO, memory, queued requests CPU, IOPS, DB locking contention Bandwidth throughput Load injection Monitoring
  19. 19. 19© 2013 SOASTA. All rights reserved. Customers Systems Engineer SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e
  20. 20. 20© 2013 SOASTA. All rights reserved. “Conductor” Analytics CloudTest Databas e Main Systems Engineer Perf Tester 1 SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e
  21. 21. 21© 2013 SOASTA. All rights reserved. Analytics CloudTest Databas e Main Systems Engineer Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e
  22. 22. 22© 2013 SOASTA. All rights reserved. Analytics CloudTest Databas e Main Systems Engineer Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e
  23. 23. 23© 2013 SOASTA. All rights reserved. Analytics CloudTest Databas e Main Systems Engineer Perf Tester 1 SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,…. “Conductor” Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS)
  24. 24. 24© 2013 SOASTA. All rights reserved. Analytics CloudTest Databas e Main Systems Engineer Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e Analytics Load GeneratorLoad GeneratorLoad Generator San Francisco (GoGrid) Analytics Load GeneratorLoad GeneratorLoad Generator Chicago (Rackspace) Analytics Load GeneratorLoad GeneratorLoad Generator Tokyo (AWS) Analytics Load GeneratorLoad Generator Amsterdam (Azure) Analytics Load GeneratorLoad Generator Virginia (IBM)
  25. 25. 25© 2013 SOASTA. All rights reserved. Analytics CloudTest Databas e Main Systems Engineer Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e Analytics Load GeneratorLoad GeneratorLoad Generator San Francisco (GoGrid) Analytics Load GeneratorLoad GeneratorLoad Generator Chicago (Rackspace) Analytics Load GeneratorLoad GeneratorLoad Generator Tokyo (AWS) Analytics Load GeneratorLoad Generator Amsterdam (Azure) Analytics Load GeneratorLoad Generator Virginia (IBM) SOASTA Global Test Cloud
  26. 26. 26© 2013 SOASTA. All rights reserved. Analytics CloudTest Databas e Main Systems Engineer Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) Analytics Load GeneratorLoad GeneratorLoad Generator San Francisco (GoGrid) Analytics Load GeneratorLoad GeneratorLoad Generator Chicago (Rackspace) Analytics Load GeneratorLoad GeneratorLoad Generator Tokyo (AWS) Analytics Load GeneratorLoad Generator Amsterdam (Azure) Analytics Load GeneratorLoad Generator Virginia (IBM) SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e SOASTA Global Test Cloud Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,…. “Conductor”
  27. 27. 27© 2013 SOASTA. All rights reserved. Analytics CloudTest Databas e Main Systems Engineer Perf Tester 1 Analytics Load GeneratorLoad GeneratorLoad Generator San Francisco (GoGrid) Analytics Load GeneratorLoad GeneratorLoad Generator Tokyo (AWS) Analytics Load GeneratorLoad Generator Amsterdam (Azure) Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,…. SUT/AUT Web Server Web Server App ServerWeb Server Web Server App Server App Server App Server Load Balancer Cache Cache Databas e Analytics Load GeneratorLoad Generator Virginia (IBM) Analytics Load GeneratorLoad GeneratorLoad Generator Chicago (Rackspace) SOASTA Global Test Cloud Access for all Analytics Load GeneratorLoad GeneratorLoad Generator East Coast (AWS) “Conductor”
  28. 28. 28© 2013 SOASTA. All rights reserved. o Collect: Response times, errors, resources, anecdotes o Aggregate: average, max, 95th percentile, end-to-end, at varying granularities o Visualize: Response times, resources, bandwidth “over load” o Interpret: Make observations, create and test hypotheses, support with data, draw conclusions o Assess: Compare to acceptable results, make recommendations o Report: Executive summary, supporting detail; assemble stakeholders and do read-outs SKILLS: Pattern recognition, Excel pivots, visualization tools, basic statistics Data Analyst
  29. 29. 29© 2013 SOASTA. All rights reserved. Data Analyst
  30. 30. 30© 2013 SOASTA. All rights reserved. Great performance testing needs to master four sets of complementary skills – recruit your team accordingly! o Without solid business requirements much hard work may yield little value o Poorly designed scripts make results analysis harder and maintenance difficult o Testing without monitoring is like flying an airplane without instruments o Well interpreted, actionable results delivers the business value – this is where you earn your stripes
  31. 31. 31© 2013 SOASTA. All rights reserved. Thanks Contact SOASTA: info@soasta.com 866.344.8766 Follow us: twitter.com/cloudtest facebook.com/cloudtest Knowledge Center White Papers Webinar Recordings Case Studies Additional Resources CloudLink Community Support Tutorials Video Contact Mentora: info@mentora.com 866.636.8672 Follow us: twitter.com/MentoraGroup Blog.Mentora.com www.mentora.com For whitepapers, presentations and resources visit Mentora.com/Resources www.SOASTA.com www.soasta.com/FREE Get Our Free Products
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×