Best Practices inPerformance Testing   Issues Uncovered as                      Scale IncreasesWebinar Series
The Test LabGoals:• Isolate bugs in code• Optimize page performance• Plug memory leaks• Tune databases• Establish performa...
The Staging EnvironmentGoals:• Verify capacity• Establish configurations• Stress available resources• Verify performance e...
The Full Production EnvironmentGoals:• Test to and beyond expected limits• Verify entire infrastructure• Confirm load bala...
Performance isn’t One-Dimensional“You have a fever.” Now what?                         External web load                  ...
Performance is Multi-DimensionalFind problems fast when metrics are viewed in context, at scale                           ...
Performance is Multi-DimensionalFind problems fast when metrics are viewed in context, at scale                           ...
Extending Performance Beyond the Lab        Discovering issues across teams, time and scaleStage / Team                   ...
Extending Performance Beyond the Lab        Discovering issues across teams, time and scaleStage / Team                   ...
Extending Performance Beyond the Lab        Discovering issues across teams, time and scale                               ...
CloudTest LiveReal-world Issue ResolutionCompany Profile                Project ProfileCompany     Toy Manufacturer   Envi...
CloudTest Live Real-world Issue Resolution• First test cycle in the lab • Some servers hitting 100% CPU   Lab• 239 hits pe...
CloudTest Live Real-world Issue Resolution• Second test cycle in the lab   • Some servers hitting 100% CPU    Lab• Higher ...
CloudTest Live Real-world Issue Resolution• Duration problem was crippling – could not recover   Lab
CloudTest Live Real-world Issue Resolution• Bandwidth – ouch!                               Staging• CPU at 100%
CloudTest Live Real-world Issue Resolution• 767 Home pages served at 650 megs total transfer.• That’s an 850kb page weight...
CloudTest LiveReal-world Issue Resolution           Mission Debriefing - Fortune 500 Toy Manufacturer              Lab    ...
CloudTest LiveReal-world Issue ResolutionCompany Profile                      Project ProfileCompany     Media and Broadca...
CloudTest Live Real-world Issue Resolution• Steadily increasing response times• Constant stream of errors (< 1%)     Produ...
CloudTest LiveReal-world Issue Resolution• One rogue page – streaming the TV episode                                      ...
CloudTest LiveReal-world Issue Resolution• Connection reset errors starting the video stream• Entitlement license server c...
CloudTest LiveReal-world Issue Resolution          Mission Debriefing - Fortune 500 Media Company     Production          ...
CloudTest LiveReal-world Issue ResolutionCompany Profile                   Project Profile Company    Advanced Technology ...
CloudTest LiveReal-world Issue Resolution   • Recurring error pattern during scenario completion   Production
CloudTest LiveReal-world Issue Resolution             Awesome outcome in Production!
CloudTest LiveReal-world Issue Resolution         Mission Debriefing - Fortune 50 Technology Company      Production      ...
Q&APlease join us for the next webinar in this series:               Testing Mobile Applications            May 3rd at 10A...
Upcoming SlideShare
Loading in...5
×

Webinar issues we_find_slideshare

229

Published on

SOASTA's tens-of-thousands of tests in customer labs and production environments have uncovered issues that range from code level bugs to issues in 3rd party services. Testing early, often and at real scale is the only way to be fully prepared.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
229
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Good Morning and welcome to the second in our new series, Best Practices in Performance Testing. This series is designed to share with you the best practices our performance engineering team is developing as they work with world-class customers of all sizes. Today we will be exploring what issues are uncovered as customers test applications starting a small scale in development and moving to full-scale testing in production. My name is Betsy Zikakis, I’m the VP, Marketing for Soasta and I’ll be your MC today.
  • Thank you so much for attending this webinar. The archive will be available on our website by the end of this week. Please tell others about it. Our next webinar in this series is focused on testing mobile applications. You can go to the link on this slide to register now. It will be up on the SOASTA home page to register tomorrow morning. And you will receive an email invitation next week.
  • Webinar issues we_find_slideshare

    1. 1. Best Practices inPerformance Testing Issues Uncovered as Scale IncreasesWebinar Series
    2. 2. The Test LabGoals:• Isolate bugs in code• Optimize page performance• Plug memory leaks• Tune databases• Establish performance expectationsLimitations:• Production-quality hardware• IT support and resources• Bandwidth and network infrastructure•3rd party application components• Time
    3. 3. The Staging EnvironmentGoals:• Verify capacity• Establish configurations• Stress available resources• Verify performance expectationsLimitations:• Availability / scheduling• Not full-scale• Excludes key components• Time
    4. 4. The Full Production EnvironmentGoals:• Test to and beyond expected limits• Verify entire infrastructure• Confirm load balancing and firewall integrity• Certify performance in normal and extreme conditionsBenefits:• All components• 3rd party apps• All integrations• Continuous• Global latencies• Reality
    5. 5. Performance isn’t One-Dimensional“You have a fever.” Now what? External web load User Experience QA OPS
    6. 6. Performance is Multi-DimensionalFind problems fast when metrics are viewed in context, at scale External and Internal user load Complex Web Applications : AJAX, HTML5, video, mobile, web services User Experience Multi-Dimensional Views
    7. 7. Performance is Multi-DimensionalFind problems fast when metrics are viewed in context, at scale External and Internal user load Complex Web Applications : AJAX, HTML5, video, mobile, web services Application Deployment Deployment Component & Infrastructure Infrastructure Network User Experience Code (Internal) (External) Extract Transport Load Real-Time End to End The CloudTest Performance Data platform is built for scalability and real- In-Memory time identification OLAP of issues Multi-Dimensional Views
    8. 8. Extending Performance Beyond the Lab Discovering issues across teams, time and scaleStage / Team Debugging code Garbage collection Memory leaks Memory leaks Database thread counts Dev & Test Slow pages Inefficient database queries Method-level tuning Test Lab (a few thousand users) Scale of Test
    9. 9. Extending Performance Beyond the Lab Discovering issues across teams, time and scaleStage / Team Configuration Setting & Sizing Auto-scaling failures Release Max sockets exceeded Global latency variance & Deploy Slow third-party plug-ins Security bottlenecks Default configuration settings Search technology limits Inadequate server resources Garbage collection Memory leaks Memory leaks Database thread counts Dev & Test Slow pages Inefficient database queries Method-level tuning Test Lab Staging (thousands to 10’s of thousands) Scale of Test
    10. 10. Extending Performance Beyond the Lab Discovering issues across teams, time and scale Tuning for full-scale…and well beyond Latency between systems Network Network configuration & Ops Network bandwidth CDN file placement Conflict with other apps DNS routingStage / Team Load Balancer configuration Firewall max capacity Auto-scaling failures Unbalanced web servers Release & Deploy Max sockets exceeded s Global latency variance Security bottlenecks Slow third-party plug-ins Default configuration settings Search technology limits Inadequate server resources Garbage collection Memory leaks Memory leaks Database thread counts Dev & Test Slow pages Inefficient database queries Method-level tuning Test Lab Staging Production (100% +++) Scale of Test
    11. 11. CloudTest LiveReal-world Issue ResolutionCompany Profile Project ProfileCompany Toy Manufacturer Environment Lab and StagingSize Fortune 500 Application Drupal (Apache, PHP, MySQL) Location Private Hosted Cloud Objective Identify Breaking Points Project Length Five Days (non-consecutive) Lab Staging
    12. 12. CloudTest Live Real-world Issue Resolution• First test cycle in the lab • Some servers hitting 100% CPU Lab• 239 hits per second • Bottleneck in throughput• 40.5 MBit/second bandwidth
    13. 13. CloudTest Live Real-world Issue Resolution• Second test cycle in the lab • Some servers hitting 100% CPU Lab• Higher concurrency target • Performance problem over time• Bandwidth stayed the same • Load balancer flapping servers
    14. 14. CloudTest Live Real-world Issue Resolution• Duration problem was crippling – could not recover Lab
    15. 15. CloudTest Live Real-world Issue Resolution• Bandwidth – ouch! Staging• CPU at 100%
    16. 16. CloudTest Live Real-world Issue Resolution• 767 Home pages served at 650 megs total transfer.• That’s an 850kb page weight! Yikes! Staging
    17. 17. CloudTest LiveReal-world Issue Resolution Mission Debriefing - Fortune 500 Toy Manufacturer Lab Staging• Poor throughput on the initial tests • Massive page weight on the home page• CPU hits 100% • Application unresponsive at ~2000 users• Bandwidth cap identified in the lab• Database connections leaking over time• Load balancer dropping servers Outcome • Customer happy with the breaking point identified • Optimized web assets in the home page • Offloaded assets to the CDN • Load balancer algorithm optimizations
    18. 18. CloudTest LiveReal-world Issue ResolutionCompany Profile Project ProfileCompany Media and Broadcasting Environment ProductionSize Fortune 500 Application Java Location Private Data Center Objective 25,000 Concurrent Project Length Three Days Production
    19. 19. CloudTest Live Real-world Issue Resolution• Steadily increasing response times• Constant stream of errors (< 1%) Production• Variety of error types
    20. 20. CloudTest LiveReal-world Issue Resolution• One rogue page – streaming the TV episode Production
    21. 21. CloudTest LiveReal-world Issue Resolution• Connection reset errors starting the video stream• Entitlement license server calls failing Production
    22. 22. CloudTest LiveReal-world Issue Resolution Mission Debriefing - Fortune 500 Media Company Production Outcome• Connection issues with CDN • Worked with CDN provider to reinforce certain geographies• One problem page • Added capacity to the license server cluster• Massive bandwidth usage
    23. 23. CloudTest LiveReal-world Issue ResolutionCompany Profile Project Profile Company Advanced Technology Environment Production Size Fortune 50 Application Java (Struts), Oracle Multiple HostedPrivate Location Data Centers Objective 172,000 Concurrent Project Length Two Months Production
    24. 24. CloudTest LiveReal-world Issue Resolution • Recurring error pattern during scenario completion Production
    25. 25. CloudTest LiveReal-world Issue Resolution Awesome outcome in Production!
    26. 26. CloudTest LiveReal-world Issue Resolution Mission Debriefing - Fortune 50 Technology Company Production Outcome• Load balancer firmware bug with 2048 bit cert • Worked with hardware LB manufacturer on a fix• Database issues with the login process • Really low error rate • Successful launch!
    27. 27. Q&APlease join us for the next webinar in this series: Testing Mobile Applications May 3rd at 10AM PDT, 1PM EDT Register at: http://bit.ly/ezq3fP Contact SOASTA: www.soasta.com/cloudtest/ info@soasta.com 866.344.8766 Follow us: twitter.com/cloudtest facebook.com/cloudtest
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×