Presentation from TACT QA in Israel around Load testing methods

1. Gomez Web Load Testing
Load Testing 1.0 Fails Web 2.0
Elad Altstein – CTO Software Products Division
Matrix

4. The Browser Is Becoming THE Integration Platform
Number of hosts accessed directly by the browser,
per user transaction, averaged across 3,000 companies 10.6

5. The Challenge of Delivering Web Applications
Systems …user is
management NOT happy
tools: “OK”
The Web Application Delivery Chain
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web
Servers
Mobile
Components
App
Servers Internet
DB Major
Servers ISP
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier
Traditional zone
of control

6. The Challenge of Ensuring Quality Web Experiences
Systems …user is
management NOT happy
tools: “OK”
The Web Application Delivery Chain
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
• Inconsistent geo performance
Web • Bad performance under load • Network peering • Poorly
Servers • Blocking content delivery problems performing
• Incorrect geo-targeted content • Bandwidth JavaScript
Mobile throttling • Browser/device
Components • Inconsistent incompatibility
• Configuration • Network peering connectivity • Page size
App errors problems too big
Servers • Application • Outages Internet • Too many
design issues • Network resource objects
DB
• Code defects Major shortage
Servers ISP • Low cache
• Insufficient • Faulty content hit rate
Storage infrastructure transcoding
• SMS routing /
Mainframe latency issues
• Configuration issues
Network • Oversubscribed POP Content Delivery Mobile
• Poor routing optimization Networks Carrier
• Low cache hit rate
Traditional zone
Traditional zone Zone of customer expectation of customer expectation
Zone of customer expectation
of control
of control Zone

7. The Business Impact of Poor Web Experiences
Systems …user is
management NOT happy
tools: “OK”
The Web Application Delivery Chain
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web Ineffective SLA’s and • Lost
Servers partner relationships revenue
• Brand
Mobile damage
Components • Dissatisfied
customers
App Ineffective SLA’s and • Increased
Servers Over investment partner relationships
Internet call center
in infrastructure volume
DB Major • Increased
Servers ISP costs
Storage Over spending
on CDNs
Mainframe
Network Content Delivery Mobile
Networks Carrier
Traditional zone Zone of customer expectation
of control

8. Pinpointing Problems and Optimizing Performance
The Web Application Delivery Chain
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web
Servers
Is it my
Mobile
Components Is it an Is it a Is it a
App
Servers
data ISP or the 3rd party
Internet
browser or
DB center? Internet?
Major provider? device?
Servers ISP
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier

9. The Answer: Adopt an “Outside-in” Customer Point of
View
The Web Application Delivery Chain
Enterprise Internet
3rd Party/ Browsers
Load and resolve
Isolate Test/monitor your site the Services customers use it:
Cloud way your Local ISP and devices Users
Balancers
performance • What they do (key pages and transactions)
problems across the • Where they do it (geographic locations)
Web
application
Servers • How they do it (browsers and mobile devices)
infrastructure • When they do it (normal and peak usage)
Mobile Determine the impact on their behavior and your business
Components
App
Servers “Outside-in” customer point of view
Internet
DB Major
ServersVantage ISP Gomez Platform
Storage
Web
Enterprise
Cross- Web Web Web Web
Mainframe
Application Cross- Load and Performance Performance
Browser
Browser Performance Management
Content Delivery MobileBusiness
NetworkTesting
Performance Testing Testing Analysis
Management Networks Carrier
Traditional zone Your zone of control withZone of customer expectation
Gomez
of control

10. Gomez Network: The World’s Most Comprehensive
Performance and Testing Network
Backbone Gomez Last Mile Virtual Test Bed Your Actual Users
Web Performance Web Performance Cross-Browser Testing Real User
Management and Management and 500+ browser/OS Monitoring
Load Testing Load Testing combo’s Worldwide, wherever
150+ locations 150,000+ locations 5,000+ supported your users are
devices

11. Traditional Behind-the-firewall Testing (Load Testing
1.0)
• Pros:
Load generation – Best of breed visibility into resources
behind the firewall
behind the firewall – Optimal for stress testing internal systems
• Cons
– Offered as a product  you need to install
SW, provision hardware, apply patches,
etc
– Requires hundreds of servers for high
volume load testing
3rd Party/ Browsers
Load Cloud Services Local ISP & devices Users – Typically harder to script
Balancers
Web – No visibility into end-user experience
Servers
Mobile
– No visibility into third-party external
Components components
App
Servers Internet
– No visibility into geographical response
time discrepancies that may surface under
DB
Servers
Major
ISP
load
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier

12. Cloud Testing (Load Testing 1.5)
Load generation • Pros:
from the cloud – Generate large volume of load
– Offered as a SaaS (no software
to maintain, or hardware to
provision)
• Cons
3rd Party/ Browsers – No visibility into end-user
Load
Balancers
Cloud Services Local ISP & devices Users
experience
Web
Servers
– No visibility into all third-party
Mobile
external components (e.g
Components CDNs)
App
Servers Internet – You can’t find problems across
DB Major the entire Web Application
Servers ISP Delivery Chain
Storage
Mainframe – No visibility into geographical
Network Content Delivery Mobile response time discrepancies
Networks Carrier
that can surface under load

13. Gomez’s Web Load Testing (Load Testing 2.0)
High volume cloud load & real-world • Pros:
load from +100k Last Mile computers – SaaS
– Generate high volume load
– Able to find user experience
breaking points
– Offer full visibility across all
external components
3rd Party/
Cloud Services
Browsers
Local ISP & devices Users
– You can find and resolve
Load
Balancers problems across the entire
Web WADC
Servers
Mobile
Components
App • Cons
Servers Internet
DB Major – Offers less visibility within the
Servers ISP
datacenter (when compared
Storage with traditional load testing)
Mainframe
Network Content Delivery Mobile
Networks Carrier

14. Load Testing from the cloud misses the point… end users
matter
Company: Online Gaming Site
Testing a new rollout in support of a new sports season
• Support anticipated traffic increases
• Load tested using cloud and Last Mile to validate performance for real
users in new geographies.
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web
Servers
Mobile
Components
App
Servers Internet
DB Major
Servers ISP
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier

15. View from the Cloud
• First 20 minutes Cloud testing
shows acceptable performance
• After 2500 users Response time
climbs Availability drops Error
rate climbs

16. View from the Last Mile
• Last Mile shows
different story
• Availability is terrible
even at minimal load
for real users
Summary:
Cloud-only testing may give
misleading availability data
Cloud starts with 100%
availability
Less than 25% for the Last
Mile
1.0 1.5 2.0


17. Difference in browsers
• Varied commercial browsers
show different response
times
• Response time change as a
function of load is
dependent on the browser.
Summary:
Understanding response
time under load requires
the use of multiple
browsers
1.0 1.5 2.0


18. 3 Steps To Success - Realistically Load Test Your
Websites And Applications From The Cloud
Record Action
Easily record Website and application transactions across 5,000+ device
profiles with the Gomez Recorder
Generate Load
Generate high volume load from multiple locations outside the data center to
find infrastructure breaking points with Gomez Web Load Testing
Analyze Results
Identify performance bottlenecks and issues with Gomez Web Load Testing

19. Record Your Business Process With The Gomez Recorder
Record your script once and use it in Gomez
Web Load Testing across all browsers and
devices
Record your website or
application transactions across
5,000+ device profiles

20. Generate Load From The Cloud With Reality Load
Easily upload your scripts across cloud
locations and providers for greater accuracy

21. Identify Issues Under Peak Conditions Before End-Users Are
Impacted
Drilldown to
diagnose and
quickly resolve
issues with object
level detail and
screen capture on
error Use your scripts
to understand how end-users
experience your service during peak traffic
times with Gomez Web Load Testing

22. Test Across the Entire Web Application Delivery Chain
The Need: Load Testing the Entire Web App Delivery Chain
The Web Application Delivery Chain
Load Testing 2.0 (Gomez)
Load Testing 1.5
Load Testing 1.0
3rd Party/ Browsers
Load Cloud Services Local ISP and devices Users
Balancers
Web
Servers
Mobile
Components
App
Servers Internet
DB Major
Servers ISP
Storage
Mainframe
Network Content Delivery Mobile
Networks Carrier

23. Gomez Load Testing: Most Accurate Load Test for User Experience
Load Test 1.0 Load Test 1.5 Load Test 2.0
HTTP : Behind the HTTP : Data Centers Browser : Data Real World
Firewall Centers Desktops
Last Mile
Traditional
Client/
Datacenter Testing
Server Test
Only Gomez Spans
Accuracy of End-User Incomplete Incomplete Indicative Most Accurate
Response Time
Accuracy of Invalid Indicative Indicative Most Accurate
Application Availability
Ability to drive large Yes-requires Best Better Good
load volume substantial
hardware
Understand CDN No Misleading Misleading Most
Impact Accurate
Understand 3rd Party No Minimal Some Most
(ads, feeds, etc…) Accurate
Realistic object No No Yes Yes
download Static Only
Visibility behind the Best Good Good Good
firewall

24. Thank You
Elad Altstein – CTO Software Products Division
Matrix
elada@matrix.co.il
054-4420390