Web and mobile performance has become business critical in today’s world. Website failures are costly in terms of lost revenue and tarnished reputation. Quality user experience has taken center stage. Are you ready to deliver? To help prepare for web traffic spikes associated with seasonal readiness, promotions or special online events, proper planning and execution is a must. This means that the combination of technology tools, implementation best practices and expertise needs to be carefully managed and optimized. Since 2008, SOASTA has conducted tens-of-thousands of tests, and logged over a million hours in the cloud. During this time we have accelerated web and mobile testing with scale and affordability for the world’s leading brands. During our upcoming webinar, we will share key SOASTA testing strategies needed to avoid epic web fails. They include the need to: - Test continuously - Test end-to-end - And, test at scale Join SOASTA experts Peter Galvin, SVP SOASTA Marketing and Brad Johnson, SOASTA VP Product Marketing as they share the best practices you need to follow to ensure your web and mobile apps are real-world ready. We will also show you how SOASTA’s industry leading CloudTest platform enables you to streamline and automate performance tests, so you can deliver quality user experiences every time.

1. 0© 2013 SOASTA CONFIDENTIAL - All rights reserved.Webinar
8

2. 1© 2013 SOASTA CONFIDENTIAL - All rights reserved.
What have we learned from recent epic fails?
TODAY’S SOASTA PRESENTERS
Peter Galvin: Brad Johnson:
SVP Marketing VP Product Marketing
Agenda:
• Poll question
• Lessons Learned
• Some Best Practices
Questions:
Submit in chat box during event

3. Poll Question

4. 3© 2013 SOASTA CONFIDENTIAL - All rights reserved.

5. 4© 2013 SOASTA CONFIDENTIAL - All rights reserved.

6. 5© 2013 SOASTA CONFIDENTIAL - All rights reserved.
“We didn’t have enough
testing for a very
complicated project”
US Health and Human Services
Secretary, Kathleen Sebelius

7. 6© 2013 SOASTA CONFIDENTIAL - All rights reserved.
• Millions of uninsured citizens
• Healthcare program mistrust
• President’s approval rating dropped
• Congress tied up in legislation
• Public cynicism
• Press and social media circus
• Uncertainty for the future
Brought QUALITY and TESTING to the public forefront!

8. 7© 2013 SOASTA CONFIDENTIAL - All rights reserved.
RECOMMENDATIONS
“Name a single implementation
leader”
“Lock down” requirements by April 30
“Streamline decision making”
“End-to-end integrated operations and
IT testing”
http://www.businessweek.com/news/2013-11-19/obamacare-
consultants-warned-of-health-website-failure-in-march
WHAT HAPPENED
“No single leader”
“Shifting requirements”
“Multiple definitions of success
“Insufficient time and scope of end-to-
end testing”
http://www.npr.org/blogs/alltechconsidered/2013/11/19/246132770/this
-slide-shows-why-healthcare-gov-wouldnt-work-at-launch
In March 2013

9. 8© 2013 SOASTA CONFIDENTIAL - All rights reserved.

10. 9© 2013 SOASTA CONFIDENTIAL - All rights reserved.

11. 10© 2013 SOASTA CONFIDENTIAL - All rights reserved.

12. 11© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Source: Venture Beat & Ray Wang at
Constellation Research 2012
“Twitter’s down…and it has cost the economy
$1 billion already?”

13. 12© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Source: Venture Beat & Ray Wang at
Constellation Research 2012
OK...reduce Twitter usage assumptions 50%.
Then, only $400,641,025.00 was lost.
“Twitter’s down…and it has cost the economy
$1 billion already?”

14. 13© 2013 SOASTA CONFIDENTIAL - All rights reserved.

15. 14© 2013 SOASTA CONFIDENTIAL - All rights reserved.
“Mass Panic As Common App Crashes”

16. 15© 2013 SOASTA CONFIDENTIAL - All rights reserved.
o Aug. 14: New York Times
• 2 hours - Stock dropped - opened paywall (lost $$) –
Twitter storm
o Aug 17: Google
• Less than 5 minutes – lost $500,000 – Internet traffic
dropped 40%
o Sept 2: HSBC (UK)
• Internal and external systems – “Weekend of
disruption”
o Nov 12: Kayazoonga.com (India)
• Last cricket match of Sachin Tendulkar - 19.7M
hits/hour – crashed all day – “Chaos”
o Nov 18: YouTube
• 35 minutes – est. 100,000 users affected

17. 16© 2013 SOASTA CONFIDENTIAL - All rights reserved.

18. 17© 2013 SOASTA CONFIDENTIAL - All rights reserved.

19. 18© 2013 SOASTA CONFIDENTIAL - All rights reserved.
500 Tests
400,000 concurrent users
431 Million Visitors
4.73 Billion Page Views
Access from 201 Countries
“We began working with SOASTA
CloudTest six months prior to the
games.”
Paul Bunnell,
Lead Architect,
London 2012

20. 19© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Prep for…

21. 20© 2013 SOASTA CONFIDENTIAL - All rights reserved.
“ We were on very
tight timelines…the
SOASTA team was
able to run multiple
tests very quickly
over a two-day
period and get us
going on issues to
immediately fix.”
Kunal Sarda,
VerbalizeIt
COO and Co-
Founder
Plan based on 6-8 million TV viewers
Tested & tuned for 10 million hits
250 to 30,000 visits/day
7x customer growth
Zero downtime

22. 21© 2013 SOASTA CONFIDENTIAL - All rights reserved.

23. 22© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Tested to 7x Initial load test goals
Reached 9x previous year’s peak
Validated dynamic caching investment
Reduced hardware capacity needs
Validated Akamai to 4x beyond peak
“One of the best
investments to our website
this year .”
Nancy Jordan,
IT Middleware
& Integration
Manager

24. 23© 2013 SOASTA CONFIDENTIAL - All rights reserved.

25. 24© 2013 SOASTA CONFIDENTIAL - All rights reserved.
“Website performance really
affects your brand image
when you have
problems…The inherent
savings to our brand by
using CloudTest is
priceless.”
—Cheryl Davis,
digital technical
operations manager
Hallmark Cards, Inc.
Tested to 120% of expected load
Saved 50% on testing expenses
Reduced staff costs by 66%
Eliminated test software and hw costs

26. 25© 2013 SOASTA CONFIDENTIAL - All rights reserved.
“No amount of
testing within
reasonable time
limits can adequately
replicate a live
environment of this
nature”

27. 26© 2013 SOASTA CONFIDENTIAL - All rights reserved.
“No amount of
testing within
reasonable time
limits can adequately
replicate a live
environment of this
nature”

28. 27© 2013 SOASTA CONFIDENTIAL - All rights reserved.

29. 28© 2013 SOASTA CONFIDENTIAL - All rights reserved.
o 55 Contractors
o Not enough time in schedule to conduct adequate performance
testing
o No Integration testing between systems
o No end-to-end testing simulating real life experiences of end users
o Followed waterfall development with parallel/disconnected projects
o Locked at 2000 users at the 1st step
http://www.washingtonpost.com/politics/house-panel-grills-contractors-
on-troubled-health-insurance-web-site/2013/10/24/8f42c748-3ca7-
11e3-b7ba-503fb5822c3e_story.html
http://www.cnn.com/2013/10/29/politics/obamacare-warning/

30. 29© 2013 SOASTA CONFIDENTIAL - All rights reserved.
o Test Continuously

31. 30© 2013 SOASTA CONFIDENTIAL - All rights reserved.
CI system runs load tests
against a pre-prod site
…while unit, functional and
mobile device tests run in parallel
Daily (or more frequent) updates
keep teams “performance focused”
Identify bottlenecks & inform
teams, just like unit and functional
tests in CI

32. 31© 2013 SOASTA CONFIDENTIAL - All rights reserved.
o Test End To End

33. 32© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Customers
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
CDN
Credit
Card
Web
analytic
s
Other
3rd pty
svcs

34. 33© 2013 SOASTA CONFIDENTIAL - All rights reserved.
“Conductor”
Analytics
CloudTest
Databas
e
Main
Perf Tester 1
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
CDN
Credit
Card
Web
analytic
s
Other
3rd pty
svcs

35. 34© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
CDN
Credit
Card
Web
analytic
s
Other
3rd pty
svcs

36. 35© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
CDN
Credit
Card
Web
analytic
s
Other
3rd pty
svcs

37. 36© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Databas
e
Main
Perf Tester 1
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
“Conductor”
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
CDN
Credit
Card
Web
analytic
s
Other
3rd pty
svcs

38. 37© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Chicago
(Rackspace)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)
SOASTA
Global
Test Cloud
CDN
Credit
Card
Web
analytic
s
Other
3rd pty
svcs

39. 38© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
SOASTA
Global
Test Cloud
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
“Conductor”
CDN
Credit
Card
Web
analytic
s
Other
3rd pty
svcs
Chicago
(Rackspace)

40. 39© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
SOASTA
Global
Test Cloud
Access
for all
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
“Conductor”
CDN
Credit
Card
Web
analytic
s
Other
3rd pty
svcs
Chicago
(Rackspace)

41. 40© 2013 SOASTA CONFIDENTIAL - All rights reserved.
o Test to Any Scale

42. 41© 2013 SOASTA CONFIDENTIAL - All rights reserved.
s
Memory leaks
Scale of Test
Stage/Team
Dev
& Test
Release
& Deploy
Network
& Ops
CDN file placement
Load Balancer configuration
Network bandwidth
Network configuration
DNS routing
Inadequate server resources
Default configuration settings
Unbalanced web serversAuto-scaling failures
Latency between systems
Slow third-party plug-ins
Garbage collection
Database thread counts
Inefficient database queriesSlow pages
Conflict with other apps
Test Lab
Search technology limits
Method-level tuning
Max sockets exceeded
Firewall max capacity
Global latency variance
Security bottlenecks
Scale is relative – bugs are everywhere
Tuning for full-scale…and well beyond
Frequent Testing from Lab to Production
Staging Production (100% +++)

43. 42© 2013 SOASTA CONFIDENTIAL - All rights reserved.
s
Scale of Test
Stage/Team
Dev
& Test
Release
& Deploy
Network
& Ops
Test Lab Staging Production (100% +++)
Where do we typically find the issues?
Top 10 Performance Problems
Architecture
Configuration Settings
Load Balancer
Database Performance
Bandwidth
Third Party
Services
Culture and Mindset
Application Issues
Shared
Environments
Connectivity

44. 43© 2013 SOASTA CONFIDENTIAL - All rights reserved.
o HHS mistakes have quality at the forefront (again)
o Risks to revenue and reputation are measurable and high profile
o Speed and quality do coexist
o Best practices are well established (follow them)
o There IS time, and you CAN test realistically
o Start now

45. Thanks
Contact SOASTA:
www.soasta.com/cloudtest/
info@soasta.com
866.344.8766
Follow us:
twitter.com/cloudtest
facebook.com/cloudtest
RESOURCES
Case studies: soasta.com/customers
Upcoming webinars: soasta.com/knowledge-center
- This Friday, Nov. 22: “Getting Started with CloudTest”
- Dec. 11: “Continuous Testing for Optimal Performance”
Need a test, fast? CloudTest On-Demand.
- email info@soasta.com
- You’ll hear from us by tomorrow

46. www.SOASTA.com