Your SlideShare is downloading. ×
0
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov

758

Published on

Web and mobile performance has become business critical in today’s world. Website failures are costly in terms of lost revenue and tarnished reputation. Quality user experience has taken center stage. …

Web and mobile performance has become business critical in today’s world. Website failures are costly in terms of lost revenue and tarnished reputation. Quality user experience has taken center stage. Are you ready to deliver?

To help prepare for web traffic spikes associated with seasonal readiness, promotions or special online events, proper planning and execution is a must. This means that the combination of technology tools, implementation best practices and expertise needs to be carefully managed and optimized.

Since 2008, SOASTA has conducted tens-of-thousands of tests, and logged over a million hours in the cloud. During this time we have accelerated web and mobile testing with scale and affordability for the world’s leading brands.

During our upcoming webinar, we will share key SOASTA testing strategies needed to avoid epic web fails. They include the need to:

- Test continuously
- Test end-to-end
- And, test at scale

Join SOASTA experts Peter Galvin, SVP SOASTA Marketing and Brad Johnson, SOASTA VP Product Marketing as they share the best practices you need to follow to ensure your web and mobile apps are real-world ready. We will also show you how SOASTA’s industry leading CloudTest platform enables you to streamline and automate performance tests, so you can deliver quality user experiences every time.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
758
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • http://www.forbes.com/sites/theapothecary/2013/10/23/now-she-tells-us-sebelius-says-obamacares-exchange-website-needed-six-years-of-development-instead-of-two/
  • http://www.forbes.com/sites/theapothecary/2013/10/23/now-she-tells-us-sebelius-says-obamacares-exchange-website-needed-six-years-of-development-instead-of-two/
  • CMS – Centers for Medicare / MedicaidHHA – Home Health Agencies
  • Ray Wand estimated 500M users, 2% online, and avg salary $125k.We reworked with 250M (recent estimate) and $100k salary.
  • http://news.cnet.com/8301-1023_3-57612283-93/livingsocial-suffers-two-day-long-site-and-mobile-app-outage/
  • http://www.thedailybeast.com/articles/2013/11/01/the-2014-common-app-is-a-glitch-ridden-nightmare-for-college-seniors.html
  • http://www.telegraph.co.uk/finance/personalfinance/10280250/HSBC-customers-hit-by-IT-glitch.htmlhttp://www.zdnet.com/quantifying-the-impact-of-downtime-7000019595/
  • Visitors/Hr: Final Test was over 7x initial load test Orders/Hr: Reached over 9x previous year’s peak Validated dynamic page caching investmentResponse time actually went down over 4x after implementation Akamai Peak Views/Sec: Generated over 4x peak day’s traffic Results of Engagement:18 Recommendations Implemented: caching (8), application code (3), database (2), WAS/HTTP configuration (5)Significant reduction in Hardware Capacity requirements
  • Visitors/Hr: Final Test was over 7x initial load test Orders/Hr: Reached over 9x previous year’s peak Validated dynamic page caching investmentResponse time actually went down over 4x after implementation Akamai Peak Views/Sec: Generated over 4x peak day’s traffic Results of Engagement:18 Recommendations Implemented: caching (8), application code (3), database (2), WAS/HTTP configuration (5)Significant reduction in Hardware Capacity requirements
  • Transcript

    • 1. 8 Webinar © 2013 SOASTA CONFIDENTIAL - All rights reserved. 0
    • 2. What have we learned from recent epic fails? TODAY’S SOASTA PRESENTERS Peter Galvin: SVP Marketing Brad Johnson: VP Product Marketing Agenda: • Poll question • Lessons Learned • Some Best Practices Questions: Submit in chat box during event © 2013 SOASTA CONFIDENTIAL - All rights reserved. 1
    • 3. Poll Question
    • 4. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 3
    • 5. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 4
    • 6. “We didn’t have enough testing for a very complicated project” US Health and Human Services Secretary, Kathleen Sebelius © 2013 SOASTA CONFIDENTIAL - All rights reserved. 5
    • 7. • • • • • • • Millions of uninsured citizens Healthcare program mistrust President’s approval rating dropped Congress tied up in legislation Public cynicism Press and social media circus Uncertainty for the future Brought QUALITY and TESTING to the public forefront! © 2013 SOASTA CONFIDENTIAL - All rights reserved. 6
    • 8. In March 2013 RECOMMENDATIONS WHAT HAPPENED ―Name a single implementation leader‖ ―No single leader‖ ―Lock down‖ requirements by April 30 ―Streamline decision making‖ ―End-to-end integrated operations and IT testing‖ http://www.businessweek.com/news/2013-11-19/obamacareconsultants-warned-of-health-website-failure-in-march ―Shifting requirements‖ ―Multiple definitions of success ―Insufficient time and scope of end-toend testing‖ http://www.npr.org/blogs/alltechconsidered/2013/11/19/246132770/this -slide-shows-why-healthcare-gov-wouldnt-work-at-launch © 2013 SOASTA CONFIDENTIAL - All rights reserved. 7
    • 9. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 8
    • 10. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 9
    • 11. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 10
    • 12. ―Twitter’s down…and it has cost the economy $1 billion already?‖ Source: Venture Beat & Ray Wang at Constellation Research 2012 © 2013 SOASTA CONFIDENTIAL - All rights reserved. 11
    • 13. OK...reduce Twitter usage assumptions 50%. Then, only $400,641,025.00 was lost. ―Twitter’s down…and it has cost the economy $1 billion already?‖ Source: Venture Beat & Ray Wang at Constellation Research 2012 © 2013 SOASTA CONFIDENTIAL - All rights reserved. 12
    • 14. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 13
    • 15. ―Mass Panic As Common App Crashes‖ © 2013 SOASTA CONFIDENTIAL - All rights reserved. 14
    • 16. o Aug. 14: New York Times • 2 hours - Stock dropped - opened paywall (lost $$) – Twitter storm o Aug 17: Google • Less than 5 minutes – lost $500,000 – Internet traffic dropped 40% o Sept 2: HSBC (UK) • Internal and external systems – “Weekend of disruption” o Nov 12: Kayazoonga.com (India) • Last cricket match of Sachin Tendulkar - 19.7M hits/hour – crashed all day – “Chaos” o Nov 18: YouTube • 35 minutes – est. 100,000 users affected © 2013 SOASTA CONFIDENTIAL - All rights reserved. 15
    • 17. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 16
    • 18. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 17
    • 19. 500 Tests 400,000 “We began working with SOASTA CloudTest six months prior to the games.” Paul Bunnell, Lead Architect, London 2012 concurrent users 431 Million Visitors 4.73 Billion Page Views Access from 201 Countries © 2013 SOASTA CONFIDENTIAL - All rights reserved. 18
    • 20. Prep for… © 2013 SOASTA CONFIDENTIAL - All rights reserved. 19
    • 21. Plan based on Tested & “ We were on very tight timelines…the SOASTA team was able to run multiple tests very quickly 6-8 million TV viewers over a two-day period and get us tuned for 10 million hits going on issues to immediately fix.” Kunal Sarda, VerbalizeIt 250 to 30,000 visits/day COO and CoFounder 7x customer growth Zero downtime © 2013 SOASTA CONFIDENTIAL - All rights reserved. 20
    • 22. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 21
    • 23. “One of the best investments to our website this year .” Nancy Jordan, Tested to 7x Initial load test goals IT Middleware & Integration Reached 9x previous year’s peak Manager Validated dynamic caching investment Reduced hardware capacity needs Validated Akamai to 4x beyond peak © 2013 SOASTA CONFIDENTIAL - All rights reserved. 22
    • 24. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 23
    • 25. Tested to 120% of expected Saved 50% on testing “Website performance really affects your brand image when you have problems…The inherent savings to our brand by using CloudTest is load priceless.” —Cheryl Davis, expenses digital technical operations manager Hallmark Cards, Inc. Reduced staff costs by 66% Eliminated test software and hw costs © 2013 SOASTA CONFIDENTIAL - All rights reserved. 24
    • 26. ―No amount of testing within reasonable time limits can adequately replicate a live environment of this nature‖ © 2013 SOASTA CONFIDENTIAL - All rights reserved. 25
    • 27. ―No amount of testing within reasonable time limits can adequately replicate a live environment of this nature‖ © 2013 SOASTA CONFIDENTIAL - All rights reserved. 26
    • 28. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 27
    • 29. o 55 Contractors o Not enough time in schedule to conduct adequate performance testing o No Integration testing between systems o No end-to-end testing simulating real life experiences of end users o Followed waterfall development with parallel/disconnected projects o Locked at 2000 users at the 1st step http://www.washingtonpost.com/politics/house-panel-grills-contractorson-troubled-health-insurance-web-site/2013/10/24/8f42c748-3ca711e3-b7ba-503fb5822c3e_story.html http://www.cnn.com/2013/10/29/politics/obamacare-warning/ © 2013 SOASTA CONFIDENTIAL - All rights reserved. 28
    • 30. o Test Continuously © 2013 SOASTA CONFIDENTIAL - All rights reserved. 29
    • 31. Identify bottlenecks & inform teams, just like unit and functional tests in CI CI system runs load tests against a pre-prod site …while unit, functional and mobile device tests run in parallel Daily (or more frequent) updates keep teams ―performance focused‖ © 2013 SOASTA CONFIDENTIAL - All rights reserved. 30
    • 32. o Test End To End © 2013 SOASTA CONFIDENTIAL - All rights reserved. 31
    • 33. Credit Card Web analytic s CDN Other 3rd pty svcs SUT/AUT Web Server App Server Cache Web Server App Server Cache Web Server App Server Web Server App Server Load Balancer Databas e Customers © 2013 SOASTA CONFIDENTIAL - All rights reserved. 32
    • 34. CloudTest Credit Card Analytics Main Databas e Web analytic s CDN Other 3rd pty svcs SUT/AUT Web Server App Server Cache Web Server App Server Cache Web Server App Server Web Server App Server Load Balancer ―Conductor‖ Databas e Perf Tester 1 © 2013 SOASTA CONFIDENTIAL - All rights reserved. 33
    • 35. CloudTest Credit Card Analytics Main Databas e Analytics Load Generator Load Web analytic s CDN Other 3rd pty svcs SUT/AUT Generator Load Generator Web Server App Server Cache Web Server App Server Cache Web Server App Server Web Server App Server East Coast (AWS) Load Balancer Databas e Perf Tester 1 © 2013 SOASTA CONFIDENTIAL - All rights reserved. 34
    • 36. CloudTest Credit Card Analytics Main Databas e Analytics Load Generator Load Web analytic s CDN Other 3rd pty svcs SUT/AUT Generator Load Generator Web Server App Server Cache Web Server App Server Cache Web Server App Server Web Server App Server East Coast (AWS) Load Balancer Databas e Perf Tester 1 © 2013 SOASTA CONFIDENTIAL - All rights reserved. 35
    • 37. CloudTest Credit Card Analytics Main Databas e Analytics Load Generator Load Web analytic s CDN Other 3rd pty svcs SUT/AUT Generator Load Generator Web Server App Server Cache Web Server App Server Cache Web Server App Server Web Server App Server East Coast (AWS) Load Balancer Databas e Perf Tester 1 ―Conductor‖ Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,…. © 2013 SOASTA CONFIDENTIAL - All rights reserved. 36
    • 38. Analytics Load Generator Load CloudTest Analytics Main Analytics Load Generator Load Generator Amsterdam (Azure) Generator Load Generator Analytics Load Generator Load Tokyo (AWS) Generator Load Generator Analytics Load Generator Load Web analytic s CDN Other 3rd pty svcs Generator San Francisco (GoGrid) Databas e Analytics Load Generator Load Credit Card Analytics Load Generator Load Virginia (IBM) SUT/AUT SOASTA Generator Load Global Generator Test Cloud East Coast (AWS) App Server Cache Web Server App Server Web Server App Server Web Server Chicago (Rackspace) Web Server Cache Generator Load Generator App Server Load Balancer Databas e Perf Tester 1 © 2013 SOASTA CONFIDENTIAL - All rights reserved. 37
    • 39. Analytics Load Generator Load CloudTest Analytics Main Analytics Load Generator Load Generator Amsterdam (Azure) Generator Load Generator Analytics Load Generator Load Tokyo (AWS) Generator Load Generator Analytics Load Generator Load Web analytic s CDN Other 3rd pty svcs Generator San Francisco (GoGrid) Databas e Analytics Load Generator Load Credit Card Analytics Load Generator Load Virginia (IBM) SUT/AUT SOASTA Generator Load Global Generator Test Cloud East Coast (AWS) App Server Cache Web Server App Server Web Server App Server Web Server Chicago (Rackspace) Web Server Cache Generator Load Generator App Server Load Balancer Databas e Perf Tester 1 ―Conductor‖ Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,…. 38 © 2013 SOASTA CONFIDENTIAL - All rights reserved.
    • 40. Analytics Load Generator Load CloudTest Analytics Main Analytics Load Generator Load Generator Amsterdam (Azure) Generator Load Generator Analytics Load Generator Load Tokyo (AWS) Generator Load Generator Analytics Load Generator Load Web analytic s CDN Other 3rd pty svcs Generator San Francisco (GoGrid) Databas e Analytics Load Generator Load Credit Card Analytics Load Generator Load Virginia (IBM) SUT/AUT SOASTA Generator Load Global Generator Test Cloud East Coast (AWS) Access for all App Server Cache Web Server App Server Web Server App Server Web Server Chicago (Rackspace) Web Server Cache Generator Load Generator App Server Load Balancer Databas e Perf Tester 1 ―Conductor‖ Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,…. 39 © 2013 SOASTA CONFIDENTIAL - All rights reserved.
    • 41. o Test to Any Scale © 2013 SOASTA CONFIDENTIAL - All rights reserved. 40
    • 42. Frequent Testing from Lab to Production Scale is relative – bugs are everywhere Tuning for full-scale…and well beyond Latency between systems Network & Ops Network configuration Network bandwidth CDN file placement Stage / Team Conflict with other apps Load Balancer configuration Auto-scaling failures Release & Deploy Max sockets exceeded Global latency variance Security bottlenecks Search technology limits Default configuration settings Inadequate server resources Database thread counts Memory leaks Dev & Test Firewall max capacity Unbalanced web servers s Slow third-party plug-ins Garbage collection DNS routing Inefficient database queries Slow pages Method-level tuning Test Lab Staging Production (100% +++) Scale of Test © 2013 SOASTA CONFIDENTIAL - All rights reserved. 41
    • 43. Top 10 Performance Problems Where do we typically find the issues? Architecture Network & Ops Bandwidth Stage / Team Load Balancer Third Party Services Connectivity Release & Deploy s Database Performance Shared Environments Configuration Settings Application Issues Dev & Test Culture and Mindset Test Lab Staging Production (100% +++) Scale of Test © 2013 SOASTA CONFIDENTIAL - All rights reserved. 42
    • 44. o HHS mistakes have quality at the forefront (again) o Risks to revenue and reputation are measurable and high profile o Speed and quality do coexist o Best practices are well established (follow them) o There IS time, and you CAN test realistically o Start now © 2013 SOASTA CONFIDENTIAL - All rights reserved. 43
    • 45. Thanks RESOURCES Case studies: soasta.com/customers Upcoming webinars: soasta.com/knowledge-center - This Friday, Nov. 22: ―Getting Started with CloudTest‖ - Dec. 11: ―Continuous Testing for Optimal Performance‖ Need a test, fast? CloudTest On-Demand. - email info@soasta.com - You’ll hear from us by tomorrow Contact SOASTA: www.soasta.com/cloudtest/ info@soasta.com 866.344.8766 Follow us: twitter.com/cloudtest facebook.com/cloudtest
    • 46. www.SOASTA.com

    ×