Testing In Production (TiP) Advances with Big Data & the Cloud


Published on

Testing in Production (TiP) has moved from taboo to accepted practice owing to its ability to measure reality and provide actionable feedback. These risks can be mitigated by using proven methodologies, methodologies borne of experience and tools built specifically to handle TiP’s unique requirements.

Big Data enables TIP by analyzing user behavior then creating realistic tests. During testing, cloud-based resources are used for the huge data volumes and processed within-memory technology specifically designed for this process

Microsoft’s Seth Eliot is a TiP pioneer and SOASTA’s Rob Holcomb has helped evolve the practice with hundreds of SOASTA customers. Catch this webinar, now on-demand, as they dig into:

How to leverage both active and passive monitoring for TiP
Testing and measuring system stress in production
Experimentation and iterative improvement
How SOASTA CloudTest facilitates TiP for organizations of all sizes

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Early and often: TargetDon’t wait until the last minute: American GirlTIP for real results: Intuit TurboTax (hundreds of thousands of users) – in a real production environmentTest mix: spike (Nike—shoe release)Integrated monitoring data: Dillard’s DynaTrace—pushing app servers to the limit; stop!
  • Testing In Production (TiP) Advances with Big Data & the Cloud

    1. 1. PresentsWebinar © 2012 SOASTA. All rights reserved. 1
    2. 2. Methodologies and technology for Testing in Production (TiP)TODAY’S PRESENTERSSeth Eliot: Sr. Knowledge Engineer in Test, Microsoft- @setheliotRob Holcomb: VP Performance Engineering, SOASTA - @rcholcombModerator: Brad Johnson - @bradjohnsonsvAgenda:• Poll question• Leveraging active and passive monitoring for TiP• Testing and measuring system stress in production• Experimentation and iterative improvement• SOASTA CloudTest for TiP• Closing PollQuestions:Submit in the question box during event © 2012 SOASTA. All rights reserved. October 30, 2012 2
    3. 3. © 2012 SOASTA. All rights reserved. October 30, 2012 3
    4. 4. Seth EliotSr. Knowledge Engineer in Test © 2012 SOASTA. All rights reserved. October 30, 2012 4
    5. 5. About Setho Currently with Microsoft Engineering Excellence focused on helping teams transition to The Cloudo Previously with Bing, and before that Amazon.com The author is an employee of Microsoft Corporation. The views expressed in this presentation are those of the author and do not necessarily reflect any views or positions of Microsoft nor imply any relationship between Microsoft and SOASTA.o Seth wishes to thank Brad Johnson, Rob Holcomb and SOASTA for this opportunity 5
    6. 6. IntroductionTesting in Production (TiP) TestOps Big Data 6
    7. 7. Testing at Microsoft 1985o Design, execute and document testso Generate Test Scripts and automatic testing packages
    8. 8. What Testing Usually Is…
    9. 9. What Can Testing Be?Big Data
    10. 10. The Three (or more) V’s of Big Data [Strata Jan 2012]
    11. 11. TestOpso Monitoring: What Ops doeso Testing: What Test Doeso TestOps: Change (augment) the “signal” used for quality From Test Results… …to Big Data
    12. 12. The Big Data Signalo Is often found in Productiono May not always be “Big” o The Quality Insights however should be Bigo TestOps: use this Big Data for quality assessmento Big Data is in production o Therefore we Test in Production
    13. 13. © 2012 SOASTA. All rights reserved. 13
    14. 14. The Big Data Pipelineo Facebook: Developers Instrument Everythingo Amazon: Central Monitoring o Add some config  Trending and Alertso Netflix: Custom libraries + AWS CloudWatch Servers CPU
    15. 15. How does TiP fit into Teststrategy? Does TiP Replace Up-Front Testing (UFT)? The Death of BUFT (Big UFT)? Test BUFT Strat Test UFT TiP Strat
    16. 16. Four Categories of TiPo Passive Monitoring o with Real Datao Active Monitoring o with Synthetic Transactionso Experimentation o on Real Userso System Stress o of the Service and Environment 17
    17. 17. Passive Monitoring with Real Data 18
    18. 18. Facebook Mines Big Data for QualityGanglia “5 million metrics” CPU, network usage [Cook, June 2010]
    19. 19. User Performance Testingo Collect specific telemetry about how long stuff takes from user point of viewo Real User Data – Real User Experienceo End to End = complete request and response cycle o From user to back-end round-trip o Include traffic to partners, dependency response time o Measured from the user point of viewo From around the worldo From diversity of browsers, OS, devices
    20. 20. Hotmail JSI User Performance Testing Big Data?o Hotmails JavaScript Instrumentation (JSI) o Budget for 500 Million measurements / month o Scale for backend collection and analysiso PLT by browser, OS, country, cluster, etc.. o As experienced by Millions of Real Users
    21. 21. Hotmail JSI User Performance Testing • PLT by browser, OS, country, cluster, etc..
    22. 22. User Performance Testing Exampleso Hotmail o Re-architected from the ground up around performance o Read messages are 50% fastero Windows Azure™ o Every API: Tracks how many calls were made; how many succeeded, and how long each call took to process
    23. 23. Active Monitoring with Synthetic Transactions 24
    24. 24. TiP Test Executiono From the Inside o Against internal APIs o Automatedo From the Outside o From User Entry Point o E2E Scenario in Production o Automated o or Manual 25
    25. 25. This looks like thisbut in Productionwhich is OK, but…Can we leverage Big Data? 26
    26. 26. Active Monitoringo Microsoft Exchange o Instead of pass/fail signal look at thousands of continuous runs. o Did we meet the "five nines" (99.999%) availability for scenario? o Is scenario slower this release than last? - performance [Deschamps, Johnston, Jan 2012] 27
    27. 27. Test Data Handlingo Synthetic Tests + Real Data = Potential Trouble o Avoid it o Tag it o Clean it upo Example: Facebook Test Users o Cannot interact with real users o Can only friend other Test Users o Create 100s o Programmatic Control 28
    28. 28. Experimentation on Real Users 29
    29. 29. Experimentation“To have a great idea,have a lot of them” -- Thomas Edison o Try new things… in production o Build on successes o Cut your losses… before they get expensive
    30. 30. Mitigate Risk with Exposure Control o Launch a new Service – Everyone sees it o Exposure Control – only some see it By Browser By Location By Percent (scale) 31
    31. 31. Example: Controlled Test Flight: Netflix 1B API requests per day “Canary” Deployment[Cockcroft, March 2012]
    32. 32. Dogfood and Beta
    33. 33. System Stress of the Service and Environment 37
    34. 34. Load Testing in Productiono Injects load on top of real user traffico Monitors for performanceoTo assess system capabilities and scalabilityo Big Data o Traffic mix: real user queries, simulate scenarios o Real time telemetry: Monitor and Back-Off o After the fact Analysis o Tune SLAs/Targets o Tune real-time monitors and alerts 38
    35. 35. Load Testing in Productiono Rob will discuss some SOASTA case studies o Identified issues that only could be found in production o Agile approach to implementation 39
    36. 36. Destructive Testing in Productiono Google first year of a new data center [Google DC, 2008] o 20 rack failures, 1000 server failures and thousands of hard drive failureso High Availability means you must embrace failure o How do you test this? 40
    37. 37. Netflix Tests its “Rambo Architecture” o …system has to be able to succeed, no matter what, even all on its own o Test with Fault Injection [Netflix Army, July 2011] o Netflix Simian Army o Chaos monkey randomly kills production instance in AWS o Chaos Gorilla simulates an outage of an entire Amazon AZ o Janitor Monkey, Security Monkey, Latency Monkey….. 41
    38. 38. Changing theQuality Signal 42
    39. 39. What Can Testing Be?Change the signal fromTest Results to…
    40. 40. Big Data Quality Signal aka TestOps Big DataKPI: Key Performance Indicator• Request latency• RPS• Availability / MTTR 44
    41. 41. Seth Eliotseth.eliot@microsoft.com Twitter: @setheliotBlog: http://bit.ly/seth_qa © 2012 SOASTA. All rights reserved. October 30, 2012 45
    42. 42. Rob HolcombVP Performance Engineering, Founder © 2012 SOASTA. All rights reserved. October 30, 2012 46
    43. 43. o Start testing early and often!o Don’t wait until the last minuteo Test in production for real resultso Test mix: baseline, stress, spike, endurance, failover, diagnostic • Start with a baseline to understand general performance characteristics • Test types chosen depend on the defined goalso Test case selection: performance testing is not functional testingo Integrated monitoring data; know when to say wheno Define a clear test strategy with test plans, goals, and deliverable dateso Focus on actionable results! © 2012 SOASTA. All rights reserved. October 30, 2012 47
    44. 44. © 2012 SOASTA. All rights reserved. October 30, 2012 48
    45. 45. Thank You! Next Webinar: Nov. 8, 2010 - 10 a.m. PST “RUM Expert Roundtable” * Buddy Brewer & Philip Tellis (LogNormal founders); Aaron Kulick (WalmartLabs): Moderator - Cliff Crocker (SOASTA) * Register at www.soasta.com/knowledge-center/webinars White Papers, Webinar Recordings, Case Studies www.soasta.com - Knowledge Center Contact SOASTA: Contact Seth: www.soasta.com/cloudtest/ seth.eliot@microsoft.co info@soasta.com m@setheliot 866.344.8766 Follow us: Contact Rob: twitter.com/cloudtest rholcomb@soasta.com @rcholcomb facebook.com/cloudtest © 2012 SOASTA. All rights reserved. October 30, 2012 49
    46. 46. References[Google Talk, June 2007] Google: Seattle Conference on Scalability: Lessons In Building Scalable Systems, Reza Behforooz http://video.google.com/videoplay?docid=6202268628085731280[Unpingco, Feb 2011] Edward Unpingco; Bug Miner; Internal Microsoft Presentation, Bing Quality Day[Barranco, Dec 2011] René Barranco; Heuristics-Based Testing; Internal Microsoft Presentation[Dell, 2012] http://whichtestwon.com/dell%e2%80%99s-site-wide-search-box-test[Microsoft.com, TechNet] http://technet.microsoft.com/en-us/library/cc627315.aspx[Cockcroft, March 2012] http://perfcap.blogspot.com/2012/03/ops-devops-and-noops-at-netflix.html[Deschamps, Johnston, Jan Experiences of Test Automation; Dorothy Graham; Jan 2012; ISBN 0321754069; Chapter: “Moving to the Cloud: The2012] Evolution of TiP, Continuous Regression Testing in Production”; Ken Johnston, Felix Deschamps[Google DC, 2008] http://content.dell.com/us/en/gen/d/large-business/google-data-center.aspx?dgc=SM&cid=57468&lid=1491495 http://perspectives.mvdirona.com/2008/06/11/JeffDeanOnGoogleInfrastructure.aspx[Kohavi, Oct 2010] Tracking Users’ Clicks and Submits: Tradeoffs between User Experience and Data Loss http://www.exp-platform.com/Pages/TrackingClicksSubmits.aspx[Strata Jan 2012] What is big data? - An introduction to the big data landscape http://radar.oreilly.com/2012/01/what-is-big-data.html 50
    47. 47. References, continued[Netflix Army, July 2011] The Netflix Simian Army; July 2011 http://techblog.netflix.com/2011/07/netflix-simian-army.html[Google-Wide Profiling, 2010] Ren, Gang, et al. Google-wide Profiling: A Continuous Profiling Infrastructure for Data Centers. [Online] July 30, 2010. research.google.com/pubs/archive/36575.pdf[Facebook ships, 2011] http://framethink.blogspot.com/2011/01/how-facebook-ships-code.html[Google BusinessWeek, April How Google Fuels Its Idea Factory, BusinessWeek, April 29, 2008;2008] http://www.businessweek.com/magazine/content/08_19/b4083054277984.htm[IBM 2011] http://www.ibm.com/developerworks/websphere/techjournal/1102_supauth/1102_supauth.html[Kokogiak, 2006] http://www.kokogiak.com/gedankengang/2006/08/amazons-digital-video-sneak-peek.html[Google GTAC 2010] Whittaker, James. GTAC 2010: Turning Quality on its Head. [Online] October 29, 2010. http://www.youtube.com/watch?v=cqwXUTjcabs&feature=BF&list=PL1242F05D3EA83AB1&index=16.[Google, JW 2009] http://googletesting.blogspot.com/2009/07/plague-of-homelessness.html[STPCon, 2012] STPCon Spring 2012 - Testing Wanted: Dead or Alive – March 26, 2012[Cook, June 2010] Ganglia, OSD: Cook, Tom. A Day in the Life of Facebook Operations. Velocity 2010. [Online] June 2010. http://www.youtube.com/watch?v=T-Xr_PJdNmQ 51