Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automated Analytics Testing


Published on

Selenium Committer Day 2017
Marcus Merrell さんによる講演資料です。

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Automated Analytics Testing

  1. 1. Automated Analytics Testing MARCUS MERRELL, RETAILMENOT, INC @MMERRELL
  2. 2. What We’re Talking about  Conservancy  Overview of Web/User Analytics  Explanation of A/B Testing  Why this matters to QA personnel  Examples  How to get started
  3. 3. Who Is Conservancy?  Software Freedom Conservancy  Protects the branding, copyrights, and licenses for over 40 free and open source projects  Includes Selenium, git, Samba, PyPy, and many others   Non-profit organization  The reason Jim, Manoj, and I are here
  4. 4. Overview of User Analytics  Primarily B2C: Banks, shopping, high-volume apps  Applies to all platforms, including video game consoles  Tells the ”story” of user interaction with a product
  5. 5. User Analytics Hits Sessions Users “If you are not paying for it… you are the product being sold” —Andrew Lewis (blue_beetle)
  6. 6. A/B Testing  Don’t change everything at once  Major source of regressions in analytics:  Analytics will need to change as a result of a test, but sometimes those changes are forgotten (  We used to just release changes after basic QA  “The Highball Incident”
  7. 7. Real-life RMN Example ...&b21=17&… ...&b21=18&… ...&b21=0&… (wait, shouldn’t that say 18?)
  8. 8. How Is This Related?  Analytics have historically not been included in test design  Major source of regression defects  In this case, no A/B testing was done  Data loss was permanent, irrecoverable
  9. 9. Real-world Examples  Shopping cart—shipping & tax calculation  Suggesting products and content based on cookie, not login
  10. 10. Real-world Examples  RetailMeNot’s email program  Category and Merchant “affinity”
  11. 11. Classic Approach Marketing Business Intelligence Product Manager EngineeringQA (Usually the newest person on the team)
  12. 12. Why You Should Care  What if the analytics contain the wrong information?  But furthermore…  This is everywhere  It is only growing  Companies are becoming smart  (Really really smart)  QA is being left behind
  13. 13. Why You Should Care
  14. 14. Why You Should Really Care  As a tester:  There is a team of people working on this  It gets worked into features as they are developed  It is rarely called out separately in a scheduled task  It rarely receives QA outside of the Product Management and Business Intelligence people who really care about it
  15. 15. Fortunately, It’s Easy  Usually one extra HTTP request, made during a navigation event  Intercept this request, then verify the data within it
  16. 16. Process  Break down into key-value pairs  Build a Map of “expected results”  Assert on expected vs actual results sequence=12 sequence=12 category=food category=food type=code type=sale
  17. 17. Test Cycle @Test Launch Browser Navigate to Position Start Proxy Perform Main Action Stop Proxy Clean-up Test
  18. 18. Execution  Send JSON to the “expectation” end-point  Includes a “timer” and an alert email  Later (asynchronously), more JSON is sent to the “result”  If there’s no match before the timer runs out, ALERT Expectation Action Result
  19. 19. Reporting  Report to a dashboard  Indicates “PASS”, “FAIL”, and “Staleness”
  20. 20. Alternatives  Anand Bagmar   Tool for testing Omniture Analytics  Uses Omniture’s built-in web interface to examine attributes emitted by page, rather than using a proxy
  21. 21. Conclusion  User Analytics are your CEO’s favorite subject!  Deliver real value—million-dollar decisions are made with this data  Can be implemented with just as many bugs as any other kind of software
  22. 22. Questions?  Once again, thank you so very much  Twitter: @mmerrell   I’m also on LinkedIn