Recommended
Recommended
More Related Content
Similar to Automated Analytics Testing
Similar to Automated Analytics Testing (20)
More from Naoya Kojima
More from Naoya Kojima (10)
Recently uploaded
Recently uploaded (20)
Automated Analytics Testing
- 1. Automated Analytics Testing MARCUS MERRELL, RETAILMENOT, INC @MMERRELL
- 2. What We’re Talking about Conservancy Overview of Web/User Analytics Explanation of A/B Testing Why this matters to QA personnel Examples How to get started http://bit.ly/2u6tJMa http://bit.ly/2urc80O
- 3. Who Is Conservancy? Software Freedom Conservancy Protects the branding, copyrights, and licenses for over 40 free and open source projects Includes Selenium, git, Samba, PyPy, and many others www.sfconservancy.org Non-profit organization The reason Jim, Manoj, and I are here
- 4. Overview of User Analytics Primarily B2C: Banks, shopping, high-volume apps Applies to all platforms, including video game consoles Tells the ”story” of user interaction with a product
- 5. User Analytics Hits Sessions Users “If you are not paying for it… you are the product being sold” —Andrew Lewis (blue_beetle)
- 6. A/B Testing Don’t change everything at once Major source of regressions in analytics: Analytics will need to change as a result of a test, but sometimes those changes are forgotten ( We used to just release changes after basic QA “The Highball Incident” http://bit.ly/2ttj1RK
- 8. Real-life RMN Example ...&b21=17&… ...&b21=18&… ...&b21=0&… (wait, shouldn’t that say 18?)
- 9. How Is This Related? Analytics have historically not been included in test design Major source of regression defects In this case, no A/B testing was done Data loss was permanent, irrecoverable
- 10. Real-world Examples Shopping cart—shipping & tax calculation Suggesting products and content based on cookie, not login
- 11. Real-world Examples RetailMeNot’s email program Category and Merchant “affinity”
- 12. Classic Approach Marketing Business Intelligence Product Manager EngineeringQA (Usually the newest person on the team)
- 13. Why You Should Care What if the analytics contain the wrong information? But furthermore… This is everywhere It is only growing Companies are becoming smart (Really really smart) QA is being left behind
- 14. Why You Should Care
- 15. Why You Should Really Care As a tester: There is a team of people working on this It gets worked into features as they are developed It is rarely called out separately in a scheduled task It rarely receives QA outside of the Product Management and Business Intelligence people who really care about it http://bit.ly/2u6j4AW
- 16. Fortunately, It’s Easy Usually one extra HTTP request, made during a navigation event Intercept this request, then verify the data within it
- 17. Process Break down into key-value pairs Build a Map of “expected results” Assert on expected vs actual results sequence=12 sequence=12 category=food category=food type=code type=sale
- 18. Test Cycle @Test Launch Browser Navigate to Position Start Proxy Perform Main Action Stop Proxy Clean-up Test
- 19. Execution Send JSON to the “expectation” end-point Includes a “timer” and an alert email Later (asynchronously), more JSON is sent to the “result” If there’s no match before the timer runs out, ALERT Expectation Action Result
- 20. Reporting Report to a dashboard Indicates “PASS”, “FAIL”, and “Staleness”
- 21. Alternatives Anand Bagmar https://github.com/anandbagmar/WAAT Tool for testing Omniture Analytics Uses Omniture’s built-in web interface to examine attributes emitted by page, rather than using a proxy
- 22. Conclusion User Analytics are your CEO’s favorite subject! Deliver real value—million-dollar decisions are made with this data Can be implemented with just as many bugs as any other kind of software
- 23. Questions? Once again, thank you so very much Twitter: @mmerrell mmerrell@gmail.com I’m also on LinkedIn