Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

QA Fest 2018. Slavik Pashanin. Analytics Data Driven Testing

95 views

Published on

"Without data, you are just another person with an opinion". How can we use data to help us test our product better and be more efficient? In this lecture, we'll discuss the use of BI in testing, implementing BI using Google Analytics or an internal company BI system, TPAs (Anodot, Grafana, TrackJS), and making smart decisions using data.

Published in: Education
  • Be the first to comment

QA Fest 2018. Slavik Pashanin. Analytics Data Driven Testing

  1. 1. Slavik Pashanin slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa Data-driven Testing
  2. 2. Let’s play a game
  3. 3. Testing your product
  4. 4. All Most Common
  5. 5. How BI* Works *Business Intelligence
  6. 6. Implement BI events (methods)
  7. 7. example Implement BI events (methods)
  8. 8. Implement BI events (methods) Collect data
  9. 9. Implement BI events (methods) Collect data Query for information
  10. 10. Make decisions past present future
  11. 11. Make decisions past present future historic
  12. 12. Make decisions past present future real time
  13. 13. Make decisions past present future predictive
  14. 14. Why BI can help your testing
  15. 15. Without data, you are just another person with an opinion. W. Edwards Deming
  16. 16. Checkout form KPI (Key Performance Indicator) # of hits on the checkout button
  17. 17. Checkout form # of hits recorded in the database Success rate
  18. 18. Checkout form Error rate # of hits not recorded in the database
  19. 19. Getting BI into testing
  20. 20. In productionFeature Release Feature development
  21. 21. In production Feature Release Feature development
  22. 22. In productionFeature development Feature Release
  23. 23. 3 ways to get BI into your testing 1. Google Analytics 2. Internal tools 3. TPAs
  24. 24. Google Analytics SETUP Get the tracking number e.g., UA-XXXXX-Y and add the code <!-- Google Analytics --> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-XXXXX-Y', 'auto'); ga('send', 'pageview'); </script> <!-- End Google Analytics --> downloads analytics.js allows you to schedule commands to be run
  25. 25. Google Analytics ADD EVENTS Event hits are sent by using the Send command and specifying a hitType <button id="addButton"> Click me </button> <script type="text/javascript"> const button = document.getElementById('addButton') button.addEventListener('click',function (e) { ga('send', { hitType: 'event', eventCategory: 'buttonClick', eventAction: 'mainButtonClick', eventLabel: 'nameOfThePage' }); }); </script>
  26. 26. Google Analytics ▪ Page tracking ▪ Event tracking (user interactions) ▪ Social interactions ▪ User timing ▪ Error tracking
  27. 27. Example Mobile devices
  28. 28. Example Users’ common flows
  29. 29. Without big data analytics, companies are blind and deaf, wandering out onto the web like deer on a freeway. Geoffrey Moore, partner at MDV
  30. 30. Google Analytics ▪ It’s free ▪ Big community ▪ Fast integration ▪ Custom reports - upgrading is $150,000 - may be overwhelming
  31. 31. Internal BI Tools ▪ Tailored events ▪ Track history or real-time ▪ Trends AND users
  32. 32. Catalog for all Wix BI events
  33. 33. User explorer (history)
  34. 34. User explorer (real-time)
  35. 35. Quix (stored / custom queries)
  36. 36. Internal BI Tools ▪ Flexible & scalable ▪ TPAs Integration ▪ Single user flows ▪ Time frames - Takes time to develop and define
  37. 37. TPAs ▪ Anomaly detection ▪ Reports ▪ Automatic notifications ▪ Server-side status ▪ Client-side status
  38. 38. Anodot Real-time anomaly detection
  39. 39. Tableau reports
  40. 40. Slack automatic messages
  41. 41. 1. Drop in success rate 2. Automatic message sent to Slack channel 3. Dev teams looking for new GA’s open experiments 4. Rollback/ closing new experiments 5. Fix issue and GA 6. Back to normal
  42. 42. New Relic Server-side status
  43. 43. Grafana Client-side status
  44. 44. TrackJS Client-side status
  45. 45. Sentry.io Fix crashes in real-time
  46. 46. BI focuses you on the right things to test. As part of any feature development, you should rely on BI data to make better decisions.
  47. 47. Thank You slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa
  48. 48. Q&A slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa

×