Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Slavik Pashanin slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa Data-driven Testing
Let’s play a game
Testing your product
All Most Common
How BI* Works *Business Intelligence
Implement BI events (methods)
example Implement BI events (methods)
Implement BI events (methods) Collect data
Implement BI events (methods) Collect data Query for information
Make decisions past present future
Make decisions past present future historic
Make decisions past present future real time
Make decisions past present future predictive
Why BI can help your testing
Without data, you are just another person with an opinion. W. Edwards Deming
Checkout form KPI (Key Performance Indicator) # of hits on the checkout button
Checkout form # of hits recorded in the database Success rate
Checkout form Error rate # of hits not recorded in the database
Getting BI into testing
In productionFeature Release Feature development
In production Feature Release Feature development
In productionFeature development Feature Release
3 ways to get BI into your testing 1. Google Analytics 2. Internal tools 3. TPAs
Google Analytics SETUP Get the tracking number e.g., UA-XXXXX-Y and add the code <!-- Google Analytics --> <script> (funct...
Google Analytics ADD EVENTS Event hits are sent by using the Send command and specifying a hitType <button id="addButton">...
Google Analytics ▪ Page tracking ▪ Event tracking (user interactions) ▪ Social interactions ▪ User timing ▪ Error tracking
Example Mobile devices
Example Users’ common flows
Without big data analytics, companies are blind and deaf, wandering out onto the web like deer on a freeway. Geoffrey Moor...
Google Analytics ▪ It’s free ▪ Big community ▪ Fast integration ▪ Custom reports - upgrading is $150,000 - may be overwhel...
Internal BI Tools ▪ Tailored events ▪ Track history or real-time ▪ Trends AND users
Catalog for all Wix BI events
User explorer (history)
User explorer (real-time)
Quix (stored / custom queries)
Internal BI Tools ▪ Flexible & scalable ▪ TPAs Integration ▪ Single user flows ▪ Time frames - Takes time to develop and d...
TPAs ▪ Anomaly detection ▪ Reports ▪ Automatic notifications ▪ Server-side status ▪ Client-side status
Anodot Real-time anomaly detection
Tableau reports
Slack automatic messages
1. Drop in success rate 2. Automatic message sent to Slack channel 3. Dev teams looking for new GA’s open experiments 4. R...
New Relic Server-side status
Grafana Client-side status
TrackJS Client-side status
Sentry.io Fix crashes in real-time
BI focuses you on the right things to test. As part of any feature development, you should rely on BI data to make better ...
Thank You slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa
Q&A slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa
Upcoming SlideShare
Loading in …5
×

QA Fest 2018. Slavik Pashanin. Analytics Data Driven Testing

162 views

Published on

"Without data, you are just another person with an opinion". How can we use data to help us test our product better and be more efficient? In this lecture, we'll discuss the use of BI in testing, implementing BI using Google Analytics or an internal company BI system, TPAs (Anodot, Grafana, TrackJS), and making smart decisions using data.

Published in: Education
no profile picture user

  • Be the first to comment

QA Fest 2018. Slavik Pashanin. Analytics Data Driven Testing

  1. 1. Slavik Pashanin slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa Data-driven Testing
  2. 2. Let’s play a game
  3. 3. Testing your product
  4. 4. All Most Common
  5. 5. How BI* Works *Business Intelligence
  6. 6. Implement BI events (methods)
  7. 7. example Implement BI events (methods)
  8. 8. Implement BI events (methods) Collect data
  9. 9. Implement BI events (methods) Collect data Query for information
  10. 10. Make decisions past present future
  11. 11. Make decisions past present future historic
  12. 12. Make decisions past present future real time
  13. 13. Make decisions past present future predictive
  14. 14. Why BI can help your testing
  15. 15. Without data, you are just another person with an opinion. W. Edwards Deming
  16. 16. Checkout form KPI (Key Performance Indicator) # of hits on the checkout button
  17. 17. Checkout form # of hits recorded in the database Success rate
  18. 18. Checkout form Error rate # of hits not recorded in the database
  19. 19. Getting BI into testing
  20. 20. In productionFeature Release Feature development
  21. 21. In production Feature Release Feature development
  22. 22. In productionFeature development Feature Release
  23. 23. 3 ways to get BI into your testing 1. Google Analytics 2. Internal tools 3. TPAs
  24. 24. Google Analytics SETUP Get the tracking number e.g., UA-XXXXX-Y and add the code <!-- Google Analytics --> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-XXXXX-Y', 'auto'); ga('send', 'pageview'); </script> <!-- End Google Analytics --> downloads analytics.js allows you to schedule commands to be run
  25. 25. Google Analytics ADD EVENTS Event hits are sent by using the Send command and specifying a hitType <button id="addButton"> Click me </button> <script type="text/javascript"> const button = document.getElementById('addButton') button.addEventListener('click',function (e) { ga('send', { hitType: 'event', eventCategory: 'buttonClick', eventAction: 'mainButtonClick', eventLabel: 'nameOfThePage' }); }); </script>
  26. 26. Google Analytics ▪ Page tracking ▪ Event tracking (user interactions) ▪ Social interactions ▪ User timing ▪ Error tracking
  27. 27. Example Mobile devices
  28. 28. Example Users’ common flows
  29. 29. Without big data analytics, companies are blind and deaf, wandering out onto the web like deer on a freeway. Geoffrey Moore, partner at MDV
  30. 30. Google Analytics ▪ It’s free ▪ Big community ▪ Fast integration ▪ Custom reports - upgrading is $150,000 - may be overwhelming
  31. 31. Internal BI Tools ▪ Tailored events ▪ Track history or real-time ▪ Trends AND users
  32. 32. Catalog for all Wix BI events
  33. 33. User explorer (history)
  34. 34. User explorer (real-time)
  35. 35. Quix (stored / custom queries)
  36. 36. Internal BI Tools ▪ Flexible & scalable ▪ TPAs Integration ▪ Single user flows ▪ Time frames - Takes time to develop and define
  37. 37. TPAs ▪ Anomaly detection ▪ Reports ▪ Automatic notifications ▪ Server-side status ▪ Client-side status
  38. 38. Anodot Real-time anomaly detection
  39. 39. Tableau reports
  40. 40. Slack automatic messages
  41. 41. 1. Drop in success rate 2. Automatic message sent to Slack channel 3. Dev teams looking for new GA’s open experiments 4. Rollback/ closing new experiments 5. Fix issue and GA 6. Back to normal
  42. 42. New Relic Server-side status
  43. 43. Grafana Client-side status
  44. 44. TrackJS Client-side status
  45. 45. Sentry.io Fix crashes in real-time
  46. 46. BI focuses you on the right things to test. As part of any feature development, you should rely on BI data to make better decisions.
  47. 47. Thank You slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa
  48. 48. Q&A slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa

×