Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Slavik Pashanin
slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa
Data-driven Testing
Let’s play a game
Testing
your product
All
Most
Common
How BI*
Works
*Business Intelligence
Implement
BI events (methods)
example
Implement
BI events (methods)
Implement
BI events (methods)
Collect
data
Implement
BI events (methods)
Collect
data
Query for
information
Make
decisions
past present future
Make
decisions
past present future
historic
Make
decisions
past present future
real time
Make
decisions
past present future
predictive
Why BI can help
your testing
Without data,
you are just another
person with an opinion.
W. Edwards Deming
Checkout form
KPI (Key Performance Indicator)
# of hits on the
checkout button
Checkout form
# of hits recorded
in the database
Success rate
Checkout form
Error rate
# of hits not recorded
in the database
Getting BI
into testing
In productionFeature Release
Feature
development
In production
Feature
Release
Feature development
In productionFeature development Feature Release
3 ways to get BI into your testing
1. Google Analytics
2. Internal tools
3. TPAs
Google Analytics
SETUP
Get the tracking number e.g., UA-XXXXX-Y and add the
code
<!-- Google Analytics -->
<script>
(funct...
Google Analytics
ADD EVENTS
Event hits are sent by using the Send command and specifying a hitType
<button id="addButton">...
Google
Analytics
▪ Page tracking
▪ Event tracking (user interactions)
▪ Social interactions
▪ User timing
▪ Error tracking
Example
Mobile
devices
Example
Users’
common
flows
Without big data analytics,
companies are blind and
deaf, wandering out
onto the web like
deer on a freeway. Geoffrey Moor...
Google
Analytics
▪ It’s free
▪ Big community
▪ Fast integration
▪ Custom reports
- upgrading is $150,000
- may be overwhel...
Internal
BI Tools
▪ Tailored events
▪ Track history or real-time
▪ Trends AND users
Catalog
for all Wix
BI events
User
explorer
(history)
User
explorer
(real-time)
Quix (stored
/ custom
queries)
Internal
BI Tools
▪ Flexible & scalable
▪ TPAs Integration
▪ Single user flows
▪ Time frames
- Takes time to
develop and
d...
TPAs
▪ Anomaly detection
▪ Reports
▪ Automatic notifications
▪ Server-side status
▪ Client-side status
Anodot
Real-time
anomaly
detection
Tableau
reports
Slack
automatic
messages
1. Drop in success rate
2. Automatic message
sent to Slack channel
3. Dev teams looking for
new GA’s open experiments
4. R...
New Relic
Server-side
status
Grafana
Client-side
status
TrackJS
Client-side
status
Sentry.io
Fix crashes
in real-time
BI focuses
you on the
right things
to test.
As part of any
feature
development, you
should rely on BI
data to make better
...
Thank You
slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa
Q&A
slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa
Upcoming SlideShare
Loading in …5
×

QA Fest 2018. Slavik Pashanin. Analytics Data Driven Testing

165 views

Published on

"Without data, you are just another person with an opinion". How can we use data to help us test our product better and be more efficient? In this lecture, we'll discuss the use of BI in testing, implementing BI using Google Analytics or an internal company BI system, TPAs (Anodot, Grafana, TrackJS), and making smart decisions using data.

Published in: Education
  • Be the first to comment

QA Fest 2018. Slavik Pashanin. Analytics Data Driven Testing

  1. 1. Slavik Pashanin slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa Data-driven Testing
  2. 2. Let’s play a game
  3. 3. Testing your product
  4. 4. All Most Common
  5. 5. How BI* Works *Business Intelligence
  6. 6. Implement BI events (methods)
  7. 7. example Implement BI events (methods)
  8. 8. Implement BI events (methods) Collect data
  9. 9. Implement BI events (methods) Collect data Query for information
  10. 10. Make decisions past present future
  11. 11. Make decisions past present future historic
  12. 12. Make decisions past present future real time
  13. 13. Make decisions past present future predictive
  14. 14. Why BI can help your testing
  15. 15. Without data, you are just another person with an opinion. W. Edwards Deming
  16. 16. Checkout form KPI (Key Performance Indicator) # of hits on the checkout button
  17. 17. Checkout form # of hits recorded in the database Success rate
  18. 18. Checkout form Error rate # of hits not recorded in the database
  19. 19. Getting BI into testing
  20. 20. In productionFeature Release Feature development
  21. 21. In production Feature Release Feature development
  22. 22. In productionFeature development Feature Release
  23. 23. 3 ways to get BI into your testing 1. Google Analytics 2. Internal tools 3. TPAs
  24. 24. Google Analytics SETUP Get the tracking number e.g., UA-XXXXX-Y and add the code <!-- Google Analytics --> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-XXXXX-Y', 'auto'); ga('send', 'pageview'); </script> <!-- End Google Analytics --> downloads analytics.js allows you to schedule commands to be run
  25. 25. Google Analytics ADD EVENTS Event hits are sent by using the Send command and specifying a hitType <button id="addButton"> Click me </button> <script type="text/javascript"> const button = document.getElementById('addButton') button.addEventListener('click',function (e) { ga('send', { hitType: 'event', eventCategory: 'buttonClick', eventAction: 'mainButtonClick', eventLabel: 'nameOfThePage' }); }); </script>
  26. 26. Google Analytics ▪ Page tracking ▪ Event tracking (user interactions) ▪ Social interactions ▪ User timing ▪ Error tracking
  27. 27. Example Mobile devices
  28. 28. Example Users’ common flows
  29. 29. Without big data analytics, companies are blind and deaf, wandering out onto the web like deer on a freeway. Geoffrey Moore, partner at MDV
  30. 30. Google Analytics ▪ It’s free ▪ Big community ▪ Fast integration ▪ Custom reports - upgrading is $150,000 - may be overwhelming
  31. 31. Internal BI Tools ▪ Tailored events ▪ Track history or real-time ▪ Trends AND users
  32. 32. Catalog for all Wix BI events
  33. 33. User explorer (history)
  34. 34. User explorer (real-time)
  35. 35. Quix (stored / custom queries)
  36. 36. Internal BI Tools ▪ Flexible & scalable ▪ TPAs Integration ▪ Single user flows ▪ Time frames - Takes time to develop and define
  37. 37. TPAs ▪ Anomaly detection ▪ Reports ▪ Automatic notifications ▪ Server-side status ▪ Client-side status
  38. 38. Anodot Real-time anomaly detection
  39. 39. Tableau reports
  40. 40. Slack automatic messages
  41. 41. 1. Drop in success rate 2. Automatic message sent to Slack channel 3. Dev teams looking for new GA’s open experiments 4. Rollback/ closing new experiments 5. Fix issue and GA 6. Back to normal
  42. 42. New Relic Server-side status
  43. 43. Grafana Client-side status
  44. 44. TrackJS Client-side status
  45. 45. Sentry.io Fix crashes in real-time
  46. 46. BI focuses you on the right things to test. As part of any feature development, you should rely on BI data to make better decisions.
  47. 47. Thank You slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa
  48. 48. Q&A slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa

×