1.
Slavik Pashanin
slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa
Data-driven Testing

2.
Let’s play a game

3.
Testing
your product

4.
All
Most
Common

5.
How BI*
Works
*Business Intelligence

6.
Implement
BI events (methods)

7.
example
Implement
BI events (methods)

8.
Implement
BI events (methods)
Collect
data

9.
Implement
BI events (methods)
Collect
data
Query for
information

10.
Make
decisions
past present future

11.
Make
decisions
past present future
historic

12.
Make
decisions
past present future
real time

13.
Make
decisions
past present future
predictive

14.
Why BI can help
your testing

15.
Without data,
you are just another
person with an opinion.
W. Edwards Deming

16.
Checkout form
KPI (Key Performance Indicator)
# of hits on the
checkout button

17.
Checkout form
# of hits recorded
in the database
Success rate

18.
Checkout form
Error rate
# of hits not recorded
in the database

19.
Getting BI
into testing

20.
In productionFeature Release
Feature
development

21.
In production
Feature
Release
Feature development

22.
In productionFeature development Feature Release

23.
3 ways to get BI into your testing
1. Google Analytics
2. Internal tools
3. TPAs

24.
Google Analytics
SETUP
Get the tracking number e.g., UA-XXXXX-Y and add the
code
<!-- Google Analytics -->
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-XXXXX-Y', 'auto');
ga('send', 'pageview');
</script>
<!-- End Google Analytics -->
downloads
analytics.js
allows you to
schedule commands
to be run

25.
Google Analytics
ADD EVENTS
Event hits are sent by using the Send command and specifying a hitType
<button id="addButton"> Click me </button>
<script type="text/javascript">
const button = document.getElementById('addButton')
button.addEventListener('click',function (e) {
ga('send', {
hitType: 'event',
eventCategory: 'buttonClick',
eventAction: 'mainButtonClick',
eventLabel: 'nameOfThePage'
});
});
</script>

26.
Google
Analytics
▪ Page tracking
▪ Event tracking (user interactions)
▪ Social interactions
▪ User timing
▪ Error tracking

27.
Example
Mobile
devices

28.
Example
Users’
common
flows

29.
Without big data analytics,
companies are blind and
deaf, wandering out
onto the web like
deer on a freeway. Geoffrey Moore,
partner at MDV

30.
Google
Analytics
▪ It’s free
▪ Big community
▪ Fast integration
▪ Custom reports
- upgrading is $150,000
- may be overwhelming

31.
Internal
BI Tools
▪ Tailored events
▪ Track history or real-time
▪ Trends AND users

32.
Catalog
for all Wix
BI events

33.
User
explorer
(history)

34.
User
explorer
(real-time)

35.
Quix (stored
/ custom
queries)

36.
Internal
BI Tools
▪ Flexible & scalable
▪ TPAs Integration
▪ Single user flows
▪ Time frames
- Takes time to
develop and
define

37.
TPAs
▪ Anomaly detection
▪ Reports
▪ Automatic notifications
▪ Server-side status
▪ Client-side status

38.
Anodot
Real-time
anomaly
detection

39.
Tableau
reports

40.
Slack
automatic
messages

41.
1. Drop in success rate
2. Automatic message
sent to Slack channel
3. Dev teams looking for
new GA’s open experiments
4. Rollback/ closing new
experiments
5. Fix issue and GA
6. Back to normal

42.
New Relic
Server-side
status

43.
Grafana
Client-side
status

44.
TrackJS
Client-side
status

45.
Sentry.io
Fix crashes
in real-time

46.
BI focuses
you on the
right things
to test.
As part of any
feature
development, you
should rely on BI
data to make better
decisions.

47.
Thank You
slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa

48.
Q&A
slavikp@wix.com twitter@slavik_pashanin github.com/slavikpa