This document provides tips and best practices for using data-driven product development. It outlines a 4-step process: 1) understand business objectives and define key performance indicators, 2) build mock reports and define analytics events, 3) launch experiences and continuously improve through A/B testing and reports, 4) rethink the entire product to achieve global maxima. It also provides 7 tips for effective A/B testing including looking at leading indicators, managing expectations, adding common sense, getting clean samples, using "ghost links" to understand demand, using analytics for debugging, and identifying targeted audiences.

1. 1
Data-Driven Product Development
Nitzan Achsaf

2. 2
A Bit About Myself
General Manager,
Consumer
Experience
Business Unit
CEO & Founder,
SafetyNest
VP Product & Eng,
Sonicbids
(acquired by
BackStage)
Head of Next
Generation Product
Team, Yahoo
Search
Strategy Consultant,
Business Consulting
Services, IBM
Marketing Strategy
Manager, Consumer
Electronics Division,
msystems
8200, Bachelor in
Computer Science
MBA, Harvard
Business School

3. 3
Yahoo Search: $3B, 150UU/M

4. 4
Wix: $200M, 60M+ Reg Users

5. 5
There are different ways
to gather data on users

6. 6
Interviews
or just talking w/ them (-:
Observations
e.g. usability tests
Surveys
Usage Data
BI-Based

7. 7
Qualitative
Quantitative
Interviews
or just talking w/ them (-:
Observations
e.g. usability tests
Surveys
Usage Data
BI-Based

8. 8
Qualitative
Quantitative
Can be
biased
No
bias
Interviews
or just talking w/ them (-:
Observations
e.g. usability tests
Surveys
Usage Data
BI-Based

9. 9
First you should know your KPIs –
what are you trying to measure?
STEP ONE
(This is actually the second step after understanding your business (-: )

10. 10
Increased traffic
Increased conversion
Increased engagement
Increased signups
Reduced churn
How will you know that you were successful?
Etc…
Basically, the question here is

11. 11
Example
% Abandonment Rate% Login Success % Sign-Up Success % Reg-to-Anonymous
KPIs:

12. 12
Then, before you define which BI
events to add, build a mock report
based on your hypothesis
STEPTWO

13. 13
Example
64% 82% 12 Months
Premium
Wix users
Wix users already
used ShoutOut
AverageWix users’ age
1.
2.
3.
4.
5.

14. 14
After you launch your product,
continue to improve via abTests &
product analysis reports
STEPTHREE

15. 15
What is an abTest? (1 min overview)
A/BTesting = Running a simultaneous
experiment between two or more
product variants to see which
performs better
statistically significant better

16. 16
Change is signal and not noise
Use of 4 numbers
(sample sizes for a & b, converted in a & b)
Confidence >95% to say that A or B won
A. B.

17. 17
Example
100
80
60
40
20
0
2
Phase Completed
%ofVisited
Visited Clicked Phase 1 Phase 2 Phase 3
21%
90%
95%
55%
1
4 5
3

18. 18
Example
Baseline Test
20%
Increase

19. 19
To reach a global max
(e.g. 3x conversion), re-think
your entire product
STEP FOUR

20. 20
Continuing to improve the current experience / funnel will
get you close to an optimal local max (i.e. Evolution)
In order to reach a global max, you need to do
something significantly different (i.e. Revolution),
and not to continue to improve the current experience

21. 21
Example
A place where you build your site A place where you manage
and grow your business

22. 22
Understand your
business / product
objectives
Define your KPIs
1

23. 23
Understand your
business / product
objectives
Define your KPIs
Build a mock report
Define your
BI events
1
2

24. 24
Understand your
business / product
objectives
Define your KPIs
Continue to improve and iterate
(via abTests & Reports) till you
find a better baseline
Define your
BI events
Launch the experience,
and measure it;
this is your baseline
Then, it will become
your new baseline
1
2
3
Build a mock report

25. 25
Understand your
business / product
objectives
Define your KPIs
Build a mock report
Continue to improve and iterate
(via abTests & Reports) till you
find a better baseline
To reach a global max, re-
think your entire product
Define your
BI events
Launch the experience,
and measure it;
this is your baseline
Then, it will become
your new baseline
1
2
3
4

26. 26
Tip #1
Don’t have enough traffic?
Look at leading-indicators
and/or delta trend over time

27. 27
Anonymous
Free
Open
Publish
Package Picker
Save
Purchase Page
Example

28. 28
Example
-100
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Prem Diff OverTime
Days since Launch
PremiumDifferencebetweenTest&Baseline

29. 29
Tip #2
Realize that most tests
won’t impact $, especially
in the short-term

30. 30
Unless you add many features / new product
/ top funnel / payment / package picker
Sanity tests are totally fine, but set your
expectations accordingly
Tests (in most cases) won’t show you an
impact on churn ( use reports / monthly)

31. 31
Tip #3
Always add common sense

32. 32
Remember, 95% confidence, can still mean that
5 out of 100 tests can show a false-positive or
false-negative result
If the results don’t make sense, try to give it
more time (and use the tip #1)

33. 33
Tip #4
Try to get a clean sample as much
as possible

34. 34
The more precise your target audience is, the
less you “dilute” your signal and risk seeing
nothing even when there is a change
Usually start only from “new users”
Run the test on the specific page (vs. the entire
product) – i.e. only users that were actually
exposed to your new feature

35. 35
Tip #5
BI can help you understand
“demand” prior to actually building
the product/feature via “Ghost
Links”

36. 36
Example

37. 37
Example

38. 38
Example

39. 39
Tip #6
BI is also great for “debugging”

40. 40
Example

41. 41
Tip #7
If you want to perform interviews /
surveys / observations with a
targeted audience, BI reports will
help you identify it

42. 42
For instance, only users who used
feature X but not featureY
Better understand the rational
behind usage data
Investigate what is the most needed
feature or biggest pain point

43. 43
WhichTestWon.com
Cool site to follow

44. 44
ThankYou