В данной сессии рассмотрим, какие популярные подходы к анализу продукта зачастую ведут к принятию неправильных решений, а также обсудим способы их избежать.

1. 10 MAJOR MISTAKES
IN PRODUCT ANALYSIS
Vera Karpova Product Owner at devtodev

2. 2
INTRODUCTION TO THE TOPIC
The goal of product analysis is to help the team
to develop the product, increase metrics and
improve UX

3. 3
TOO LATE INTEGRATION &
LACK OF TESTING
Problems:
● A new version has been released, but
the data is missing
● The data is being tracked, but it is incorrect
● Experiment was launched, but we don’t
have data for evaluating results

4. 4
CAUSES
● Team is out of sync and not all members
are aware of plans and releases
● Self-organized team without a leader and
not everyone appreciates analytics
● The user of analytics is absent in the team

5. 5
WHEN TO START
CONCEPT DEVELOPMENT
ANALYTICS SOFT LAUNCH
IMPROVEMENTS
PROTOTYPING ANALYTICS TESTING

6. 6
POOR TESTING
Mistakes
identification
INTEGRATION
Data collection
Release
Bug fixing

7. 7
TRACKING EVERYTHING
Event name Parameters and values
level_passed successful (1/0)
gems_balance (100, 300), ...
coins_balance (105, 430, ...)
energy_balance (7, 13, ...)
messages_tap messages_number (1, 2, ...)
language_changed from_to (fr_de, es_pt, ...)
user_level (1, 2, ...)
music_muted user_level (1, 2, ...)
authorization method (fb, email, ...)

8. 8
TRACKING EVERYTHING
Define the goals and ways of analysis:
● What information is important for business
● What are the near-term directions of product
development
● What questions will be answered by the
specific events
● What decisions can be taken based on the
acquired data

9. 9
ONE METRIC FOR FOCUSING
Advantage:
● All efforts are directed at one goal, the result is
quickly achieved
Disadvantages:
● The rest of metrics could be decreased dramatically
● What if this goal is wrong?

10. 10
OTHER OPTIONS
● Use the metrics tree
● Think over the risks and track them
Marketing
Development
Costs
New users
“Old users”
1st session
engagement
Variety of
locations
Location
difficulty
Tutorial
conversion
% completed
X levels during
1st session
Revenue
● Change the directions periodically
● Use several KPIs instead of one

11. 11
Metric Users Conversion Price
New users 12,000
Finished tutorial 9,000 75%
Starter pack 1,800 20% $2.99
Small gem pack 1,080 12% $4.99
Medium gem pack 540 6% $9.99
Huge gem pack 450 5% $14.99
Revenue $22,911
WORKING WITH USELESS METRICS
Hypotheses
Revenue
change
Increase tutorial conversion from 75% to 80% +7%
Reduce price of Starter Pack to $1.99, which will
lead the conversion increase by 10%
-6%
Increase number of new users from 12,000 to
15,000
+25%

12. 12
ADDITIONAL CALCULATION BONUSES
● Define how different metrics influence the key one and select the metriс which is worth to work on
● Build a model of user behavior and metric connections
● Save time on working on metrics that have little impact
● Estimate the audience size which will be affected by the experiment

13. 13
HIDE METRICS FROM THE TEAM
Problems:
● Lack of result demotivates
● Goal is to increase the metric which
is unknown
● It creates an atmosphere of distrust

14. 14
NOT SMART
Goal — run a marathon
● Run a marathon somehow, just to meet the time limit
● Run a marathon in less than 4 hours
● Run a marathon faster than a colleague
● Run a marathon and be in the first 10 finishers
SMART
Increase MAU
● of the project X
● by 20%
● for 2 months

15. 15
GOALS
● Increase the loyalty of the customers
● Optimize the development process
● Launch the product on a new market
● Reduce customer response time

16. 16
RISKS OF UNCERTAIN GOALS
● Move towards the goal very slowly
● Do it endlessly
No criteria of achieving the goal

17. 17
DO NOT AUTOMATE
How to optimize the reporting process
● Create the dashboards with the most
in-demand information
● Use scripts to automate all possible reports
Results:
● Reduce time spent making reports
● The analyst will no longer be a bottleneck
● If the analyst is unavailable, their colleague
will be able to easily build reports

18. 18
DO NOT MAKE CONCLUSIONS
The common way of A/B tests estimation — defining which variation is a winner.
Studying users to find additional information
● Send the information about the experiment to custom user properties, create a segment
● Then send parameters to custom events which are valuable for experiment

19. 19
DO NOT RISK
DATA-DRIVEN APPROACH
● Making strategic decisions based on data analysis
vs INTUITION
● Intuition is a form of knowledge that appears in consciousness without obvious
deliberation. It is not magical but rather a faculty in which hunches are generated by
the unconscious mind rapidly sifting through past experience and cumulative knowledge

20. 20
INTUITION
● Not always
● Based on huge experience
● After gaining experience with using the data
driven approach

21. 21
CONCLUSIONS
Don’t postpone analytics
integration and test it afterwards
Define measurable goals
◉ Keep an eye on
◉ several key metrics
Engage your team
Draw conclusions after
the experiments
Use your experience
for decision making (sometimes)

22. 22
VERA KARPOVA
Product Owner at devtodev
karpova@devtodev.com
www.devtodev.com
THANK YOU!