This document discusses using analytics in the product development lifecycle. It covers: 1. Different business models and stages of growth that determine the appropriate metrics to track, such as empathy, stickiness, virality, revenue, and scale. 2. What makes a good metric, including being comparative, understandable, a rate or ratio, and changing user behavior. It warns against "vanity metrics" like page views or followers. 3. Different types of metrics including exploratory, qualitative vs. quantitative, leading vs. lagging, and correlated vs. causal. 4. How to use segments, cohorts, A/B testing, and multivariate testing to test changes and see what correlates with desired

1. Using Analytics in the
Product development
LifeCycle

2. HELLO!
I am Muthoni
I am here because I love to make sense
of data.
2

3. “
Everyone has data, the key is figuring
out what pieces will improve your
learning and decision making.
Everyone knows they need metrics, but
finding ones that are specific,
measurable, actionable, relevant and
timely is a huge challenge.
Zach Nies, CTO Rally Software
3

4. Analytics & Product
development
▸ Business/ Product models
▸ Stages in growth
▸ Analytics
▸ Thinking like a data scientist
▸ Q/A
4

5. Product
Development
life cycles
1

6. 1

7. Business
Models
What business are you in?
What stage of growth are you in?
7

8. What business are
you in?
E-commerce
Selling things to customers
through your platform.
SAAS
Offering software as a
service to customers.
Mobile Applications
Mobile applications that use
in-app purchases to generate
revenue
8
User-Generated Content
Primary focus is getting your
users to generate content on
your platform
Content creation
Media sites & revenue
generation through
advertising
Two-sided marketplaces
Buyers & sellers come
together on your platform

9. What stage of
growth are you in?
1. Empathy
What is important to the
customers?
2. Stickiness
Is what you have built sticky
and will customers engage?
3. Virality
At this stage, it’s time to
focus on user acquisition &
growth
9
4. Revenue
Can you make money in a
scalable, consistent and
self-sustaining way?
5. Scale
Wider audiences & entry into
new markets
Model + Stage drives the
metrics you track

10. Metrics &
Analytics
▸ What makes a good metric?
▸ What are vanity metrics?
▸ Types of metrics
▸ A/B testing vs Multivariate
testing
▸ Segments and Cohorts
10

11. What makes a
good metric?
1. A good metric is comparative
2. A good metric is
understandable
3. A good metric is a rate or ratio
4. A good metric changes the way
you behave.
11

12. A good metric?
“Instagram gave it’s users 3 simple
measures for how they were
performing. A follower count, a
following count, and likes on their
photos.”
Source: No filter
12

13. A good metric?
“These feedback scores were
enough to make the experience
thrilling, even addicting. With
every like and follow a user
would get a little rush of
satisfaction, sending dopamine
to the brain’s reward centres.”
Source: No filter
13

14. A good metric?
“Facebook automatically
catalogued every tiny action
from it’s users, not just their
comments and clicks but words
they typed and did not send,
posts they hovered over while
scrolling and did not click, and
the people’s name they
searched and did not befriend.”
Source: No filter
14

15. A good metric?
“They could use that data to
figure out who your closest
friends were, defining the
strength of your relationship
with a constantly changing
number between 0 and 1 they
called a friend coefficient.”
Source: No filter
15

16. Types of Metrics
16
Exploratory &
Reporting metrics
Qualitative VS
Quantitative Metrics
Leading & Lagging
metrics
Correlated & Casual
Metrics

17. Vanity Metrics &
How to avoid them
17

18. 1. Number of hits
2. Number of page views
3. Number of visits
4. Number of unique visitors
5. Number of followers, friends, likes
6. Time on site
7. Emails collected
8. Number of downloads
Source: Lean Analytics
8 Vanity Metrics to watch out for:
18

19. Segments &
Cohorts
A B C
Yellow 10 20 7
Blue 30 15 10
Orange 5 24 16
19

20. Segments & Cohorts
20
Jan Feb March April May
Total
Customers
1000 2000 3000 4000 5000
Average
revenue per
customer
Ksh.500 Ksh.450 Ksh.440 Ksh.425 KSh.450

21. Segments & Cohorts
21
Jan Feb March April May
New
users
1000 1000 1000 1000 1000
Total
users
1000 2000 3000 4000 5000
Month 1 Ksh.500 Ksh.300 Ksh.200 Ksh.100 Ksh.50
Month 2 Ksh.600 Ksh.400 Ksh.200 Ksh.100
Month 3 Ksh.700 Ksh.600 Ksh.500
Month 4 Ksh.800 Ksh.700
Month 5 Ksh.900

22. Segments & Cohorts
22
Month of use
Cohort 1 2 3 4 5
Jan Ksh.500 Ksh. 300 Ksh. 200 Ksh. 100 Ksh. 50
Feb Ksh.600 Ksh. 400 Ksh.200 Ksh.100
March Ksh.700 Ksh. 600 Ksh.500
April Ksh.800 Ksh. 700
May Ksh.900
June
Average Ksh.700 Ksh.500 Ksh.300 Ksh.100 Ksh.50

23. Segments
Comparing all people
divided by some attribute
eg. age, gender, region
Segments vs A/B Testing Vs
Multivariate testing
A/B Testing
Changing one thing eg.
color, shape and measuring
the result eg. clicks,
revenue
23
Multivariate Testing
Changing several things
(color, text, pictures) at
once to see which one
correlates with a metric of
interest eg. clicks, revenue

24. 89,526,124What is the one metric that matters to you?
24

25. Review...
25
XXXX
Identify
What are the top 3-5
metrics you track &
review frequently?
XXXX
Actionable
Which are good
metrics & why?
XXXX
Data-driven decisions
Which one’s do you use
to make decisions &
which are vanity
metrics
XXXX
Actions:
1. Which ones will you
eliminate ?
2. Which ones will you
add to the list that are
more meaningful?

26. Ksh. 89,526,124
That’s a lot of money
100%
Total success!
185,244 users
And a lot of users
26

27. Thinking like a data scientist:
Pitfalls to avoid as you dig into the data-
1. Assuming the data is
clean
Simple data cleaning can often reveal
interesting patterns. Eg. Is a bug
causing 20% of the data to be null?
2. Excluding outliers
Ignoring the 20 people who use your
product 1000 times a day might be a
mistake if they are actually bots & not
humans
3. Including outliers
When building a general model, to
inform product development including
the 20 people, might not be productive.
27
4. Ignoring seasonality
Is “intern” the fastest-growing job of
the year? Ensure you consider time of
day, day of week and monthly changes
when looking at patterns
5. Ignoring size when
reporting growth
Context is key. “ Your dad signing up,
doesn’t count as doubling your user
base”
6. Data Vomit
A dashboard isn’t of much use if you
don’t know where to look .
Source: Monica Ragati, Data Scientist-
Linkedin

28. Closing thoughts
28

29. 29
THANKS!
Any questions?
You can find me at:
▸ @_mzonn (Twitter)
▸ muthoniwanyoike@gmail.com

30. Credits
Special thanks to all the people who made and released these
awesome resources for free:
▸ Presentation template by SlidesCarnival
▸ Illustrations by Sergei Tikhonov
▸ Photographs by Unsplash
▸ Lean Startups & Lean Analytics
30