In this presentation, Will Hamlin, covers the following: - How to choose the right metric - How to balance different metrics in opposition to each other - How to choose the right time horizons for your metrics - When to use other tools, e.g., human evaluation, to measure instead - When to trust your gut. This will not be a scientific answer

1. www.productschool.com
How to Know When Metrics Are Not Right
by Pinterest PM

2. FREE INVITE
Join 23,000+ Product Managers on

3. COURSES
Product Management
Learn the skills you need to land a product manager job

4. COURSES
Coding for Managers
Build a website and gain the technical knowledge to lead software engineers

5. COURSES
Data Analytics for Managers
Learn the skills to understand web analytics, SQL and machine learning concepts

6. COURSES
Digital Marketing for Managers
Learn how to acquire more users and convert them into clients

7. COURSES
Blockchain for Managers
Learn how to trade cryptocurrencies and build products using the blockchain

8. Will Hamlin
TONIGHT’S SPEAKER

9. September
26, 2018
Product
Manager @
PinterestWill Hamlin
The data isn’t
always right

10. Tell me about you

11. What is Pinterest?
Our Mission
Help you discover & do what you
love

13. Email & Notifications at Pinterest
Our Philosophy
Email & Notifications are an
extension of the product
Our Goal
Engage Pinterest users
outside the product

14. Email & Notifications at Pinterest

15. Email & Notifications at Pinterest

16. Why Metrics?
“If we have data, let’s look at
data. If all we have are opinions,
let’s go with mine.”
– Jim Barksdale, co-founder of Netscape
Said best:

17. 5 pitfalls
You don’t control the metric
Data can be too simple
There’s a time mismatch
Data can incentivize poorly
Some things are hard to measure
1
2
3
4
5

18. What’s more important?
The really important
measurement you
don’t control
The secondary
measurement you
have full control
of

19. You don’t always control the key metric
Will start date
Will’s key
metric
1

20. Consider multiple goals
The really important
measurement you
don’t control
The secondary
measurement you
have full control
of
1

21. Consider multiple goals
Weekly Active Users
(WAU) coming from
an email/notification
Incremental Weekly
Active Users from
experiments
1

22. Which users should we focus on?
Low-engagedCore Casual

23. Low-engagedCore Casual
Data can be too simple
Clicks /
Sends
2

24. Low-engagedCore Casual
Optimize for direct impact
Incr-
emental
impact
1

25. Not everything happens in real-time

26. There can be a time mismatch
Users Opt-outs
1 week
User loss from
frequency tests
Opt-out
decrease from
frequency tests
3

27. There can be a time mismatch
Users Opt-outs
User loss from
frequency tests
Opt-out
decrease from
frequency tests
User gain from
decreased opt-
outs
1 week
3

28. Measure for the long-term
Users Opt-outs
User loss
from
frequency
tests
Opt-out
decrease
from
frequency
tests
User gain
from
decreased
opt-outs
1 week 52 week
3

29. What could go wrong b/c of our metric?
Our Goal
Engage Pinterest users
outside the product
Our Metric
Weekly Active Users
(WAU) coming from an
email/notification

30. Data can incentivize the wrong things
Our Metric
Weekly Active Users
(WAU) coming from an
email/notification
The Flaw
WAU come from clicks.
Clickbait drives clicks.
Clickbait sucks.
4

31. Add Constraints4
Content Levels
Banned OK on Pinterest,
not in email
OK in email, if
you’re interested Totally OK

32. Some things are hard to measure
Original New
pandas panda art
tattoo ideas panda
deviantART art
white walls panda painting
paintings animal art
painting
artwork
panda tattoo
watercolor
5

33. Some things are hard to measure
Engagement Relevance
panda art
panda
art
panda painting
animal art
painting
artwork
panda tattoo
watercolor
5

34. Look at examples
These are “pandas” These are pandas
5

35. 5 things to remember from today
1
2
3
4
5
Consider multiple goals
Optimize for direct impact
Measure for the long-term
Add constraints
Look at examples

36. www.productschool.com
Part-time Product Management, Coding, Data, Digital
Marketing and Blockchain courses in San Francisco, Silicon
Valley, New York, Santa Monica, Los Angeles, Austin, Boston,
Boulder, Chicago, Denver, Orange County, Seattle, Bellevue,
Toronto, London and Online