Talk at Social Media & Web Analytics LinkedIn's homepage contains content from over 40 product areas and has evolved over hundreds of experiments. For modern websites this is not an unusual phenomena. To parallelize website development and work in harmony, product teams rely on two guidance systems, organizational cohesion and analytical feedback. Our focus is analytics and in particular, metrics. Unfortunately, not all metrics are created equal. Common metrics such as mean average precision and engagement stickiness have massive downsides if used incorrectly. Here we explore criteria to align optimizing metrics with improving user experience and reaching company goals.

1. Search, Growth & Engagement Metrics:
Snake Oil or North Stars
Dr. June Andrews
April 29, 2015

2. Table of Contents
1 Product Evolution
2 Metric Case Studies
Growth
Engagement
Search

3. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

4. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

5. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

6. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

7. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

8. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

9. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

10. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

11. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

12. Website Evolution
One Team, One Metric
Not the complete picture.

13. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

14. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

15. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

16. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

17. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

18. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

19. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

20. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

21. Website Evolution
Figure : Evolution of LinkedIn’s Signup page. Metric values are fake.

22. Website Evolution
One Team, One Multiple Metrics
Not the complete picture.

23. Website Evolution
Figure : 6 teams impact first 5 pages in registration flow: Security,
Profile, Connections, Channels, Mobile, Registration

24. Website Evolution
A/B Testing
The registration team alone has performed over 50
experiments in the past 6 months.
One Team, One Metric
Many Teams, Many Metrics
Add hundreds of A/B tests . . . now we have the picture.

25. Worst Case
Automated Algorithm
Exploit algorithm’s weaknesses, potentially by spamming the
algorithm with increasing amounts of data.
Figure : Content and Sharing teams reposition each other’s content.

26. Worst Case
Figure : Teams undo each other’s work.
Figure : Content and Sharing teams reposition each other’s content.

27. Enter Metrics
Figure : Metrics act as conductors for website development.
(De La Parra in SF)

28. Metrics
Goals
• Track progress towards vision.
• Enable multiple teams.
• Alert to changes in site health.
• Summarize copious amounts of data.
• Support executive knowledge.
• Domain and Org specific . . .
Framework
Organize the metrics required to achieve all goals.

29. Tiered Metrics

30. Progress
1 Product Evolution
2 Metric Case Studies
Growth
Engagement
Search

31. Growth
Vision
Connect the world’s professionals to make them more
productive and successful.
Growth’s Vision
Reach the world’s professionals.

32. Growth Metrics
Tier Metric
Company Number of Users
Organization Market Penetration
Mobile App Installs
Product Viral Coefficient
Growth Channels
Unique Inviters
Operational Signup Rates
CTR
Delivered Rates
Site Speed
User Growth Projections

33. Growth Metrics
Tier Metric Concern
Company Number of Users See simulation
Organization Market Penetration Prioritizes well developed markets
Product Viral Coefficient Unstable in early markets
Operational Signup Rates Growth of low engagement users
Site Speed Uncanny Valley
Table : Metrics to be handled with care.

34. Number of Users
Fire ring model:
Figure : Number of Users is continually growing, until the space is
exhausted. Snake oil or north star?

35. Number of Users
Fire ring model:
Figure : Number of Users is continually growing, until the space is
exhausted. Snake oil or north star?

36. Number of Users
Fire ring model:
Figure : Number of Users is continually growing, until the space is
exhausted. Snake oil or north star?

37. Number of Users
Fire ring model:
Figure : Number of Users is continually growing, until the space is
exhausted. Snake oil or north star?

38. Progress
1 Product Evolution
2 Metric Case Studies
Growth
Engagement
Search

39. Engagement
Vision
Connect the world’s professionals to make them more
productive and successful.
Engagement’s Vision
Foster interactions among the world’s professionals.

40. Engagement Metrics
Tier Metric
Company Monthly Active Users
Stickiness (DAU/MAU)
Page Views
Organization Engagement Distribution (4x4, 1x3, 1x1)
Interactions among Users
Product Organic Visits
Transactional Visits
Operational Session Duration
Session Pageviews
Landing Page
Frustration Rates
Interactions per session

41. Engagement Metrics
Tier Metric Concern
Company Stickiness DAU/MAU See Simulation
Organization 4x4, 1x3, 1x1 Time Delay
Product Transactional Visits Encourages Spam
Operational Session Duration Encourages Burnout
Table : Metrics to be handled with care.

42. Stickiness: DAU/MAU
Figure : Set of (MAU, DAU) with same Stickiness score.

43. Stickiness: DAU/MAU
Figure : Two sets of (MAU, DAU) values with same Stickiness score.

44. Stickiness: DAU/MAU
Figure : Improving between the two.

45. Stickiness: DAU/MAU
Snake Oil
Decrease in MAU leads to an increase in Stickiness. Exclusivity
was not part of the engagement vision. Optimizing the metric
does not lead to achieving the vision.

46. Stickiness: Alternative
North Star
MAU + 4 · DAU, promotes growth of members at all
engagement levels with rewarding increased engagement.

47. Progress
1 Product Evolution
2 Metric Case Studies
Growth
Engagement
Search

48. Search
Vision
Connect the world’s professionals to make them more
productive and successful.
Search’s Vision
Provide fast efficient search to enable professional to be more
productive.

49. Search Metrics
Tier Metric
Company Page Views
Organization Number of Searches or Sessions
Unique Searchers
Time to First Click
Product MAP (Mean Average Precision)
DCG (Discounted Cumulative Gain)
Search Intent to Result
Operational Dead End Rate
Result Load Time
Number of Results
Session Time
Successful Search Rate

50. Search Metrics
Tier Metric Concern
Company Page Views Search improvements can ↓
Organization Number of Searches Move ↑ or ↓?
Product MAP See simulation
Operational Number of Results Balance Pertinence
Table : Metrics to be handled with care.

51. MAP
MAP combines:
• Precision - Give me only what I want
• Recall - Give me everything I want
Figure : (Precision, Recall) values with same MAP score.

52. MAP
Figure : Two sets of (Precision, Recall) values with same MAP score.

53. MAP
Figure : Improve a search algorithm from point a with either small
increase in Recall or a large increase in Precision.

54. MAP
Figure : Additional points.

55. MAP
Snake Oil
When Precision and Recall values are not balanced, MAP only
responds to changes in the lower one.
North Star
When Precision and Recall values are balanced, promotes
improvement of both Precision and Recall.

56. Metric Check List
Vision
Optimized Metric Achieves Vision.
Usable
Operates in the Metric Framework to enable a many metric,
many team, many experiments environment.
Responsive
Metric moves if and only if progress towards vision changes.
Current
Metric is updated as company vision changes.

57. Thankyou!
References:
• Fire Ring - Johan Ugander
• Measure Me
• Mean Average Precision Isn’t So Nice
juneandrews.com