The document outlines a process called the "Data Driven Decision Engine" which combines engineering, data science, and product management to make better decisions. It describes generating ideas from many sources, sizing opportunities, developing products in a controlled way, communicating recommendations, and continuously learning and improving the process. The goal is to use data-driven insights along with human perspectives to build products that balance growth, quality, and member experience over the long term.

1. Data Driven Decision Engine
Engineer + Data Science + Product Manager = More Power!
Dr. June Andrews
February 10, 2015
June Andrews Data Driven Decision Engine February 10, 2015 1 / 42

2. Professional Social Graph
June Andrews Data Driven Decision Engine February 10, 2015 2 / 42

3. LinkedIn’s Vision
June Andrews Data Driven Decision Engine February 10, 2015 3 / 42

4. LinkedIn for Members
June Andrews Data Driven Decision Engine February 10, 2015 4 / 42

5. LinkedIn for Customers
June Andrews Data Driven Decision Engine February 10, 2015 5 / 42

6. Diverse Product Portfolio
Figure : Over 40 products integrated into the homepage.
June Andrews Data Driven Decision Engine February 10, 2015 6 / 42

7. June Andrews
My Role
Understand:
User Ecosystem
Engineering & Data
Constraints
Vision
Recommend Roadmaps
Support Roadmaps
Figure : Which is better? Routes via
Google Maps
June Andrews Data Driven Decision Engine February 10, 2015 7 / 42

8. Roadmaps Worked On
June Andrews Data Driven Decision Engine February 10, 2015 8 / 42

9. Roadmap Process
1 Generate Ideas
2 Project Sizing
Project Impact
Details for Maximal Impact
3 Project Development Playbook
4 Communicate Recommendations
5 Learn
Note
Use the same process for yearly, quarterly, internal, and external
roadmaps.
June Andrews Data Driven Decision Engine February 10, 2015 9 / 42

10. Generate Ideas
1 Generate Ideas
2 Project Sizing
Project Impact
Details for Maximal Impact
3 Project Development Playbook
4 Communicate Recommendations
5 Learn
June Andrews Data Driven Decision Engine February 10, 2015 10 / 42

11. Sourcing Ideas
Source from Everywhere:
External Inspiration: books, research, sociologists, etc.
Brain storming sessions with PMs, Designers, & Engineers
Deep Dive Analysis
Friends & Family
Details
This is the top of the funnel. Make it big.
Limit how long you listen, not where you listen.
Do not interject ’We tried that.’
June Andrews Data Driven Decision Engine February 10, 2015 11 / 42

12. Edit Ideas
Example (My Mother)
Products can come from a personal space. Translate the example into
salient points.
Example (Generalize Products)
Products can come from a product specific space. Generalize
requests for UI changes to the broader product base.
Example (Unify Products)
When a particularly big change is desired it will appear as many small
suggestions. Find the focus of what product wants change.
June Andrews Data Driven Decision Engine February 10, 2015 12 / 42

13. Support Visionary Goals
Figure : Line Ideas up with Vision, check for full coverage.
June Andrews Data Driven Decision Engine February 10, 2015 13 / 42

14. Batch Process Ideas
1 Generate Ideas
2 Project Sizing
Project Impact
Details for Maximal Impact
3 Project Development Playbook
4 Communicate Recommendations
5 Learn
June Andrews Data Driven Decision Engine February 10, 2015 14 / 42

15. Idea Sizing - Back of the Envelope
Goal
For each idea find:
1 How many members involved
2 How frequently involved
3 Magnitude of involvement
Physics:
Force = Mass · Acceleration
Work = Force · Distance
Products:
Impact = Number of People · ∆Metric
Work = Impact · Product Cost
June Andrews Data Driven Decision Engine February 10, 2015 15 / 42

16. Idea Sizing - Back of the Envelope
User error propagation to temper expectations:
Ideal Impact = Number of People · ∆Metric
Estimated Impact = 1
2 · Number of People · 1
2 · ∆Metric
Estimated Impact = 1
4 · Ideal Impact
Note
These estimates will be checked after the product is built. Be
conservative - under promise over deliver.
June Andrews Data Driven Decision Engine February 10, 2015 16 / 42

17. Batch Process Idea Sizing
Idea Success Ratio
For every idea in production, there are ≈ 7 ideas that did not make the
cut. A single roadmap involves 5 to 20 major projects.
Figure : By adding dimensions to Hadoop queries can batch process ideas.
June Andrews Data Driven Decision Engine February 10, 2015 17 / 42

18. Find the Giants
Find the Maximal Impact
A big idea is first seen from multiple angles as small ideas.
Figure : Glimpses of the big picture. The Godzilla!
June Andrews Data Driven Decision Engine February 10, 2015 18 / 42

19. Product Development Playbook
1 Generate Ideas
2 Project Sizing
Project Impact
Details for Maximal Impact
3 Project Development Playbook
4 Communicate Recommendations
5 Learn
June Andrews Data Driven Decision Engine February 10, 2015 19 / 42

20. New Product
Quantity or Quality
Blank page effect is quantity without quality.
Diamond in the rough effect is quality without quantity.
Recommendation:
Improve Quality
Improve Quantity
Iterate
Goal
Long Term Growth. Virality easily controls Quantity, Quality is hard.
Spend 80 to 20 on quality to quantity.
June Andrews Data Driven Decision Engine February 10, 2015 20 / 42

21. Order Matters - Social Products
Figure : Fire burns outward as a ring. A metric of burn length increases until
the fire burns out.
June Andrews Data Driven Decision Engine February 10, 2015 21 / 42

22. Fire Ring Examples
Figure : Farmville hit 80M users in 1 year. Google+ hit 25M users in 24 days
with an average of 7 min per month per user.
Fatal Flaw
Turned on uncontrolled viral mechanisms before creating a solid
member experience.
June Andrews Data Driven Decision Engine February 10, 2015 22 / 42

23. Controlled Virality
Figure : Facebook hit 6M users at 1 year. Gmail spent 3 years as an
invitation only service.
Key Components
1 Released in stages to new populations.
2 Release delay allowed for quality improvements.
3 You were hungry for it, before you could get it.
June Andrews Data Driven Decision Engine February 10, 2015 23 / 42

24. Order Matters - Local or Global
Local Domain
Question is still Quantity or Quality.
Growth Mechanisms:
Community Managers
Power Users or Elites
All Social Viral Mechanisms
June Andrews Data Driven Decision Engine February 10, 2015 24 / 42

25. Order Matters - Local or Global?
Figure : Citysearch developed many reviewers with few reviews. Thomas
Brothers spent 94 years being the expert map makers of the west coast.
Fatal Flaw
Balance. Citysearch went global before understanding local drivers.
Thomas Brothers fought to stay local.
June Andrews Data Driven Decision Engine February 10, 2015 25 / 42

26. Order Matters - Controlled Local & Global
Figure : Both Yelp and Uber grow one city at a time.
Key Components
Community Managers
Rewarded initial users with parties and discounts
Word of Mouth Virality - slow and controlled
June Andrews Data Driven Decision Engine February 10, 2015 26 / 42

27. Order Matters - Established Product Shifts
Established Domain
Goal is to protect power user base and create new opportunities
Growth Mechanisms:
Grandfathering of old members
Layering of new and old product
App specialization
June Andrews Data Driven Decision Engine February 10, 2015 27 / 42

28. Shifts without Grandfathering
Figure : Netflix split DVD mailings from base subscriptions. Foursquare
ported checkins over to Swarm.
Fatal Flaw
Final outcome is still to be seen. Power users provided copious
negative feedback about having to adapt their experience.
June Andrews Data Driven Decision Engine February 10, 2015 28 / 42

29. Shifts with grandfathering
Figure : Pandora grandfathered in yearly contracts to their now monthly
subscription. Gmail’s introduction of tabs can be set to old experience.
Key Components
Positive messaging for power users
Notice of changes far in advance
Expanded opportunity for connecting with new members
June Andrews Data Driven Decision Engine February 10, 2015 29 / 42

30. Simulate Growth
Stochastic Processes
Can accurately prodict a year out. Simulate changes in virality
coefficients and engagement.
June Andrews Data Driven Decision Engine February 10, 2015 30 / 42

31. Humanize the Data
1 Generate Ideas
2 Project Sizing
Project Impact
Details for Maximal Impact
3 Project Development Playbook
4 Communicate Recommendations
5 Learn
June Andrews Data Driven Decision Engine February 10, 2015 31 / 42

32. Member Base Perspectives
Figure : Data Driven & Intuitive perspectives of the member base.
June Andrews Data Driven Decision Engine February 10, 2015 32 / 42

33. Member Base Perspective
Figure : Perseption of member base change.
Members as Data
Advantage is well defined tracking for all members.
Disadvantage is limited understanding of emotional impact.
June Andrews Data Driven Decision Engine February 10, 2015 33 / 42

34. Member Base Perspective
Train Intuitive Thinking
Find a manageable set of representative users.
Interview these members. UEX team.
Case Study these members’ experiences and long term behavior
Figure : Use Data to train Intuitive Thinking.
June Andrews Data Driven Decision Engine February 10, 2015 34 / 42

35. Reflect
1 Generate Ideas
2 Project Sizing
Project Impact
Details for Maximal Impact
3 Project Development Playbook
4 Communicate Recommendations
5 Learn
June Andrews Data Driven Decision Engine February 10, 2015 35 / 42

36. Reflect
Refine
Compare Project Sizing estimates and launch results
Compare Playbook with release strategy
Adapt elements to work with the company you are at
Preserve past ideas and sizing for future considerations
June Andrews Data Driven Decision Engine February 10, 2015 36 / 42

37. Recap - Generate Ideas
No voice is too small.
June Andrews Data Driven Decision Engine February 10, 2015 37 / 42

38. Recap - Size Opportunity
Bound the future.
June Andrews Data Driven Decision Engine February 10, 2015 38 / 42

39. Recap - Release Playbook
Build with balance and pivots.
Figure : Seahawks’ playbook did not include Lynch in the final 2 minutes.
June Andrews Data Driven Decision Engine February 10, 2015 39 / 42

40. Recap - Communicate
Make conclusions relatable and memorable.
Figure : Humanize the Data
June Andrews Data Driven Decision Engine February 10, 2015 40 / 42

41. Recap - Reflect
Tune the Data Driven Decision Engine!
Figure : It takes a village to run this engine.
June Andrews Data Driven Decision Engine February 10, 2015 41 / 42

42. We’re Hiring
Drive the Data Driven Decision Engine!
June Andrews Data Driven Decision Engine February 10, 2015 42 / 42