With millions of new data points every moment, how are Product Managers expected to make sense of it all? This talk outlined the steps required to distill and synthesize data to drive actionable product decisions. The most effective Product Managers are those who know their data: they can justify product priority and roadmap changes, calibrate resource asks and manage their own time more effectively. This lecture equipped the audience with the tools necessary to draw insight from unstructured data using Google’s cloud analytics suite.
8. Data and Analytics for
Product Managers
Joao Fiadeiro, PM @ Google
June 2018
9. A little about me...
● Product Manager at Google,
where I work in the YouTube
Music team
○ Involved in social/community
efforts for artists and analytics
● I began my career as a data
scientist 5 years ago
● Proud generalist!
10. Today’s content
Part I: Analytics Concepts for PMs
● The key concepts within analytics
Part II: Implementing Analytics
● From vision to metrics
Part III: Experimentation
● Data-driven product roadmaps
Part IV: Reporting
● Let your analysis be heard
High-level overview of
the role of analytics in a
Product Manager’s day-
to-day…
...Followed by a hands-
on demo of some
useful tools
12. Deriving value from data: key concepts
Data Points
Segmentation Funnels Cohorts
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
13. Data Points
Data points are the individual points of collected data that are
measurements of particular items within the platform
● Data usually exists somewhere; though it may be hard to find
○ Is the right data being collected?
○ Do you have access to it?
● As a PM, your job is to know what data is being collected and
where it lives
● If you don’t have measurements, you don’t have anything…
Get creative.
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
14. Segmentation
Segmentation is about grouping together people by a
common characteristic and seeing what the usage patterns of
the product are as a group
● Segments must be measurable; typical segments are
behavioral, technical or demographic
● Segmentation slices the analytics, allowing underlying
patterns in behaviour and usage to be observed, rather than
be hidden by noise and averaging
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
15. Gangnam Style: A study in Attention Span
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
Watch Time (%) by Gender and Age Avg. % Completion by Age
16. Funnels
A funnel is made up of the measurement of the key event at
each step of the flow or user journey
● Users don’t just do something in isolation. Instead, they
perform a series of actions to accomplish a task or goal.
These flows or user journeys can be measured using funnels
● Optimize for improving every step of the funnel. A solid
conceptual funnel allows for rich metrics between every layer
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
17. Funnels at YouTube
Each layer expressed a
ratio of the previous
layer… is worth more than
a thousand words.
Find the golden path!
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
18. Cohorts
The primary purpose of cohort analysis is for comparative analysis
to answer the question of how users’ behaviour changes over
time
● Segment users into buckets and explore differences in
behavior.
○ For example: how does the behaviour of users who registered a
week ago differ from that of users who registered a month ago?
● Used to understand retention and churn
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
21. Implementing Analytics
Product Managers must establish a clear product vision but
should also be able to articulate key performance indicators
The process of planning consists of these steps:
1. Define the product vision
2. Define the KPIs that meet the product vision
3. Define the metrics that allow you to hit your KPIs
4. Define the funnels (via user journeys) that affect your metrics
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
22. The Vision
● By starting with the vision, you ensure that what you measure
will help you achieve the product vision
● Avoid the trap of vanity metrics by tying everything that is
measured to what you are trying to achieve
● It is the filter that allows you to ignore the potential mass of
data you can collect
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
23. Key Performance Indicators (KPIs)
● KPIs are derived from the product vision and tell you how well
your product is meeting the vision.
○ They are product focused and only indicate the performance of
the product.
● KPIs are used to set targets for the performance of the
product
● The KPIs need to reflect the current stage your product is at
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
24. Metrics: what should you be measuring?
Engagement vs Transactional Apps
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
26. Growth:
Gaining new users
How many new users do you have
and where do they come from?
● # of daily/weekly/monthly new
user signups
● Metrics by acquisition channel
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
Growth
● DAU/WAU/MAU
● Trial Users/Paying subscribers
27. Transactions & Engagement:
Increasing usage of the app
How much are your users engaging with
the product?
● Engagement: Typically
consumption
● Transactions: Average Order Value
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
Transactions
● Views/Watch Time (per user)
● Likes/Shares/Comments
28. Retention:
Ensuring that existing users come back
Are your users coming back for more?
● How many of your users are
coming back within 24 hrs, 7 days
and 28 days?
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
● Avg. Session length
● DAU/MAU
Retention
29. Monetization:
Converting usage into dollars
How effectively are you converting
usage into revenue?
● Ad-supported or Subscription
m10n
● Revenue share/margins
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
● Revenue per Watch Hr (RPH)
● Avg Revenue Per User (ARPU)
Monetization
30. A word of caution...
Most of these metrics, on their own, might make you feel good,
but they don’t offer clear guidance for what to do…
Every new feature that is being considered should move the
needle on the key metrics in A/B tests otherwise the feature
may be of questionable value
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
31. Experiments (aka. A/B Tests)
A/B experiments produce the most actionable of all metrics,
because they explicitly refute or confirm a specific hypothesis
Analytics → “What is going on with x?”
Experiments → “How do we improve x?”
Experimentation informs:
Planning of (new)
experiments
Product backlog
Development
prioritisation
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
32. Experimentation Cycle
Plan
Monitor Implement
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
Start with a question and
formulate a hypothesis.
Identify (in)dependent
variables
Implement in code (or tool)
and be patient
Do not change the control!
Segment, segment,
segment
Ensure statistical validity
33. Closing the loop
Ask yourself these two questions:
● What do these results mean for development prioritisation?
● Why did I get these results?
The result whether positive or negative is immaterial, what is
material is that you learnt something from the test
Keep experimenting - ensure it’s in the product & engineering
culture
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
35. Reporting:
The danger zone
● Don’t get caught up in the data/analytics! If you don’t report
your findings, you’re wasting your time
● Beautiful visualizations are great, but less is more
● Some tips:
○ Do it in a manner that is easily grasped by everyone in the
company
○ Allow for as much input/interaction from the audience as
possible
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
36. Reporting:
Managing stakeholders
● When presenting to senior stakeholders, you’ll likely get many
“what if?” follow up questions
● Advice: come prepared with multiple scenarios. If possible,
complement with a spreadsheet that allows custom inputs
Projected
MAU
Watch Hours
per MAU RPH
Total
Revenue Probability
Expected
Revenue
Best Case 300M 20 $0.020 $120M 10% $12M
Expected Case 280M 15 $0.015 $63M 70% $44M
Worst Case 200M 12 $0.012 $29M 20% $6M
TOTAL $62M
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
37. Reporting:
What about dashboards?
● Dashboards can be very useful, but also very dangerous to PMs
● Pros:
○ Audience can self-serve data needs
○ PM does not waste time with data requests
○ Can be widely published and shared
● Cons:
○ Audience can make incorrect assumptions about data
○ PM may have to maintain them → Huge time sink
○ Orphaned data is a real problem
Analytics Concepts
Implementing Analytics
Experimentation
Reporting
39. Google Big Data Solutions
Google offers an integrated, serverless Big Data
platform for data-driven applications
40. Today we’ll look at three products
A Product Manager can use these products to analyze, explore,
and present data in an effective way:
1. BigQuery: fully managed, low cost analytics data warehouse.
Use SQL to query massive datasets.
2. Dataprep: data service for visually exploring, cleaning, and
preparing structured and unstructured data for analysis
3. Data Studio: turns data into dashboards and reports that are
easy to read, share, and customize.
42. Dataprep Demo
Goals:
● Import a CSV file
● Create a ‘Flow’ then a recipe
● Explore data transformations
● Export dataset
Link
43. Data Studio Demo
Goals:
● Import data source
● Create a few charts
● Make and customize a table
● Add filters
● Publish
Link
44. Part-time Product Management Courses in
San Francisco, Silicon Valley, Los Angeles, New York, Austin,
Boston, Seattle, Chicago, Denver, London, Toronto
www.productschool.com
Editor's Notes
understand web analytics, learn SQL, and machine learning concepts
Real world example:
Data not there at all: Need to start measuring what exactly is a post impression
Data aggregation: information is data in context → Need to generate data points that are aggregation of individual elements
External data: using APIs to do competitor research
Gender is hard to measure! Age!
When you hear companies doing PR about the billions of messages sent using their product, or the total GDP of their economy, think vanity metrics. But there are examples closer to home. Consider the most basic of all reports: the total number of “hits” to your website. Let’s say you have 10,000. Now what? Do you really know what actions you took in the past that drove those visitors to you, and do you really know which actions to take next? In most cases, I don’t think it’s very helpful.
At a high level, engagement apps are focused on getting their users to perform actions within the app. For example, Facebook would like you to post on your wall, like other posts and share posts. Spotify, on the other hand, would like you to spend more hours listening to music within the app.
Transactional apps are focused on getting users to complete a transaction. For example, Amazon would like you to buy as many things from them as possible. Uber, would like you to order a car and complete your journey as quickly as possible.
Used by Spotify, Evernote, SnapChat
SELECT
category,
COUNT(DISTINCT unique_key)
FROM
`bigquery-public-data.san_francisco.sfpd_incidents`
GROUP BY 1