Stop Refreshing Vanity Metrics & Start Focusing on the Metrics that Inform Decisions
There is a propensity to focus on vanity metrics; metrics that show you the score: How many new views, new daily active users, how much revenue last week. You may slice these by different attributes - geography, platform, user demographics. While this can help you understand the high level trends in your business, it does little to tell you how to get better.
This slide deck looks at how vanity metrics can distract you from focusing on the analysis that matters, which is identifying and measuring the metrics that drive decisions. There are several real examples of how companies (Venmo, Simply Business, and Looker) have used event data in highly customized ways to make better decisions about their products.
4. The Case for Dashboards
Increased Engagement
Using visual to draws in the less data inclined
Clarity and Priority
Teams driving towards a singular goal set
“Data Culture”
Data as the source of truth
5. Too Broad Too Rigid Too Static
The Not-So-Good
Individuals don’t feel
empowered to drive
top-level metrics
The difficult-to-quantify
is often eschewed for
the simplest metrics
Can’t creating
alignment around
metrics that are
changing
6. Define Behavior
Quantify
Act and Refine
“Custom” Analytics
• How can we understand this behavior with data?
• Find correlated action if necessary.
• Set up tracking as needed
• Define ongoing metrics
• What are we actually trying to measure?
• Create more targeted, individualized dashboards and
views
• Track over time and monitor performance
• Build better metrics
7. Four Examples
1. Quantifying bad user experience at Venmo
2. Prioritizing new market launches at a delivery start-up
3. Better understanding of the best performing content at
Upworthy
4. Driving deeper retention in mobile gaming
15. Attention Minutes
• What action are we really trying to drive – Engagement
• Then measure engagement
• Upworthy’s attention minutes aim to do just that
• Total Attention on Site (per hour, day, week, month, whatever) — that tells us (like total uniques or total
pageviews) how good of a job Upworthy is doing overall at drawing attention to important topics.
• And Total Attention per Piece, which is a combination of how many people watch something on Upworthy
and how much of it they actually watch. Pieces with higher Total Attention should be promoted more.
Source: http://blog.upworthy.com/post/89621755036/the-code-literally-to-what-lies-between-
17. Gaming Analytics
• DAU and MAU are de facto
success in gaming
• Getting underneath the
surface is where operational
success comes from
• Cohort analysis is a crucial
tool
22. Takeaways
Define the behaviors you want to quantify
Measure them and create tangible progress
And keep the TV Dashboards, at least you’ll get people excited
about data
At HotelTonight we transitioned from a focus on bookings to a focus on room nights – everyone is driving at the same team goals
Who created the metrics – developers weren’t the ones doing this, analysts and product were doing this on their own. Observing a problem and then quantifying.
For services growing the business locally, finding the right markets to serve can be a challenge. Any scaled back service in a given market leads to impatient customers that may expect better coverage. This is acutely true for applications in the app store, where ratings are a crucial growth mechanism. When folks download the app and can’t get service, poor ratings follow.
For services growing the business locally, finding the right markets to serve can be a challenge. Any scaled back service in a given market leads to impatient customers that may expect better coverage. This is acutely true for applications in the app store, where ratings are a crucial growth mechanism. When folks download the app and can’t get service, poor ratings follow.
Knowing we now have a proxy for bad user experience, we can set goals of driving these down to zero. In this example, we look at % of users that open the app more than 5 miles from a service provider, on the upper left on different continents, ranging from the purple (not launched in South America), to the Green – coverage in the United States. Obviously all of this can be sliced up and down, even to sub-regions. On the lower right we have the launch in Puerto Rico, where coverage went from 0% quickly to near 100%.
This can be replicated everywhere, and even moving the quality bar from 5 miles down to 1 mile, etc.
This could also be done for things like response time by location on web, or search page load time. We can simply look at each users response time rather than something less granular than average conversion rate.
How many of you have run in to a online slideshow recently?
How many of you thought it was a good user experience?
Why do they exist?
Pageviews
Unfortunately advertisers are actually paying for views sometimes
This is changing and ad performance is improving both lots of businesses are still using these blunt metrics
Chartbeat, Medium have also talked about this at length – engagement over vanity metrics
Level success rates
actively rebalancing the game to drive continued usage
Level success rates
actively rebalancing the game to drive continued usage
Level success rates
actively rebalancing the game to drive continued usage