We are a team of 12 comprising of analysts, data scientists and leads. At smule, I led performance mktg analytics as well as brand marketing analytics so think TV and other brand campaigns Prior to Smule I was at 2 other tech startups(flipboard&gofundme) as part of their growth team. We worked on A/B testing and optimizing our onboarding flows, understanding the value of performance marketing across digital channels and brand marketing Prior to these startups I was at Vmware for 7 yrs when I joined we were 500 and when I left we were 15000 so it was a great learning experience for me. Working on growth initiatives has been a strong theme in my career and I am excited to continue down that path
Steer clear of vanity metrics. We read about these all the time on tech publications like 100M registered users, 50M active users. So stop here and ask what does an “active” user really mean? I can give you an example from my time at flipboard. Flipboard is an ad supported business that prides itself on content personalization. It curates stories based on your interests from across news sources, social feeds and other leading publications. So you can imagine we want users to be consuming a lot of content. DAU was a common growth metric at that time. Definition of active meant a user that launches the app. So there are issues with counting app launch as a growth metric. For starters We were pre-installed on Samsung and sometime users did not even know they were launching the app or reading flipboard when they were actually using Samsung Everytime we send push notifications we see spike in active user activity as users launch the app as noticed in this blue line where the peaks co-incide with push notifications. Now for some business app launch maybe a fine metric if they can monetize those push notifications.
However in our case we refined our metric to truly understand when does a user derive value? So we defined daily active reader. We monetize users once they hit past 4 page flips..so we started tracking daily active reader as a growth metric. You can see that is a lot more stable and does not fluctuate as much as the blue line which is impacted by push notifications. So I highly encourage you to think how you define an active user.
So now that we have setup our growth dashboard, figured out kpi’s we want to go deeper into understanding what makes our product sticky. What have our power users figure out that makes them keep coming back? We wanted to be the interest graph just like facebook is your friends graph. Well to find out make an inventory of all your product features and then see what % of your occasional users, moderate users and heavy users use those features.. This analysis helped us uncover a few interesting usage patterns from our heaviest users. For example what you see on the left was our original onboarding flow for flipboard where we gave you a bunch of pre-selected tiles and then users could always customize ..add more /remove. We then summarized all stories across your tiles into coverstories to create a mix. However we noticed that our most engaged users had no more than 3 titles..national geographic and some very cool sports journals were amongst the top picks by our most engaged users. So what we learnt was incorporated into the new onboarding flow. We ask you to select no more than 3 topics…so that the summarized view is a lot cleaner and does a much better job at truly being your interest graph.
As we continue to talk about digging deeper I want to talk about the usual tools at our disposal. It starts with forming hypothesis for why a certain business problem maybe happening. Like lower conversion from sign up to AVOD conversion. Then exploring that hypothesis using data exploration or A/B tests. This is good at telling u the what. Folks sign-up on living room convert to better AVOD users than those who signup on web. However notice it does not tell you the why. Well so we designed quick UX research survey sending questions to a bunch of customers that signed up on the web but did not convert. Why? One big insight was that a lot of signups happen as users come to redeem digital movie codes. However they just redeem these codes and log out because we only talk about our TVOD titles here. No mention of AVOD titles. Similarly thru data exploration we knew that only few folks that signup on Web actually hook up Vudu on their TV. Why? Survey revealed that users were unaware that we were available on living room. So we changed our copy to move our TV messaging to the top.
Diving Deeper Into Growth Metrics by Priya Singhee (Vudu at Walmart)
Diving deeper into growth KPI'S 1
Head, Marketing analytics @Smule2
Growth Analytics @ Flipboard & GofundMe
Director, Customer insights & Data Science @Vudu1Current
Customer Insights @ VMware4
Growth Analytics @ Flipboard & GofundMe3
Please swing by for some Vudu treats at the end of the talk
Build a growth model
Define, Refine KPI’S
Steer clear of vanity
Know your user lifecycle
Digging for Gold
Do UX research
Identify what sticks
Beware of empty calories
By Mktg channel, Geography, Device Platform
Anomaly detection tool
Diving deeper into growth KPI'S 4
Total Mins Watched
# of Users
Stream free movie
Mins watched /User
Identify true ”North Star” metric that explains core value of your business
Unpack the variables that drive it
Build a multi-variate regression model that explains how delta change in one metric impacts the overall outcome
New user re-activation
Day X retention
Steer clear of vanity metrics
Diving deeper into growth KPI'S 5
What does an ”active user” imply?
DAU: A user that launches the app
DAR: A user that reads past 4 pages because that is when they see the first ad opportunity
DAU vs DAR
Know your user lifecycle: Why does it matter?
Diving deeper into growth KPI'S 6
Visit Signup Convert
2 3 4
1 2 3 4 5 6 7 8 9 10 11 12 13
% of conversion that happen on Day X
fast conversion slow conversion
Identify the few features/actions that make your product sticky
Diving deeper into growth KPI'S 7
If you could have your user take only one action what would that be?
UX research is useful to understand “Why”
Diving deeper into growth KPI'S 8
Data exploration and A/B testing can explain “What” is happening..
To understand the “Why” go to UX research
If possible tie back research results to “actual” user activity (transactions, streams etc)
Beware of empty calories
Acquisition & Conversion (Traffic, New Accounts)
Re-engagement (Active, Dormant, Lapsed customers)
Monetary ( Pre/Post :ARPU, Margin, CLTV)✓
It’s critical to evaluate long term benefits of your partnership/promotional programs
Clean reliable tracking OR instrumentation is key to this.