Your SlideShare is downloading. ×
0
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Daily Active Users Don't Matter! | Raj Singh, Tempo AI
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Daily Active Users Don't Matter! | Raj Singh, Tempo AI

1,411

Published on

Raj Singh, CEO, Tempo AI …

Raj Singh, CEO, Tempo AI

A blueprint for collecting & analyzing actionable mobile metrics

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,411
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
32
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • There are 2 kinds of metrics for the blueprints and I’m going to walk you thru that process in this preso
  • Pick the stuff we want to highlight and bold it
  • Transcript

    • 1. Daily Active Users Don’t Matter A Blueprint for Collecting & Analyzing Actionable Mobile Metrics Raj Singh CEO, Tempo AI, Inc. 510-282-4229 (M) raj@tempo.ai @mobileraj
    • 2. About Me – Mobile First Dude • 20+ mobile titles since ‘99 • 7+ apps hit #1 in the App Store/Catalog (Vanity Metric!) • MobiTV, Kodak Mobile, Dell Mobile, Skyfire, Recipes, Tempo • Mobile blogger for O’reilly; guest blogger at numerous
    • 3. 2 Kinds of Metrics Vanity or Actionable? Examples # of webpages indexed by Google % of users uploading a video on YouTube DAU/MAU (Daily Actives / Monthly Actives) Registration vs skip registration CTR (click-thru rate) over time
    • 4. Growth Hacking The process is EXACTLY the same on mobile or any screen for that matter! • Test • Measure • Repeat
    • 5. But Mobile Has Caveats • 3-4 week App Review cycles suck! • No HTML; native code is slow! • Disconnected data across screens; stats from web to App Store to app are hard to connect Prioritize your A/B tests!
    • 6. Start With the Framework AARRR (Dave McClure’s Pirate Framework) • Acquisition • Activation • Retention • Referral • Revenue Other frameworks like Acquisition, Activation, Engagement, Virality are the same.
    • 7. Identify What Type of App You Are • Free / Advertising • 1-time Paid • In-App (Trending) • Subscription (Trending)
    • 8. Why App Type Matters? Retention Time Retention Time Retention Time Retention Time Productivity Smile Curve? Network Effect? Common…Paid App?
    • 9. Case Study: Tempo Smart Calendar App Type: Subscription I care about longer-term retention App Category: Productivity I care about unique acquisition strategies since we are not social Goal for Launch: MVP MVP = Minimum Viable Product Viable = Until 50% of our Alpha group were DAUs
    • 10. Tempo’s Timeline Brainstorming 12/10 8/11 Founded 3/12 First Proto 2/13 Public Launch Seeking Core Product Value 7/12 Optimize Activation & Engagement 6/13 1. Digest Data 2. Peaks & Valleys 3. Acquisition 4. Activation Engagement / Retention AARRR is a great framework but you don’t necessarily need to attack it in that order as demonstrated with our timeline.
    • 11. Seeking Core Product Value • Created an Alpha Group: ~150 users ‒ Initially 50 users; moved to iOS Enterprise acct (~150 was what we could manage) ‒ Friends, advisors, investors (too biased) ‒ Added friends of friends, non-tech types, high school buddies, former colleagues • Test, learn & iterate w/ Alpha Group ‒ Monthly builds for 9 months (until Launch) ‒ Interviewed each member of the Alpha group regularly ‒ Except for identifying peaks and valleys, data wasn’t useful (sample too small) ‒ Interviews were a MUST; lots of inductive learning • Searching for Hooks and Must-Haves ‒ Hook = 3 Days (of active use); initial delight and activation ‒ Must-Have = 3 Weeks (of active use)
    • 12. Seeking Core Product Value (Cont) • Conducted surveys to identify Must-Haves ‒ Started with fill-in-the-blank Qs (eg What do use Tempo for) ‒ Ended with “Choose from the list” and rank 1-5 ‒ Must-Haves = When >50% of people in our survey said they “Can’t live without this feature” Survey #1 1. Why do you use Tempo? [Fill in the Blank] I like how it looks; The auto-dialing feature is great; 100+ responses 2. Why don’t you use Tempo? *Fill in the Blank+ It doesn’t have the date on the icon; I can’t set 2 alerts; 100+ responses Survey #2 • Distill results to just 5-8 things 1. What features you can’t live without (Rate 1-5)
    • 13. Seeking Core Product Value (Cont) • Perform Craigslist UX testing ‒ 50/50 split between target demographic and random ‒ Testing focused on Hooks & Education ‒ Always completed a test with a simple Q&A and a word game (ie What words amongst the following adjectives best describe Tempo for you) Accessible Desirable Gets in the way Patronizing Stressful Appealing Easy to use Hard to use Personal Time-consuming Attractive Efficient High quality Predictable Time-saving Busy Empowering Inconsistent Relevant Too technical Collaborative Exciting Intimidating Reliable Trustworthy Complex Familiar Inviting Rigid Uncontrollable Comprehensive Fast Motivating Simplistic Unconventional Confusing Flexible Not valuable Slow Unpredictable Connected Fresh Organized Smart Usable Consistent Frustrating Overbearing Stimulating Useful Customizable Fun Overwhelming Straight Forward Valuable
    • 14. We Identified Core Product Value? Tempo Alpha Group Users loves Auto-Conference Calls and I’m Late! But… Alpha data can be misleading • Small # of Users = Small Learnings • Big # of Users = Big Learnings • Goal for Launch was to get Big Users (10K) to learn big things • Why 10K? ‒ That’s a big # in the productivity category! ‒ Productivity apps are generally not social ‒ We thought we could achieve it!
    • 15. • Actionable Stats ‒ Funnel drop-off rate by screen? ‒ Time spent on each screen (speed is important) ‒ Download to conversion rate • Alpha group wasn’t good data b/c they all activated! (Don’t let friends test apps) So how did we test activation pre-launch? Performed an out-of-market test in Canada • Used a different application name: Teempo Smart Calendar • Spent $500 (total) to acquire ~2-3K users with mobile ads; we optimized the ad buying as well Why Canada? • Because it was convenient Results • Our Alpha group connected all their data but the Canadians did not? Privacy? • Retention wasn’t very good Still Pre-Launch: Moved Focus to Activation
    • 16. Canada Taught Us 2 Things 1. We need to improve the “don’t connect anything” education in Tempo 2. Users acquired via cheap mobile ads was not the right segment for us; mostly gamers We improved education in Tempo but we didn’t have enough time to run another Canadian test (running out of money) • Focused on blind Craigslist testing and heuristics usability testing • Heuristics Usability = Asking experts for feedback
    • 17. Still Pre-Launch: Moved Focus to ASO (App Store Optimization) • Identified what we could A/B test ‒ App Store description ‒ App Store images ‒ App Title ‒ App Keywords • Created 10s of permutations and tested • Couldn’t measure acquisition to retention (unfortunately)
    • 18. Buying Ads – We Tested A Lot! We even created a fake HTML App Store landing page to test different descriptions Results • Not enough statistical variance between words: “Smart,” “AI,” “Intelligent,” “Magic” etc (not conclusive) • Users preferred icons that looked like calendars • App screenshots with descriptions outperformed screenshots without descriptions Living Cal Calendar Assistant Useful Calendar Smart Cal Smart Calendar Smart Calendar #3 Cal Assistant Intelligent Calendar 0.50 0.50 Actionable Cal 0.60 Smart Calendar #4 Magic Calendar 0.51 0.52 Smart Calendar #1 0.61 0.53 Living Calendar 0.55 0.63 Smart Calendar #2 Learning Cal 0.50 0.40 0.43 0.45 0.46 0.59
    • 19. We’re Ready – Gearing Up For Launch My Pre-Launch Blueprint • Identify Core Product Value (Hooks & Must-Haves: UI, Conference Calls & I’m Late) • Optimize Activation / On-Boarding (Reduced the # of screens and clicks) • Identify Peaks and Valleys from Alpha data (Improved in-app education for the more “private user”) • Optimize Acquisition / Discovery (ASO: A/B test keywords, app title, icon, description) (See slide 10 for the timeline)
    • 20. Launch Day • Exceeded our goal of 10K users • What do we focus on now??? ‒ Look for surprises: Users were 24X more data than our Canada Alpha! ‒ New Peaks & Valleys ‒ Activation, On-Boarding • It takes 3 months before you have enough data to really understand engagement and retention
    • 21. Post-Launch: Studied On-Boarding • 30-40% bounce rate on registration :/ • Possible A/B Tests: ‒ Facebook or LinkedIn registration ‒ Skip • Users confused between Tempo registration and connecting their calendar More tests planned for future (but not the priority at present) 100% 70% 62% 62% 60% 59% Downloads* Account Creation Attempted Account Creation Successful
    • 22. Post-Launch: Studied Initial Bounce Rate • 34% of users left after the 1st day • Data did not answer Day 1 bounce rates, so: ‒ Identified 100 users from support who bounced after 1-3 days ‒ Sent a personal email and spoke to a handful on the phone ‒ Considered a survey but it doesn’t work with dis-engaged users. Surveys yield responses from your fans! Results From Discussions • Email connectivity issues • Sync issues • Privacy • Confusion / Education We can fix these things!
    • 23. Post-Launch: Seeking Peaks & Valleys Examples • The Settings button was the most popular button? (Peak) • FB connect errors as a % of FB actions were really high (Peak) • Adding Event was the most popular action (Peak) • Few customized their Agenda view image (Valley) • Few clicked on Foursquare or Yelp (Valley)
    • 24. Post-Launch: Seeking Peaks & Valleys (Cont) Valleys • Discovery/education or uninteresting feature? • Create simple A/B tests to answer Example A/B Test Hypothesis: Users didn’t customize Agenda View photos because they didn’t know they could Test: Sent an email with tips after Tempo user registration Result: Was no longer a valley. Test succeeded
    • 25. Post-Launch: Seeking Peaks & Valleys (Cont) Peaks • Unexpected or a feature • Create simple A/B tests to answer Example A/B Test Hypothesis: Users clicked Settings because they didn’t know how to change views Test: Move the menu from the top-middle to the top-left Result: Settings was no longer a peak. Test succeeded
    • 26. Post Launch: Seeking Peaks & Valleys (Cont) Not all A/B tests are as cut and dry A/B Test Hypothesis: “Post Happy B-Day” is a popular action Test: Let people “Post Happy B-Day” from the Calendar View Result: No change. Test was not successful? Prioritize your A/B tests! Test, Measure, Repeat
    • 27. 3 Months After Launch: Understanding Retention • Defined an internal goal of 20% after 3 months ‒ Got comps from other apps in the category (eg Evernote etc) ‒ It’s a great # and it’s good to have goals • Tempo: 10-20% Retention ‒ Current priority! And so we started with the questions to answer: • How many days until we see the biggest drop-off in retention? • Why after using Tempo for X weeks, do you leave? • Are certain features in the app more or less used by more or less engaged users? • Who is our user?
    • 28. 3 Months After Launch: Understanding Retention (Cont) The process is exactly the same! • Identify users from support and other channels and contact them! • Create a survey but it doesn’t substitute for speaking! • Looked for peaks & valleys but on a per active user and per session basis over time (actionable stats) • Created A/B tests and executed But one difference: • Look for correlations
    • 29. 3 Months After Launch: Looking For Correlations Correlation = Ratios Ratios ~= Actionable Data Tempo Examples Users with Exchange are more engaged Users who use conference calling are more engaged Users with Facebook events only are less engaged Create hypothesis and measure the data over time
    • 30. My Post-Launch Blueprint • Look for surprises (Fixed settings button click rate) • Study activation workflow in aggregate (Fixed email registration confusion) • Identify, interview & talk to users (Improved education) • Find peaks and valleys (A/B tested Facebook birthday post) • Look for correlations (Lots of Excel!) • A/B test and prioritize your tests
    • 31. What’s Next? (Conversation For Next Year?) • Achieve our 20% 3 month retention goal • Then we focus on acquisition (Virality = Frequency X Reach X Yield) • Then we focus on monetization No matter where you are in AARRR, the process is the same: Test, Measure, Repeat Talk to your users Prioritize your A/B tests

    ×