Successfully reported this slideshow.

Lean Analytics workshop (from Lean Startup Conf)



Loading in …3
1 of 166
1 of 166

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Lean Analytics workshop (from Lean Startup Conf)

  1. 1. Lean Analytics Use data to build a better business faster. @byosko | @acroll @leananalytics
  2. 2. Who you are, who we are, what we’ll cover
  3. 3. Case studies to explore E- 2-sided Mobile User-gen SaaS Media commerce market app content Empathy Stickiness Virality Revenue Scale Plus...
  4. 4. All attendees Hashtag 12 clicks 28 responses 3 responses
  5. 5. You’re all liars
  6. 6. What you asked for
  7. 7. Most startups don’t know what they’ll be when they grow up. Freshbooks Mitel was invoicing Wikipedia was a Paypal for a web was to be lawnmower first built for company design firm written by Palmpilots experts only Flickr Hotmail Twitter Autodesk was going to was a was a made desktop be an MMO database podcasting automation company company
  8. 8. Kevin Costner is a lousy entrepreneur. Don’t sell what you can make. Make what you can sell.
  9. 9. Analytics keeps you honest.
  10. 10. Analytics is the measurement of movement towards your business goals.
  12. 12. In a startup, the purpose of analytics is to iterate to a product/market fit before the money runs out.
  13. 13. Good metrics, bad metrics
  14. 14. What makes a good metric? • It’s comparable to another time period, group, competitor, etc. • It’s understandable in a way the target audience will understand • It’s a ratio or rate • Which means it’s easier to act on (acquisition cost per customer) • It allows you to represent the tension between two things (ads shown versus bounce rate, for example) • It’s targeted to the right audience (Internal business, developers, marketers, investors, media) • It changes the way you behave • “Accounting” metrics make your predictions more accurate • “Experimental” metrics make your future behavior more effective
  15. 15. Think about a car • You know 60MPH is twice as fast as 30MPH • In a country, speed limits and mileage are well understood • Kilometers are conveniently decimal; miles map to hours • Miles travelled is good; miles per hour is better; accelerating or decelerating changes your gas pedal • You can measure “MPH divided by speeding tickets” as a metric of “driving fast without losing my license”
  16. 16. Vanity metrics A metric from the early, foolish days of the Web. If you have a site with many objects on Hits it, this will be a big number. Count people instead. Only slightly better than hits, since it counts the number of times someone requests a Page views page. Unless you’re displaying ad inventory you should count people instead. Is this one person visiting a hundred times, or are a hundred people visiting once? Fail. Visits The only thing this shows you is how many people saw your home page. It tells you Unique visitors nothing about what they did, why they stuck around, or if they left. Counting followers rather than actions is a bad idea. Once you know how many Followers/friends/likes followers will do your bidding when asked, you’ve got something. Time on site, or pages Poor substitute for actual engagement or activity. If customers spend a lot of time on per visit your support or complaints pages, that could be a bad thing. Until you know how many will open your mails (and act on what’s inside them) this isn’t Emails collected useful. Test some of them and see. While it sometimes affects your place in app stores and rankings, downloads alone Number of downloads don’t lead to lifetime value. Measure activations, account creations, or something else.
  17. 17. 5 essential dimensions in analytics Qualitative Quantitative Unstructured, anecdotal, revealing, hard to Numbers and stats; hard facts but less insight. aggregate. Vanity Actionable Make you feel good, but don’t change how you’ll Change your behavior by helping you pick a act. course of action. Exploratory Reporting Speculative, trying to find unexpected or interesting Predictable, keeping you abreast of normal, insights. managerial operations. Leading Lagging Number today that shows metric tomorrow— Historical metric that shows how you’re doing— makes the news reports the news Correlated Causal Two variables that change in similar ways , perhaps An independent factor that directly impacts a because they’re linked to something else dependent one
  18. 18. Donald Rumsfeld on analytics Are facts which may be wrong and we know should be checked against data. know we don’t Are questions we can answer by reporting, which we should baseline know & automate. Things we Are intuition which we should we know quantify and teach to improve don’t effectiveness, efficiency. know we don’t Are exploration which is where unfair advantage and interesting know epiphanies live. (Or rather, Avinash Kaushik channeling Rumsfeld)
  19. 19. Segments, cohorts, A/B, and multivariates Cohort: Comparison of similar groups along a timeline. Segment: A/B test: ☀ Multivariate Cross-sectional ☀ Changing one analysis comparison of all thing (i.e. color) ☁ Changing several people divided by and measuring ☀ things at once to some attribute ☁ the result (i.e. see which correlates ☁ (age, gender, etc.) revenue.) with a result.
  20. 20. Why use cohorts? Here’s an example. Averages   January February March April May hide the pattern Rev/customer CA$5.00 CA$4.50 CA$4.33 CA$4.25 CA$4.50 Cohort 1 2 3 4 5 Cohorts show the January CA$5 CA$3 CA$2 CA$1 CA$0.5 revenue drop as February CA$6 CA$4 CA$2 CA$1   customers age March CA$7 CA$6 CA$5     April CA$8 CA$7       May CA$9         Averages CA$7 CA$5 CA$3 CA$1 CA$0.5
  21. 21. A few words on causality
  22. 22. 50 37.5 25 12.5 0 1 2 3 4 5 6 7 8 9 10 Seat rentals
  23. 23.
  24. 24.
  25. 25. 10000 1000 100 10 1 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Ice cream consumption Drownings
  26. 26.
  27. 27.
  28. 28.
  29. 29. 10000 1000 100 10 1 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Ice cream consumption Drownings Temperature
  30. 30.
  31. 31.
  32. 32.
  33. 33. Mostly, you’re easily distracted Startup founder
  34. 34. One metric to rule them all After all this, you should focus on one metric.
  35. 35. The right information in the right place just changes your life. Stewart Brand
  36. 36. The right information in the right place at the right time just changes your life. Stewart Brand
  37. 37. Two dimensions show you the One Metric That Matters E- 2-sided Mobile User-gen SaaS Media commerce market app content Business model Empathy Stickiness Stage Virality Revenue Scale
  38. 38. An example you might not have expected. • Stage: Revenue • Model: Retailer • Solare is an Italian fine-dining restaurant under new management. The new team is trying to identify the key metrics and leading indicators
  39. 39. Solare watches the numbers • One Metric That Matters: Gross Revenue to Labor Cost • Under 30% is good • Below 24% is great • Lower than 20% and you may be under-staffing, leading to dissatisfied customers • A leading indicator to optimize: Total covers is 5x reservations at 5PM • If you have 50 reservations at 5, you’ll have 250 covers that night. • This ratio varies by restaurant. • If the reservations cause the covers, then focusing on more reservations will grow the business. How would they test this?
  40. 40. The 5 stages of Lean Analytics
  41. 41. Dave McClure’s Pirate metrics How do your users become aware of you? Acquisition AARRR SEO, SEM, widgets, email, PR, campaigns, blogs ... Do drive-by visitors subscribe, use, etc? Activation Features, design, tone, compensation, affirmation ... Does a one-time user become engaged? Retention Notifications, alerts, reminders, emails, updates... Do you make money from user activity? Revenue Transactions, clicks, subscriptions, DLC, analytics... Do users promote your product? Referral Email, widgets, campaigns, likes, RTs, affiliates...
  42. 42. Eric Ries’ Three engines Stickiness Virality Price Approach Keep people Make people Spend revenue coming back invite friends getting customers Math that Get customers How many they Customers are matters faster than you tell, how fast worth more than lose them they tell them they cost to get
  43. 43. Long Funnel • Inject signal at the top • Measure results at the bottom • Includes multi-channel interactions
  44. 44. Social visitors flow report
  45. 45. Multi-channel funnels overview
  46. 46. Ash Maurya’s Lean canvas Lean Canvas box Some relevant metrics Respondents who have this need; respondents who are aware of having Problem the need. Respondents who try the MVP; engagement; churn; most-used/least- Solution used features; people willing to pay. Feedback scores; independent ratings; sentiment analysis; customer- Unique Value Proposition worded descriptions; surveys; search and competitive analysis. How easy it is to find groups of prospects; unique keyword segments; Customer Segments targeted funnel traffic from a particular source. Leads/cust per channel; viral coefficient and cycle; net promoter score; Channels affiliate margins; open, click-through rate; PageRank; message reach. Respondents’ understanding of the USP; patents; brand equity; barriers Unfair Advantage to entry; number of new entrants; exclusivity of relationships. Lifetime customer value; Average revenue per user; Conversion rate; Revenue Streams Shopping cart size; Click-through rate. Fixed costs; cost of customer acquisition; cost of servicing the nth Cost Structure customer; support costs; keyword costs.
  47. 47. Sean Ellis’ Startup growth pyramid Step on the gas in Scale new markets, growth products, channels. Find a defensible Stack the odds unfair advantage and tweak it. Decide what you Product/market fit sell to whom, then prove it.
  48. 48. McClure! Sean Ellis Maurya Lean Lean Lean Analytics Pirate Growth Canvas! Startup! Metrics! Pyramid! phase! Problem Acquisition! Problem validation (of testers, prospects, Empathy Customer Solution etc.) segments validation Product/ market fit Unique value MVP building proposition Activation MVP Stickiness Startup lifecycle stage! Solution iteration, Retention sticky engine (Natural) channels Organic Stacking the growth, viral Referral Virality Growth rate engine odds Revenue stream Monetization, price engine Revenue Revenue Cost structure Scale (Formal) growth channels Inorganic Attention! growth, (at scale, of Scale Unfair beyond Lean customers) advantage
  49. 49. Pause #1 What stage are you at? What model are you in? If you could have only one number what would it be?
  50. 50. What business are you in?
  51. 51. Business model flipbook Revenue model: How you take money from someone Product type: What you give them in return Together, these Delivery model: How you get it to them make up a Acquisition channel: How they learn about you business model Selling tactic: How you convince them to buy
  52. 52. Paid advertising Banner on Search Engine Mgmt. High pagerank for ELC in kid’s toys Acquisition Social media outreach Active on Twitter i.e. Kissmetrics channel How the visitor, Inherent virality Inviting team member to Asana customer, or user finds out about the startup. Artificial virality Rewarding Dropbox user for others’ signups Affiliate marketing Sharing a % of sales with a referring blogger Public relations Speaker submission to SXSW App/ecosystem mkt. Placement in the Android market Simple purchase Buying a PC on What the startup does Discounts & incentives Black Friday discount, loss leader, free ship Selling tactic to convince the visitor Free trial Time-limited trial such as fitbit Premium or user to become a Freemium Free tier, relying on upgrades, like Evernote paying customer. Pay for privacy Free account content is public, like Slideshare Free-to-play Monetize in-app purchases, like Airmech One-time transaction Single purchase from Fab How the startup Recurring subscription Monthly charge from Freshbooks Revenue model extracts money from its Consumption charges Compute cycles from Rackspace visitors, users, or Advertising clicks PPC revenue on customers. Re-sale of user data Twitter’s firehose license Donation Wikipedia’s annual campaign Software Oracle’s accounting suite What the startup does Platform Amazon’s EC2 cloud in return. May be a Product Merchandising Thinkgeek’s retail store type product or service; may be hardware or User-generated content Facebook’s status update software; may be a Marketplace AirBnB’s list of house rentals mixture. Media/content CNN’s news page Service A hairstylist Delivery Hosted service’s CRM model How the product gets Digital delivery Valve purchase of desktop game to the customer. Physical delivery Knife shipped from Sur La Table
  53. 53. Business Flipbook Dropbox example aspect page(s) Acquisition Inherent virality. Sharing files with others. channel Artificial virality. Free storage when others sign up. Selling Limited-capacity accounts are free; Freemium. tactic subscribe when you need more. Revenue Recurring $99/year, monthly fees, enterprise model subscription. tiers. Product Storage-as-a-service with APIs, Platform. type collaboration, synchronization tools. Delivery Hosted service. Cloud storage, web interface. model Digital delivery. Desktop client software.
  54. 54. E-commerce TL;DR: • Are you focused on loyalty or acquisition? • Pricing matters more than you think • Don’t overlook logistics, delays, and ratings • Old “average conversion rates”
  55. 55. Loyalty or acquisition? How many of your customers Then you are in Your customers You are just Focus on buy a second this mode will buy from you like time in 90 days? Low CAC, 1-15% Acquisition Once 70% high of retailers checkout 15-30% Hybrid 2-2.5 20% Increasing per year of retailers returns Loyalty, >30% Loyalty >2.5 10% inventory per year of retailers expansion (Thanks to Kevin Hilstrom for this.)
  56. 56. Pricing has a huge impact
  57. 57. Don’t forget the real world Shipping time, stock availability, logistics, ratings, and other factors have a real impact on most e- commerce companies.
  58. 58. E-commerce model: WineExpress increases revenues • Stage: Revenue • Model: E-commerce • Exclusive wine shop partner of the Wine Enthusiast catalog and website. • “Wine of the day” page is highly trafficked, needed optimization
  59. 59. WineExpress: before and after • Tested 3 design variations, primarily focused on layout • One version was a clear winner • Conversion went up for a heavily discounted shipping promotion • Real key was a 41% increase in revenue per visitor
  60. 60. Software-as-a-Service TL;DR: • Eventually, focus on Customer Acquisition Payback • Engagement varies by intended use of the app (i.e. CRM versus travel booking) • Credit cards up front have a huge effect • Freemium is a sales tactic, not a business model • Churn, acquisition cost, and lifetime value • Subscriptions may be a bad thing (per-transaction pricing is an option too.)
  61. 61. Some sample churn calculations • Active users have logged in at least once in the month after signing up • New users are growing at 20% a month • 30% use the service at least once (in the month after signing up) • 2% convert into paid customers • Example churn calculation for February • Lost / Starting with (Paying Users) * 100 • 26 / 1035 * 100 = 2.5% • If 2.5% of customers churn every month, it means that the average customer stays around for 40 months (100 / 2.5). • This is how you can start to calculate the Lifetime Value of a customer (40 months * Cost of the Service.)
  62. 62. SaaS model: Backupify’s customer lifecycle • Stage: Scale • Model: SaaS • Leading backup provider for cloud based data. • The company was founded in 2008 by Robert May and Vik Chadha • Has gone on to raise $19.5M in several rounds of financing.
  63. 63. Shifting to Customer Acquisition Payback as a key metric • Initially focused on site visitors • Then focused on trials • Then switched to signups • Today, MRR • In early 2010, CAC was $243 and ARPU was only $39 • Pivoted to target business users • CLV-to-CAC today is 5-6x • Now they track Customer Acquisition Payback • Target is less than 12 months
  64. 64. Mobile TL;DR: • App stores make or break you • In-app revenue is the only way to make a living
  65. 65. Not all churn is equal “Track churn at 1 day, 1 week and 1 month, because users leave at different times for different reasons. After one day it could be you have a lousy tutorial or just aren’t hooking users. After a week it could be that your game isn’t ‘deep enough,’ and after a month it could be poor update planning.” Keith Katz, co-founder of Execution Labs and former VP of Monetization for OpenFeint (Knowing when users churn gives you an indication of why they’re churning and what you can try in order to keep them longer.)
  66. 66. Mobile app model: Localmind hacks Twitter • Stage: Empathy • Model: UGC/mobile • Real-time question and answer platform tied to locations. • Needed to find out if a core behavior—answering questions about a place— happened enough to make the business real
  67. 67. Localmind hacks Twitter • Before writing a line of code, Localmind was concerned that people would never answer questions. • This was their biggest risk: if questions went unanswered users would have a terrible experience and stop using Localmind. • Ran an experiment on Twitter • Tracked geolocated tweets in Times Square • Sent @ messages to people who had just tweeted, asking questions about the area: how busy is it; is the subway running on time; is something open; etc. • The response rate to their tweeted questions was very high. • Good enough proxy to de-risk the solution, and convince the team and investors that it was worth building Localmind.
  68. 68. Media TL;DR: • Advertising is a complicated, many-faced beast
  69. 69. Paywalls and baselines • Paywalls might work if done right. Jury’s still out. • A June, 2012 study by the Advertising Research Foundation conducted across a half-million ad impressions showed that blank ads bearing no information had a click-through rate of roughly 0.08%—comparable to that of some paid campaigns.
  70. 70. Media model: Just For Laughs and Youtube • Stage: Revenue • Model: Media • Gags program launched in 2000 shows visual pranks with no words, licensed widely through traditional TV channels. Recently, the organization learned about its popularity on YouTube and decided to invest time in a channel.
  71. 71. Just for Laughs Gags • Had been a YouTube partner since 2009, but intended to build its own site • Noticed some unauthorized activity online, so chose to bulk upload 2,000 2- minute prank clips. • Experimented with overlay, pre- and post-roll ads. Short format meant intro video clips drove people away. • 10-15 second intro resulted in a 30% drop-off. • Tested longer versus short-form videos • In 24h, both long- and short-form clips got 30-40K views. • Long-form clip had 5x that of a short clip—but a long clip had 12 short ones, so overall the long one pays less. • Long-form video has a longer viewing tail. • Long-form has intro, so there’s a 40% audience drop-off halfway into an episode, versus a 15% drop-off halfway into a single short video.
  72. 72. Just for Laughs Gags • Big growth in UGC since the focus on YouTube • Could have done a DCMA takedown but decided to monetize instead • Less lucrative than on-channel content, but volume makes up for it • 100,000 user-generated videos generate 40-50% of total monthly views • 2h mash-ups can get millions of views • Today, JFL tries to learn from and emulate what users show them is working
  73. 73. User-generated content TL;DR: • Content virality and user virality • Trying to move users up the engagement funnel • More time on fraud than you expect • Notifications and email are the real user interface • Passive content creation is on the horizon
  74. 74. UGC engagement funnels
  75. 75. Notifications are the real UI “… notifications become the primary way I use the phone and the apps. I rarely open twitter directly. I see that I have ‘10 new @mentions’ and I click on the notification and go to twitter @mention tab. I see that I have ‘20 new checkins’ and I click on the notification and go to the foursquare friends tab…” - Fred Wilson • Allows use of many more engagement apps on my phone without them being on the main page • Have as many communications apps as I want. Don’t need to use the apps, just the notification inbox • Notification inbox is the new home screen
  76. 76. UGC model: Reddit goes from links to community • Stage: Virality • Model: UGC • A graduate of the first YCombinator class, reddit was acquired by Conde Nast but left largely to its own devices. Thanks to a vibrant community and some good guidance by its founders, it’s a traffic powerhouse.
  77. 77. Reddit goes from links to community • Product evolution • Started as a simple link-sharing site with voting • Then added the ability to comment, with votes on comments • Then created the ability to make “self-posts” rather than only comment on off- site traffic • Now self-posts are more than half of all posts • Revenue from ads and “reddit gold” • Started as a joke, but turned into a revenue source • One person paid $1000 for a month; some paid $0.01. Avg. around $4. • Paying users get early access to features, since they’re an engaged beta • Spam is a big issue • Human flagging isn’t good enough • 50% of the company’s development time is devoted to beating spammers • Having a 7y “training set” helps them develop and test better algorithms
  78. 78. 2-sided market TL;DR: • Focus on the money side • Breaking the chicken-and-egg
  79. 79. Plenty of examples • Real estate listing services • Crowdfunders like Indiegogo and Kickstarter • Charities like Donors Choose • Seller markets like eBay, Craigslist, and Etsy • App stores • Dating sites • Excess inventory travel like Hotwire and Priceline • They all: • Include a shared inventory model • Have two stakeholders—buyers and sellers; creators and supporters; prospective partners; or hotels and travellers • Make money when the two stakeholders come together • Differentiate based on a particular set of search parameters or qualifications (apartments that have been vetted; seller ratings.) • Need an inventory to get started
  80. 80. Chickens and eggs Seed the buyer side: demand Seed the seller side: artificial creation inventory • When Uber launched in Seattle, it • Amazon started selling books; then paid drivers $30 an hour to drive broadened its offering; then launched passengers around a marketplace for goods from many • Only switched to a commission other suppliers. model once they had enough • They created inventory; then demand to make it worthwhile for the demand; and then a two-sided drivers. marketplace. • Knew when to switch by: • Knew when to switch by: Measuring how much drivers would Measuring loyal buyers; identifying be making on a commission basis, unmet search needs and pent-up as well as the inventory and the time demand. it took a driver to pick up a customer.
  81. 81. 2-sided market model: AirBnB and photography • Stage: Revenue • Model: 2-sided marketplace • Rental-by-owner marketplace that allows property owners to list and market their houses. Offers a variety of related services as well.
  82. 82. AirBnB tests a hypothesis • The hypothesis: “Hosts with professional photography will get more business. And hosts will sign up for professional photography as a service.” • Built a concierge MVP • Found that professionally photographed listings got 2-3x more bookings than the market average. • In mid-to-late 2011, AirBnB had 20 photographers in the field taking pictures for hosts.
  83. 83. NIGHTS BOOKED 10 million 8 million 6 million 20 photographers 4 million 2 million 2008 2009 2010 2011 2012
  84. 84. Pause #2 Which models would you like to see more about?
  85. 85. What stage are you at?
  86. 86. Lifecycle stage Discover a known need within a sizeable market Identify a solution reachable people will pay for Develop and validate a viable product to sell profitably Create a sustainable business model Attract and appease investors Find a successful exit
  87. 87. Where is the risk? Real need? Key: Empathy Right solution? Key: Stickiness Key: Growth rate Good product? Key: Virality Sustainable biz? Key: Revenue Healthy market? Successful exit? Key: Scale
  88. 88. Lean Analytics “Gate” needed! Key metrics for this stage! Rationale! stage! to move forward! I’ve found a real, poorly-met Qualitative responses need a reachable market Identifying a real problem and a real solution is Meetings held Empathy faces. The cheapest thing to do (since it’s just the Scored qualitative price of coffee.) It also addresses the riskiest I’ve figured out how to solve Surveys, bulk feedback question—will anyone care? It comes first. the problem in a way they will adopt and pay for. Signup rates Will the dogs eat the dog food? Make your Activation on invite/beta Stickiness mistakes with a small, friendly audience you MVP adoption can love and nurture before throwing the I’ve built the right product/ features/functionality that Retention, churn unwashed masses at it. keeps users around. Message open rates Inherent sharing Sharing helps grow, but also verifies that what Virality you’ve made is good. Word of mouth is Growth rate Viral coefficient/cycle time endorsement. And virality is a “force The users and features fuel growth organically and Word of Mouth sharing multiplier” for paid customer acquisition. artificially. Key goal conversion rates Will people open their pocketbooks? And can Customer lifetime value you charge them enough to fund your Revenue I’ve found a sustainable, Customer acquisition cost ongoing operations, plus your artificial scalable business with the Margins acquisition of users? right margins in a healthy ecosystem. Access to employees Access to capital You need channels to amortize the cost of Scale Competition sales and distribution. You need an I can achieve a successful ecosystem to cross the “hole in the middle” exit for the right terms. Visibility/SEO/SEM from niche player to big company Channel health Exit!
  89. 89. Empathy You need to get inside your target market’s head. You need to know you’re solving a problem people care about. This is really risky, so getting out of the building makes a lot of sense. Few people would dispute this.
  90. 90. Signs you’ve found a problem worth tackling Good signs Bad signs • They want to pay you right away • They’re distracted • They’re actively trying to (or have • Their shoulders are slumped or tried to) solve the problem in they’re slouching in their chair (poor question body language) • They talk a lot and ask a lot of • They talk a lot but it’s not about the questions demonstrating a passion problem or the issues at-hand for the problem (they’re rambling) • They lean forward and are animated (positive body language)
  91. 91. How to avoid leading the witness Avoid biased wording, preconceptions, Don’t tip your hand or a giveaway appearance. Word your surveys carefully to be neutral. Get them to purchase. Ask them to pay. Demand real introductions. Or ask them “how Make the question real many of your friends would say X” to avoid self-effacement Ask “why” several times. Leave lingering, uncomfortable pauses in the conversation Keep digging and let them fill them. Have a colleague make notes of when they react, or of their body language. Look for other clues
  92. 92. Creating an answers-at-scale campaign • Know why you’re doing a survey in the first place Ask what existing Ask how customers Ask what kind of Test which tagline brands come to try to find a product money people or unique value mind in an industry or service spend on a proposition problem resonates best with customers Market alongside Help you plan Choose the them? Address marketing Shape your pricing winning one, or competitors? campaigns and strategy just take that as Choose partners? choice of media advice
  93. 93. Creating an answers-at-scale campaign • Know why you’re doing a survey in the first place • Design the survey Demographic Quantifiable Qualitative, segmentation answers to your open-ended questions research problem feedback
  94. 94. Creating an answers-at-scale campaign • Know why you’re doing a survey in the first place • Design the survey • Test it (you’ll always have mistakes)
  95. 95. Creating an answers-at-scale campaign • Know why you’re doing a survey in the first place • Design the survey • Test it (you’ll always have mistakes) • Send it out Via your NW To a paid list As an ad campaign Beware of Give the solution or Beware of Audience plea Name the problem respondent bias, unique value spamminess, low misrepresentation open rates of the larger market “Are you a single “Can’t sleep? We’re “Our accounting mom? Take this brief trying to fix that, and software automatically survey and help us want your input.”) finds tax breaks. Help address a big us plan the product challenge.” roadmap.”
  96. 96. Creating an answers-at-scale campaign • Know why you’re doing a survey in the first place • Design the survey • Test it (you’ll always have mistakes) • Send it out Were you able to capture the attention of the market? Did they click on your ads and links? Which ones worked best? • Collect the results Are you on the right track? What decisions can you now make with the data you’ve collected? By • Analyze the data segment! Will people try out your solution/product? How many of your respondents were willing to be contacted? How many agreed to join a forum or a beta? How many asked for access in their open-ended responses?
  97. 97. Empathy stage: LikeBright’s mechanical turk • Stage: Empathy • Model: 2-sided marketplace • Early stage startup in the dating space that joined TechStars Seattle in 2011 • Initially rejected, saying, “We don’t think you understand your customer well enough.”
  98. 98. Talking to a hundred single women in a day • Used Mechanical Turk and Google Voice to speak with 100 single women; paid $2. The interviews lasted typically around 10-15 minutes. • Simple interview script with open-ended questions, since he was digging into the problem validation stage of his startup. • Founder Nick Soman: “I was amazed at the feedback I got. We were able to speak with one hundred single women that met our criteria in four hours on one evening.” • Went back to TechStars and got accepted. • LikeBright’s website is now live with a 50% female user base, and recently raised a round of funding. • “Since that first foray into interviewing customers, I’ve probably spoken with over a thousand people through Mechanical Turk,”
  99. 99. How to score problem interviews Teach the controversy
  100. 100. How to score problem interviews • Did the interviewee successfully rank the problems you presented? • Are they actively trying to solve the problems (or have they done so in the past)? • Were they engaged and focused throughout the interview? • Did they agree to a follow-up meeting or interview (to present your solution)? • Did they offer to refer others to you for interviews? • Did they offer to pay you immediately for the solution?
  101. 101. Stickiness This comes from a good product. This is the same as the Eric Ries’ Stickiness engine of growth. You need to find out if you can build a solution to the problem you’ve discovered. There’s no point promoting something awful if your visitors will bounce right off it in disgust. Companies like Color that attempted to scale prematurely, without having proven stickiness, haven’t fared well.
  102. 102. 1995 Hits 1997 Visits 1999 Visitors 2002 Conversions Who did you add? Where from? Why? 2010 Engagement What did they do? How did it benefit? Who did you lose? Why did they leave?
  103. 103. What time on page can tell you about goals & behaviors • People spend ~1m on a page when they’re engaged with it • If a page has high traffic and low engagement, why are people leaving: • Did they come expecting something else? • Is the layout working? • Is it simply a page that isn’t designed to keep users for long? • Show off your good stuff. If a page has a high engaged time but few visitors, promote it. (Thanks to Chartbeat for the analysis.)
  104. 104. More interesting when broken down by business model (Thanks to Chartbeat for the analysis.)
  105. 105. Days since last visit 1200 25000 Number of users January February 1000 20000 800 15000 600 10000 400 5000 200 0 0 1 2 3 4 5 6 7 8 9 Disengaged Days since last engagement (>10 days)
  106. 106. Watch out for the dumb stuff There’s a large number of ways to break an MVP
  107. 107. How to stay focused on the right stuff
  108. 108. Virality Once you’ve got a product or service that’s sticky, it’s time to use inherent virality— word-of-mouth that’s tied into the use of the product. That way, you’ll test out your acquisition and onboarding processes on visitors who are motivated to try you, because you have an implied endorsement from an existing user. You can also think of virality as a force multiplier for paid promotion—so you want to maximize that multiplier before you start spending money on customer acquisition through inorganic methods like advertising.
  109. 109. 3 kinds of virality • Inherent virality is built into the product, and happens as a function of use. • Artificial virality is forced, and often built into a reward system. • Word of mouth virality is simply conversations generated by satisfied users.
  110. 110. Plagues for fun and profit
  111. 111. Viral coefficient ------------------------------------------------------ Get your free private email at ------------------------------------------------------
  112. 112. Viral coefficient v ≠ 1, pt  = δp0 (1 – vt+1) / (1 – v) + p0
  113. 113. Or simpler x Viral - Churn & > Users coefficient abandonment 1
  114. 114. How to calculate it • First calculate the invitation rate, which is the number of invites sent divided by the number of users you have. • Then calculate the acceptance rate, which is the number of signups or enrollments divided by the number of invites. • Then multiply the two together. • Consider, for example • Your 2,000 customers have sent out 5,000 invitations during their lifetime on your site. • Your invitation rate is 2.5. • For every ten invitations received, one gets clicked. • Your acceptance rate is 0.1. • Multiply the two, and you have your viral coefficient: 0.25. Every customer you add will add an addition 25% of a customer.
  115. 115. Don’t forget cycle time How long until they invite someone? • After 20 days with a cycle time of two days, you will have 20,470 users. • If you half that cycle time to one day, you would have over 20 million users!
  116. 116. Virality stage: Circle of Moms finds an engaged market • Stage: Stickiness • Model: UGC • Launched as Circle of Friends in 2007, it was a way for small groups to interact atop Facebook’s platform; but when engagement wasn’t good enough, the founders decided to dig deeper.
  117. 117. The problem: Not enough engagement • Too few people were actually using the product • Less than 20% of any circles had any activity after their initial creation • A few million monthly uniques from 10M registered users, but no sustained traction
  118. 118. What Circle of Moms found • They found moms were far more engaged • Their messages to one another were on average 50% longer • They were 115% more likely to attach a picture to a post they wrote • They were 110% more likely to engage in a threaded (i.e. deep) conversation • Circle owners’ friends were 50% more likely to engage with the circle • They were 75% more likely to click on Facebook notifications • They were 180% more likely to click on Facebook news feed items • They were 60% more likely to accept invitations to the app • Pivoted to the new market, including a name change • By late 2009, 4.5M users and strong engagement • Sold to Sugar, inc. in early 2012
  119. 119. Revenue You’ll want to monetize things at this point. That doesn’t mean you haven’t already been charging—for many businesses, even the first customer has to pay. It just means that earlier on, you’re less focused on revenue than on growth. You’re giving away free trials, free drinks, or free copies. Now you’re focused on maximizing and optimizing revenue. This phase is closely tied to the Lean Startup’s Price engine of growth.
  120. 120. Sergio Zyman’s many “mores” • If you’re dependent on physical, per-transaction costs (like direct sales, or shipping products to a buyer, or signing up merchants) then more efficiently will figure prominently on either the supply or demand side of your business model. • If you’ve found a high viral coefficient, then more people makes sense, because you’ve got a strong force multiplier added to every dollar you pour into customer acquisition. • If you’ve got a loyal, returning set of customers who buy from you every time, then more often makes sense, and you’re going to emphasize getting them to come back more frequently. • If you’ve got a one-time, big-ticket transaction, then more money will help a lot, because you’ve only got one chance to extract revenue from the customer and need to leave as little money as possible on the table. • If you’re a subscription model, and you’re fighting churn, then upselling customers to higher-capacity packages with broader features to additional subscribers within their organization is your best way of growing existing
  121. 121. What’s a customer worth? BAD FOR YOU GOOD FOR YOU Visitor User Customer Spam/fraud Support/resources Ad impressions Usage data Content virality Invite virality Inherent virality Voting/flagging Victim/critical mass Revenue
  122. 122. Market/product fit Most people’s first instinct when things aren’t going incredibly well is to build more features. Instead, try pivoting into a new market. • Assume the product isn’t the problem, it’s the target customer. • It may be easier to change markets than products.
  123. 123. Scale With revenues coming in, it’s time to move from growing your business to growing your market. You need to acquire more customers from new markets. You can invest in channels and distribution to help grow your user base, since direct interaction with individual customers is less critical—you’re close to product/ market fit and you’re analyzing things quantitatively. This phase is closely tied to the acquisition cost optimization side of the Lean Startup’s Price engine of growth.
  124. 124. The hole in the middle Differentiation Efficiency Apple Costco Here be dragons Your local sustainable gluten-free cupcake shop Niche
  125. 125. Growth hacking Finding your way up and to the right. Human approaches, mathematical approaches
  126. 126. The leading indicator • A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya) • If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt) • A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang) • Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman) • A LinkedIn user getting to X connections in Y days (Elliot Schmukler) • These kinds of criteria also make great segments to analyze. (from the 2012 Growth Hacking conference)
  127. 127. But wait: correlated or causal? Correlation lets you Causality lets you predict the future change the future “I will have 420 “If I can make more engaged users and first-time visitors stay 75 paying customers on for 17 minutes I next month.” will increase sales in 90 days.” Optimize the Find correlation Test causality causal factor
  128. 128. The growth hack • Growth hacking is simply what marketing should have been doing, but it fell in love with Don Draper and opinions along the way • At its most basic: Optimize a factor you think is correlated with growth
  129. 129. An example from Reddit Logged-in users All users Days since last visit Visits Pageviews Pg/visit Visits Pageviews Pg/visit 0 127,797,781 1.925B 15.06 242,650,914 3.478B 14.33 1 5,816,594 87,339,766 15.02 13,021,131 187,992,129 14.44 2 1,997,585 27,970,618 14 4,958,931 69,268,831 13.97 Causality: When we make Correlation: If you look at 13.88 3 955,029 13,257,404 changes that get someone 2,620,037 34,047,741 13 >15 pages today you’ll to look at 20,644,331 15 pages 12.32 they’ll come back tomorrow. 14.23 4 625,976 8,905,483 1,675,476 come back. 5 355,643 4,256,639 11.97 1,206,731 14,162,572 11.74
  130. 130. Pause #3 What stage are you at? What is your gating metric?
  131. 131. Asymptotes What to do when there’s no line in the sand
  132. 132. How to draw the line yourself • Sample company trying to drive enrollment • At first, out of over 1,200 visitors, only 4 (0.3%) sign up. • By the end of the month, the site is converting 8.2% of its 1,462 visitors. • Should the company keep optimizing?
  133. 133. How to draw the line yourself • By plotting a trendline we see that in the current situation the line is around 9%. • To really move the line will require a revolutionary, not evolutionary, change.
  134. 134. Finding your One Metric That Matters
  135. 135. Your OMTM must fit Your basic The stage your business model startup is at (monetization) (lifecycle) • E-Commerce • Empathy • UGC • Stickiness • Media • Virality • SaaS • Revenue • Mobile App • Scale • 2-sided market
  136. 136. What’s your OMTM? E- 2-sided Mobile User-gen SaaS Media commerce market app content Empathy Interviews; qualitative results; quantitative scoring; surveys Loyalty, Inventory, Engagement, Downloads, Content, Traffic, visits, Stickiness conversion listings churn churn, virality spam returns CAC, shares, Inherent WoM, app Invites, Content Virality reactivation SEM, sharing virality, CAC ratings, CAC sharing virality, SEM (Money from transactions) (Money from active users) (Money from ad clicks) Transaction, Transactions, Upselling, CLV, Ads, CPE, affiliate Revenue CLV commission CAC, CLV ARPDAU donations %, eyeballs Affiliates, Other API, magic Spinoffs, Analytics, Syndication, Scale white-label verticals #, mktplace publishers user data licenses
  137. 137. Choose only one metric.
  138. 138. It will soon change.
  139. 139. In a startup, focus is hard to achieve.
  140. 140. Having only one metric addresses this problem.
  141. 141. Metrics are like squeeze toys.
  142. 142. Pause #4
  143. 143. B2B: Selling to the enterprise
  144. 144. Lean Analytics lifecycle for an enterprise-focused startup Stage Do this Fear this Consulting to test ideas and Lock-in, IP Empathy bootstrap the business control, overfitting Standardization and integration; Ability to Stickiness shift from custom to generic integrate; support Word of mouth, references, case Bad vibes; Virality studies exclusivity Growing direct sales, professional Pipeline, revenue Revenue services, support recognition, comp Channels, analysts, ecosystems, Crossing the Scale APIs, vertically targeted products chasm; Gorillas
  145. 145. Intrapreneur
  146. 146. Intrapreneur example: P&G changes the mop instead of the soap • Stage: Empathy • Model: Retail/consumer packaged goods • P&G is constantly looking for better soaps. But innovation was slowing. Frustrated, they hired a design team to help them.
  147. 147. P&G changes the mop instead of the soap • Heavy internal investment in R&D, but limited results • Brought in an outside agency (Continuum) to help • The team watched people as they mopped, recording and iterating their research approach • Watched someone pick up spilled coffee. Rather than mopping, the person swept up with a broom, then wiped with a cloth • Realized the mop, not the liquid, mattered • Studied the makeup of floor dirt; realized much of it is dust • Swiffer is a $500M innovation in a stalled industry
  148. 148. The Lean Analytics lifecycle for an Intrapreneur Stage Do this Fear this Get buy-in Political fallout Beforehand Find problems; don’t test demand. Entitled, aggrieved Empathy Skip the business case, do analytics customers Know your real minimum based on Hidden “must haves”, Stickiness expectations, regulations feature creep Build inherent virality in from the Luddites who don’t Virality start; attention is the new currency understand sharing Consider the ecosystem, channels, Channel conflict, Revenue and established agreements resistance, contracts Hand the baton to others gracefully Hating what happens Scale to your baby
  149. 149. In the end: Ask good questions The OMTM should answer the most important question.
  150. 150. Once, a leader convinced others in the absence of data.
  151. 151. Now, a leader knows what questions to ask.
  152. 152. Ben Yoskovitz @byosko Alistair Croll @acroll