Lean Analytics workshop (from Lean Startup Conf)


Published on

A comprehensive (but not complete!) review of the Lean Analytics book (http://leananalyticsbook.com), which was presented at the Lean Startup Conference in 2012. Focuses on the

Published in: Technology
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Lean Analytics workshop (from Lean Startup Conf)

  1. 1. Lean Analytics Use data to build a better business faster. www.leananalyticsbook.com @byosko | @acroll @leananalytics
  2. 2. Who you are, who we are, whatwe’ll cover
  3. 3. Case studies to explore E- 2-sided Mobile User-gen SaaS Media commerce market app content EmpathyStickiness Virality Revenue Scale Plus...
  4. 4. All attendees Hashtag 12 clicks28 responses 3 responses
  5. 5. You’re all liars
  6. 6. What you asked for
  7. 7. Most startups don’t know what they’ll bewhen they grow up. Freshbooks Mitel was invoicing Wikipedia was a Paypal for a web was to be lawnmowerfirst built for company design firm written by Palmpilots experts only Flickr Hotmail Twitter Autodesk was going to was a was a made desktop be an MMO database podcasting automation company company
  8. 8. Kevin Costner is a lousy entrepreneur. Don’t sell what you can make. Make what you can sell.
  9. 9. Analytics keeps you honest.
  10. 10. Analytics is the measurementof movement towards yourbusiness goals. http://www.flickr.com/photos/itsgreg/446061432/
  12. 12. In a startup, the purpose ofanalytics is to iterate to aproduct/market fit beforethe money runs out.
  13. 13. Good metrics, bad metrics
  14. 14. What makes a good metric?• It’s comparable to another time period, group, competitor, etc.• It’s understandable in a way the target audience will understand• It’s a ratio or rate • Which means it’s easier to act on (acquisition cost per customer) • It allows you to represent the tension between two things (ads shown versus bounce rate, for example)• It’s targeted to the right audience (Internal business, developers, marketers, investors, media)• It changes the way you behave • “Accounting” metrics make your predictions more accurate • “Experimental” metrics make your future behavior more effective
  15. 15. Think about a car• You know 60MPH is twice as fast as 30MPH• In a country, speed limits and mileage are well understood • Kilometers are conveniently decimal; miles map to hours• Miles travelled is good; miles per hour is better; accelerating or decelerating changes your gas pedal• You can measure “MPH divided by speeding tickets” as a metric of “driving fast without losing my license”
  16. 16. Vanity metrics A metric from the early, foolish days of the Web. If you have a site with many objects on Hits it, this will be a big number. Count people instead. Only slightly better than hits, since it counts the number of times someone requests a Page views page. Unless you’re displaying ad inventory you should count people instead. Is this one person visiting a hundred times, or are a hundred people visiting once? Fail. Visits The only thing this shows you is how many people saw your home page. It tells you Unique visitors nothing about what they did, why they stuck around, or if they left. Counting followers rather than actions is a bad idea. Once you know how manyFollowers/friends/likes followers will do your bidding when asked, you’ve got something.Time on site, or pages Poor substitute for actual engagement or activity. If customers spend a lot of time on per visit your support or complaints pages, that could be a bad thing. Until you know how many will open your mails (and act on what’s inside them) this isn’t Emails collected useful. Test some of them and see. While it sometimes affects your place in app stores and rankings, downloads aloneNumber of downloads don’t lead to lifetime value. Measure activations, account creations, or something else.
  17. 17. 5 essential dimensions in analytics Qualitative QuantitativeUnstructured, anecdotal, revealing, hard to Numbers and stats; hard facts but less insight.aggregate. Vanity ActionableMake you feel good, but don’t change how you’ll Change your behavior by helping you pick aact. course of action. Exploratory ReportingSpeculative, trying to find unexpected or interesting Predictable, keeping you abreast of normal,insights. managerial operations. Leading LaggingNumber today that shows metric tomorrow— Historical metric that shows how you’re doing—makes the news reports the news Correlated CausalTwo variables that change in similar ways , perhaps An independent factor that directly impacts abecause they’re linked to something else dependent one
  18. 18. Donald Rumsfeld on analytics Are facts which may be wrong and we know should be checked against data. know we don’t Are questions we can answer by reporting, which we should baseline know & automate.Things we Are intuition which we should we know quantify and teach to improve don’t effectiveness, efficiency. know we don’t Are exploration which is where unfair advantage and interesting know epiphanies live. (Or rather, Avinash Kaushik channeling Rumsfeld)
  19. 19. Segments, cohorts, A/B, and multivariates Cohort: Comparison of similar groups along a timeline. Segment: A/B test: ☀ Multivariate Cross-sectional ☀ Changing one analysis comparison of all thing (i.e. color) ☁ Changing several people divided by and measuring ☀ things at once to some attribute ☁ the result (i.e. see which correlates ☁(age, gender, etc.) revenue.) with a result.
  20. 20. Why use cohorts? Here’s an example.Averages   January February March April May hide the pattern Rev/customer CA$5.00 CA$4.50 CA$4.33 CA$4.25 CA$4.50 Cohort 1 2 3 4 5 Cohorts show the January CA$5 CA$3 CA$2 CA$1 CA$0.5 revenue drop as February CA$6 CA$4 CA$2 CA$1  customers age March CA$7 CA$6 CA$5     April CA$8 CA$7       May CA$9         Averages CA$7 CA$5 CA$3 CA$1 CA$0.5
  21. 21. A few words on causality http://www.flickr.com/photos/roryfinneren/65729247
  22. 22. 5037.5 2512.5 0 1 2 3 4 5 6 7 8 9 10 Seat rentals http://www.rvca.com/anp/wp-content/plugins/wp-o-matic/cache/57226_07+proof+1a+hb+beach+day.jpg
  23. 23. http://www.imdb.com/media/rm3768753408/tt0073195
  24. 24. http://www.flickr.com/photos/kapungo/2287237966
  25. 25. 100001000 100 10 1 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Ice cream consumption Drownings
  26. 26. http://www.flickr.com/photos/25159787@N07/3766111564
  27. 27. http://www.flickr.com/photos/wheressteve/3284532080
  28. 28. http://www.flickr.com/photos/wtlphotos/1086968783
  29. 29. 100001000 100 10 1 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Ice cream consumption Drownings Temperature
  30. 30. http://www.flickr.com/photos/stuttermonkey/57096884
  31. 31. http://www.flickr.com/photos/germanuncut77/3785152581
  32. 32. http://www.flickr.com/photos/fasteddie42/2421039207
  33. 33. Mostly, you’re easily distracted Startup founder
  34. 34. One metric to rule them allAfter all this, you should focus on one metric.
  35. 35. The right informationin the right placejust changes your life. Stewart Brand
  36. 36. The right informationin the right placeat the right timejust changes your life. Stewart Brand
  37. 37. Two dimensions show youthe One Metric That Matters E- 2-sided Mobile User-gen SaaS Media commerce market app content Business model EmpathyStickiness Stage Virality Revenue Scale
  38. 38. An example you might nothave expected.• Stage: Revenue• Model: Retailer• Solare is an Italian fine-dining restaurant under new management. The new team is trying to identify the key metrics and leading indicators
  39. 39. Solare watches the numbers• One Metric That Matters: Gross Revenue to Labor Cost • Under 30% is good • Below 24% is great • Lower than 20% and you may be under-staffing, leading to dissatisfied customers• A leading indicator to optimize: Total covers is 5x reservations at 5PM • If you have 50 reservations at 5, you’ll have 250 covers that night. • This ratio varies by restaurant.• If the reservations cause the covers, then focusing on more reservations will grow the business. How would they test this?
  40. 40. The 5 stages of Lean Analytics
  41. 41. Dave McClure’s Pirate metrics How do your users become aware of you? Acquisition AARRR SEO, SEM, widgets, email, PR, campaigns, blogs ... Do drive-by visitors subscribe, use, etc? Activation Features, design, tone, compensation, affirmation ... Does a one-time user become engaged? Retention Notifications, alerts, reminders, emails, updates... Do you make money from user activity? Revenue Transactions, clicks, subscriptions, DLC, analytics... Do users promote your product? Referral Email, widgets, campaigns, likes, RTs, affiliates...
  42. 42. Eric Ries’Three engines Stickiness Virality PriceApproach Keep people Make people Spend revenue coming back invite friends getting customersMath that Get customers How many they Customers are matters faster than you tell, how fast worth more than lose them they tell them they cost to get
  43. 43. Long Funnel• Inject signal at the top• Measure results at the bottom• Includes multi-channel interactions
  44. 44. Social visitors flow report
  45. 45. Multi-channel funnels overview
  46. 46. Ash Maurya’sLean canvas Lean Canvas box Some relevant metrics Respondents who have this need; respondents who are aware of having Problem the need. Respondents who try the MVP; engagement; churn; most-used/least- Solution used features; people willing to pay. Feedback scores; independent ratings; sentiment analysis; customer- Unique Value Proposition worded descriptions; surveys; search and competitive analysis. How easy it is to find groups of prospects; unique keyword segments; Customer Segments targeted funnel traffic from a particular source. Leads/cust per channel; viral coefficient and cycle; net promoter score; Channels affiliate margins; open, click-through rate; PageRank; message reach. Respondents’ understanding of the USP; patents; brand equity; barriers Unfair Advantage to entry; number of new entrants; exclusivity of relationships. Lifetime customer value; Average revenue per user; Conversion rate; Revenue Streams Shopping cart size; Click-through rate. Fixed costs; cost of customer acquisition; cost of servicing the nth Cost Structure customer; support costs; keyword costs.
  47. 47. Sean Ellis’Startup growth pyramid Step on the gas in Scale new markets, growth products, channels. Find a defensible Stack the odds unfair advantage and tweak it. Decide what you Product/market fit sell to whom, then prove it.
  48. 48. McClure! Sean Ellis Maurya Lean Lean Lean Analytics Pirate Growth Canvas! Startup! Metrics! Pyramid! phase! Problem Acquisition! Problem validation (of testers, prospects, Empathy Customer Solution etc.) segments validation Product/ market fit Unique value MVP building proposition Activation MVP StickinessStartup lifecycle stage! Solution iteration, Retention sticky engine (Natural) channels Organic Stacking the growth, viral Referral Virality Growth rate engine odds Revenue stream Monetization, price engine Revenue Revenue Cost structure Scale (Formal) growth channels Inorganic Attention! growth, (at scale, of Scale Unfair beyond Lean customers) advantage
  49. 49. Pause #1What stage are you at?What model are you in?If you could have only one number what would it be?
  50. 50. What business are you in?
  51. 51. Business model flipbookRevenue model: How you take money from someoneProduct type: What you give them in return Together, theseDelivery model: How you get it to them make up aAcquisition channel: How they learn about you business modelSelling tactic: How you convince them to buy
  52. 52. Paid advertising Banner on Informationweek.com Search Engine Mgmt. High pagerank for ELC in kid’s toysAcquisition Social media outreach Active on Twitter i.e. Kissmetrics channel How the visitor, Inherent virality Inviting team member to Asana customer, or user finds out about the startup. Artificial virality Rewarding Dropbox user for others’ signups Affiliate marketing Sharing a % of sales with a referring blogger Public relations Speaker submission to SXSW App/ecosystem mkt. Placement in the Android market Simple purchase Buying a PC on Dell.com What the startup does Discounts & incentives Black Friday discount, loss leader, free shipSellingtactic to convince the visitor Free trial Time-limited trial such as fitbit Premium or user to become a Freemium Free tier, relying on upgrades, like Evernote paying customer. Pay for privacy Free account content is public, like Slideshare Free-to-play Monetize in-app purchases, like Airmech One-time transaction Single purchase from Fab How the startup Recurring subscription Monthly charge from FreshbooksRevenue model extracts money from its Consumption charges Compute cycles from Rackspace visitors, users, or Advertising clicks PPC revenue on CNET.com customers. Re-sale of user data Twitter’s firehose license Donation Wikipedia’s annual campaign Software Oracle’s accounting suite What the startup does Platform Amazon’s EC2 cloud in return. May be aProduct Merchandising Thinkgeek’s retail store type product or service; may be hardware or User-generated content Facebook’s status update software; may be a Marketplace AirBnB’s list of house rentals mixture. Media/content CNN’s news page Service A hairstylistDelivery Hosted service Salesforce.com’s CRM model How the product gets Digital delivery Valve purchase of desktop game to the customer. Physical delivery Knife shipped from Sur La Table
  53. 53. Business Flipbook Dropbox example aspect page(s) Acquisition Inherent virality. Sharing files with others. channel Artificial virality. Free storage when others sign up. Selling Limited-capacity accounts are free; Freemium. tactic subscribe when you need more. Revenue Recurring $99/year, monthly fees, enterprise model subscription. tiers. Product Storage-as-a-service with APIs, Platform. type collaboration, synchronization tools. Delivery Hosted service. Cloud storage, web interface. model Digital delivery. Desktop client software.
  54. 54. E-commerceTL;DR:• Are you focused on loyalty or acquisition?• Pricing matters more than you think• Don’t overlook logistics, delays, and ratings• Old “average conversion rates”
  55. 55. Loyalty or acquisition? How many of your customers Then you are in Your customers You are just Focus on buy a second this mode will buy from you liketime in 90 days? Low CAC, 1-15% Acquisition Once 70% high of retailers checkout 15-30% Hybrid 2-2.5 20% Increasing per year of retailers returns Loyalty, >30% Loyalty >2.5 10% inventory per year of retailers expansion (Thanks to Kevin Hilstrom for this.)
  56. 56. Pricing has a huge impact http://hbr.org/1992/09/managing-price-gaining-profit/ar/8
  57. 57. Don’t forget the real world Shipping time, stock availability, logistics, ratings, and other factors have a real impact on most e- commerce companies.
  58. 58. E-commerce model:WineExpress increasesrevenues• Stage: Revenue• Model: E-commerce• Exclusive wine shop partner of the Wine Enthusiast catalog and website.• “Wine of the day” page is highly trafficked, needed optimization
  59. 59. WineExpress: before and after• Tested 3 design variations, primarily focused on layout• One version was a clear winner • Conversion went up for a heavily discounted shipping promotion • Real key was a 41% increase in revenue per visitor
  60. 60. Software-as-a-ServiceTL;DR:• Eventually, focus on Customer Acquisition Payback• Engagement varies by intended use of the app (i.e. CRM versus travel booking)• Credit cards up front have a huge effect• Freemium is a sales tactic, not a business model• Churn, acquisition cost, and lifetime value• Subscriptions may be a bad thing (per-transaction pricing is an option too.)
  61. 61. Some sample churn calculations• Active users have logged in at least once in the month after signing up• New users are growing at 20% a month• 30% use the service at least once (in the month after signing up)• 2% convert into paid customers• Example churn calculation for February • Lost / Starting with (Paying Users) * 100 • 26 / 1035 * 100 = 2.5%• If 2.5% of customers churn every month, it means that the average customer stays around for 40 months (100 / 2.5).• This is how you can start to calculate the Lifetime Value of a customer (40 months * Cost of the Service.)
  62. 62. SaaS model:Backupify’s customerlifecycle• Stage: Scale• Model: SaaS• Leading backup provider for cloud based data.• The company was founded in 2008 by Robert May and Vik Chadha• Has gone on to raise $19.5M in several rounds of financing.
  63. 63. Shifting to Customer AcquisitionPayback as a key metric• Initially focused on site visitors• Then focused on trials• Then switched to signups• Today, MRR• In early 2010, CAC was $243 and ARPU was only $39 • Pivoted to target business users • CLV-to-CAC today is 5-6x• Now they track Customer Acquisition Payback • Target is less than 12 months
  64. 64. MobileTL;DR:• App stores make or break you• In-app revenue is the only way to make a living
  65. 65. Not all churn is equal“Track churn at 1 day, 1 week and 1 month, because usersleave at different times for different reasons. After one day itcould be you have a lousy tutorial or just aren’t hookingusers. After a week it could be that your game isn’t ‘deepenough,’ and after a month it could be poor updateplanning.” Keith Katz, co-founder of Execution Labs and former VP of Monetization for OpenFeint(Knowing when users churn gives you an indication of why they’re churning andwhat you can try in order to keep them longer.)
  66. 66. Mobile app model:Localmind hacks Twitter• Stage: Empathy• Model: UGC/mobile• Real-time question and answer platform tied to locations.• Needed to find out if a core behavior—answering questions about a place— happened enough to make the business real
  67. 67. Localmind hacks Twitter• Before writing a line of code, Localmind was concerned that people would never answer questions. • This was their biggest risk: if questions went unanswered users would have a terrible experience and stop using Localmind.• Ran an experiment on Twitter • Tracked geolocated tweets in Times Square • Sent @ messages to people who had just tweeted, asking questions about the area: how busy is it; is the subway running on time; is something open; etc.• The response rate to their tweeted questions was very high. • Good enough proxy to de-risk the solution, and convince the team and investors that it was worth building Localmind.
  68. 68. MediaTL;DR:• Advertising is a complicated, many-faced beast
  69. 69. Paywalls and baselines• Paywalls might work if done right. Jury’s still out.• A June, 2012 study by the Advertising Research Foundation conducted across a half-million ad impressions showed that blank ads bearing no information had a click-through rate of roughly 0.08%—comparable to that of some paid campaigns.
  70. 70. Media model:Just For Laughs andYoutube• Stage: Revenue• Model: Media• Gags program launched in 2000 shows visual pranks with no words, licensed widely through traditional TV channels. Recently, the organization learned about its popularity on YouTube and decided to invest time in a channel.
  71. 71. Just for Laughs Gags• Had been a YouTube partner since 2009, but intended to build its own site• Noticed some unauthorized activity online, so chose to bulk upload 2,000 2- minute prank clips.• Experimented with overlay, pre- and post-roll ads. Short format meant intro video clips drove people away. • 10-15 second intro resulted in a 30% drop-off.• Tested longer versus short-form videos • In 24h, both long- and short-form clips got 30-40K views. • Long-form clip had 5x that of a short clip—but a long clip had 12 short ones, so overall the long one pays less. • Long-form video has a longer viewing tail. • Long-form has intro, so there’s a 40% audience drop-off halfway into an episode, versus a 15% drop-off halfway into a single short video.
  72. 72. Just for Laughs Gags• Big growth in UGC since the focus on YouTube• Could have done a DCMA takedown but decided to monetize instead • Less lucrative than on-channel content, but volume makes up for it• 100,000 user-generated videos generate 40-50% of total monthly views • 2h mash-ups can get millions of views• Today, JFL tries to learn from and emulate what users show them is working
  73. 73. User-generated contentTL;DR:• Content virality and user virality• Trying to move users up the engagement funnel• More time on fraud than you expect• Notifications and email are the real user interface• Passive content creation is on the horizon
  74. 74. UGC engagement funnels
  75. 75. Notifications are the real UI “… notifications become the primary way I use thephone and the apps. I rarely open twitter directly. I see thatI have ‘10 new @mentions’ and I click on the notificationand go to twitter @mention tab. I see that I have ‘20 newcheckins’ and I click on the notification and go to thefoursquare friends tab…” - Fred Wilson• Allows use of many more engagement apps on my phone without them being on the main page• Have as many communications apps as I want. Don’t need to use the apps, just the notification inbox• Notification inbox is the new home screen
  76. 76. UGC model:Reddit goes from links tocommunity• Stage: Virality• Model: UGC• A graduate of the first YCombinator class, reddit was acquired by Conde Nast but left largely to its own devices. Thanks to a vibrant community and some good guidance by its founders, it’s a traffic powerhouse.
  77. 77. Reddit goes from links to community• Product evolution • Started as a simple link-sharing site with voting • Then added the ability to comment, with votes on comments • Then created the ability to make “self-posts” rather than only comment on off- site traffic • Now self-posts are more than half of all posts• Revenue from ads and “reddit gold” • Started as a joke, but turned into a revenue source • One person paid $1000 for a month; some paid $0.01. Avg. around $4. • Paying users get early access to features, since they’re an engaged beta• Spam is a big issue • Human flagging isn’t good enough • 50% of the company’s development time is devoted to beating spammers • Having a 7y “training set” helps them develop and test better algorithms
  78. 78. 2-sided marketTL;DR:• Focus on the money side• Breaking the chicken-and-egg
  79. 79. Plenty of examples• Real estate listing services• Crowdfunders like Indiegogo and Kickstarter• Charities like Donors Choose• Seller markets like eBay, Craigslist, and Etsy• App stores• Dating sites• Excess inventory travel like Hotwire and Priceline• They all: • Include a shared inventory model • Have two stakeholders—buyers and sellers; creators and supporters; prospective partners; or hotels and travellers • Make money when the two stakeholders come together • Differentiate based on a particular set of search parameters or qualifications (apartments that have been vetted; seller ratings.) • Need an inventory to get started
  80. 80. Chickens and eggsSeed the buyer side: demand Seed the seller side: artificialcreation inventory• When Uber launched in Seattle, it • Amazon started selling books; then paid drivers $30 an hour to drive broadened its offering; then launched passengers around a marketplace for goods from many• Only switched to a commission other suppliers. model once they had enough • They created inventory; then demand to make it worthwhile for the demand; and then a two-sided drivers. marketplace.• Knew when to switch by: • Knew when to switch by: Measuring how much drivers would Measuring loyal buyers; identifying be making on a commission basis, unmet search needs and pent-up as well as the inventory and the time demand. it took a driver to pick up a customer.
  81. 81. 2-sided market model:AirBnB and photography• Stage: Revenue• Model: 2-sided marketplace• Rental-by-owner marketplace that allows property owners to list and market their houses. Offers a variety of related services as well.
  82. 82. AirBnB tests a hypothesis• The hypothesis: “Hosts with professional photography will get more business. And hosts will sign up for professional photography as a service.”• Built a concierge MVP• Found that professionally photographed listings got 2-3x more bookings than the market average.• In mid-to-late 2011, AirBnB had 20 photographers in the field taking pictures for hosts.
  83. 83. NIGHTS BOOKED10 million 8 million 6 million 20 photographers 4 million 2 million 2008 2009 2010 2011 2012
  84. 84. Pause #2Which models would you like to see more about?
  85. 85. What stage are you at?
  86. 86. Lifecycle stage Discover a known need within a sizeable market Identify a solution reachable people will pay for Develop and validate a viable product to sell profitably Create a sustainable business model Attract and appease investors Find a successful exit
  87. 87. Where is the risk? Real need? Key: Empathy Right solution? Key: Stickiness Key: Growth rate Good product? Key: Virality Sustainable biz? Key: Revenue Healthy market? Successful exit? Key: Scale
  88. 88. Lean Analytics “Gate” needed! Key metrics for this stage! Rationale! stage! to move forward! I’ve found a real, poorly-met Qualitative responses need a reachable market Identifying a real problem and a real solution is Meetings held Empathy faces. The cheapest thing to do (since it’s just the Scored qualitative price of coffee.) It also addresses the riskiest I’ve figured out how to solve Surveys, bulk feedback question—will anyone care? It comes first. the problem in a way they will adopt and pay for. Signup rates Will the dogs eat the dog food? Make your Activation on invite/beta Stickiness mistakes with a small, friendly audience you MVP adoption can love and nurture before throwing the I’ve built the right product/ features/functionality that Retention, churn unwashed masses at it. keeps users around. Message open rates Inherent sharing Sharing helps grow, but also verifies that what Virality you’ve made is good. Word of mouth isGrowth rate Viral coefficient/cycle time endorsement. And virality is a “force The users and features fuel growth organically and Word of Mouth sharing multiplier” for paid customer acquisition. artificially. Key goal conversion rates Will people open their pocketbooks? And can Customer lifetime value you charge them enough to fund your Revenue I’ve found a sustainable, Customer acquisition cost ongoing operations, plus your artificial scalable business with the Margins acquisition of users? right margins in a healthy ecosystem. Access to employees Access to capital You need channels to amortize the cost of Scale Competition sales and distribution. You need an I can achieve a successful ecosystem to cross the “hole in the middle” exit for the right terms. Visibility/SEO/SEM from niche player to big company Channel health Exit!
  89. 89. EmpathyYou need to get inside your target market’s head. You need to know you’re solvinga problem people care about. This is really risky, so getting out of the buildingmakes a lot of sense. Few people would dispute this.
  90. 90. Signs you’ve found a problem worth tackling Good signs Bad signs• They want to pay you right away • They’re distracted• They’re actively trying to (or have • Their shoulders are slumped or tried to) solve the problem in they’re slouching in their chair (poor question body language)• They talk a lot and ask a lot of • They talk a lot but it’s not about the questions demonstrating a passion problem or the issues at-hand for the problem (they’re rambling)• They lean forward and are animated (positive body language)
  91. 91. How to avoid leading the witness Avoid biased wording, preconceptions, Don’t tip your hand or a giveaway appearance. Word your surveys carefully to be neutral. Get them to purchase. Ask them to pay. Demand real introductions. Or ask them “howMake the question real many of your friends would say X” to avoid self-effacement Ask “why” several times. Leave lingering, uncomfortable pauses in the conversation Keep digging and let them fill them. Have a colleague make notes of when they react, or of their body language. Look for other clues
  92. 92. Creating an answers-at-scale campaign• Know why you’re doing a survey in the first place Ask what existing Ask how customers Ask what kind of Test which tagline brands come to try to find a product money people or unique value mind in an industry or service spend on a proposition problem resonates best with customers Market alongside Help you plan Choose the them? Address marketing Shape your pricing winning one, or competitors? campaigns and strategy just take that as Choose partners? choice of media advice
  93. 93. Creating an answers-at-scale campaign• Know why you’re doing a survey in the first place• Design the survey Demographic Quantifiable Qualitative, segmentation answers to your open-ended questions research problem feedback
  94. 94. Creating an answers-at-scale campaign• Know why you’re doing a survey in the first place• Design the survey• Test it (you’ll always have mistakes)
  95. 95. Creating an answers-at-scale campaign• Know why you’re doing a survey in the first place• Design the survey• Test it (you’ll always have mistakes)• Send it out Via your NW To a paid list As an ad campaign Beware of Give the solution or Beware of Audience plea Name the problem respondent bias, unique value spamminess, low misrepresentation open ratesof the larger market “Are you a single “Can’t sleep? We’re “Our accounting mom? Take this brief trying to fix that, and software automatically survey and help us want your input.”) finds tax breaks. Help address a big us plan the product challenge.” roadmap.”
  96. 96. Creating an answers-at-scale campaign• Know why you’re doing a survey in the first place• Design the survey• Test it (you’ll always have mistakes)• Send it out Were you able to capture the attention of the market? Did they click on your ads and links? Which ones worked best?• Collect the results Are you on the right track? What decisions can you now make with the data you’ve collected? By• Analyze the data segment! Will people try out your solution/product? How many of your respondents were willing to be contacted? How many agreed to join a forum or a beta? How many asked for access in their open-ended responses?
  97. 97. Empathy stage:LikeBright’s mechanicalturk• Stage: Empathy• Model: 2-sided marketplace• Early stage startup in the dating space that joined TechStars Seattle in 2011• Initially rejected, saying, “We don’t think you understand your customer well enough.”
  98. 98. Talking to a hundred single women in aday• Used Mechanical Turk and Google Voice to speak with 100 single women; paid $2. The interviews lasted typically around 10-15 minutes.• Simple interview script with open-ended questions, since he was digging into the problem validation stage of his startup. • Founder Nick Soman: “I was amazed at the feedback I got. We were able to speak with one hundred single women that met our criteria in four hours on one evening.”• Went back to TechStars and got accepted.• LikeBright’s website is now live with a 50% female user base, and recently raised a round of funding. • “Since that first foray into interviewing customers, I’ve probably spoken with over a thousand people through Mechanical Turk,”
  99. 99. How to score problem interviewsTeach the controversy
  100. 100. How to score problem interviews• Did the interviewee successfully rank the problems you presented?• Are they actively trying to solve the problems (or have they done so in the past)?• Were they engaged and focused throughout the interview?• Did they agree to a follow-up meeting or interview (to present your solution)?• Did they offer to refer others to you for interviews?• Did they offer to pay you immediately for the solution?
  101. 101. StickinessThis comes from a good product. This is the same as the Eric Ries’ Stickinessengine of growth. You need to find out if you can build a solution to the problemyou’ve discovered. There’s no point promoting something awful if your visitors willbounce right off it in disgust. Companies like Color that attempted to scaleprematurely, without having proven stickiness, haven’t fared well.
  102. 102. 1995 Hits1997 Visits1999 Visitors2002 Conversions Who did you add? Where from? Why?2010 Engagement What did they do? How did it benefit? Who did you lose? Why did they leave?
  103. 103. What time on pagecan tell you about goals & behaviors• People spend ~1m on a page when they’re engaged with it• If a page has high traffic and low engagement, why are people leaving: • Did they come expecting something else? • Is the layout working? • Is it simply a page that isn’t designed to keep users for long? • Show off your good stuff. If a page has a high engaged time but few visitors, promote it. (Thanks to Chartbeat for the analysis.)
  104. 104. More interesting when brokendown by business model (Thanks to Chartbeat for the analysis.)
  105. 105. Days since last visit 1200 25000Number of users January February 1000 20000 800 15000 600 10000 400 5000 200 0 0 1 2 3 4 5 6 7 8 9 Disengaged Days since last engagement (>10 days)
  106. 106. Watch out for the dumb stuffThere’s a large number of ways to break an MVP
  107. 107. How to stay focused on the right stuff
  108. 108. ViralityOnce you’ve got a product or service that’s sticky, it’s time to use inherent virality—word-of-mouth that’s tied into the use of the product. That way, you’ll test out youracquisition and onboarding processes on visitors who are motivated to try you,because you have an implied endorsement from an existing user. You can alsothink of virality as a force multiplier for paid promotion—so you want to maximizethat multiplier before you start spending money on customer acquisition throughinorganic methods like advertising.
  109. 109. 3 kinds of virality• Inherent virality is built into the product, and happens as a function of use.• Artificial virality is forced, and often built into a reward system.• Word of mouth virality is simply conversations generated by satisfied users.
  110. 110. Plagues for fun and profit
  111. 111. Viral coefficient ------------------------------------------------------ Get your free private email at http://www.hotmail.com ------------------------------------------------------
  112. 112. Viral coefficient v ≠ 1, pt  = δp0 (1 – vt+1) / (1 – v) + p0  http://robert.zubek.net/blog/2008/01/30/viral-coefficient-calculation/
  113. 113. Or simpler x Viral - Churn & > Users coefficient abandonment 1
  114. 114. How to calculate it• First calculate the invitation rate, which is the number of invites sent divided by the number of users you have.• Then calculate the acceptance rate, which is the number of signups or enrollments divided by the number of invites.• Then multiply the two together.• Consider, for example • Your 2,000 customers have sent out 5,000 invitations during their lifetime on your site. • Your invitation rate is 2.5. • For every ten invitations received, one gets clicked. • Your acceptance rate is 0.1. • Multiply the two, and you have your viral coefficient: 0.25. Every customer you add will add an addition 25% of a customer.
  115. 115. Don’t forget cycle timeHow long until they invite someone?• After 20 days with a cycle time of two days, you will have 20,470 users.• If you half that cycle time to one day, you would have over 20 million users!
  116. 116. Virality stage:Circle of Moms finds anengaged market• Stage: Stickiness• Model: UGC• Launched as Circle of Friends in 2007, it was a way for small groups to interact atop Facebook’s platform; but when engagement wasn’t good enough, the founders decided to dig deeper.
  117. 117. The problem: Not enough engagement• Too few people were actually using the product• Less than 20% of any circles had any activity after their initial creation• A few million monthly uniques from 10M registered users, but no sustained traction
  118. 118. What Circle of Moms found• They found moms were far more engaged • Their messages to one another were on average 50% longer • They were 115% more likely to attach a picture to a post they wrote • They were 110% more likely to engage in a threaded (i.e. deep) conversation • Circle owners’ friends were 50% more likely to engage with the circle • They were 75% more likely to click on Facebook notifications • They were 180% more likely to click on Facebook news feed items • They were 60% more likely to accept invitations to the app• Pivoted to the new market, including a name change• By late 2009, 4.5M users and strong engagement• Sold to Sugar, inc. in early 2012
  119. 119. RevenueYou’ll want to monetize things at this point. That doesn’t mean you haven’t alreadybeen charging—for many businesses, even the first customer has to pay. It justmeans that earlier on, you’re less focused on revenue than on growth. You’regiving away free trials, free drinks, or free copies. Now you’re focused onmaximizing and optimizing revenue. This phase is closely tied to the Lean Startup’sPrice engine of growth.
  120. 120. Sergio Zyman’s many “mores”• If you’re dependent on physical, per-transaction costs (like direct sales, or shipping products to a buyer, or signing up merchants) then more efficiently will figure prominently on either the supply or demand side of your business model.• If you’ve found a high viral coefficient, then more people makes sense, because you’ve got a strong force multiplier added to every dollar you pour into customer acquisition.• If you’ve got a loyal, returning set of customers who buy from you every time, then more often makes sense, and you’re going to emphasize getting them to come back more frequently.• If you’ve got a one-time, big-ticket transaction, then more money will help a lot, because you’ve only got one chance to extract revenue from the customer and need to leave as little money as possible on the table.• If you’re a subscription model, and you’re fighting churn, then upselling customers to higher-capacity packages with broader features to additional subscribers within their organization is your best way of growing existing
  121. 121. What’s a customer worth? BAD FOR YOU GOOD FOR YOU Visitor User Customer Spam/fraud Support/resources Ad impressions Usage data Content virality Invite virality Inherent virality Voting/flagging Victim/critical mass Revenue
  122. 122. Market/product fitMost people’s first instinct when things aren’t going incredibly well is to build morefeatures. Instead, try pivoting into a new market.• Assume the product isn’t the problem, it’s the target customer.• It may be easier to change markets than products.
  123. 123. ScaleWith revenues coming in, it’s time to move from growing your business to growingyour market. You need to acquire more customers from new markets. You caninvest in channels and distribution to help grow your user base, since directinteraction with individual customers is less critical—you’re close to product/market fit and you’re analyzing things quantitatively. This phase is closely tied tothe acquisition cost optimization side of the Lean Startup’s Price engine of growth.
  124. 124. The hole in the middle Differentiation Efficiency Apple Costco Here be dragons Your local sustainable gluten-free cupcake shop Niche
  125. 125. Growth hackingFinding your way up and to the right.Human approaches, mathematical approaches
  126. 126. The leading indicator• A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya)• If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt)• A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang)• Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman)• A LinkedIn user getting to X connections in Y days (Elliot Schmukler)• These kinds of criteria also make great segments to analyze. (from the 2012 Growth Hacking conference)
  127. 127. But wait: correlated or causal? Correlation lets you Causality lets you predict the future change the future“I will have 420 “If I can make moreengaged users and first-time visitors stay75 paying customers on for 17 minutes Inext month.” will increase sales in 90 days.” Optimize theFind correlation Test causality causal factor
  128. 128. The growth hack• Growth hacking is simply what marketing should have been doing, but it fell in love with Don Draper and opinions along the way• At its most basic: Optimize a factor you think is correlated with growth
  129. 129. An example from Reddit Logged-in users All usersDays since last visit Visits Pageviews Pg/visit Visits Pageviews Pg/visit 0 127,797,781 1.925B 15.06 242,650,914 3.478B 14.33 1 5,816,594 87,339,766 15.02 13,021,131 187,992,129 14.44 2 1,997,585 27,970,618 14 4,958,931 69,268,831 13.97 Causality: When we makeCorrelation: If you look at 13.88 3 955,029 13,257,404 changes that get someone 2,620,037 34,047,741 13 >15 pages today you’ll to look at 20,644,331 15 pages 12.32 they’ll come back tomorrow. 14.23 4 625,976 8,905,483 1,675,476 come back. 5 355,643 4,256,639 11.97 1,206,731 14,162,572 11.74
  130. 130. Pause #3What stage are you at?What is your gating metric?
  131. 131. AsymptotesWhat to do when there’s no line in the sand
  132. 132. How to draw the line yourself• Sample company trying to drive enrollment • At first, out of over 1,200 visitors, only 4 (0.3%) sign up. • By the end of the month, the site is converting 8.2% of its 1,462 visitors.• Should the company keep optimizing?
  133. 133. How to draw the line yourself• By plotting a trendline we see that in the current situation the line is around 9%.• To really move the line will require a revolutionary, not evolutionary, change.
  134. 134. Finding yourOne Metric That Matters
  135. 135. Your OMTM must fit Your basic The stage your business model startup is at (monetization) (lifecycle) • E-Commerce • Empathy • UGC • Stickiness • Media • Virality • SaaS • Revenue • Mobile App • Scale • 2-sided market
  136. 136. What’s your OMTM? E- 2-sided Mobile User-gen SaaS Media commerce market app content Empathy Interviews; qualitative results; quantitative scoring; surveys Loyalty, Inventory, Engagement, Downloads, Content, Traffic, visits,Stickiness conversion listings churn churn, virality spam returns CAC, shares, Inherent WoM, app Invites, Content Virality reactivation SEM, sharing virality, CAC ratings, CAC sharing virality, SEM (Money from transactions) (Money from active users) (Money from ad clicks) Transaction, Transactions, Upselling, CLV, Ads, CPE, affiliate Revenue CLV commission CAC, CLV ARPDAU donations %, eyeballs Affiliates, Other API, magic Spinoffs, Analytics, Syndication, Scale white-label verticals #, mktplace publishers user data licenses
  137. 137. Choose only one metric.
  138. 138. It will soon change.
  139. 139. In a startup,focus is hard to achieve.
  140. 140. Having only one metricaddresses this problem.
  141. 141. Metrics are like squeeze toys.http://www.flickr.com/photos/connortarter/4791605202/
  142. 142. Pause #4
  143. 143. B2B: Selling to the enterprise
  144. 144. Lean Analytics lifecyclefor an enterprise-focused startup Stage Do this Fear this Consulting to test ideas and Lock-in, IP Empathy bootstrap the business control, overfitting Standardization and integration; Ability to Stickiness shift from custom to generic integrate; support Word of mouth, references, case Bad vibes; Virality studies exclusivity Growing direct sales, professional Pipeline, revenue Revenue services, support recognition, comp Channels, analysts, ecosystems, Crossing the Scale APIs, vertically targeted products chasm; Gorillas
  145. 145. Intrapreneur
  146. 146. Intrapreneur example:P&G changes the mopinstead of the soap• Stage: Empathy• Model: Retail/consumer packaged goods• P&G is constantly looking for better soaps. But innovation was slowing. Frustrated, they hired a design team to help them.
  147. 147. P&G changes the mopinstead of the soap• Heavy internal investment in R&D, but limited results• Brought in an outside agency (Continuum) to help• The team watched people as they mopped, recording and iterating their research approach• Watched someone pick up spilled coffee. Rather than mopping, the person swept up with a broom, then wiped with a cloth• Realized the mop, not the liquid, mattered• Studied the makeup of floor dirt; realized much of it is dust• Swiffer is a $500M innovation in a stalled industry
  148. 148. The Lean Analytics lifecyclefor an Intrapreneur Stage Do this Fear this Get buy-in Political fallout Beforehand Find problems; don’t test demand. Entitled, aggrieved Empathy Skip the business case, do analytics customers Know your real minimum based on Hidden “must haves”, Stickiness expectations, regulations feature creep Build inherent virality in from the Luddites who don’t Virality start; attention is the new currency understand sharing Consider the ecosystem, channels, Channel conflict, Revenue and established agreements resistance, contracts Hand the baton to others gracefully Hating what happens Scale to your baby
  149. 149. In the end:Ask good questionsThe OMTM should answer the most important question.
  150. 150. Once, a leader convinced others inthe absence of data.
  151. 151. Now, a leader knows whatquestions to ask.
  152. 152. Ben Yoskovitzbyosko@gmail.com@byoskoAlistair Crollacroll@gmail.com@acroll