Startup metrics toronto March 19

  • 3,384 views
Uploaded on

Slides from the first Startup Metrics Meetup in Toronto in March 2013

Slides from the first Startup Metrics Meetup in Toronto in March 2013

More in: Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,384
On Slideshare
0
From Embeds
0
Number of Embeds
9

Actions

Shares
Downloads
0
Comments
0
Likes
17

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Lean Analytics Use data to build a better business faster. www.leananalyticsbook.com @byosko | @acroll @leananalyticsSunday, 24 March, 13
  • 2. Some housekeeping.Sunday, 24 March, 13
  • 3. Sunday, 24 March, 13
  • 4. Sunday, 24 March, 13
  • 5. Kevin Costner is a lousy entrepreneur. Don’t sell what you can make. Make what you can sell.Sunday, 24 March, 13
  • 6. Analytics is the measurement of movement towards your business goals. http://www.flickr.com/photos/itsgreg/446061432/Sunday, 24 March, 13
  • 7. Small business example: Solare watches the numbers • Stage: Revenue • Model: Retailer • Solare is an Italian fine-dining restaurant under new management. The new team is trying to identify the key metrics and leading indicatorsSunday, 24 March, 13
  • 8. Solare watches the numbers • A line in the sand: Gross Revenue to Labor Cost • Under 30% is good • Below 24% is great • Lower than 20% and you may be under-staffing, leading to dissatisfied customers • A leading indicator: Total covers is 5x reservations at 5PM • If you have 50 reservations at 5, you’ll have 250 covers that night. • This ratio varies by restaurant.Sunday, 24 March, 13
  • 9. In a startup, the purpose of analytics is to iterate to a product/market fit before the money runs out.Sunday, 24 March, 13
  • 10. Most startups don’t know what they’ll be when they grow up. Freshbooks Mitel was invoicing Wikipedia was a Paypal for a web was to be lawnmower first built for company design firm written by Palmpilots experts only Flickr Hotmail Twitter Autodesk was going to was a was a made desktop be an MMO database podcasting automation company companySunday, 24 March, 13
  • 11. Five things you need to know •What makes a good metric •Understanding cohorts and segments •The business model flipbook •The Lean Analytics framework •Picking One Metric That MattersSunday, 24 March, 13
  • 12. Clear, comparable ratios Tied to your business model A good metric is Actionable, not vain Correlated or Causal Leading or LaggingSunday, 24 March, 13
  • 13. Comparable ratios: think about a car • Clear: You know 60MPH is twice as fast as 30MPH • In a country, speed limits and mileage are well understood • Kilometers are conveniently decimal; miles map to hours • Rates: Miles travelled is good; miles per hour is better; accelerating or decelerating changes your gas pedal • Business model: You can measure “MPH divided by speeding tickets” as a metric of “driving fast without losing my license”Sunday, 24 March, 13
  • 14. Vanity Actionable Picks a direction. Makes you feel good, but doesn’t change how you’ll act. http://www.flickr.com/photos/lostseouls/807253220/ http://www.flickr.com/photos/aussiegall/6382775153/Sunday, 24 March, 13
  • 15. A metric from the early, foolish days of the Web. Hits Count people instead. Marginally better than hits. Unless you’re displaying Page views ad inventory, count people. Is this one person visiting a hundred times, or are a Visits hundred people visiting once? Fail. This tells you nothing about what they did, why they Unique visitors stuck around, or if they left. Followers/ Count actions instead. Find out how many followers friends/likes will do your bidding. Time on site, or Poor version of engagement. Lots of time spent on pages/visit support pages is actually a bad sign. How many recipients will act on what’s in them? Emails collected Number of Outside app stores, downloads alone don’t lead to downloads lifetime value. Measure activations/active accounts.Sunday, 24 March, 13
  • 16. http://www.flickr.com/photos/circasassy/7858155676/ If it won’t change how you behave, it’s a bad metric. Sunday, 24 March, 13
  • 17. 2-sided market model: AirBnB and photography • Stage: Revenue • Model: 2-sided marketplace • Rental-by-owner marketplace that allows property owners to list and market their houses. Offers a variety of related services as well.Sunday, 24 March, 13
  • 18. AirBnB tests a hypothesis • The hypothesis: “Hosts with professional photography will get more business. And hosts will sign up for professional photography as a service.” • Built a concierge MVP • Found that professionally photographed listings got 2-3x more bookings than the market average. • In mid-to-late 2011, AirBnB had 20 photographers in the field taking pictures for hosts.Sunday, 24 March, 13
  • 19. NIGHTS BOOKED 10 million 8 million 6 million 20 photographers 4 million 2 million 2008 2009 2010 2011 2012Sunday, 24 March, 13Friday, November 9, 12
  • 20. A few words on causality. http://www.flickr.com/photos/roryfinneren/65729247Sunday, 24 March, 13
  • 21. 50 37.5 25 12.5 0 1 2 3 4 5 6 7 8 9 10 Seat rentals http://www.rvca.com/anp/wp-content/plugins/wp-o-matic/cache/57226_07+proof+1a+hb+beach+day.jpgSunday, 24 March, 13
  • 22. http://www.imdb.com/media/rm3768753408/tt0073195Sunday, 24 March, 13
  • 23. http://www.flickr.com/photos/kapungo/2287237966Sunday, 24 March, 13
  • 24. 10000 1000 100 10 1 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Ice cream consumption DrowningsSunday, 24 March, 13
  • 25. http://www.flickr.com/photos/25159787@N07/3766111564Sunday, 24 March, 13
  • 26. http://www.flickr.com/photos/wheressteve/3284532080Sunday, 24 March, 13
  • 27. http://www.flickr.com/photos/wtlphotos/1086968783Sunday, 24 March, 13
  • 28. 10000 1000 100 10 1 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Ice cream consumption Drownings TemperatureSunday, 24 March, 13
  • 29. http://www.flickr.com/photos/stuttermonkey/57096884Sunday, 24 March, 13
  • 30. http://www.flickr.com/photos/germanuncut77/3785152581Sunday, 24 March, 13
  • 31. http://www.flickr.com/photos/fasteddie42/2421039207Sunday, 24 March, 13
  • 32. Correlated Causal Two variables that An independent change in similar factor that directly ways , perhaps impacts a because they’re dependent one. linked to something else. Summer al Ca us us Ca Correlated al Drowning Ice cream consumptionSunday, 24 March, 13
  • 33. http://www.flickr.com/photos/bootbearwdc/1243690099/Sunday, 24 March, 13
  • 34. Causality is a superpower, because it lets you change the future. Correlation lets you Causality lets you predict the future change the future “I will have 420 “If I can make more engaged users and first-time visitors stay 75 paying customers on for 17 minutes I next month.” will increase sales in 90 days.” Optimize the Find correlation Test causality causal factorSunday, 24 March, 13
  • 35. Leading Lagging Number today that Historical metric that shows metric shows how you’re tomorrow—makes doing—reports the the news. news.Sunday, 24 March, 13
  • 36. What mode of e-commerce are you? How many of your customers Then you are in Your customers You are just Focus on buy a second this mode will buy from you like time in 90 days? Low CAC, 1-15% Acquisition Once 70% high of retailers checkout 15-30% Hybrid 2-2.5 20% Increasing per year of retailers returns Loyalty, >30% Loyalty >2.5 10% inventory per year of retailers expansion (Thanks to Kevin Hilstrom for this.)Sunday, 24 March, 13
  • 37. • A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya) • If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt) • A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang) • Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman) • A LinkedIn user getting to X connections in Y days (Elliot Schmukler) (These are also great segments to analyze.) (from the 2012 Growth Hacking conference)Sunday, 24 March, 13
  • 38. So how do you test things? Segmentation.http://www.flickr.com/photos/zlakfoto/5294803278/Sunday, 24 March, 13
  • 39. Segments, cohorts, A/B, and multivariates Cohort: Comparison of similar groups along a timeline. Segment: A/B test: ☀ Multivariate Cross-sectional ☀ Changing one analysis comparison of all thing (i.e. color) ☁ Changing several people divided by and measuring ☀ things at once to some attribute ☁ the result (i.e. see which correlates ☁ (age, gender, etc.) revenue.) with a result.Sunday, 24 March, 13
  • 40. Why use cohorts? Here’s an example. Is this   January February March April May companygrowing or Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50stagnating? Cohort 1 2 3 4 5 January $5 $3 $2 $1 $0.5 How about February $6 $4 $2 $1 now? March $7 $6 $5 April   $8 $7 May       $9Sunday, 24 March, 13
  • 41. Why use cohorts? Here’s an example. Cohort 1 2 3 4 5 January $5 $3 $2 $1 $0.5 Look at the February $6 $4 $2 $1   same data in cohorts March $7 $6 $5     April $8 $7       May $9         Averages $7 $5 $3 $1 $0.5Sunday, 24 March, 13
  • 42. What business are you in?Sunday, 24 March, 13
  • 43. Business model flipbook Revenue model: How you take money from someone Product type: What you give them in return Together, these Delivery model: How you get it to them make up a Acquisition channel: How they learn about you business model Selling tactic: How you convince them to buySunday, 24 March, 13
  • 44. Paid advertising Banner on Informationweek.com Search Engine Mgmt. High pagerank for ELC in kid’s toys Acquisition Social media outreach Active on Twitter i.e. Kissmetrics channel How the visitor, Inviting team member to Asana Inherent virality customer, or user finds out about the startup. Artificial virality Rewarding Dropbox user for others’ signups Affiliate marketing Sharing a % of sales with a referring blogger Public relations Speaker submission to SXSW App/ecosystem mkt. Placement in the Android market Simple purchase Buying a PC on Dell.com What the startup does Discounts & incentives Black Friday discount, loss leader, free ship Selling tactic to convince the visitor Free trial Time-limited trial such as fitbit Premium or user to become a Freemium Free tier, relying on upgrades, like Evernote paying customer. Pay for privacy Free account content is public, like Slideshare Free-to-play Monetize in-app purchases, like Airmech One-time transaction Single purchase from Fab How the startup Recurring subscription Monthly charge from Freshbooks Revenue model extracts money from its Consumption charges Compute cycles from Rackspace visitors, users, or Advertising clicks PPC revenue on CNET.com customers. Re-sale of user data Twitter’s firehose license Donation Wikipedia’s annual campaign Software Oracle’s accounting suite What the startup does Platform Amazon’s EC2 cloud in return. May be a Product Merchandising Thinkgeek’s retail store type product or service; may be hardware or User-generated content Facebook’s status update software; may be a Marketplace AirBnB’s list of house rentals mixture. Media/content CNN’s news page Service A hairstylist Delivery Hosted service Salesforce.com’s CRM model How the product gets Digital delivery Valve purchase of desktop game to the customer. Physical delivery Knife shipped from Sur La TableSunday, 24 March, 13
  • 45. Business Flipbook Dropbox example aspect page(s) Acquisition Inherent virality. Sharing files with others. channel Artificial virality. Free storage when others sign up. Selling Limited-capacity accounts are free; Freemium. tactic subscribe when you need more. Revenue Recurring $99/year, monthly fees, enterprise model subscription. tiers. Product Storage-as-a-service with APIs, Platform. type collaboration, synchronization tools. Delivery Hosted service. Cloud storage, web interface. model Digital delivery. Desktop client software.Sunday, 24 March, 13
  • 46. The Lean Analytics Framework.Sunday, 24 March, 13
  • 47. Eric Ries’ Three engines Stickiness Virality Price Approach Keep people Make people Spend revenue coming back. invite friends. getting customers. Math that Get customers How many they Customers are matters faster than you tell, how fast worth more than lose them. they tell them. they cost to get.Sunday, 24 March, 13
  • 48. The five Stages of Lean Analytics The business you’re in E- 2-sided Mobile User-gen SaaS Media commerce market app content Empathy The stage you’re at One Metric Stickiness Virality Revenue That Matters. ScaleSunday, 24 March, 13
  • 49. Example: a restaurant • Empathy: Before opening, the owner first learns about the diners in its area, their desires, what foods aren’t available, and trends in eating. • Stickiness: Then he develops a menu and tests it out with consumers, making frequent adjustments until tables are full and patrons return regularly. He’s giving things away, testing things, asking diners what they think. Costs are high because of variance and uncertain inventory. • Virality: He starts loyalty programs to bring frequent diners back, or to encourage people to share with their friends. He engages on Yelp and Foursquare. • Revenue: With virality kicked off, he works on margins—fewer free meals, tighter controls on costs, more standardization. • Scale: Finally, knowing he can run a profitable business, he pours some of the revenues into marketing and promotion. He reaches out to food reviewers, travel magazines, and radio stations. He launches a second restaurant, or a franchise based on the initial one.Sunday, 24 March, 13
  • 50. Example: a software company • Empathy: The founder finds an unmet need, often because she has a background in a particular industry or has worked with existing solutions that are being disrupted. • Stickiness: She meets with an initial group of prospects, and signs contracts that look more like consulting agreements, which she uses to build an initial product. She’s careful not to commit to exclusivity, and tries to steer customers towards standardized solutions, charging heavily for custom features. She supports the customers directly from the engineering team until the product is stable and usable. • Virality: Product in hand, she asks for references from satisfied customers, and uses them as testimonials. She starts direct sales, and grows the customer base. She launches a user group, and starts to automate support. She releases an API, encouraging third-party development and scaling potential market size without direct development. • Revenue: She focuses on growing the pipeline, sales margins, and revenues while controlling costs. Tasks are automated, outsourced, or offshored. Feature enhancements are scored based on anticipated payoff and development cost. Recurring license and support revenue becomes an increasingly large component of overall revenues. • Scale: She signs deals with large distributors, and works with global consulting firms to have them deploy and integrate her tool. She attends trade shows to collect leads, carefully measuring cost of acquisition against close rate and lead value.Sunday, 24 March, 13
  • 51. Empathy stage: Localmind hacks Twitter • Stage: Empathy • Model: UGC/mobile • Real-time question and answer platform tied to locations. • Needed to find out if a core behavior—answering questions about a place— happened enough to make the business realSunday, 24 March, 13
  • 52. Localmind hacks Twitter • Before writing a line of code, Localmind was concerned that people would never answer questions. • This was their biggest risk: if questions went unanswered users would have a terrible experience and stop using Localmind. • Ran an experiment on Twitter • Tracked geolocated tweets in Times Square • Sent @ messages to people who had just tweeted, asking questions about the area: how busy is it; is the subway running on time; is something open; etc. • The response rate to their tweeted questions was very high. • Good enough proxy to de-risk the solution, and convince the team and investors that it was worth building Localmind.Sunday, 24 March, 13
  • 53. Stickiness stage: WP Engine discovers the 2% cancellation rate • Stage: Stickiness • Model: SaaS • Wordpress hosting company founded in July 2010, it raised $1.2M in November 2011Sunday, 24 March, 13
  • 54. WP-Engine discovers the 2% cancellation rate • All companies have cancellations, but founder Jason Cohen was alarmed that he was losing a quarter of customers every year. • Jason called customers himself. “Not everyone wanted to speak with me, but enough people were willing to talk, even after they had left, that I learned a lot about why they were leaving.” • Asked around. Turns out 2% is best case for most hosting companies. • Without this, the company would have been getting diminishing returns over- optimizing churn; instead, they could focus on maximizing revenues or lowering acquisition costs.Sunday, 24 March, 13
  • 55. Virality stage: qidiq streamlines invites • Stage: Virality • Model: SaaS • Tool to poll small groups, built in the Year One Labs acceleratorSunday, 24 March, 13
  • 56. Initial design Redesigned workflow Survey owner adds recipient to group Survey owner adds recipient to group 70-90% RESPONSE RATE Survey owner asks question Survey owner asks question Recipient gets invite Recipient reads survey question 10-25% RESPONSE RATE Recipient installs mobile app Recipient responds to question Recipient sees survey results Recipient creates account, profile Recipient can edit profile, view past (Later, if needed…) questions, etc. Recipient visits website Recipient reads survey question Recipient has no password! Recipient responds to question Recipient does password recovery Recipient sees survey results One-time link sent to email Recipient creates password Recipient can edit profile, view past questions, etc.Sunday, 24 March, 13
  • 57. Revenue stage: Backupify’s customer lifecycle • Stage: Scale • Model: SaaS • Leading backup provider for cloud based data. • The company was founded in 2008 by Robert May and Vik Chadha • Has gone on to raise $19.5M in several rounds of financing.Sunday, 24 March, 13
  • 58. Shifting to Customer Acquisition Payback as a key metric • Initially focused on site visitors • Then focused on trials • Then switched to signups • Today, MRR • In early 2010, CAC was $243 and ARPU was only $39 • Pivoted to target business users • CLV-to-CAC today is 5-6x • Now they track Customer Acquisition Payback • Target is less than 12 monthsSunday, 24 March, 13
  • 59. What’s your OMTM? E- 2-sided Mobile User-gen SaaS Media commerce market app content Empathy Interviews; qualitative results; quantitative scoring; surveys Loyalty, Inventory, Engagement, Downloads, Content, Traffic, visits, Stickiness conversion listings churn churn, virality spam returns CAC, shares, Inherent WoM, app Invites, Content Virality reactivation SEM, sharing virality, CAC ratings, CAC sharing virality, SEM (Money from transactions) (Money from active users) (Money from ad clicks) Transaction, Transactions, Upselling, CLV, Ads, CPE, affiliate Revenue CLV commission CAC, CLV ARPDAU donations %, eyeballs Affiliates, Other API, magic Spinoffs, Analytics, Syndication, Scale white-label verticals #, mktplace publishers user data licensesSunday, 24 March, 13
  • 60. B2B and intrapreneursSunday, 24 March, 13
  • 61. The B2B stereotype • Domain expert knows industry and the problem domain. Has a Rolodex; proxy for customers. http://www.techdigest.tv/2007/02/im_a_pc_im_a_ma.html • Disruption expert knows tech that will produce a change Sees beyond the current model. Domain Disruption expert expert OperationsSunday, 24 March, 13
  • 62. Three typical approaches Create a popular consumer Dropbox Enterprise pivot product then pivot to tackle the enterprise Take an existing consumer or Yammer, Copy and rebuild open source idea and make it MapR enterprise-ready Convince the enterprise to Taleo, Disrupt a problem discard the old way because of Google overwhelming advantages. AppsSunday, 24 March, 13
  • 63. Lean Analytics lifecycle for an enterprise-focused startup Stage Do this Fear this Consulting to test ideas and Lock-in, IP Empathy bootstrap the business control, overfitting Standardization and integration; Ability to Stickiness shift from custom to generic integrate; support Word of mouth, references, case Bad vibes; Virality studies exclusivity Growing direct sales, professional Pipeline, revenue Revenue services, support recognition, comp Channels, analysts, ecosystems, Crossing the Scale APIs, vertically targeted products chasm; GorillasSunday, 24 March, 13
  • 64. The Zero Overhead principle A central theme to this new wave of innovation is the application of core product tenets from the consumer space to the enterprise. In particular, a universal lesson that I keep sharing with all entrepreneurs building for the enterprise is the Zero Overhead Principle: no feature may add training costs to the user. DJ PatilSunday, 24 March, 13
  • 65. IntrapreneurSunday, 24 March, 13
  • 66. Skunk Works for intrapreneurs • The Lockheed Martin Skunk WorksSunday, 24 March, 13
  • 67. Span of control and the railroads • Daniel C. McCallumSunday, 24 March, 13
  • 68. The BCG matrix • How businesses think about products or Question marks! increase
 Pivot to 
 Stars! companies (low market share, market
 (high growth rate, share
 high growth rate) through
 high market share) May be the next big thing. virality,
 What everyone wants. As • Lean is about moving Consumes investment, but attention market invariably stops will require money to growing, should become up and to the right Growth rate increase market share. cash cows. Milk with
 Pivot to
 Pivot to
 revenue
 redefine problem/
 increase growth
 optimization as
 solution through
 rate through
 growth slows empathy disruption Dogs! Cash cows! (low market share, (high market share, low growth rate) low growth rate) Barely breaks even, may Boring sources of cash, to be a distraction from better be milked but not worth opportunities. Sell off or additional investment. shut down. Market shareSunday, 24 March, 13
  • 69. Intrapreneur example: P&G changes the mop instead of the soap • Stage: Empathy • Model: Retail/consumer packaged goods • P&G is constantly looking for better soaps. But innovation was slowing. Frustrated, they hired a design team to help them.Sunday, 24 March, 13
  • 70. P&G changes the mop instead of the soap • Heavy internal investment in R&D, but limited results • Brought in an outside agency (Continuum) to help • The team watched people as they mopped, recording and iterating their research approach • Watched someone pick up spilled coffee. Rather than mopping, the person swept up with a broom, then wiped with a cloth • Realized the mop, not the liquid, mattered • Studied the makeup of floor dirt; realized much of it is dust • Swiffer is a $500M innovation in a stalled industrySunday, 24 March, 13
  • 71. The Lean Analytics lifecycle for an Intrapreneur Stage Do this Fear this Get buy-in Political fallout Beforehand Find problems; don’t test demand. Entitled, aggrieved Empathy Skip the business case, do analytics customers Know your real minimum based on Hidden “must haves”, Stickiness expectations, regulations feature creep Build inherent virality in from the Luddites who don’t Virality start; attention is the new currency understand sharing Consider the ecosystem, channels, Channel conflict, Revenue and established agreements resistance, contracts Hand the baton to others gracefully Hating what happens Scale to your babySunday, 24 March, 13
  • 72. “The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.” Lloyd S. NelsonSunday, 24 March, 13
  • 73. Choose only one metric.Sunday, 24 March, 13
  • 74. Yes, one metric.Sunday, 24 March, 13
  • 75. It will soon change.Sunday, 24 March, 13
  • 76. In a startup, focus is hard to achieve.Sunday, 24 March, 13
  • 77. Having only one metric addresses this problem.Sunday, 24 March, 13
  • 78. Metrics are like squeeze toys. http://www.flickr.com/photos/connortarter/4791605202/Sunday, 24 March, 13
  • 79. ARCHIMEDES HAD TAKEN BATHS BEFORE.Sunday, 24 March, 13
  • 80. Once, a leader convinced others in the absence of data.Sunday, 24 March, 13
  • 81. Now, a leader knows what questions to ask.Sunday, 24 March, 13
  • 82. Ben Yoskovitz byosko@gmail.com @byosko Alistair Croll acroll@gmail.com @acrollSunday, 24 March, 13