Lean Analytics Use data to build a better business faster. www.leananalyticsbook.com @byosko | @acroll @leananalytics
Kevin Costner is a lousy entrepreneur. Don’t sell what you can make. Make what you can sell.
Analytics is the measurementof movement towards yourbusiness goals. http://www.ﬂickr.com/photos/itsgreg/446061432/
Small business example:Solare watches thenumbers• Stage: Revenue• Model: Retailer• Solare is an Italian ﬁne-dining restaurant under new management. The new team is trying to identify the key metrics and leading indicators
Solare watches the numbers• A line in the sand: Gross Revenue to Labor Cost • Under 30% is good • Below 24% is great • Lower than 20% and you may be under-stafﬁng, leading to dissatisﬁed customers• A leading indicator: Total covers is 5x reservations at 5PM • If you have 50 reservations at 5, you’ll have 250 covers that night. • This ratio varies by restaurant.
In a startup, the purpose ofanalytics is to iterate to aproduct/market ﬁt beforethe money runs out.
What I’ll cover•What makes a good metric•Understanding cohorts and segments•The Lean Analytics cycle•The Stages of Lean Analytics•Picking One Metric That Matters
Qualitative or Quantitative5 things you Exploratory or Reportingneed to know Vanity or Actionableabout metrics Correlated or Causal Leading or Lagging
Qualitative Quantitative Unstructured, Numbers and stats; anecdotal, hard facts but less revealing, hard to insight. aggregate. Warm and fuzzy. Cold and hard.http://www.ﬂickr.com/photos/zooboing/8388257248/ http://www.ﬂickr.com/photos/x1brett/4665645157/
Simply: you can’t count smiles.Discover qualitatively, prove quantitatively.Qualitative is inspiration, quantitative is veriﬁcation.
Exploratory Reporting Speculative, trying Predictable, keeping to ﬁnd unexpected you abreast of or interesting normal, managerial insights. operations.http://www.ﬂickr.com/photos/50755773@N06/5415295449/ http://www.ﬂickr.com/photos/elwillo/4737933662/
Donald Rumsfeld on analytics Are facts which may be wrong and we know should be checked against data. know we don’t Are questions we can answer by reporting, which we should baseline know & automate.Things we Are intuition which we should we know quantify and teach to improve don’t effectiveness, efﬁciency. know we don’t Are exploration which is where unfair advantage and interesting know epiphanies live. (Or rather, Avinash Kaushik channeling Rumsfeld)
Vanity Actionable Picks a direction.Makes you feelgood, but doesn’tchange how you’llact.http://www.ﬂickr.com/photos/lostseouls/807253220/ http://www.ﬂickr.com/photos/aussiegall/6382775153/
A metric from the early, foolish days of the Web. Hits Count people instead. Marginally better than hits. Unless you’re displaying Page views ad inventory, count people. Is this one person visiting a hundred times, or are a Visits hundred people visiting once? Fail. This tells you nothing about what they did, why theyUnique visitors stuck around, or if they left. Followers/ Count actions instead. Find out how many followers friends/likes will do your bidding.Time on site, or Poor version of engagement. Lots of time spent on pages/visit support pages is actually a bad sign. How many recipients will act on what’s in them?Emails collected Number of Outside app stores, downloads alone don’t lead to downloads lifetime value. Measure activations/active accounts.
http://www.ﬂickr.com/photos/circasassy/7858155676/ it’s a how you behave, If it won’t changebadmetric.
2-sided market model:AirBnB and photography• Stage: Revenue• Model: 2-sided marketplace• Rental-by-owner marketplace that allows property owners to list and market their houses. Offers a variety of related services as well.
AirBnB tests a hypothesis• The hypothesis: “Hosts with professional photography will get more business. And hosts will sign up for professional photography as a service.”• Built a concierge MVP• Found that professionally photographed listings got 2-3x more bookings than the market average.• In mid-to-late 2011, AirBnB had 20 photographers in the ﬁeld taking pictures for hosts.
NIGHTS BOOKED10 million 8 million 6 million 20 photographers 4 million 2 million 2008 2009 2010 2011 2012
A few words on causality. http://www.ﬂickr.com/photos/roryﬁnneren/65729247
Correlated CausalTwo variables that An independentchange in similar factor that directlyways , perhaps impacts abecause they’re dependent one.linked to somethingCausalelse. Summer al Ca us us Ca Correlated al Drowning Ice creamconsumption
Causality is a superpower, because it lets youchange the future. Correlation lets you Causality lets you predict the future change the future“I will have 420 “If I can make moreengaged users and ﬁrst-time visitors stay75 paying customers on for 17 minutes Inext month.” will increase sales in 90 days.” Optimize theFind correlation Test causality causal factor
Leading LaggingNumber today that Historical metric thatshows metric shows how you’retomorrow—makes doing—reports thethe news. news.
What mode of e-commerce are you? How many of your customers Then you are in Your customers You are just Focus on buy a second this mode will buy from you liketime in 90 days? Low CAC, 1-15% Acquisition Once 70% high of retailers checkout 15-30% Hybrid 2-2.5 20% Increasing per year of retailers returns Loyalty, >30% Loyalty >2.5 10% inventory per year of retailers expansion (Thanks to Kevin Hilstrom for this.)
• A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya)• If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt)• A Dropbox user who puts at least one ﬁle in one folder on one device (ChenLi Wang)• Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman)• A LinkedIn user getting to X connections in Y days (Elliot Schmukler) (These are also great segments to analyze.) (from the 2012 Growth Hacking conference)
So how do you test things? Segmentation.http://www.ﬂickr.com/photos/zlakfoto/5294803278/
Segments, cohorts, A/B, and multivariates Cohort: Comparison of similar groups along a timeline. Segment: A/B test: ☀ Multivariate Cross-sectional ☀ Changing one analysis comparison of all thing (i.e. color) ☁ Changing several people divided by and measuring ☀ things at once to some attribute ☁ the result (i.e. see which correlates ☁(age, gender, etc.) revenue.) with a result.
Why use cohorts? Here’s an example. Is this January February March April May companygrowing or Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50stagnating? Cohort 1 2 3 4 5 January $5 $3 $2 $1 $0.5How about February $6 $4 $2 $1 now? March $7 $6 $5 April $8 $7 May $9
Why use cohorts? Here’s an example. Cohort 1 2 3 4 5 January $5 $3 $2 $1 $0.5Look at the February $6 $4 $2 $1 same data in cohorts March $7 $6 $5 April $8 $7 May $9 Averages $7 $5 $3 $1 $0.5
Eric Ries’Three engines Stickiness Virality PriceApproach Keep people Make people Spend revenue coming back. invite friends. getting customers.Math that Get customers How many they Customers are matters faster than you tell, how fast worth more than lose them. they tell them. they cost to get.
The ﬁve Stages of Lean Analytics The business you’re in E- 2-sided Mobile User-gen SaaS Media commerce market app content EmpathyThe stage you’re at One Metric Stickiness Virality Revenue That Matters. Scale
Mobile app model:Localmind hacks Twitter• Stage: Empathy• Model: UGC/mobile• Real-time question and answer platform tied to locations.• Needed to ﬁnd out if a core behavior—answering questions about a place— happened enough to make the business real
Localmind hacks Twitter• Before writing a line of code, Localmind was concerned that people would never answer questions. • This was their biggest risk: if questions went unanswered users would have a terrible experience and stop using Localmind.• Ran an experiment on Twitter • Tracked geolocated tweets in Times Square • Sent @ messages to people who had just tweeted, asking questions about the area: how busy is it; is the subway running on time; is something open; etc.• The response rate to their tweeted questions was very high. • Good enough proxy to de-risk the solution, and convince the team and investors that it was worth building Localmind.
Stickiness stage:WP Engine discovers the2% cancellation rate• Stage: Stickiness• Model: SaaS• Wordpress hosting company founded in July 2010, it raised $1.2M in November 2011
WP-Engine discovers the 2%cancellation rate• All companies have cancellations, but founder Jason Cohen was alarmed that he was losing a quarter of customers every year.• Jason called customers himself. “Not everyone wanted to speak with me, but enough people were willing to talk, even after they had left, that I learned a lot about why they were leaving.”• Asked around. Turns out 2% is best case for most hosting companies.• Without this, the company would have been getting diminishing returns over- optimizing churn; instead, they could focus on maximizing revenues or lowering acquisition costs.
Virality stage:qidiq streamlines invites• Stage: Virality• Model: SaaS• Tool to poll small groups, built in the Year One Labs accelerator
Initial design Redesigned workﬂow Survey owner adds recipient to group Survey owner adds recipient to group 70-90% RESPONSE RATE Survey owner asks question Survey owner asks question Recipient gets invite Recipient reads survey question10-25% RESPONSE RATE Recipient installs mobile app Recipient responds to question Recipient sees survey results Recipient creates account, proﬁle Recipient can edit proﬁle, view past (Later, if needed…) questions, etc. Recipient visits website Recipient reads survey question Recipient has no password! Recipient responds to question Recipient does password recovery Recipient sees survey results One-time link sent to email Recipient creates password Recipient can edit proﬁle, view past questions, etc.
Revenue stage:Backupify’s customerlifecycle• Stage: Scale• Model: SaaS• Leading backup provider for cloud based data.• The company was founded in 2008 by Robert May and Vik Chadha• Has gone on to raise $19.5M in several rounds of ﬁnancing.
Shifting to Customer AcquisitionPayback as a key metric• Initially focused on site visitors• Then focused on trials• Then switched to signups• Today, MRR• In early 2010, CAC was $243 and ARPU was only $39 • Pivoted to target business users • CLV-to-CAC today is 5-6x• Now they track Customer Acquisition Payback • Target is less than 12 months
What these have in common:The Lean Analytics CycleSuccess! Pick a KPI Draw a line in the sand Pivot or give up Draw a new line Find a potential Try again improvementDid we movethe needle? Without With data: data: make a ﬁnd a good guess commonality Design a test Measure the results Hypothesis Make changes in production
What’s your OMTM? E- 2-sided Mobile User-gen SaaS Media commerce market app content Empathy Interviews; qualitative results; quantitative scoring; surveys Loyalty, Inventory, Engagement, Downloads, Content, Trafﬁc, visits,Stickiness conversion listings churn churn, virality spam returns CAC, shares, Inherent WoM, app Invites, Content Virality reactivation SEM, sharing virality, CAC ratings, CAC sharing virality, SEM (Money from transactions) (Money from active users) (Money from ad clicks) Transaction, Transactions, Upselling, CLV, Ads, CPE, afﬁliate Revenue CLV commission CAC, CLV ARPDAU donations %, eyeballs Afﬁliates, Other API, magic Spinoffs, Analytics, Syndication, Scale white-label verticals #, mktplace publishers user data licenses