Startup KPIs and A/B Testing

516 views

Published on

What startup KPIs are, how they should be used, and how to set them up. Intro slides plus a couple of workshop activities for students.

Published in: Business
  • Be the first to comment

Startup KPIs and A/B Testing

  1. 1. Startup KPIs & A/B Testing What they are, why you should use them, & how to set them up jeff.mcclelland@yahoo.ca www.linkedin.com/in/jeffmcclelland
  2. 2. Hi, I’m Jeff! Integral Designs Bell Canada ING Bank Skype → Microsoft TransferWise MSc - Informa- tion Science Bachelors - Business data → data → data → data → data → data → data → data → data → data → data → data → data
  3. 3. Workshop Topics What are you trying to do? Why focus on KPIs? What are well-defined KPIs? Where does A/B testing fit in?
  4. 4. KPIs + A/B testing support the org’s learning You win if you learn faster ● Smart KPIs are widely applicable across your business ● A/B testing is a specialized tool: great for a small number of problems Data-driven organization Focus on learning Smart KPIs A/B Testing You Win
  5. 5. What are you trying to do?
  6. 6. MVP Learning Cycle MVP - does not mean “minimum viable piece of crap” ● You need to understand who your customers are and what they want. ● Can the customers use your product? ● Can you do for the customer what you actually promise?
  7. 7. Metrics → feedback → learning You need to understand your metrics and set realistic targets! This will help you define success and start iterating. For instance: ● New users ● NPS ● Cancellations ● Conversion
  8. 8. Iterate Usually the one that iterates fastest wins! ● Prioritising is they key here ● Make sure you know what you effect and measure it! ● A/B testing
  9. 9. Why focus on KPIs?
  10. 10. You have a hard job ● You have a limited opportunity to become successful ● Your success depends on how quickly you learn. ● Specifically, how quickly you learn how to deliver: ○ A product customers value ○ A sustainable business at scale
  11. 11. You’ll learn faster with good KPIs Using well-defined KPI’s help you learn faster: ● Which products/features customers value ● How effectively you’re scaling: ○ marketing channels ○ costs ○ operations ● Which activities you should focus on
  12. 12. You are here You want to get here Why you should focus on KPIs ● You have a mountain to climb to achieve your vision - KPIs are your guideposts ● Validation: It’s proof that you’re moving forward ● Investors/valuations
  13. 13. What are well-defined KPIs?
  14. 14. The best metric/KPI We did another thing! time the best metric We did a thing! Metric is stable and flat when you’re not influencing it
  15. 15. 5 principles of well-defined KPIs Directional & Not Ambiguous IntuitiveImportant Reflects reality In control of the group it’s measured against if performance is good, it's good for the user and for the company Can be explained why it's important for the user and to the company Proportional to reality, dynamic enough to raise attention (not flat day-after-day), but not so jumpy that you can’t discern signal from the noise higher or lower is always better all other things being equal so that the team can commit to and be accountable for improvement (for targets in particular)
  16. 16. Early on in your startup Do Not Focus On Vanity metrics i.e. total registered users, total volume transferred Getting statistical significance, split testing (A/B testing) More than a handful of metrics Focus on The 3 or 4 most important metrics (KPIs) that indicate your progress. Think about: ● Growth & Conversion ● How you’re adding value ● How much runway you have Helpful to work backwards from some target state (i.e. 10K new users) and validate how you get there
  17. 17. KPI Examples Growth - Signups, Downloads, New users, Active Users, Revenue, Profit Engagement/conversion - feature use, frequency of use, # features Quality - Net Promoter Score (NPS), Customer Satisfaction (CSAT) Profitability/Sustainability/cost per user (marketing and/or OPEX) Churn - users stop using your product/service, downgrade Virality/network effects - spreading the word, building the network, market share Speed/Efficiency/Backlog/Queues/Time to X - efficiency of your machine Risk/Exposure - Leverage, FX Exposure, Value at Risk, SLA, cash/runway
  18. 18. Task - Define KPIs for your business Pick a handful of KPIs that drive your business (or area of the company) Pick a handful of segments Present it back; give and get feedback
  19. 19. Where does A/B testing fit in?
  20. 20. What are A/B tests? Everyone believes they know what’s right...but they’re often wrong. ● Everyone has biases ● Scientific experiments ● A/B Tests can help you learn ● If done well they give you unbiased “proof” of whether A is better than B
  21. 21. When should you use A/B testing? Work well for optimisation problems - small iterative learning ● Product, pricing, messaging, support all possible But even if you could, you sometimes shouldn’t as they can be expensive. Learning is expensive.
  22. 22. ● Get high volumes and be patient waiting on results ● Don’t corrupt with other overlapping tests ● Know basic statistics AB testing 101 Are your results significant? tl;dr: no It’s difficult to achieve statistical significance You need to: ● Plan/design in advance ● Pick the right KPI(s) & scope ● Random or pseudo-random allocation
  23. 23. AB Testing - ballpark testing Useful: http://www.thumbtack.co m/labs/abba Add your variations here Find out whether it’s significant here
  24. 24. Task - A/B testing 1. Pick up one of each colour at random from the front. These are the parameters of an AB experiment you’ve set up 2. Use those numbers to figure out what numbers to put in each box on here: thumbtack.com/labs/abba/ 3. Figure out whether your experiment will be “significant” or not 4. Figure out which variables to change to make it significant (i.e. longer duration?, more visitors?, bigger change?)
  25. 25. Small sample but huge performance increase A/B Test 1 ✔
  26. 26. A/B Test 2 ✔ Huge sample but small performance increase
  27. 27. Summary: How to get to significance? One set is too small ✖
  28. 28. The gap is too small for the number of trials Summary: How to get to significance? ✖
  29. 29. Summary: How to get to significance? Success is too rare ✖
  30. 30. The gap is too small for the number of trials Small sample but huge performance increase Summary: How to get to significance? ✔ Huge sample but small performance increase ✔ One set is too small ✖ ✖ Success is too rare ✖
  31. 31. Final notes
  32. 32. Recap Data-driven organization Focus on learning Smart KPIs A/B Testing You Win ● Your goal is to learn quickly ● KPIs are important for learning what to focus on ● Well-defined KPIs have five attributes: important, intuitive, directional, reflect reality, & in your control ● AB testing is powerful when done right, but you have to set it up in a smart way
  33. 33. Learn more Read On how to set up KPIs for SaaS companies http://www.forentrepreneurs.com/saas-metrics-2/ and any relevant post by Avinash Kaushik The evolution of data-driven A/B testing & product development at Etsy: http://mcfunley.com/data-driven-products-now Get in touch jeff.mcclelland@yahoo.ca Skype: jrmcclel LinkedIn +372 5845 1455
  34. 34. APPENDIX: presenting & using KPIs
  35. 35. Tip: should you use a graph or a table? Graphs To focus on one metric and a handful of splits Trends, change Missing important context Tables Many metrics, many splits Point-to-point comparison & where you need context Focusing attention Best use Challenge Use when you need
  36. 36. Tip: Colours and Lines General rule: As little as possible When you’ve got your data... ● Remove all lines/colours and add logically ● Add only when it makes “so what” more clear ● Every single line, colour, metric and split must have a meaning and add value ● Create an “all other” category ● Test: Hide/remove a metric or a split - see if it changes your interpretation How?
  37. 37. Useful for any processes that happen over time ● sales, engagement, customer retention, etc. Help you compare users fairly and isolate behaviour changes from portfolio/mix changes Tip: Use Cohort metrics when you can user cohorts time
  38. 38. Tip: be mindful of signal vs noise Even the same data can tell a completely different story Choosing what you focus on, and how you look at it matters
  39. 39. Tip: Avoid common KPI pitfalls If you must denominate, choose the least-bad denominator Divide by something or target the total? Too broad Too complex Unclear link to business goals Higher & lower values could both be good? Be careful with ratiosCommon KPI errors
  40. 40. Task - Sketch a one-page dashboard Using the metrics and slices derived earlier: Sketch out what you would want to see in a compact (one page or screen) dashboard. Think about: Frequency (daily, weekly, monthly) & amount of history Prioritise the most important information Balance the need for a high-level overview with actionable information Effective charts/tables What decisions you will make based on the dashboard (if KPI x does this, I will do that) Present it back; give and get feedback
  41. 41. APPENDIX: A/B Testing activity Print out for activity on slide 24
  42. 42. 100 visitors / week 1,000 visitors / week 10,000 visitors / week 50,000 visitors / week 1,000,000 visitors / week 1% improvement 3% improvement 5% improvement 7% improvement 15% improvement 1 day 1 week 2 weeks 4 weeks 24 weeks 100,000 visitors / week 10% improvement 4 weeks 1% convert 3% convert 10% convert 15% convert 50% convert 25% convert 10% treated 30% treated 50% treated 90% treated

×