What startup KPIs are, how they should be used, and how to set them up. Intro slides plus a couple of workshop activities for students.

1. Startup KPIs & A/B Testing
What they are, why you should use them, & how to set them up
jeff.mcclelland@yahoo.ca
www.linkedin.com/in/jeffmcclelland

2. Hi, I’m Jeff!
Integral
Designs Bell Canada ING Bank Skype → Microsoft TransferWise
MSc -
Informa-
tion
Science
Bachelors - Business
data → data → data → data → data → data → data → data → data → data → data → data → data

3. Workshop Topics
What are you trying to do?
Why focus on KPIs?
What are well-defined KPIs?
Where does A/B testing fit in?

4. KPIs + A/B testing support the org’s learning
You win if you learn faster
● Smart KPIs are widely
applicable across your
business
● A/B testing is a
specialized tool: great
for a small number of
problems
Data-driven
organization
Focus on
learning
Smart KPIs
A/B
Testing
You
Win

5. What are you trying to do?

6. MVP Learning Cycle
MVP - does not mean “minimum
viable piece of crap”
● You need to understand who
your customers are and what
they want.
● Can the customers use your
product?
● Can you do for the customer
what you actually promise?

7. Metrics → feedback → learning
You need to understand your
metrics and set realistic targets!
This will help you define success
and start iterating.
For instance:
● New users
● NPS
● Cancellations
● Conversion

8. Iterate
Usually the one that iterates
fastest wins!
● Prioritising is they key here
● Make sure you know what
you effect and measure it!
● A/B testing

9. Why focus on KPIs?

10. You have a hard job
● You have a limited opportunity
to become successful
● Your success depends on how
quickly you learn.
● Specifically, how quickly you
learn how to deliver:
○ A product customers value
○ A sustainable business at scale

11. You’ll learn faster with good KPIs
Using well-defined KPI’s help
you learn faster:
● Which products/features
customers value
● How effectively you’re
scaling:
○ marketing channels
○ costs
○ operations
● Which activities you
should focus on

12. You are
here
You want
to get here
Why you should focus on KPIs
● You have a mountain to climb to achieve
your vision - KPIs are your guideposts
● Validation: It’s proof that you’re moving
forward
● Investors/valuations

13. What are well-defined KPIs?

14. The best metric/KPI
We did
another thing!
time
the
best
metric
We did
a thing!
Metric is stable and flat
when you’re not
influencing it

15. 5 principles of well-defined KPIs
Directional &
Not
Ambiguous
IntuitiveImportant
Reflects
reality
In control of
the group it’s
measured
against
if performance is good, it's
good for the user and for
the company
Can be explained why it's
important for the user and
to the company
Proportional to reality, dynamic
enough to raise attention (not flat
day-after-day), but not so jumpy
that you can’t discern signal from
the noise
higher or lower is always
better all other things
being equal
so that the team can commit
to and be accountable for
improvement (for targets in
particular)

16. Early on in your startup
Do Not Focus On
Vanity metrics
i.e. total registered users, total volume transferred
Getting statistical significance,
split testing (A/B testing)
More than a handful of metrics
Focus on
The 3 or 4 most important metrics
(KPIs) that indicate your progress. Think
about:
● Growth & Conversion
● How you’re adding value
● How much runway you have
Helpful to work backwards from some
target state (i.e. 10K new users) and
validate how you get there

17. KPI Examples
Growth - Signups, Downloads, New users, Active Users, Revenue, Profit
Engagement/conversion - feature use, frequency of use, # features
Quality - Net Promoter Score (NPS), Customer Satisfaction (CSAT)
Profitability/Sustainability/cost per user (marketing and/or OPEX)
Churn - users stop using your product/service, downgrade
Virality/network effects - spreading the word, building the network, market
share
Speed/Efficiency/Backlog/Queues/Time to X - efficiency of your machine
Risk/Exposure - Leverage, FX Exposure, Value at Risk, SLA, cash/runway

18. Task - Define KPIs for your business
Pick a handful of KPIs that drive your business (or area of the
company)
Pick a handful of segments
Present it back; give and get feedback

19. Where does A/B testing fit in?

20. What are A/B tests?
Everyone believes they know
what’s right...but they’re often
wrong.
● Everyone has biases
● Scientific experiments
● A/B Tests can help you learn
● If done well they give you
unbiased “proof” of whether
A is better than B

21. When should you use A/B testing?
Work well for optimisation
problems - small iterative
learning
● Product, pricing, messaging,
support all possible
But even if you could, you
sometimes shouldn’t as they can
be expensive. Learning is
expensive.

22. ● Get high volumes and be
patient waiting on results
● Don’t corrupt with other
overlapping tests
● Know basic statistics
AB testing 101
Are your results significant? tl;dr: no
It’s difficult to achieve statistical
significance
You need to:
● Plan/design in advance
● Pick the right KPI(s) & scope
● Random or pseudo-random
allocation

23. AB Testing - ballpark testing
Useful:
http://www.thumbtack.co
m/labs/abba
Add your variations
here
Find out whether it’s
significant here

24. Task - A/B testing
1. Pick up one of each colour at random from the front. These
are the parameters of an AB experiment you’ve set up
2. Use those numbers to figure out what numbers to put in
each box on here: thumbtack.com/labs/abba/
3. Figure out whether your experiment will be “significant” or
not
4. Figure out which variables to change to make it significant
(i.e. longer duration?, more visitors?, bigger change?)

25. Small sample but huge
performance increase
A/B Test 1
✔

26. A/B Test 2
✔
Huge sample but small
performance increase

27. Summary: How to get to significance?
One set is too small
✖

28. The gap is too small for
the number of trials
Summary: How to get to significance?
✖

29. Summary: How to get to significance?
Success is too rare
✖

30. The gap is too small for
the number of trials
Small sample but huge
performance increase
Summary: How to get to significance?
✔
Huge sample but small
performance increase
✔
One set is too small ✖
✖
Success is too rare
✖

31. Final notes

32. Recap
Data-driven
organization
Focus on
learning
Smart KPIs
A/B
Testing
You
Win
● Your goal is to learn quickly
● KPIs are important for
learning what to focus on
● Well-defined KPIs have five
attributes: important,
intuitive, directional, reflect
reality, & in your control
● AB testing is powerful when
done right, but you have to
set it up in a smart way

33. Learn more
Read
On how to set up KPIs for SaaS companies
http://www.forentrepreneurs.com/saas-metrics-2/
and any relevant post by Avinash Kaushik
The evolution of data-driven A/B testing & product
development at Etsy:
http://mcfunley.com/data-driven-products-now
Get in touch
jeff.mcclelland@yahoo.ca
Skype: jrmcclel
LinkedIn
+372 5845 1455

34. APPENDIX:
presenting & using KPIs

35. Tip: should you use a graph or a table?
Graphs
To focus on one metric
and a handful of splits
Trends, change
Missing important context
Tables
Many metrics,
many splits
Point-to-point comparison &
where you need context
Focusing attention
Best use
Challenge
Use when
you need

36. Tip: Colours and Lines
General rule: As little as possible
When you’ve got your data...
● Remove all lines/colours and
add logically
● Add only when it makes “so
what” more clear
● Every single line, colour, metric
and split must have a meaning
and add value
● Create an “all other” category
● Test: Hide/remove a metric or a
split - see if it changes your
interpretation
How?

37. Useful for any processes that
happen over time
● sales, engagement,
customer retention, etc.
Help you compare users fairly
and isolate behaviour
changes from portfolio/mix
changes
Tip: Use Cohort metrics when you can
user
cohorts time

38. Tip: be mindful of signal vs noise
Even the same data can tell a completely different story
Choosing what you focus on, and how you look at it matters

39. Tip: Avoid common KPI pitfalls
If you must denominate,
choose the least-bad
denominator
Divide by something or
target the total?
Too broad
Too
complex
Unclear link to
business goals
Higher & lower values
could both be good?
Be careful with ratiosCommon KPI errors

40. Task - Sketch a one-page dashboard
Using the metrics and slices derived earlier:
Sketch out what you would want to see in a compact (one page
or screen) dashboard. Think about:
Frequency (daily, weekly, monthly) & amount of history
Prioritise the most important information
Balance the need for a high-level overview with actionable
information
Effective charts/tables
What decisions you will make based on the dashboard (if KPI x does this, I
will do that)
Present it back; give and get feedback

41. APPENDIX:
A/B Testing activity
Print out for activity on slide 24

42. 100 visitors / week
1,000 visitors / week
10,000 visitors / week
50,000 visitors / week
1,000,000 visitors / week
1% improvement
3% improvement
5% improvement
7% improvement
15% improvement
1 day
1 week
2 weeks
4 weeks
24 weeks
100,000 visitors / week 10% improvement 4 weeks
1% convert
3% convert
10% convert
15% convert
50% convert
25% convert
10% treated
30% treated
50% treated
90% treated