"Measure and Learn" Slides for the Communication Design Lab from IED Master Madrid.

1. MEASURE & LEARN
IED - COMMUNICATION DESIGN LABS

2. HELLO !

3. about.me
Psychologist, full-stack digital
marketer, data native, founder
and much more buzz words! connect about.me/andykaiser

4. “Data Scientist:The Sexiest Job
of the 21st Century”Perhaps NOT! We all will
work as Data-Specialist.
https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century/

6. • Most startups don’t know what they’ll be
when they grow up.
• Markets get disrupted by Data
• Evaluate data to understand what’s working
and what’s not.

7. first build for Palmpilots

8. was a database company

9. was written by experts only

10. made desktop automation

11. was going to be a MMO

12. was a podcasting company

13. –Avinash Kaushik
“Analytics is the analysis of qualitative and
quantitative data from your business and
competition to drive continual improvement
of the online experience that your customers
and potential customers have which translates to
your desired outcomes, both online and offline.”

14. A DAY ATTHE RACES
Data Driven Decisions

16. –Pat Symonds
“Bloody hell, it's so different these days. I used to be
able to have dinner with the drivers on Friday night!
Nowadays they spend evenings mulling over graphs
and spreadsheets to inform their race preparation.”

17. • 150 sensors per car
• 1.000 data points per lap
• 200 Gigabytes move from track and factory
every race weekend
• 50 Data Engineers per race

20. –Kenny O’Donnell
“You will make the best decision you can make.”

21. Gut Feeling Data-driven
DecisionIntuition Scientific

22. METRICS 📈

23. A good metric is…
• comparable to another time period, group,
competitor,…
• understandable for the target audience.
• a ratio or rate
• targeted to the right audience (developers, marketers,
business development,…)
• a behaviour changer.

24. • Qualitative metrics are unstructured,
anecdotal, revealing, hard to aggregate.
• Quantitative metrics are numbers and stats,
hard facts but less insights.
Discover
Proof

25. https://www.google.com/trends/

26. https://www.youtube.com/trendsdashboard

27. VANITY METRICS
• Vanity Metrics makes you feel good but doesn’t
change how you’ll act.
• Vanity Metrics are bad!

28. • Hits
• PageViews
• Visits
• Users
• Followers/friends/likes
• Logins
Only for Ad Inventory
Count People
Cross it with People
Why they stuck or left?
Count actions!
What are the actions and value?

29. “STATISTICS” 🎲

30. CORRELATION
• Understand Correlation/Causality
• What is the relationship between two variables?
• Correlation does not imply causation.

31. chart courtesy fromTylerVigen | www.tylervigen.com

32. chart courtesy fromTylerVigen | www.tylervigen.com

33. chart courtesy fromTylerVigen | www.tylervigen.com

34. CORRELATION
• Use R, Excel, Numbers or online tools to calculate the linear relationship between
two variables.
• Interpret a Correlation Coefficient r
• +- 1:A perfect linear relationship
• +- 0.70 strong linear relationship
• +- 0.50 moderate linear relationship
• +- 0.30: weak linear relationship
• +- 0: no linear relationship

35. STANDARD DEVIATION
• Standard Deviation (SD) is a measure that is used to
quantify the amount of variation or dispersion of a set
of data values.
• A standard deviation close to 0 indicates that the data
points tend to be very close to the mean of the set,
while a high standard deviation indicates that the data
points are spread out over a wider range of values.

36. STATISTICAL SIGNIFICANCE
• Statistical significance is fundamental to statistical
hypothesis testing (determine whether a null
hypothesis should be rejected or retained)
• In any experiment or observation that involves
drawing a sample from a population, there is
always the possibility that an observed effect
would have occurred due to sampling error alone.

37. A/BTESTING 🎲

38. Find Correlation Test Causality
Optimize the
causal factor
Correlation lets you
predict the future
Causality lets you
change the future

39. • A/B testing (also known as split testing) is a
method of comparing two versions against
each other to determine which one performs
better.
• Testing takes the guesswork out of
optimization and enables data-backed decisions
that shift business conversations from “we
think” to “we know.”

40. Goal:
Drive traffic to theVodafone website.
Winner: +6%

41. Goal:
Increase sales.
Winner: +38%

42. Goal:
Drive traffic to theVodafone website.
Winner!

43. Goal:
Increase Insurance Leads
Winner!

44. Goal:
Increase hotel bookings
Winner!

45. Goal:
Increase unique opens/clicks of email
Winner: +86%

46. Goal:
Reduce Bounce Rate
Winner!

47. • Define a Hypothesis and Goal.
• Define the minimum sample size.
• Wait until the minimum sample size have run
through the test.
• Calculate if the differences between versions
are statistical significant.

48. WHICH METRIC
TO FOCUS ON?

49. start with
WHY?
How great leaders inspire action: bit.ly/OX0t5a

50. follow with
WHAT? HOW?
Metrics
Tools, Persons,…

51. THE ABCD FRAMEWORK
… HOWTO ANSWER “WHAT”

52. ACQUISITION

53. BEHAVIOUR

54. CONVERSION

55. DIMENSION & DATASOURCE

56. A B C
dimension
datasource
Why?
visit easyabcd.com for more information and analysis tools
acquisition behaviour conversion
easy|A|B|C|D|
Audience

57. THE AARRR FRAMEWORK

58. Acquisition
Activation
Retention
Referral
Revenue
Customer Intent
Fulfilment of Customer Intent
How to users find you?
Do users have a great first experience?
Do users come back?
Do users tell others?
How do you make money?

59. ONE METRICTHAT MATTERS

60. Stickiness Virality Price
Approach
Math that
matters
Keep people
coming back
Gest customers
faster than you
lose them
Make people
invite friends
How many they
tell, how fast
they tell them
Spend revenue
getting customers
Customers are
worth more than
they cost to get
THREE ENGINES
AARRR
Metric
Retention Referral Revenue

61. Facebook on early days
had only 150k users,
little revenue and many
superior competitors
…but 75% of users visit
one or more times per
day.
And within one month
of launching on a new
campus, could acquire
90% of the students.

62. COHORT ANALYSIS

63. • Cohort analysis is a subset of behavioural
analytics that takes the data from a product
and rather than looking at all users as one unit,
it breaks them into related groups for analysis.
• These related groups, or cohorts, usually share
common characteristics or experiences within
a defined timespan.

64. CohortTrend
LongitudinalTrendOnboardingTrend
Percentage of Monthly Active Users (MAU) each week for 12 weeks

65. DON’T FORGET
THE CONTEXT

66. Weather, Day of the week, Calendar Day, etc
influence the User Behaviour and so the Metrics.

67. THANKYOU! 😊