Metrics, Analytics &                             ExperimentsSaturday, 16 February 13
OutcomesSaturday, 16 February 13
Outcomes                     • TheorySaturday, 16 February 13
Outcomes                     • Theory                      • What are analytics & experimentation?Saturday, 16 February 13
Outcomes                     • Theory                      • What are analytics & experimentation?                      • ...
Outcomes                     • Theory                      • What are analytics & experimentation?                      • ...
Outcomes                     • Theory                      • What are analytics & experimentation?                      • ...
Outcomes                     • Theory                      • What are analytics & experimentation?                      • ...
Outcomes                     • Theory                      • What are analytics & experimentation?                      • ...
Sim on
 Cast                                     @simo ncastSaturday, 16 February 13
Saturday, 16 February 13
Saturday, 16 February 13
Saturday, 16 February 13
Saturday, 16 February 13
Saturday, 16 February 13
Saturday, 16 February 13
Analytics  MetricsSaturday, 16 February 13
Saturday, 16 February 13
“Saturday, 16 February 13                           ”
“        Measures the state of                               the productSaturday, 16 February 13                          ...
“What you don’t                           measure you can’t                               improve!Saturday, 16 February 13...
ConceptsSaturday, 16 February 13
Concepts                     • Data pointsSaturday, 16 February 13
Concepts                     • Data points                     • SegmentationSaturday, 16 February 13
Concepts                     • Data points                     • Segmentation                     • FunnelsSaturday, 16 Fe...
Concepts                     • Data points                     • Segmentation                     • Funnels               ...
Data PointsSaturday, 16 February 13
Data Points                     • These are the user actions that are                           measuredSaturday, 16 Febru...
Data Points                     • These are the user actions that are                           measured                  ...
Data Points: ExampleSaturday, 16 February 13
SegmentationSaturday, 16 February 13
Segmentation       • Grouping users together               based on some characteristic               of the usersSaturday...
Segmentation       • Grouping users together               based on some characteristic               of the users       •...
Segmentation       • Grouping users together               based on some characteristic               of the users       •...
Segmentation: ExampleSaturday, 16 February 13
FunnelsSaturday, 16 February 13
Funnels         • Flow: series of steps users go                 throughSaturday, 16 February 13
Funnels         • Flow: series of steps users go                 through         • Funnels measure flowsSaturday, 16 Februa...
Funnels         • Flow: series of steps users go                 through         • Funnels measure flows         • Shows wh...
Funnel: ExampleSaturday, 16 February 13
CohortsSaturday, 16 February 13
Cohorts                     • Grouping by point in timeSaturday, 16 February 13
Cohorts                     • Grouping by point in time                     • How does behaviour change over              ...
Cohorts                     • Grouping by point in time                     • How does behaviour change over              ...
Cohorts                     • Grouping by point in time                     • How does behaviour change over              ...
Cohorts: ExampleSaturday, 16 February 13
Saturday, 16 February 13
Start with a planSaturday, 16 February 13
Start with a planSaturday, 16 February 13
Saturday, 16 February 13
Product            VisionSaturday, 16 February 13
Product            VisionSaturday, 16 February 13
Product                           KPIs            VisionSaturday, 16 February 13
Product                           KPIs            VisionSaturday, 16 February 13
Product                           KPIs   Metrics            VisionSaturday, 16 February 13
Product                           KPIs   Metrics            VisionSaturday, 16 February 13
Product                           KPIs   Metrics   Funnel            VisionSaturday, 16 February 13
Product                           KPIs   Metrics   Funnel            VisionSaturday, 16 February 13
Product                           KPIs   Metrics   Funnel            Vision                               Data            ...
Vision                       “   What is the product                            suppose to do?Saturday, 16 February 13    ...
KPIsSaturday, 16 February 13
KPIs                     • The KPIs (key performance indicators) are                           derived from the product vi...
KPIs                     • The KPIs (key performance indicators) are                           derived from the product vi...
KPI: ExampleSaturday, 16 February 13
KPI: Example                     • Dollar value of checkoutsSaturday, 16 February 13
KPI: Example                     • Dollar value of checkouts                     • Number of documents storedSaturday, 16 ...
Metrics                     • Represents the KPIs                     • What can be manipulated to influence the           ...
MetricsSaturday, 16 February 13
Metrics                     • QuantitativeSaturday, 16 February 13
Metrics                     • Quantitative                     • ComparativeSaturday, 16 February 13
Metrics                     • Quantitative                     • Comparative                     • ActionableSaturday, 16 ...
Metrics: Example                     • Average checkout value                     • New documents per daySaturday, 16 Febr...
Flows  Funnels                     • Focus on the flows that directly effect the                           metrics         ...
Funnel: ExampleSaturday, 16 February 13
Data Points                     • What makes up metrics                     • What makes up funnels                     • ...
Data Points: Example                     • Number of new visitors                     • Number who register               ...
ImplementationSaturday, 16 February 13
Implementation                     • Record the data points being collectedSaturday, 16 February 13
Implementation                     • Record the data points being collected                     • In-house versus 3rd part...
Implementation: ToolsSaturday, 16 February 13
Implementation: ToolsSaturday, 16 February 13
Implementation: ToolsSaturday, 16 February 13
Implementation: ToolsSaturday, 16 February 13
Implementation: ToolsSaturday, 16 February 13
ExperimentationSaturday, 16 February 13
Why  What?Saturday, 16 February 13
Why  What?                     • Apply scientific method to get a better                           productSaturday, 16 Febr...
Why  What?                     • Apply scientific method to get a better                           product                 ...
Why  What?                     • Apply scientific method to get a better                           product                 ...
How?Saturday, 16 February 13
How?Saturday, 16 February 13
Saturday, 16 February 13
Saturday, 16 February 13
The planning processSaturday, 16 February 13
The planning process                     • QuestionSaturday, 16 February 13
The planning process                     • Question                     • HypothesisSaturday, 16 February 13
The planning process                     • Question                     • Hypothesis                     • Experiment to t...
Saturday, 16 February 13
Question                     • Related to KPIs and assumptions about the                           product                ...
Question: Example                           Why is the conversion rate on the landing                           page 10% a...
HypothesisSaturday, 16 February 13
Hypothesis                     • Proposed reason or explanation for a                           phenomenaSaturday, 16 Febr...
Hypothesis                     • Proposed reason or explanation for a                           phenomena                 ...
Hypothesis                     • Proposed reason or explanation for a                           phenomena                 ...
Hypothesis: Example                     • The call to action button should be red                     • The message is not...
Hypothesis: StructureSaturday, 16 February 13
Hypothesis: Structure                     • IF I water the plants THEN the plants will                           growSatur...
Hypothesis: Structure                     • IF I water the plants THEN the plants will                           grow     ...
Hypothesis: ExampleSaturday, 16 February 13
Hypothesis: Example                     • IF the call to action button is red THEN                           the number of...
Hypothesis: Example                     • IF the call to action button is red THEN                           the number of...
Hypothesis: Example                     • IF the call to action button is red THEN                           the number of...
ExperimentSaturday, 16 February 13
Experiment                     • TypeSaturday, 16 February 13
Experiment                     • Type                     • ControlSaturday, 16 February 13
Experiment                     • Type                     • Control                     • Variant(s)Saturday, 16 February 13
Experiment                     • Type                     • Control                     • Variant(s)                     •...
Variants                     • Flow from the hypothesis                     • They are the independent variable in the    ...
Experiment: Example                           IF the call to action button is red THEN the                           numbe...
Goals                     • Flow from hypothesis                     • The data point that is expected to changeSaturday, ...
Experiment: Example                           IF the call to action button is red THEN                           the numbe...
TypesSaturday, 16 February 13
Types                     • A/B TestSaturday, 16 February 13
Types                     • A/B Test                     • Split URLSaturday, 16 February 13
Types                     • A/B Test                     • Split URL                     • MultivariateSaturday, 16 Februa...
A/B                      A (control)         B (variant)                                    vsSaturday, 16 February 13
Split URLSaturday, 16 February 13
MulitvariateSaturday, 16 February 13
Saturday, 16 February 13
Running ExperimentSaturday, 16 February 13
Running Experiment                     • Record the experiments  resultsSaturday, 16 February 13
Running Experiment                     • Record the experiments  results                     • Segment the trafficSaturday,...
Running Experiment                     • Record the experiments  results                     • Segment the traffic         ...
Running Experiment                     • Record the experiments  results                     • Segment the traffic         ...
Running Experiment                     • Record the experiments  results                     • Segment the traffic         ...
Running Experiment                     • Record the experiments  results                     • Segment the traffic         ...
Running Experiment                     • Record the experiments  results                     • Segment the traffic         ...
Closing the LoopSaturday, 16 February 13
Closing the Loop                     • What do these results mean for the                           product  development p...
Closing the Loop                     • What do these results mean for the                           product  development p...
Closing the Loop                     • What do these results mean for the                           product  development p...
Closing the Loop                     • What do these results mean for the                           product  development p...
ToolsSaturday, 16 February 13
ToolsSaturday, 16 February 13
ToolsSaturday, 16 February 13
ToolsSaturday, 16 February 13
CaveatSaturday, 16 February 13
CaveatSaturday, 16 February 13
Upcoming SlideShare
Loading in...5
×

Ceed product workshop 2012

169

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
169
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Ceed product workshop 2012

  1. 1. Metrics, Analytics & ExperimentsSaturday, 16 February 13
  2. 2. OutcomesSaturday, 16 February 13
  3. 3. Outcomes • TheorySaturday, 16 February 13
  4. 4. Outcomes • Theory • What are analytics & experimentation?Saturday, 16 February 13
  5. 5. Outcomes • Theory • What are analytics & experimentation? • Why are they importantSaturday, 16 February 13
  6. 6. Outcomes • Theory • What are analytics & experimentation? • Why are they important • How do you do them?Saturday, 16 February 13
  7. 7. Outcomes • Theory • What are analytics & experimentation? • Why are they important • How do you do them? • PracticalSaturday, 16 February 13
  8. 8. Outcomes • Theory • What are analytics & experimentation? • Why are they important • How do you do them? • Practical • Analytics planSaturday, 16 February 13
  9. 9. Outcomes • Theory • What are analytics & experimentation? • Why are they important • How do you do them? • Practical • Analytics plan • Experiment planSaturday, 16 February 13
  10. 10. Sim on
  11. 11.  Cast @simo ncastSaturday, 16 February 13
  12. 12. Saturday, 16 February 13
  13. 13. Saturday, 16 February 13
  14. 14. Saturday, 16 February 13
  15. 15. Saturday, 16 February 13
  16. 16. Saturday, 16 February 13
  17. 17. Saturday, 16 February 13
  18. 18. Analytics MetricsSaturday, 16 February 13
  19. 19. Saturday, 16 February 13
  20. 20. “Saturday, 16 February 13 ”
  21. 21. “ Measures the state of the productSaturday, 16 February 13 ”
  22. 22. “What you don’t measure you can’t improve!Saturday, 16 February 13 ”
  23. 23. ConceptsSaturday, 16 February 13
  24. 24. Concepts • Data pointsSaturday, 16 February 13
  25. 25. Concepts • Data points • SegmentationSaturday, 16 February 13
  26. 26. Concepts • Data points • Segmentation • FunnelsSaturday, 16 February 13
  27. 27. Concepts • Data points • Segmentation • Funnels • CohortsSaturday, 16 February 13
  28. 28. Data PointsSaturday, 16 February 13
  29. 29. Data Points • These are the user actions that are measuredSaturday, 16 February 13
  30. 30. Data Points • These are the user actions that are measured • Includes a datetime stampSaturday, 16 February 13
  31. 31. Data Points: ExampleSaturday, 16 February 13
  32. 32. SegmentationSaturday, 16 February 13
  33. 33. Segmentation • Grouping users together based on some characteristic of the usersSaturday, 16 February 13
  34. 34. Segmentation • Grouping users together based on some characteristic of the users • Shows patterns otherwise hidden by noise averagesSaturday, 16 February 13
  35. 35. Segmentation • Grouping users together based on some characteristic of the users • Shows patterns otherwise hidden by noise averages • Focus on those who are most important to youSaturday, 16 February 13
  36. 36. Segmentation: ExampleSaturday, 16 February 13
  37. 37. FunnelsSaturday, 16 February 13
  38. 38. Funnels • Flow: series of steps users go throughSaturday, 16 February 13
  39. 39. Funnels • Flow: series of steps users go through • Funnels measure flowsSaturday, 16 February 13
  40. 40. Funnels • Flow: series of steps users go through • Funnels measure flows • Shows where the leakage is in a flowSaturday, 16 February 13
  41. 41. Funnel: ExampleSaturday, 16 February 13
  42. 42. CohortsSaturday, 16 February 13
  43. 43. Cohorts • Grouping by point in timeSaturday, 16 February 13
  44. 44. Cohorts • Grouping by point in time • How does behaviour change over time?Saturday, 16 February 13
  45. 45. Cohorts • Grouping by point in time • How does behaviour change over time? • Relative or absoluteSaturday, 16 February 13
  46. 46. Cohorts • Grouping by point in time • How does behaviour change over time? • Relative or absolute • Comparison between cohortsSaturday, 16 February 13
  47. 47. Cohorts: ExampleSaturday, 16 February 13
  48. 48. Saturday, 16 February 13
  49. 49. Start with a planSaturday, 16 February 13
  50. 50. Start with a planSaturday, 16 February 13
  51. 51. Saturday, 16 February 13
  52. 52. Product VisionSaturday, 16 February 13
  53. 53. Product VisionSaturday, 16 February 13
  54. 54. Product KPIs VisionSaturday, 16 February 13
  55. 55. Product KPIs VisionSaturday, 16 February 13
  56. 56. Product KPIs Metrics VisionSaturday, 16 February 13
  57. 57. Product KPIs Metrics VisionSaturday, 16 February 13
  58. 58. Product KPIs Metrics Funnel VisionSaturday, 16 February 13
  59. 59. Product KPIs Metrics Funnel VisionSaturday, 16 February 13
  60. 60. Product KPIs Metrics Funnel Vision Data pointsSaturday, 16 February 13
  61. 61. Vision “ What is the product suppose to do?Saturday, 16 February 13 ”
  62. 62. KPIsSaturday, 16 February 13
  63. 63. KPIs • The KPIs (key performance indicators) are derived from the product visionSaturday, 16 February 13
  64. 64. KPIs • The KPIs (key performance indicators) are derived from the product vision • Need to take into account at what stage your product isSaturday, 16 February 13
  65. 65. KPI: ExampleSaturday, 16 February 13
  66. 66. KPI: Example • Dollar value of checkoutsSaturday, 16 February 13
  67. 67. KPI: Example • Dollar value of checkouts • Number of documents storedSaturday, 16 February 13
  68. 68. Metrics • Represents the KPIs • What can be manipulated to influence the KPIs • Calculated from multiple data pointsSaturday, 16 February 13
  69. 69. MetricsSaturday, 16 February 13
  70. 70. Metrics • QuantitativeSaturday, 16 February 13
  71. 71. Metrics • Quantitative • ComparativeSaturday, 16 February 13
  72. 72. Metrics • Quantitative • Comparative • ActionableSaturday, 16 February 13
  73. 73. Metrics: Example • Average checkout value • New documents per daySaturday, 16 February 13
  74. 74. Flows Funnels • Focus on the flows that directly effect the metrics • What data represents each step in the flow?Saturday, 16 February 13
  75. 75. Funnel: ExampleSaturday, 16 February 13
  76. 76. Data Points • What makes up metrics • What makes up funnels • Achievable • SegmentsSaturday, 16 February 13
  77. 77. Data Points: Example • Number of new visitors • Number who register • Number who reach each step of the • Value of each checkout chart • Number of checkouts • Number of documents uploadedSaturday, 16 February 13
  78. 78. ImplementationSaturday, 16 February 13
  79. 79. Implementation • Record the data points being collectedSaturday, 16 February 13
  80. 80. Implementation • Record the data points being collected • In-house versus 3rd partySaturday, 16 February 13
  81. 81. Implementation: ToolsSaturday, 16 February 13
  82. 82. Implementation: ToolsSaturday, 16 February 13
  83. 83. Implementation: ToolsSaturday, 16 February 13
  84. 84. Implementation: ToolsSaturday, 16 February 13
  85. 85. Implementation: ToolsSaturday, 16 February 13
  86. 86. ExperimentationSaturday, 16 February 13
  87. 87. Why What?Saturday, 16 February 13
  88. 88. Why What? • Apply scientific method to get a better productSaturday, 16 February 13
  89. 89. Why What? • Apply scientific method to get a better product • Test different assumptions made about the productSaturday, 16 February 13
  90. 90. Why What? • Apply scientific method to get a better product • Test different assumptions made about the product • Test different variations to see what effects the KPIsSaturday, 16 February 13
  91. 91. How?Saturday, 16 February 13
  92. 92. How?Saturday, 16 February 13
  93. 93. Saturday, 16 February 13
  94. 94. Saturday, 16 February 13
  95. 95. The planning processSaturday, 16 February 13
  96. 96. The planning process • QuestionSaturday, 16 February 13
  97. 97. The planning process • Question • HypothesisSaturday, 16 February 13
  98. 98. The planning process • Question • Hypothesis • Experiment to test hypothesisSaturday, 16 February 13
  99. 99. Saturday, 16 February 13
  100. 100. Question • Related to KPIs and assumptions about the product • How to increase y? • Why is x so? • Is z assumption true?Saturday, 16 February 13
  101. 101. Question: Example Why is the conversion rate on the landing page 10% and not 30%?Saturday, 16 February 13
  102. 102. HypothesisSaturday, 16 February 13
  103. 103. Hypothesis • Proposed reason or explanation for a phenomenaSaturday, 16 February 13
  104. 104. Hypothesis • Proposed reason or explanation for a phenomena • An answer to the question or explanation of the questionSaturday, 16 February 13
  105. 105. Hypothesis • Proposed reason or explanation for a phenomena • An answer to the question or explanation of the question • Testable with independent variable that can be controlled and a dependent variable that can be measuredSaturday, 16 February 13
  106. 106. Hypothesis: Example • The call to action button should be red • The message is not clear about the value of registering • There are too many different call-to-actions on the pageSaturday, 16 February 13
  107. 107. Hypothesis: StructureSaturday, 16 February 13
  108. 108. Hypothesis: Structure • IF I water the plants THEN the plants will growSaturday, 16 February 13
  109. 109. Hypothesis: Structure • IF I water the plants THEN the plants will grow • IF I don’t water the plants THEN the plants will not growSaturday, 16 February 13
  110. 110. Hypothesis: ExampleSaturday, 16 February 13
  111. 111. Hypothesis: Example • IF the call to action button is red THEN the number of people registering will increaseSaturday, 16 February 13
  112. 112. Hypothesis: Example • IF the call to action button is red THEN the number of people registering will increase • IF we change the copy explaining the value of registering THEN the number of people registering will go upSaturday, 16 February 13
  113. 113. Hypothesis: Example • IF the call to action button is red THEN the number of people registering will increase • IF we change the copy explaining the value of registering THEN the number of people registering will go up • IF we remove all but one call-to-action on the page THEN the number of people registering will increaseSaturday, 16 February 13
  114. 114. ExperimentSaturday, 16 February 13
  115. 115. Experiment • TypeSaturday, 16 February 13
  116. 116. Experiment • Type • ControlSaturday, 16 February 13
  117. 117. Experiment • Type • Control • Variant(s)Saturday, 16 February 13
  118. 118. Experiment • Type • Control • Variant(s) • GoalSaturday, 16 February 13
  119. 119. Variants • Flow from the hypothesis • They are the independent variable in the hypothesisSaturday, 16 February 13
  120. 120. Experiment: Example IF the call to action button is red THEN the number of people registering will increaseSaturday, 16 February 13
  121. 121. Goals • Flow from hypothesis • The data point that is expected to changeSaturday, 16 February 13
  122. 122. Experiment: Example IF the call to action button is red THEN the number of people registering will increaseSaturday, 16 February 13
  123. 123. TypesSaturday, 16 February 13
  124. 124. Types • A/B TestSaturday, 16 February 13
  125. 125. Types • A/B Test • Split URLSaturday, 16 February 13
  126. 126. Types • A/B Test • Split URL • MultivariateSaturday, 16 February 13
  127. 127. A/B A (control) B (variant) vsSaturday, 16 February 13
  128. 128. Split URLSaturday, 16 February 13
  129. 129. MulitvariateSaturday, 16 February 13
  130. 130. Saturday, 16 February 13
  131. 131. Running ExperimentSaturday, 16 February 13
  132. 132. Running Experiment • Record the experiments resultsSaturday, 16 February 13
  133. 133. Running Experiment • Record the experiments results • Segment the trafficSaturday, 16 February 13
  134. 134. Running Experiment • Record the experiments results • Segment the traffic • Traffic (random split to remove bias)Saturday, 16 February 13
  135. 135. Running Experiment • Record the experiments results • Segment the traffic • Traffic (random split to remove bias) • How long to run?Saturday, 16 February 13
  136. 136. Running Experiment • Record the experiments results • Segment the traffic • Traffic (random split to remove bias) • How long to run? • Who to test?Saturday, 16 February 13
  137. 137. Running Experiment • Record the experiments results • Segment the traffic • Traffic (random split to remove bias) • How long to run? • Who to test? • Test well - don’t take short cutsSaturday, 16 February 13
  138. 138. Running Experiment • Record the experiments results • Segment the traffic • Traffic (random split to remove bias) • How long to run? • Who to test? • Test well - don’t take short cuts • Negative resultSaturday, 16 February 13
  139. 139. Closing the LoopSaturday, 16 February 13
  140. 140. Closing the Loop • What do these results mean for the product development prioritisation?Saturday, 16 February 13
  141. 141. Closing the Loop • What do these results mean for the product development prioritisation? • Why did I get these results? or,Saturday, 16 February 13
  142. 142. Closing the Loop • What do these results mean for the product development prioritisation? • Why did I get these results? or, • Why didn’t I get the results expected?Saturday, 16 February 13
  143. 143. Closing the Loop • What do these results mean for the product development prioritisation? • Why did I get these results? or, • Why didn’t I get the results expected? • Revamp priorisation produce the next set of experimentsSaturday, 16 February 13
  144. 144. ToolsSaturday, 16 February 13
  145. 145. ToolsSaturday, 16 February 13
  146. 146. ToolsSaturday, 16 February 13
  147. 147. ToolsSaturday, 16 February 13
  148. 148. CaveatSaturday, 16 February 13
  149. 149. CaveatSaturday, 16 February 13
  150. 150. PracticeSaturday, 16 February 13

×