Moe Kiss

1. Analysis of Competing
Hypotheses (ACH)
Moe Kiss - Product Analyst / Digital Analytics
@moemkiss #emetrics #womeninanalytics

2. Thank you!

4. How to understand an Aussie
Abbreviate everything.....

5. How to understand an Aussie
Elongate
your vowels

6. How to understand an Aussie
Skip whole letters
when you speak
MELBOURNE

8. Defence

9. Analyst at
Australia’s largest
online fashion
retailer

10. Capital of Australia = Canberra

11. Analysis of Competing Hypotheses

12. We all want to be good analysts

13. There’s a
problem

14. Why do we think what
we think?

15. Heuristics

16. Anchoring
& adjustment

17. Representativeness

18. Availability

20. Familiarity

21. Escalation of
commitment

22. Last data point

23. Personal
characteristics

24. When we speak of improving the mind we
are usually referring to the acquisition of
information or knowledge….not to the actual
functioning of the mind. We spend little
time monitoring our own thinking and
comparing it with a more sophisticated ideal.
James L. Adams, Conceptual Blockbusting: A Guide to Better Ideas (New
York: W.W. Norton, second edition, 1980), p. 3.

25. There’s
a test….

27. How many could
you remember?

30. Perception bias

33. Drawings from Gerald Fisher title “Man” and “Girl”

34. Mindsets form quickly
but are hard to change...

35. This picture was originally published in Puck magazine in
1915 as a cartoon entitled “My Wife and My
Mother-in-Law.”

36. Shifting perspectives
is hard

37. The perception pitfalls
1. We perceive what we expect to see

38. The perception pitfalls
1. We perceive what we expect to see
2. Mindsets do not see gradual change

39. The perception pitfalls
1. We perceive what we expect to see
2. Mindsets do not see gradual change
3. Shifting perspectives is difficult

40. Biases and mindsets
are inevitable
TAXES

41. Analysts should be self-conscious about
their reasoning processes. They should
think about how they make judgments
and reach conclusions, not just about the
judgments and conclusions themselves.
- Richard Heuer, The Psychology of Intelligence Analysis

42. Throwing toys out of the cot

43. I think that we should
spend more on social
instead of email

44. Let’s prove what we
already think is true

45. We all want to be good analysts

46. Why Moe
ACH

47. Disprove
something

48. Methodical
and hard to
miss a step

49. Little room for
natural biases

50. Difficult to get
side tracked by
data or ideas

51. Useful when there is
missing information

52. Hypothesis

53. ...the term hypothesis [is used] in
its broadest sense as a potential
explanation or conclusion that is
to be tested by collecting and
presenting evidence
Richard Heuer, Psychology of Intelligence Analysis, pg.32

54. ACH
Hypothesis 1 Hypothesis 2 Hypothesis 3 Hypothesis 4
Evidence 1 - + + -
Evidence 2 + + + -
Evidence 3 - + + -
Evidence 4 + + + NA
Evidence 5 - + + +

55. Will streamlining the iOS app checkout flow (the same as the
Android app) improve conversion rate on the iOS app?

56. Step 1
Brainstorm

57. Get a bunch of smart
people in a room to
brainstorm
hypotheses

59. be MECE

60. Hypotheses
1. Changing the UX will reduce CR as iOS users will be
unfamiliar with the new checkout

61. Hypotheses
1. Changing the UX will reduce CR as iOS users will be
unfamiliar with the new checkout
2. Changing the UX will immediately increase CR on the iOS
app as it will remove friction points (the design is more
streamlined)

62. Hypotheses
1. Changing the UX will reduce CR as iOS users will be
unfamiliar with the new checkout
2. Changing the UX will immediately increase CR on the iOS
app as it will remove friction points (the design is more
streamlined)
3. Initially the CR will decline, as users will be unfamiliar with
the new checkout, but once accustomed to the new UX, the CR
will improve due to reduced friction points

63. Hypotheses
1. Changing the UX will reduce CR as iOS users will be
unfamiliar with the new checkout
2. Changing the UX will immediately increase CR on the iOS
app as it will remove friction points (the design is more
streamlined)
3. Initially the CR will decline, as users will be unfamiliar with
the new checkout, but once accustomed to the new UX, the CR
will improve due to reduced friction points
4. Changing the UX will have no impact on iOS app CR

64. Step 2
Collect data

65. Collect the data you need
to prove and disprove
your hypotheses

66. Data becomes evidence
and arguments
(so make sure you can get it!)

67. Evidence
1. Android app has a higher CR than the iOS app (more than double).

68. Evidence
1. Android app has a higher CR than the iOS app (more than double).
2. Android app users have a higher ABV than iOS app users (+7%).

69. Evidence
1. Android app has a higher CR than the iOS app (more than double).
2. Android app users have a higher ABV than iOS app users (+7%).
3. The iOS app has a higher abandonment rate from all steps of the checkout
than the Android app.

70. Evidence
1. Android app has a higher CR than the iOS app (more than double).
2. Android app users have a higher ABV than iOS app users (+7%).
3. The iOS app has a higher abandonment rate from all steps of the checkout
than the Android app.
4. Customer comments indicate iOS app users love the checkout and praise
it's functionality (27% of comments).

71. Evidence
1. Android app has a higher CR than the iOS app (more than double).
2. Android app users have a higher ABV than iOS app users (+7%).
3. The iOS app has a higher abandonment rate from all steps of the checkout
than the Android app.
4. Customer comments indicate iOS app users love the checkout and praise
it's functionality (27% comments).
5. Our customer survey ranks ease of checkout on the iOS app as 3/5
(compared to 3.6 for the mobile site and 3.7 for desktop).

72. Evidence
1. Android app has a higher CR than the iOS app (more than double).
2. Android app users have a higher ABV than iOS app users (+7%).
3. The iOS app has a higher abandonment rate from all steps of the checkout
than the Android app.
4. Customer comments indicate iOS app users love the checkout and praise
it's functionality (27% comments).
5. Our customer survey ranks ease of checkout on the iOS app as 3/5
(compared to 3.6 for the mobile site and 3.7 for desktop).
6. In the sort/refine UX redesign it took users some time to become familiar
with the changes (data from customer interviews).

73. Step 3 - Prepare a matrix
Hypotheses
Hypothesis 1 Hypothesis 2 Hypothesis 3 Hypothesis 4

74. Step 3 - Prepare a matrix
Hypothesis 1 Hypothesis 2 Hypothesis 3 Hypothesis 4
Evidence 1
Evidence 2
Evidence 3
Evidence 4
Evidence 5
Evidence

75. Changing UX will
reduce CR as iOS
users will be
unfamiliar with the
new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but
once accustomed, CR
will improve
Changing the UX will
have no impact on iOS
app CR
Android app has a higher CR
than the iOS app (+ double)
Android has a higher ABV than
iOS app (+7%)
iOS has higher abandonment
from checkout than Android
27% comments of praise iOS
checkout functionality
Survey ranks ease of checkout
on iOS app as 3/5
Takes users time to become
familiar with the changes

76. Analyse the evidence/argument (work across)
Changing UX will
reduce CR as iOS users
will be unfamiliar with
the new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but once
accustomed, CR will
improve
Changing the UX will
have no impact on iOS
app CR
Android app has a higher CR
than the iOS app (+ double)
Android has a higher ABV
than iOS app (+7%)
iOS has higher abandonment
from checkout than Android
27% comments praise iOS
checkout functionality
Survey ranks ease of checkout
on iOS app as 3/5
Takes users time to become
familiar with the changes

77. Analyse the evidence/argument (work across)
Changing UX will
reduce CR as iOS users
will be unfamiliar with
the new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but once
accustomed, CR will
improve
Changing the UX will
have no impact on iOS
app CR
Android app has a higher CR
than the iOS app (+ double) -
Android has a higher ABV
than iOS app (+7%)
iOS has higher abandonment
from checkout than Android
27% comments praise iOS
checkout functionality
Survey ranks ease of checkout
on iOS app as 3/5
Takes users time to become
familiar with the changes

78. Analyse the evidence/argument (work across)
Changing UX will
reduce CR as iOS users
will be unfamiliar with
the new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but once
accustomed, CR will
improve
Changing the UX will
have no impact on iOS
app CR
Android app has a higher CR
than the iOS app (+ double) - +
Android has a higher ABV
than iOS app (+7%)
iOS has higher abandonment
from checkout than Android
27% comments praise iOS
checkout functionality
Survey ranks ease of checkout
on iOS app as 3/5
Takes users time to become
familiar with the changes

79. Analyse the evidence/argument (work across)
Changing UX will
reduce CR as iOS users
will be unfamiliar with
the new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but once
accustomed, CR will
improve
Changing the UX will
have no impact on iOS
app CR
Android app has a higher CR
than the iOS app (+ double) - + +
Android has a higher ABV
than iOS app (+7%)
iOS has higher abandonment
from checkout than Android
27% comments praise iOS
checkout functionality
Survey ranks ease of checkout
on iOS app as 3/5
Takes users time to become
familiar with the changes

80. Analyse the evidence/argument (work across)
Changing UX will
reduce CR as iOS users
will be unfamiliar with
the new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but once
accustomed, CR will
improve
Changing the UX will
have no impact on iOS
app CR
Android app has a higher CR
than the iOS app (+ double) - + + -
Android has a higher ABV
than iOS app (+7%)
iOS has higher abandonment
from checkout than Android
27% comments praise iOS
checkout functionality
Survey ranks ease of checkout
on iOS app as 3/5
Takes users time to become
familiar with the changes

81. Analyse the evidence/argument (work across)
Changing UX will
reduce CR as iOS users
will be unfamiliar with
the new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but once
accustomed, CR will
improve
Changing the UX will
have no impact on iOS
app CR
Android app has a higher CR
than the iOS app (+ double) - + + -
Android has a higher ABV
than iOS app (+7%)
iOS has higher abandonment
from checkout than Android
27% comments praise iOS
checkout functionality
Survey ranks ease of checkout
on iOS app as 3/5
Takes users time to become
familiar with the changes

82. Changing UX will
reduce CR as iOS
users will be
unfamiliar with
the new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but
once accustomed, CR
will improve
Changing the UX will
have no impact on
iOS app CR
Android app has a higher CR
than the iOS app (+ double) - + + -
Android has a higher ABV than
iOS app (+7%) NA NA NA NA
iOS has higher abandonment
from checkout than Android - + + -
27% comments praise iOS
checkout functionality + + + NA
Survey ranks ease of checkout
on iOS app as 3/5 - + + +
Takes users time to become
familiar with the changes + - + NA

83. Step 4
Analyse the diagnosticity

84. Which pieces of
evidence are key?

85. Remove evidence with
no diagnostic value

86. Changing UX will
reduce CR as iOS users
will be unfamiliar with
the new checkout
Changing UX will
immediately
increase CR as it
will remove friction
Initially CR will
decline, as users will
be unfamiliar, but
once accustomed, CR
will improve
Changing the UX will
have no impact on iOS
app CR
Android app has a higher CR
than the iOS app (+ double) - + + -
Android has a higher ABV
than iOS app (+7%) NA NA NA NA
iOS has higher abandonment
from checkout than Android - + + -
27% comments praise iOS
checkout functionality + + + NA
Survey ranks ease of checkout
on iOS app as 3/5 - + + +
Takes users time to become
familiar with the changes + - + NA

87. Changing UX will
reduce CR as iOS users
will be unfamiliar with
the new checkout
Changing UX will
immediately
increase CR as it
will remove friction
Initially CR will
decline, as users will
be unfamiliar, but
once accustomed, CR
will improve
Changing the UX will
have no impact on iOS
app CR
Android app has a higher CR
than the iOS app (+ double) - + + -
Android has a higher ABV
than iOS app (+7%) NA NA NA NA
iOS has higher abandonment
from checkout than Android - + + -
27% comments praise iOS
checkout functionality + + + NA
Survey ranks ease of checkout
on iOS app as 3/5 - + + +
Takes users time to become
familiar with the changes + - + NA

88. Reconsider / reword the
hypotheses if necessary

89. Step 5
Disprove your
hypotheses

90. You can have all the evidence in the
world to back a hypothesis - but if
there is one piece of evidence that
disproves it - you can reject it

91. Analyse the hypotheses (work down)
Changing UX will
reduce CR as iOS
users will be
unfamiliar with the
new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but
once accustomed, CR
will improve
Changing the UX will
have no impact on
iOS app CR
Android app has a higher
CR than the iOS app
(+ double)
- + + -
iOS has higher
abandonment from
checkout than Android
- + + -
Survey ranks ease of
checkout on iOS app as 3/5 - + + +
Takes users time to
become familiar with the
changes
+ - + NA

92. Analyse the hypotheses (work down)
Changing UX will
reduce CR as iOS
users will be
unfamiliar with the
new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but
once accustomed, CR
will improve
Changing the UX will
have no impact on
iOS app CR
Android app has a higher
CR than the iOS app
(+ double)
- + + -
iOS has higher
abandonment from
checkout than Android
- + + -
Survey ranks ease of
checkout on iOS app as 3/5 - + + +
Takes users time to
become familiar with the
changes
+ - + NA

93. Analyse the hypotheses (work down)
Changing UX will
reduce CR as iOS
users will be
unfamiliar with the
new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but
once accustomed, CR
will improve
Changing the UX will
have no impact on
iOS app CR
Android app has a higher
CR than the iOS app
(+ double)
- + + -
iOS has higher
abandonment from
checkout than Android
- + + -
Survey ranks ease of
checkout on iOS app as 3/5 - + + +
Takes users time to
become familiar with the
changes
+ - + NA

94. Analyse the hypotheses (work down)
Changing UX will
reduce CR as iOS
users will be
unfamiliar with the
new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but
once accustomed, CR
will improve
Changing the UX will
have no impact on
iOS app CR
Android app has a higher
CR than the iOS app
(+ double)
- + + -
iOS has higher
abandonment from
checkout than Android
- + + -
Survey ranks ease of
checkout on iOS app as 3/5 - + + +
Takes users time to
become familiar with the
changes
+ - + NA

95. Analyse the hypotheses (work down)
Changing UX will
reduce CR as iOS
users will be
unfamiliar with the
new checkout
Changing UX will
immediately increase
CR as it will remove
friction
Initially CR will
decline, as users will
be unfamiliar, but
once accustomed, CR
will improve
Changing the UX will
have no impact on
iOS app CR
Android app has a higher
CR than the iOS app
(+ double)
- + + -
iOS has higher
abandonment from
checkout than Android
- + + -
Survey ranks ease of
checkout on iOS app as 3/5 - + + +
Takes users time to
become familiar with the
changes
+ - + NA

96. Step 6
Sensitivity analysis

97. Consider if some of
your evidence were
wrong, misleading or
open to a different
interpretation

98. Step 7
Report your
findings
The best bit!

99. Analytics is about telling
a story with data
(not number crunching)

100. 1. Brainstorm hypotheses

101. 1. Brainstorm hypotheses
2. Collect data

102. 1. Brainstorm hypotheses
2. Collect data
3. Prepare the matrix

103. 1. Brainstorm hypotheses
2. Collect data
3. Prepare the matrix
4. Analyse the diagnosticity

104. 1. Brainstorm hypotheses
2. Collect data
3. Prepare the matrix
4. Analyse the diagnosticity
5. Disprove your hypotheses

105. 1. Brainstorm hypotheses
2. Collect data
3. Prepare the matrix
4. Analyse the diagnosticity
5. Disprove your hypotheses
6. Sensitivity analysis

106. 1. Brainstorm hypotheses
2. Collect data
3. Prepare the matrix
4. Analyse the diagnosticity
5. Disprove your hypotheses
6. Sensitivity analysis
7. Report your findings

107. It takes time

108. The cheat’s version...
Hypotheses Supports Disproves
Changing the layout will reduce CR as iOS users will be
unfamiliar with the new checkout
Changing the layout will immediately increase CR on the
iOS app as it will remove friction points (the design is more
streamlined)
Initially the CR will decline, as users will be unfamiliar with
the new checkout, but once accustomed to the new layout,
the CR will improve due to reduced friction points
Changing the layout will have no impact on iOS app CR

109. If you want to know more about working
@ ask me in the break
We currently have two analytics positions:
1. Director of Analytics
2. Digital Analytics Specialist
Questions?