SlideShare a Scribd company logo
1 of 100
@michelejkiss
Foundational Psychology Experiments
and why we should know them
Michele Kiss @michelejkiss
Analytics Demystified
@michelejkiss
Like many people in digital analytics, I sort of “fell in to” the industry. I went to the University of Melbourne, and I studied Law
and Psychology. Seems pretty unrelated… But the more analysis I do, the more stakeholders I work with, the more I realize
that the things I learned are directly applicable to the practice of analytics.
@michelejkiss@michelejkiss
Analytics, optimization, etc are all about using quantitative methods to understand PEOPLE, and why we do the stupid stuff
we do
Today we’re going to talk about some classic studies, and why they matter (and how they help us understand human
behaviour)
@michelejkiss@michelejkiss
THE MAGIC NUMBER 7
@michelejkiss@michelejkiss
@michelejkiss@michelejkiss@michelejkiss
Data Visualisation
Those of us working in analytics are pretty familiar with the underlying tenets of data visualization, dashboards, and likely,
with Stephen Few.
“A dashboard is a
visual display
of the most important information
needed to achieve one or more objectives;
consolidated and arranged
on a single screen
so the information can be monitored
at a glance.”
[Stephen Few]
@michelejkiss
Stephen Few defined a dashboard as…
“A dashboard is a
visual display
of the most important information
needed to achieve one or more objectives;
consolidated and arranged
on a single screen
so the information can be monitored
at a glance.”
[Stephen Few]
@michelejkiss
To me, the most important words are “on a single screen”
@michelejkiss
The reason why we need the information to be on one page or screen? The limits of our memory.
@michelejkiss@michelejkiss
[Miller, 1956]
In 1956, George A. Miller published one of the most cited papers in psychology, where he spoke of the magic number 7. For
our purposes here, Miller found that the amount of information that we can retain our working (or short term) memory is
seven, plus or minus two.
@michelejkiss@michelejkiss
R X U W B J P
However, this doesn’t mean that we can only recall 7 +/- 2 individual letters, or numbers (for example.)
The 7 +/- 2 items can be 7 +/- 2 letters, or 7 +/- 2 words, or …
@michelejkiss
Take my hand / Take my whole life too / For I can’t help / Falling in love with you
You may say I’m a dreamer / But I’m not the only one
We are young / Heartache to heartache we stand
I fell into a burning ring of fire / I went down down down and the flames went higher
All you need is love / Love is all you need
Every new beginning / Comes from some other beginning’s end.
It's the end of the world as we know it and I feel fine.
@michelejkiss
… or 7 +/- 2 song lyrics.
We are capable of “chunking” information, so that each piece of information only occupies one “item” within our memory
system.
@michelejkiss@michelejkiss@michelejkiss
whyitmatters
@michelejkiss
Because, back to Stephen Few’s definition, if you are presenting information, you need to work WITHIN the limits of our
perceptual and memory systems. Expecting users to draw connections between two data points six pages apart, or thirty
slides ago, is a recipe for failure.
@michelejkiss@michelejkiss
WHEN THE FACTS DON’T MATTER
@michelejkiss
(Or, why do facts fail to convince us)
@michelejkiss
[Festinger, 1957]
@michelejkiss
The end of the world
In 1957, Leon Festinger studied a Doomsday cult who believed that aliens would rescue them from a coming flood.
Unsurprisingly, no flood (nor aliens) eventuated. However, rather than being proven wrong, the group explained away that
they had been “spared”, and clung even more tightly to their beliefs. In their book, When Prophecy Fails, Festinger (et al)
said:
@michelejkiss
“A man with a conviction is
a hard man to change.
Show him facts or figures and
he questions your sources.
Appeal to logic and he fails to see your point.
@michelejkiss
“Suppose that he is presented with evidence,
unequivocal and undeniable evidence,
that his belief is wrong: what will happen?
@michelejkiss
“The individual will frequently emerge,
not only unshaken, but
even more convinced
of the truth of his beliefs
than ever before.”
@michelejkiss@michelejkiss
Especially if…
This might happen especially if:
• Beliefs are deeply held;
• Action has been taken, that is difficult to undo; (for example, the Doomsday cult had sold all their possessions)
• Believer has social support (the cult continues to reinforce each others’ beliefs)
@michelejkiss
cognitive
dissonance
@michelejkiss
[Festinger, 1957]
Festinger explains this by the theory of cognitive dissonance. This theory tells us that we don’t like the feeling of
inconsistency (for example, of our beliefs and our actions.) We seek to reduce this uncomfortable feeling by justifying our
beliefs, and avoiding information that might further conflict. For example, smokers who know smoking is bad for them may
choose to tell themselves, “It’s not so bad, it’s not as bad as crack cocaine.” We will change our beliefs, to match our
behaviour!
In the case of the Doomsday cult, the idea that they had sold all their possessions and believed something so asinine is an
uncomfortable disconnect, so they choose to believe they were “spared”, which justifies their actions.
@michelejkiss
It also explains why you can never convince your racist uncle of anything…
@michelejkiss@michelejkiss[Chanel et al, 2010]
discussion
vs. reading
However, Chanel, Luchini, Massoni, Vergnaud (2010) found that if we are given an opportunity to discuss the evidence and
exchange arguments with someone (rather than just reading the evidence and pondering it alone) we are more likely to
change our minds, in the face of opposing facts. (So… maybe there’s some value in talking to your racist uncle.)
@michelejkiss@michelejkiss@michelejkiss
whyitmatters
@michelejkiss@michelejkiss
more data to “prove” a
contrary viewpoint
Why this matters for analysts: Even if your data seems self-evident, if you come in with “breaking news”, that goes against
what the business has known, thought, or believed for some time, you may need more data to support your contrary
viewpoint. You may also want to allow for plenty of time for discussion, rather than simply sending out your findings, as those
discussions are critical to getting buy-in for this new viewpoint.
@michelejkiss@michelejkiss
CONFIRMATION BIAS
@michelejkiss
We know now that “the facts” may not persuade us, even when brought to our attention. However, Confirmation Bias tells us
that we intentionally seek out information that continually reinforces our beliefs, rather than searching for all evidence and
fully evaluating the possible explanations.
@michelejkiss@michelejkiss[Brock & Balloun, 1967]
we seek to
affirm
our beliefs
In a 1967 study by Brock & Balloun, subjects listened to several messages, but the recording was staticky. However, the
subjects could press a button to clear up the static. They found that people selectively chose to listen to the message that
affirmed their existing beliefs. For example, smokers chose to listen more closely when the content disputed a smoking-
cancer link.
@michelejkiss@michelejkiss
2, 4, 6
[Wason, 1960]
Wason (1960) conducted a study where participants were presented with a math problem: find the pattern in a series of
numbers, such as “2-4-6.” Participants could create three subsequent sets of numbers to “test” their theory, and the
researcher would confirm whether these sets followed the pattern or not.
@michelejkiss@michelejkiss
2, 4, 6
8, 10, 12
6, 8, 10
20, 30, 40
“Even Numbers”
Rather than collecting a list of possible patterns, and using their three “guesses” to prove or disprove each possible pattern,
Wason found that participants would come up with a single hypothesis, then seek to prove it. (For example, they might
hypothesize that “the pattern is even numbers” …
@michelejkiss@michelejkiss
2, 4, 6
8, 10, 12
6, 8, 10
20, 30, 40and check whether “8-10-12”, “6-8-10” and “20-30-40” correctly matched the pattern.)
@michelejkiss@michelejkiss
2, 4, 6
8, 10, 12
6, 8, 10
20, 30, 40
Increasing
Numbers!When it was confirmed their guesses matched the pattern, they simply stopped. However, the actual pattern was “increasing
numbers” – their hypothesis was not correct at all!
@michelejkiss@michelejkiss@michelejkiss
whyitmattersDo you do this with your analysis? When you start analyzing data, where do you start? With a hunch, that you seek to prove,
then stop your analysis there? (For example, “I think our website traffic is down because our paid search spend decreased.”)
Or with multiple hypotheses, which you seek to evaluate one by one?
@michelejkiss
Analysis of Competing Hypotheses
An alternative – Analysis of Competing Hypotheses. Analysis of Competing Hypotheses (or “ACH” was developed in the
1970’s, for use by intelligence analysis. It is an attempt to overcome the natural biases that we all have. The idea is, you
come up with multiple hypotheses that could explain the data you are seeing, and seek to evaluate each one in turn. Which
ones are more or less likely to be true? What data do you have, and which hypotheses does it support? This attempts to
avoid our analysis simply being a self-fulfilling prophecy.
@michelejkiss@michelejkiss
CONFORMITY TO THE NORM
@michelejkiss
@michelejkiss[Asch, 1951]
In 1951, Asch found that we conform to the views of others, even when they are flat-out wrong, surprisingly often!
He conducted an experiment where participants were seated in a group of eight others who were “in” on the experiment
(“confederates.”) Participants were asked to judge whether a line was most similar in length to three other lines. The task was
not particularly “grey area” – there was an obvious right and wrong answer.
Each person in the group gave their answer verbally, in turn. The confederates were instructed to give the incorrect answer,
and the participant was the sixth of the group to answer.
@michelejkiss@michelejkiss
76%
gave the wrong answer
at least once!
[Asch, 1951]
@michelejkiss@michelejkiss
5%
always conformed
to the incorrect answer
[Asch, 1951]
@michelejkiss
Only 25%
never followed the group’s
incorrect answer
[Asch, 1951]
@michelejkiss
only 13%conformity if
answers were written down
@michelejkiss@michelejkiss[Asch, 1951]
@michelejkiss@michelejkiss@michelejkiss
whyitmattersAs mentioned previously, if new findings contradict existing beliefs, it may take MORE than just presenting new data.
@michelejkiss@michelejkiss@michelejkiss
challenge the
status quo
@michelejkiss
However, these conformity studies suggest that efforts to do so may be further hampered if you are presenting information to
a group. It is less likely that people will stand up for your new findings against the norm of the group. In this case, you may be
better to discuss your findings slowly to individuals, and avoid putting people on the spot to agree/disagree within a group
setting.
@michelejkiss@michelejkiss@michelejkiss
group
think
Similarly, this argues against jumping straight to a “group brainstorming” session. Once in a group, Asch demonstrated that
76% of us will agree with the group (even if they’re wrong!) so we stand the best chance of getting more varied ideas and
minimising “group think” by allowing for individual, uninhibited brainstorming and collection of all ideas first.
@michelejkiss@michelejkiss
PRIMACY AND RECENCY EFFECTS
@michelejkiss[Ebbinghaus, 1913]
The serial position effect (so named by Ebbinghaus in 1913) finds that we are most likely to recall the FIRST and LAST items
in a list, and least likely to recall those in the middle.
@michelejkiss
@michelejkiss
For example, let’s say you are asked to recall pear, orange, banana, apple and pineapple.
@michelejkiss
The serial position effect suggests that individuals are more likely to remember pear (the first item; primacy effect) and
pineapple (the final item; recency effect) and less likely to remember the items in the middle.
@michelejkiss
Primacy
Long term memory
The theory here is that the last item is retained in short term memory …
@michelejkiss
Recency
Short term memory
Remembered in context
… while the first item is more likely to have been processed through to long term memory. Also, the last item is being
remembered “in context” – the situation in which they’re recalling it is CLOSER to when they learned the last item, than the
first.
@michelejkiss@michelejkiss[Godden & Baddeley, 1975]
In 1975, Godden & Baddeley conducted experiments where they had scuba divers memorize information, and found their
recall was better when the context in which they were recalling it was the same as when they were remembering it. (For
example, they recalled better when they memorized underwater, and recalled underwater, than when they tried to recall on
land.)
(This is also why we can’t remember what the heck we were thinking until we get up, and go back to the spot we first had the
thought. Then bam! It comes to us.)
@michelejkiss
Primacy
The longer the list, the less likely the primacy effect is to apply.
@michelejkiss
The recency effect can be reduced, by requiring attention be switched to another task, and then return to the recall. (That
way, the information was forced out of the short term memory.)
@michelejkiss
This doesn’t just affect
free recall
@michelejkiss
@michelejkiss
Steve
“is smart, diligent,
critical, impulsive,
and jealous”
@michelejkiss[Asch, 1964]
Asch (1964) found participants told “Steve is smart, diligent, critical, impulsive, and jealous” had a positive evaluation of
Steve,
@michelejkiss
Steve
“is jealous,
impulsive, critical,
diligent,
and smart”
@michelejkiss[Asch, 1964]
Whereas participants told “Steve is jealous, impulsive, critical, diligent, and smart” had a negative evaluation of Steve.
Even though the adjectives are the exact same – only the order is different!
@michelejkiss
Recency
Bias
@michelejkiss
This is related to the recency bias – the idea that we give more weight to more recent information. For example… when
shopping for a car: ‘I want the sporty convertible. Last winter wasn’t “that bad”’ (And you quickly forget the ten winters before
that, where you needed a four wheel drive to get out of our blacked-out neighborhood, due to terrible snowstorms.)
@michelejkiss
Or, I should sell everything and buy bitcoin.
@michelejkiss@michelejkiss@michelejkiss
whyitmatters
Why does this matter for analytics?
When you present information, your audience is unlikely to remember everything you tell them. So choose wisely. What do
you lead with? What do you end with? And what do you prioritize lower, and save for the middle?
These findings may also affect the amount of information you provide at one time, and the cadence with which you do so. If
you want more retained, you may wish to present smaller amounts of data more slowly, rather than rapid-firing with constant
information. For example, rather than presenting twelve different “optimisation opportunities” at once, focusing on one may
increase the likelihood that action is taken.
This is also an excellent argument against a 50-slide PowerPoint presentation – while you may have mentioned something in
it, if it was 22 slides ago, the chance of your audience remembering are slim.
@michelejkiss@michelejkiss
Primacy
&
Novelty
[Kohavi et al, 2012]
Related to optimization effects:
Primacy: The idea that people are familiar with the old experience, so there’s an advantage to the control over the new
variation.
Novelty: The idea that the new experience is something cool and different, so there is a (short term) life, that won’t sustain
over time.
@michelejkiss
“People are just used to
the old design”
However, sometimes the idea of primacy and recency can be taken too far, and be used to explain results that are simply not
driven by them. A/B test results often show dramatic differences in the early days, that seldom continue on. (This is due to
the high variance and lower sample size at the start of the test.)
However, if the product team has put their sweat and soul in to a feature, it’s easy to ascribe an actual reason to the early
results, outside of statistical fluctuation. For example, “Oh, people are just used to the old design, and the variation will
perform AMAZINGLY once they get used to it.” However, it’s rare for these early effects to completely flip the other way. So,
while the low performance may not continue, it’s also unlikely that the effects will completely reverse.
I’ve even seen the “primacy effect” be argued as the reason for the control to perform better (after a reasonable amount of
time) in sites that have little to no return visitation!
@michelejkiss@michelejkiss
THE HALO EFFECT
@michelejkiss
@michelejkiss
Demeanor
=
“Likeability”
@michelejkiss[Nisbet & Wilson, 1977]
In 1977, Nisbet and Wilson conducted an experiment with university students. The two students watched a video of the same
lecturer deliver the same material, but one group saw a warm and friendly “version” of the lecturer, while the other saw the
lecturer present in a cold and distant way. The group who saw the friendly version rated the lecturer as more attractive and
likeable.
@michelejkiss
“Attractive” students
receive higher grades
@michelejkiss
There are plenty of other examples of this. For example, “physically attractive” students have been found to receive higher
grades and/or test scores than “unattractive” students at a variety of ages, including elementary school (Salvia, Algozzine, &
Sheare, 1977; Zahr, 1985), high school (Felson, 1980) and college (Singer, 1964.)
@michelejkiss
Intelligence
=
Loyalty, Bravery
@michelejkiss[Thorndike, 1920]
Thorndike (1920) found similar effects within the military, where a perception of a subordinate’s intelligence tended to lead to
a perception of other positive characteristics such as loyalty or bravery.
The Halo Effect tells us that we extrapolate from one positive effect to another. For example, because someone is intelligent,
they are also loyal and brave. Because someone is attractive, they are also smart.
@michelejkiss@michelejkiss@michelejkiss
whyitmattersWhy this matters for analysts: The appearance of your reports/dashboards/analyses, the way you present to a group, your
presentation style, even your appearance may affect how others judge your credibility and intelligence.
@michelejkiss@michelejkiss
The Halo Effect can also influence the data you are analysing! It is common with surveys (especially in the case of lengthy
surveys) that happy customers will simply respond “10/10” for everything, and unhappy customers will rate “1/10” for
everything – even if parts of the experience differed from their overall perception.
For example, if a customer had a poor shipping experience, they may extend that negative feeling about the interaction with
the brand to all aspects of the interaction – even if only the last part was bad! (And note here: There’s a definite interplay
between the Halo Effect and the Recency Effect!)
@michelejkiss@michelejkiss
THE BYSTANDER EFFECT
@michelejkiss@michelejkiss@michelejkiss
In 1964 in New York City, a woman name Kitty Genovese was murdered. According to papers, the attack had lasted an hour,
and almost forty people witnessed the attack. Yet, no one called the police. (Later reports suggested this was an
exaggeration – that there had been fewer witnesses, and that some had, in fact, called the police.)
However, this event fascinated psychologists, and triggered several experiments. This became known as the “Bystander
Effect”, which proposes that the more bystanders that are present, the less likely it is that an individual will step in and help.
@michelejkiss@michelejkiss@michelejkiss[Darley & Latane, 1968]
the more of us
the less we help
Darley & Latane (1968) manufactured a medical emergency, where one participant was allegedly having an epileptic seizure,
and measured how long it took for participants to help. They found that the more participants, the longer it took to respond to
the emergency.
(This is actually why, if you go to CPR training, the FIRST thing you’re instructed to do is to point to a specific person, and tell
them “YOU! Go call 911. YOU! Go get me the AED machine.” Because, if you leave it up to an anonymous group, everyone
may assume someone else will call, and no one calls for help!)
@michelejkiss
Diffusion of
Responsibility
@michelejkiss@michelejkiss
Think about how hard an INDIVIDUAL works in a tug of war with 20 people on one side, versus 2.
@michelejkiss@michelejkiss@michelejkiss
whyitmatters
@michelejkiss
Who are you presenting to?
Whose responsibility
is it to act?
@michelejkiss
Why this matters for analysts: Think about how you present your analyses and recommendations. If you offer them to a large
group, without specific responsibility to any individual to act upon them, you decrease the likelihood of any action being taken
at all. So when you make a recommendation, be specific. Who should be taking action on this? If your recommendation is a
generic “we should do X”, it’s far less likely to happen.
@michelejkiss@michelejkiss
SELECTIVE ATTENTION
@michelejkiss[Simons & Chabris, 1999]
@michelejkiss@michelejkiss
In 1999, Simons and Chabris conducted an experiment in awareness at Harvard University. Participants were asked to watch
a video of basketball players, where one team was wearing white shirts, and the other team was wearing black shirts. In the
video, the white team and black team respectively were passing the ball to each other. Participants were asked to count the
number of passes between players of the white team.
@michelejkiss@michelejkiss
Go watch the video! http://bit.ly/selectiveattention
@michelejkiss@michelejkiss@michelejkiss
Did you see it??
During the video, a man dressed as a gorilla walked into the middle of the court, faced the camera and thumps his chest,
then leaves (spending a total of 9 seconds on the screen.) Amazingly? Half of the participants missed the gorilla entirely!
Since then, this has been termed “the Invisible Gorilla” experiment.
Participants were SO FOCUSED on counting the passes of the white team that they completely missed it. They were
selectively attending to some information in the video, at the exclusion of other information.
@michelejkiss@michelejkiss@michelejkiss
whyitmatters
@michelejkiss@michelejkiss@michelejkiss
Don’t get so bogged down in the
details you miss critical points
Why this matters for analysts: As you are analyzing data, there can be huge, gaping issues that you may not even notice.
When we get SO FOCUSED on a particular task (for example, counting passes by the white-shirt players only, or analyzing
one subset of our customers) we may overlook something significant at a higher level.
Take time before you finalize or present your analysis to think of what other possible explanations, or variables there could be
(what could you be missing?) -- or invite a colleague to poke holes in your work.
@michelejkiss@michelejkiss
FALSE CONSENSUS
@michelejkiss[Ross, Greene & House, 1977]
Experiments have revealed that we tend to believe in a false consensus: that others would respond similarly to the way that
we would. For example, Ross, Greene & House (1977) provided participants with a scenario, with two different possible ways
of responding. Participants were asked to explain which option THEY would choose, and guess what OTHER PEOPLE
would choose. Regardless of which option they actually chose, participants believed that other people would choose the
same one.
@michelejkiss@michelejkiss@michelejkiss
whyitmatters
@michelejkiss@michelejkiss
you are not
your customer@michelejkiss
Why this matters for analysts: As you are analyzing data, you are looking at the behaviour of real people. It’s easy to make
assumptions about how they will react, or why they did what they did, based on what YOU would do. But we are not our
customer. Our customers don’t know our site, our app, our tools like we do. (Not when we spend so much time looking at
them!) But that doesn’t that we are representative of our customers! So we need to look at what they’re actually doing – not
just assume they’re acting as we would.
@michelejkiss@michelejkiss
HOMOGENEITY OF THE OUTGROUP
@michelejkiss[Quattrone & Jones, 1980]
There is a related effect here: the Homogeneity of the Outgroup. (Quattrone & Jones, 1980.)
In short, we tend to view those who are different to us (the “outgroup”) as all being very similar, while those who are like us
(the “ingroup”) are more diverse.
For example, all women are chatty, but some men are talkative, some are quiet, some are stoic, some are more emotional,
some are cautious, others are more risky… etc.
We do this a lot in marketing! “All millenials are spoiled, entitled little brats. All Baby Boomers are old fuddies who don’t know
how to use a computer.”
@michelejkiss@michelejkiss@michelejkiss
whyitmatters
@michelejkiss@michelejkiss@michelejkiss
“everyone in [countryX]
behaves like Y”
Why this matters for analysts: Similar to the False Consensus Effect, where we may analyse user behaviour assuming
everyone thinks as we do, the Homogeneity of the Outgroup suggests that we may OVERSIMPLIFY the behaviour of
customers who are different to us, and fail to fully appreciate the nuance of varied behaviour.
This may seriously bias our analyses!
For example, if we are a large global company, an analysis of customers in another region may be seriously flawed if we are
assuming customers in the region are “all the same.” To overcome this tendency, we might consider leveraging local teams or
local analysts to conduct or vet such analyses. Or, add qualitative data to give context (and maybe an opposing viewpoint, to
our assumptions!)
@michelejkiss@michelejkiss
THE HAWTHORNE EFFECT
@michelejkiss[Landsberger, 1955]
The story of the Hawthorne Effect goes as follows - In 1955, Henry Landsberger analyzed several studies conducted
between 1924 and 1932 at the Hawthorne Works factory outside Chicago. These studies originally intended to discover
whether the level of light (among other changes) within a building changed the productivity of workers. They found that the
changes did have an impact, however they were short-term improvements, AND they seemed to happen due to the fact that
a change was made – not the result of the actual change. Aka, it wasn’t the fact that the light was brighter, but the fact that
the light had been changed.
@michelejkiss
However…
@michelejkiss
However, the study has been the subject of much criticism, including:
@michelejkiss@michelejkiss
“Glorified anecdote”
Once you've got the anecdote,
you can throw away the data.
Dr. Richard Nisbett
@michelejkiss@michelejkiss
Alternative Explanations
@michelejkiss
@michelejkiss
Demand Characteristics
@michelejkiss[Martin Orne]
One possibility is that the effects were due to “demand characteristics”, and the fact that the employees knew they were
being observed. (Martin Orne) (This can go either way – participants may be looking to “please” the researchers, or
“correctly” complete the study, or they may seek to sabotage studies)
@michelejkiss@michelejkiss[Levitt & List, 2011]
But in 2011, Levitt & List conducted a re-analysis of the original data. They actually had microfilm, so they were able to go
right to the source. What they found… all of the changes were made on a SUNDAY, as it was the only day the factor was
closed. So by…
@michelejkiss
monday
@michelejkiss[Levitt & List, 2011]
… Monday, the works were coming in, refreshed from a day off… And were more productive!
@michelejkiss@michelejkiss@michelejkiss
whyitmatters
@michelejkiss@michelejkiss
Multiple Explanations
@michelejkiss
One analysis or set of data may have multiple explanations, and be subject to very different interpretations depending on who
is doing the analysis.
@michelejkiss
People may not agree
People may not agree with your conclusions. USE THIS to produce a more comprehensive analysis. Take those viewpoints
in to account. Respond to them, disprove those hypotheses.
@michelejkiss@michelejkiss
Observation may change behaviour
@michelejkiss
Demand characteristics suggest that observing people may change their behaviour. We need to be aware of this possibility
when analyzing behaviour. For example, what data would you get from an anonymous survey, versus one that requires a
person’s name and phone number? (Or their employee ID!) How do user testing and a/b testing results differ, based on
people’s level of awareness that they are being observed?
@michelejkiss@michelejkiss@michelejkiss
So, what should you take away from this?
@michelejkiss@michelejkiss
Be aware of your biases
@michelejkiss
Be aware of your biases, and attempt to control for them.
• Are you just using the data to “prove” one particular hypothesis, rather than looking for all the possible explanations?
• Are you overgeneralizing to a certain population?
• Is your interpretation of the data being influenced by conforming to your company’s norms, and the way things have
always been done?
@michelejkiss@michelejkiss
Work with people’s natural abilities
Be aware of the limitations of yourself, and others. Work WITH people’s abilities. Don’t overburden their short term memory.
And keep in mind the factors that might be influencing their memory, and their behaviour.
@michelejkiss@michelejkiss
Get a second opinion
@michelejkiss
Get someone to keep your biases in check!
Have someone else to vet your assumptions. (Especially someone unfamiliar/who hasn’t been involved with your analysis or
experiment so far.)
@michelejkiss@michelejkiss
Our best defense is awareness
@michelejkiss
The best defense we have is our AWARENESS. While we are analysts, and we are trained to approach problems rationally
and methodically, we are human too, and our best analysis comes from being aware of the influences of our humanness on
our interpretation of the data.
@michelejkiss@michelejkiss
Questions?
@MicheleJKiss
michele@analyticsdemystified.com
join.measure.chat
Awesome community of digital
analytics professionals

More Related Content

Similar to Foundational Psychology Experiments (And Why Analysts Should Know Them)

How to Craft Your Origin Story
How to Craft Your Origin StoryHow to Craft Your Origin Story
How to Craft Your Origin StoryJordan Bower
 
020415 business storytelling by cynthia hartwig
020415 business storytelling by cynthia hartwig020415 business storytelling by cynthia hartwig
020415 business storytelling by cynthia hartwigCynthia Hartwig
 
Course 0837 assingment 1 roll no ce529580
Course 0837 assingment 1 roll no ce529580Course 0837 assingment 1 roll no ce529580
Course 0837 assingment 1 roll no ce529580shannazir1
 
Illustration Essay Example. Illustration essay means / rcalvet.com
Illustration Essay Example. Illustration essay means / rcalvet.comIllustration Essay Example. Illustration essay means / rcalvet.com
Illustration Essay Example. Illustration essay means / rcalvet.comAlexandra Saunders
 
What’s Going On in Your Brain?”
What’s Going On in Your Brain?” What’s Going On in Your Brain?”
What’s Going On in Your Brain?” Trading Game Pty Ltd
 
Discussion 2Ch 241.Compare and contrast various interpret.docx
Discussion 2Ch 241.Compare and contrast various interpret.docxDiscussion 2Ch 241.Compare and contrast various interpret.docx
Discussion 2Ch 241.Compare and contrast various interpret.docxduketjoy27252
 
Learn To Adjust To Different Models Of The World
Learn To Adjust To Different Models Of The WorldLearn To Adjust To Different Models Of The World
Learn To Adjust To Different Models Of The WorldGeorge Hutton
 
Business Storytelling by Cynthia Hartwig of Two Pens
Business Storytelling by Cynthia Hartwig of Two PensBusiness Storytelling by Cynthia Hartwig of Two Pens
Business Storytelling by Cynthia Hartwig of Two PensCynthia Hartwig
 
College Prowler No Essay Scholarship
College Prowler No Essay ScholarshipCollege Prowler No Essay Scholarship
College Prowler No Essay ScholarshipErica Robertson
 
Selling with Stories
Selling with StoriesSelling with Stories
Selling with StoriesDavid Hoffeld
 
Sample Of 300 Word Essay. Online assignment writing service.
Sample Of 300 Word Essay. Online assignment writing service.Sample Of 300 Word Essay. Online assignment writing service.
Sample Of 300 Word Essay. Online assignment writing service.Cheraghearzu Donaldson
 
Mba year 1_571216_nol
Mba year 1_571216_nolMba year 1_571216_nol
Mba year 1_571216_nolrevathigiri
 
Dodgy argumentsextendeda4
Dodgy argumentsextendeda4Dodgy argumentsextendeda4
Dodgy argumentsextendeda4Junji Kai
 
Ideas For Persuasive Essay.pdf
Ideas For Persuasive Essay.pdfIdeas For Persuasive Essay.pdf
Ideas For Persuasive Essay.pdfKatie Stewart
 
Mark Hunter: Writing up your investigation
Mark Hunter: Writing up your investigationMark Hunter: Writing up your investigation
Mark Hunter: Writing up your investigationtcij
 

Similar to Foundational Psychology Experiments (And Why Analysts Should Know Them) (15)

How to Craft Your Origin Story
How to Craft Your Origin StoryHow to Craft Your Origin Story
How to Craft Your Origin Story
 
020415 business storytelling by cynthia hartwig
020415 business storytelling by cynthia hartwig020415 business storytelling by cynthia hartwig
020415 business storytelling by cynthia hartwig
 
Course 0837 assingment 1 roll no ce529580
Course 0837 assingment 1 roll no ce529580Course 0837 assingment 1 roll no ce529580
Course 0837 assingment 1 roll no ce529580
 
Illustration Essay Example. Illustration essay means / rcalvet.com
Illustration Essay Example. Illustration essay means / rcalvet.comIllustration Essay Example. Illustration essay means / rcalvet.com
Illustration Essay Example. Illustration essay means / rcalvet.com
 
What’s Going On in Your Brain?”
What’s Going On in Your Brain?” What’s Going On in Your Brain?”
What’s Going On in Your Brain?”
 
Discussion 2Ch 241.Compare and contrast various interpret.docx
Discussion 2Ch 241.Compare and contrast various interpret.docxDiscussion 2Ch 241.Compare and contrast various interpret.docx
Discussion 2Ch 241.Compare and contrast various interpret.docx
 
Learn To Adjust To Different Models Of The World
Learn To Adjust To Different Models Of The WorldLearn To Adjust To Different Models Of The World
Learn To Adjust To Different Models Of The World
 
Business Storytelling by Cynthia Hartwig of Two Pens
Business Storytelling by Cynthia Hartwig of Two PensBusiness Storytelling by Cynthia Hartwig of Two Pens
Business Storytelling by Cynthia Hartwig of Two Pens
 
College Prowler No Essay Scholarship
College Prowler No Essay ScholarshipCollege Prowler No Essay Scholarship
College Prowler No Essay Scholarship
 
Selling with Stories
Selling with StoriesSelling with Stories
Selling with Stories
 
Sample Of 300 Word Essay. Online assignment writing service.
Sample Of 300 Word Essay. Online assignment writing service.Sample Of 300 Word Essay. Online assignment writing service.
Sample Of 300 Word Essay. Online assignment writing service.
 
Mba year 1_571216_nol
Mba year 1_571216_nolMba year 1_571216_nol
Mba year 1_571216_nol
 
Dodgy argumentsextendeda4
Dodgy argumentsextendeda4Dodgy argumentsextendeda4
Dodgy argumentsextendeda4
 
Ideas For Persuasive Essay.pdf
Ideas For Persuasive Essay.pdfIdeas For Persuasive Essay.pdf
Ideas For Persuasive Essay.pdf
 
Mark Hunter: Writing up your investigation
Mark Hunter: Writing up your investigationMark Hunter: Writing up your investigation
Mark Hunter: Writing up your investigation
 

Recently uploaded

2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group MeetingAlison Pitt
 
basics of data science with application areas.pdf
basics of data science with application areas.pdfbasics of data science with application areas.pdf
basics of data science with application areas.pdfvyankatesh1
 
Fuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyFuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyRafigAliyev2
 
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...ssuserf63bd7
 
AI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfAI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfMichaelSenkow
 
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理pyhepag
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理pyhepag
 
Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Jon Hansen
 
Easy and simple project file on mp online
Easy and simple project file on mp onlineEasy and simple project file on mp online
Easy and simple project file on mp onlinebalibahu1313
 
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理pyhepag
 
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictSupply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictJack Cole
 
社内勉強会資料  Mamba - A new era or ephemeral
社内勉強会資料   Mamba - A new era or ephemeral社内勉強会資料   Mamba - A new era or ephemeral
社内勉強会資料  Mamba - A new era or ephemeralNABLAS株式会社
 
一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理pyhepag
 
一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理cyebo
 
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfGenerative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfEmmanuel Dauda
 
Artificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfArtificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfscitechtalktv
 
一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理cyebo
 

Recently uploaded (20)

2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting
 
basics of data science with application areas.pdf
basics of data science with application areas.pdfbasics of data science with application areas.pdf
basics of data science with application areas.pdf
 
Fuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyFuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertainty
 
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
 
AI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfAI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdf
 
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
 
Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)
 
Machine Learning for Accident Severity Prediction
Machine Learning for Accident Severity PredictionMachine Learning for Accident Severity Prediction
Machine Learning for Accident Severity Prediction
 
Slip-and-fall Injuries: Top Workers' Comp Claims
Slip-and-fall Injuries: Top Workers' Comp ClaimsSlip-and-fall Injuries: Top Workers' Comp Claims
Slip-and-fall Injuries: Top Workers' Comp Claims
 
Easy and simple project file on mp online
Easy and simple project file on mp onlineEasy and simple project file on mp online
Easy and simple project file on mp online
 
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotecAbortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
 
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
 
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictSupply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
 
社内勉強会資料  Mamba - A new era or ephemeral
社内勉強会資料   Mamba - A new era or ephemeral社内勉強会資料   Mamba - A new era or ephemeral
社内勉強会資料  Mamba - A new era or ephemeral
 
一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理
 
一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理
 
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfGenerative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
 
Artificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfArtificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdf
 
一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理
 

Foundational Psychology Experiments (And Why Analysts Should Know Them)

  • 1. @michelejkiss Foundational Psychology Experiments and why we should know them Michele Kiss @michelejkiss Analytics Demystified
  • 2. @michelejkiss Like many people in digital analytics, I sort of “fell in to” the industry. I went to the University of Melbourne, and I studied Law and Psychology. Seems pretty unrelated… But the more analysis I do, the more stakeholders I work with, the more I realize that the things I learned are directly applicable to the practice of analytics.
  • 3. @michelejkiss@michelejkiss Analytics, optimization, etc are all about using quantitative methods to understand PEOPLE, and why we do the stupid stuff we do Today we’re going to talk about some classic studies, and why they matter (and how they help us understand human behaviour)
  • 4. @michelejkiss@michelejkiss THE MAGIC NUMBER 7 @michelejkiss@michelejkiss
  • 5. @michelejkiss@michelejkiss@michelejkiss Data Visualisation Those of us working in analytics are pretty familiar with the underlying tenets of data visualization, dashboards, and likely, with Stephen Few.
  • 6. “A dashboard is a visual display of the most important information needed to achieve one or more objectives; consolidated and arranged on a single screen so the information can be monitored at a glance.” [Stephen Few] @michelejkiss Stephen Few defined a dashboard as…
  • 7. “A dashboard is a visual display of the most important information needed to achieve one or more objectives; consolidated and arranged on a single screen so the information can be monitored at a glance.” [Stephen Few] @michelejkiss To me, the most important words are “on a single screen”
  • 8. @michelejkiss The reason why we need the information to be on one page or screen? The limits of our memory.
  • 9. @michelejkiss@michelejkiss [Miller, 1956] In 1956, George A. Miller published one of the most cited papers in psychology, where he spoke of the magic number 7. For our purposes here, Miller found that the amount of information that we can retain our working (or short term) memory is seven, plus or minus two.
  • 10. @michelejkiss@michelejkiss R X U W B J P However, this doesn’t mean that we can only recall 7 +/- 2 individual letters, or numbers (for example.) The 7 +/- 2 items can be 7 +/- 2 letters, or 7 +/- 2 words, or …
  • 11. @michelejkiss Take my hand / Take my whole life too / For I can’t help / Falling in love with you You may say I’m a dreamer / But I’m not the only one We are young / Heartache to heartache we stand I fell into a burning ring of fire / I went down down down and the flames went higher All you need is love / Love is all you need Every new beginning / Comes from some other beginning’s end. It's the end of the world as we know it and I feel fine. @michelejkiss … or 7 +/- 2 song lyrics. We are capable of “chunking” information, so that each piece of information only occupies one “item” within our memory system.
  • 13. @michelejkiss Because, back to Stephen Few’s definition, if you are presenting information, you need to work WITHIN the limits of our perceptual and memory systems. Expecting users to draw connections between two data points six pages apart, or thirty slides ago, is a recipe for failure.
  • 14. @michelejkiss@michelejkiss WHEN THE FACTS DON’T MATTER @michelejkiss (Or, why do facts fail to convince us)
  • 15. @michelejkiss [Festinger, 1957] @michelejkiss The end of the world In 1957, Leon Festinger studied a Doomsday cult who believed that aliens would rescue them from a coming flood. Unsurprisingly, no flood (nor aliens) eventuated. However, rather than being proven wrong, the group explained away that they had been “spared”, and clung even more tightly to their beliefs. In their book, When Prophecy Fails, Festinger (et al) said:
  • 16. @michelejkiss “A man with a conviction is a hard man to change. Show him facts or figures and he questions your sources. Appeal to logic and he fails to see your point.
  • 17. @michelejkiss “Suppose that he is presented with evidence, unequivocal and undeniable evidence, that his belief is wrong: what will happen?
  • 18. @michelejkiss “The individual will frequently emerge, not only unshaken, but even more convinced of the truth of his beliefs than ever before.”
  • 19. @michelejkiss@michelejkiss Especially if… This might happen especially if: • Beliefs are deeply held; • Action has been taken, that is difficult to undo; (for example, the Doomsday cult had sold all their possessions) • Believer has social support (the cult continues to reinforce each others’ beliefs)
  • 20. @michelejkiss cognitive dissonance @michelejkiss [Festinger, 1957] Festinger explains this by the theory of cognitive dissonance. This theory tells us that we don’t like the feeling of inconsistency (for example, of our beliefs and our actions.) We seek to reduce this uncomfortable feeling by justifying our beliefs, and avoiding information that might further conflict. For example, smokers who know smoking is bad for them may choose to tell themselves, “It’s not so bad, it’s not as bad as crack cocaine.” We will change our beliefs, to match our behaviour! In the case of the Doomsday cult, the idea that they had sold all their possessions and believed something so asinine is an uncomfortable disconnect, so they choose to believe they were “spared”, which justifies their actions.
  • 21. @michelejkiss It also explains why you can never convince your racist uncle of anything…
  • 22. @michelejkiss@michelejkiss[Chanel et al, 2010] discussion vs. reading However, Chanel, Luchini, Massoni, Vergnaud (2010) found that if we are given an opportunity to discuss the evidence and exchange arguments with someone (rather than just reading the evidence and pondering it alone) we are more likely to change our minds, in the face of opposing facts. (So… maybe there’s some value in talking to your racist uncle.)
  • 24. @michelejkiss@michelejkiss more data to “prove” a contrary viewpoint Why this matters for analysts: Even if your data seems self-evident, if you come in with “breaking news”, that goes against what the business has known, thought, or believed for some time, you may need more data to support your contrary viewpoint. You may also want to allow for plenty of time for discussion, rather than simply sending out your findings, as those discussions are critical to getting buy-in for this new viewpoint.
  • 25. @michelejkiss@michelejkiss CONFIRMATION BIAS @michelejkiss We know now that “the facts” may not persuade us, even when brought to our attention. However, Confirmation Bias tells us that we intentionally seek out information that continually reinforces our beliefs, rather than searching for all evidence and fully evaluating the possible explanations.
  • 26. @michelejkiss@michelejkiss[Brock & Balloun, 1967] we seek to affirm our beliefs In a 1967 study by Brock & Balloun, subjects listened to several messages, but the recording was staticky. However, the subjects could press a button to clear up the static. They found that people selectively chose to listen to the message that affirmed their existing beliefs. For example, smokers chose to listen more closely when the content disputed a smoking- cancer link.
  • 27. @michelejkiss@michelejkiss 2, 4, 6 [Wason, 1960] Wason (1960) conducted a study where participants were presented with a math problem: find the pattern in a series of numbers, such as “2-4-6.” Participants could create three subsequent sets of numbers to “test” their theory, and the researcher would confirm whether these sets followed the pattern or not.
  • 28. @michelejkiss@michelejkiss 2, 4, 6 8, 10, 12 6, 8, 10 20, 30, 40 “Even Numbers” Rather than collecting a list of possible patterns, and using their three “guesses” to prove or disprove each possible pattern, Wason found that participants would come up with a single hypothesis, then seek to prove it. (For example, they might hypothesize that “the pattern is even numbers” …
  • 29. @michelejkiss@michelejkiss 2, 4, 6 8, 10, 12 6, 8, 10 20, 30, 40and check whether “8-10-12”, “6-8-10” and “20-30-40” correctly matched the pattern.)
  • 30. @michelejkiss@michelejkiss 2, 4, 6 8, 10, 12 6, 8, 10 20, 30, 40 Increasing Numbers!When it was confirmed their guesses matched the pattern, they simply stopped. However, the actual pattern was “increasing numbers” – their hypothesis was not correct at all!
  • 31. @michelejkiss@michelejkiss@michelejkiss whyitmattersDo you do this with your analysis? When you start analyzing data, where do you start? With a hunch, that you seek to prove, then stop your analysis there? (For example, “I think our website traffic is down because our paid search spend decreased.”) Or with multiple hypotheses, which you seek to evaluate one by one?
  • 32. @michelejkiss Analysis of Competing Hypotheses An alternative – Analysis of Competing Hypotheses. Analysis of Competing Hypotheses (or “ACH” was developed in the 1970’s, for use by intelligence analysis. It is an attempt to overcome the natural biases that we all have. The idea is, you come up with multiple hypotheses that could explain the data you are seeing, and seek to evaluate each one in turn. Which ones are more or less likely to be true? What data do you have, and which hypotheses does it support? This attempts to avoid our analysis simply being a self-fulfilling prophecy.
  • 34. @michelejkiss[Asch, 1951] In 1951, Asch found that we conform to the views of others, even when they are flat-out wrong, surprisingly often! He conducted an experiment where participants were seated in a group of eight others who were “in” on the experiment (“confederates.”) Participants were asked to judge whether a line was most similar in length to three other lines. The task was not particularly “grey area” – there was an obvious right and wrong answer. Each person in the group gave their answer verbally, in turn. The confederates were instructed to give the incorrect answer, and the participant was the sixth of the group to answer.
  • 35. @michelejkiss@michelejkiss 76% gave the wrong answer at least once! [Asch, 1951]
  • 37. @michelejkiss Only 25% never followed the group’s incorrect answer [Asch, 1951]
  • 38. @michelejkiss only 13%conformity if answers were written down @michelejkiss@michelejkiss[Asch, 1951]
  • 39. @michelejkiss@michelejkiss@michelejkiss whyitmattersAs mentioned previously, if new findings contradict existing beliefs, it may take MORE than just presenting new data.
  • 40. @michelejkiss@michelejkiss@michelejkiss challenge the status quo @michelejkiss However, these conformity studies suggest that efforts to do so may be further hampered if you are presenting information to a group. It is less likely that people will stand up for your new findings against the norm of the group. In this case, you may be better to discuss your findings slowly to individuals, and avoid putting people on the spot to agree/disagree within a group setting.
  • 41. @michelejkiss@michelejkiss@michelejkiss group think Similarly, this argues against jumping straight to a “group brainstorming” session. Once in a group, Asch demonstrated that 76% of us will agree with the group (even if they’re wrong!) so we stand the best chance of getting more varied ideas and minimising “group think” by allowing for individual, uninhibited brainstorming and collection of all ideas first.
  • 42. @michelejkiss@michelejkiss PRIMACY AND RECENCY EFFECTS @michelejkiss[Ebbinghaus, 1913] The serial position effect (so named by Ebbinghaus in 1913) finds that we are most likely to recall the FIRST and LAST items in a list, and least likely to recall those in the middle.
  • 44. @michelejkiss For example, let’s say you are asked to recall pear, orange, banana, apple and pineapple.
  • 45. @michelejkiss The serial position effect suggests that individuals are more likely to remember pear (the first item; primacy effect) and pineapple (the final item; recency effect) and less likely to remember the items in the middle.
  • 46. @michelejkiss Primacy Long term memory The theory here is that the last item is retained in short term memory …
  • 47. @michelejkiss Recency Short term memory Remembered in context … while the first item is more likely to have been processed through to long term memory. Also, the last item is being remembered “in context” – the situation in which they’re recalling it is CLOSER to when they learned the last item, than the first.
  • 48. @michelejkiss@michelejkiss[Godden & Baddeley, 1975] In 1975, Godden & Baddeley conducted experiments where they had scuba divers memorize information, and found their recall was better when the context in which they were recalling it was the same as when they were remembering it. (For example, they recalled better when they memorized underwater, and recalled underwater, than when they tried to recall on land.) (This is also why we can’t remember what the heck we were thinking until we get up, and go back to the spot we first had the thought. Then bam! It comes to us.)
  • 49. @michelejkiss Primacy The longer the list, the less likely the primacy effect is to apply.
  • 50. @michelejkiss The recency effect can be reduced, by requiring attention be switched to another task, and then return to the recall. (That way, the information was forced out of the short term memory.)
  • 51. @michelejkiss This doesn’t just affect free recall @michelejkiss
  • 52. @michelejkiss Steve “is smart, diligent, critical, impulsive, and jealous” @michelejkiss[Asch, 1964] Asch (1964) found participants told “Steve is smart, diligent, critical, impulsive, and jealous” had a positive evaluation of Steve,
  • 53. @michelejkiss Steve “is jealous, impulsive, critical, diligent, and smart” @michelejkiss[Asch, 1964] Whereas participants told “Steve is jealous, impulsive, critical, diligent, and smart” had a negative evaluation of Steve. Even though the adjectives are the exact same – only the order is different!
  • 54. @michelejkiss Recency Bias @michelejkiss This is related to the recency bias – the idea that we give more weight to more recent information. For example… when shopping for a car: ‘I want the sporty convertible. Last winter wasn’t “that bad”’ (And you quickly forget the ten winters before that, where you needed a four wheel drive to get out of our blacked-out neighborhood, due to terrible snowstorms.)
  • 55. @michelejkiss Or, I should sell everything and buy bitcoin.
  • 56. @michelejkiss@michelejkiss@michelejkiss whyitmatters Why does this matter for analytics? When you present information, your audience is unlikely to remember everything you tell them. So choose wisely. What do you lead with? What do you end with? And what do you prioritize lower, and save for the middle? These findings may also affect the amount of information you provide at one time, and the cadence with which you do so. If you want more retained, you may wish to present smaller amounts of data more slowly, rather than rapid-firing with constant information. For example, rather than presenting twelve different “optimisation opportunities” at once, focusing on one may increase the likelihood that action is taken. This is also an excellent argument against a 50-slide PowerPoint presentation – while you may have mentioned something in it, if it was 22 slides ago, the chance of your audience remembering are slim.
  • 57. @michelejkiss@michelejkiss Primacy & Novelty [Kohavi et al, 2012] Related to optimization effects: Primacy: The idea that people are familiar with the old experience, so there’s an advantage to the control over the new variation. Novelty: The idea that the new experience is something cool and different, so there is a (short term) life, that won’t sustain over time.
  • 58. @michelejkiss “People are just used to the old design” However, sometimes the idea of primacy and recency can be taken too far, and be used to explain results that are simply not driven by them. A/B test results often show dramatic differences in the early days, that seldom continue on. (This is due to the high variance and lower sample size at the start of the test.) However, if the product team has put their sweat and soul in to a feature, it’s easy to ascribe an actual reason to the early results, outside of statistical fluctuation. For example, “Oh, people are just used to the old design, and the variation will perform AMAZINGLY once they get used to it.” However, it’s rare for these early effects to completely flip the other way. So, while the low performance may not continue, it’s also unlikely that the effects will completely reverse. I’ve even seen the “primacy effect” be argued as the reason for the control to perform better (after a reasonable amount of time) in sites that have little to no return visitation!
  • 60. @michelejkiss Demeanor = “Likeability” @michelejkiss[Nisbet & Wilson, 1977] In 1977, Nisbet and Wilson conducted an experiment with university students. The two students watched a video of the same lecturer deliver the same material, but one group saw a warm and friendly “version” of the lecturer, while the other saw the lecturer present in a cold and distant way. The group who saw the friendly version rated the lecturer as more attractive and likeable.
  • 61. @michelejkiss “Attractive” students receive higher grades @michelejkiss There are plenty of other examples of this. For example, “physically attractive” students have been found to receive higher grades and/or test scores than “unattractive” students at a variety of ages, including elementary school (Salvia, Algozzine, & Sheare, 1977; Zahr, 1985), high school (Felson, 1980) and college (Singer, 1964.)
  • 62. @michelejkiss Intelligence = Loyalty, Bravery @michelejkiss[Thorndike, 1920] Thorndike (1920) found similar effects within the military, where a perception of a subordinate’s intelligence tended to lead to a perception of other positive characteristics such as loyalty or bravery. The Halo Effect tells us that we extrapolate from one positive effect to another. For example, because someone is intelligent, they are also loyal and brave. Because someone is attractive, they are also smart.
  • 63. @michelejkiss@michelejkiss@michelejkiss whyitmattersWhy this matters for analysts: The appearance of your reports/dashboards/analyses, the way you present to a group, your presentation style, even your appearance may affect how others judge your credibility and intelligence.
  • 64. @michelejkiss@michelejkiss The Halo Effect can also influence the data you are analysing! It is common with surveys (especially in the case of lengthy surveys) that happy customers will simply respond “10/10” for everything, and unhappy customers will rate “1/10” for everything – even if parts of the experience differed from their overall perception. For example, if a customer had a poor shipping experience, they may extend that negative feeling about the interaction with the brand to all aspects of the interaction – even if only the last part was bad! (And note here: There’s a definite interplay between the Halo Effect and the Recency Effect!)
  • 66. @michelejkiss@michelejkiss@michelejkiss In 1964 in New York City, a woman name Kitty Genovese was murdered. According to papers, the attack had lasted an hour, and almost forty people witnessed the attack. Yet, no one called the police. (Later reports suggested this was an exaggeration – that there had been fewer witnesses, and that some had, in fact, called the police.) However, this event fascinated psychologists, and triggered several experiments. This became known as the “Bystander Effect”, which proposes that the more bystanders that are present, the less likely it is that an individual will step in and help.
  • 67. @michelejkiss@michelejkiss@michelejkiss[Darley & Latane, 1968] the more of us the less we help Darley & Latane (1968) manufactured a medical emergency, where one participant was allegedly having an epileptic seizure, and measured how long it took for participants to help. They found that the more participants, the longer it took to respond to the emergency. (This is actually why, if you go to CPR training, the FIRST thing you’re instructed to do is to point to a specific person, and tell them “YOU! Go call 911. YOU! Go get me the AED machine.” Because, if you leave it up to an anonymous group, everyone may assume someone else will call, and no one calls for help!)
  • 69. @michelejkiss@michelejkiss Think about how hard an INDIVIDUAL works in a tug of war with 20 people on one side, versus 2.
  • 71. @michelejkiss Who are you presenting to? Whose responsibility is it to act? @michelejkiss Why this matters for analysts: Think about how you present your analyses and recommendations. If you offer them to a large group, without specific responsibility to any individual to act upon them, you decrease the likelihood of any action being taken at all. So when you make a recommendation, be specific. Who should be taking action on this? If your recommendation is a generic “we should do X”, it’s far less likely to happen.
  • 73. @michelejkiss@michelejkiss In 1999, Simons and Chabris conducted an experiment in awareness at Harvard University. Participants were asked to watch a video of basketball players, where one team was wearing white shirts, and the other team was wearing black shirts. In the video, the white team and black team respectively were passing the ball to each other. Participants were asked to count the number of passes between players of the white team.
  • 74. @michelejkiss@michelejkiss Go watch the video! http://bit.ly/selectiveattention
  • 75. @michelejkiss@michelejkiss@michelejkiss Did you see it?? During the video, a man dressed as a gorilla walked into the middle of the court, faced the camera and thumps his chest, then leaves (spending a total of 9 seconds on the screen.) Amazingly? Half of the participants missed the gorilla entirely! Since then, this has been termed “the Invisible Gorilla” experiment. Participants were SO FOCUSED on counting the passes of the white team that they completely missed it. They were selectively attending to some information in the video, at the exclusion of other information.
  • 77. @michelejkiss@michelejkiss@michelejkiss Don’t get so bogged down in the details you miss critical points Why this matters for analysts: As you are analyzing data, there can be huge, gaping issues that you may not even notice. When we get SO FOCUSED on a particular task (for example, counting passes by the white-shirt players only, or analyzing one subset of our customers) we may overlook something significant at a higher level. Take time before you finalize or present your analysis to think of what other possible explanations, or variables there could be (what could you be missing?) -- or invite a colleague to poke holes in your work.
  • 78. @michelejkiss@michelejkiss FALSE CONSENSUS @michelejkiss[Ross, Greene & House, 1977] Experiments have revealed that we tend to believe in a false consensus: that others would respond similarly to the way that we would. For example, Ross, Greene & House (1977) provided participants with a scenario, with two different possible ways of responding. Participants were asked to explain which option THEY would choose, and guess what OTHER PEOPLE would choose. Regardless of which option they actually chose, participants believed that other people would choose the same one.
  • 80. @michelejkiss@michelejkiss you are not your customer@michelejkiss Why this matters for analysts: As you are analyzing data, you are looking at the behaviour of real people. It’s easy to make assumptions about how they will react, or why they did what they did, based on what YOU would do. But we are not our customer. Our customers don’t know our site, our app, our tools like we do. (Not when we spend so much time looking at them!) But that doesn’t that we are representative of our customers! So we need to look at what they’re actually doing – not just assume they’re acting as we would.
  • 81. @michelejkiss@michelejkiss HOMOGENEITY OF THE OUTGROUP @michelejkiss[Quattrone & Jones, 1980] There is a related effect here: the Homogeneity of the Outgroup. (Quattrone & Jones, 1980.) In short, we tend to view those who are different to us (the “outgroup”) as all being very similar, while those who are like us (the “ingroup”) are more diverse. For example, all women are chatty, but some men are talkative, some are quiet, some are stoic, some are more emotional, some are cautious, others are more risky… etc. We do this a lot in marketing! “All millenials are spoiled, entitled little brats. All Baby Boomers are old fuddies who don’t know how to use a computer.”
  • 83. @michelejkiss@michelejkiss@michelejkiss “everyone in [countryX] behaves like Y” Why this matters for analysts: Similar to the False Consensus Effect, where we may analyse user behaviour assuming everyone thinks as we do, the Homogeneity of the Outgroup suggests that we may OVERSIMPLIFY the behaviour of customers who are different to us, and fail to fully appreciate the nuance of varied behaviour. This may seriously bias our analyses! For example, if we are a large global company, an analysis of customers in another region may be seriously flawed if we are assuming customers in the region are “all the same.” To overcome this tendency, we might consider leveraging local teams or local analysts to conduct or vet such analyses. Or, add qualitative data to give context (and maybe an opposing viewpoint, to our assumptions!)
  • 84. @michelejkiss@michelejkiss THE HAWTHORNE EFFECT @michelejkiss[Landsberger, 1955] The story of the Hawthorne Effect goes as follows - In 1955, Henry Landsberger analyzed several studies conducted between 1924 and 1932 at the Hawthorne Works factory outside Chicago. These studies originally intended to discover whether the level of light (among other changes) within a building changed the productivity of workers. They found that the changes did have an impact, however they were short-term improvements, AND they seemed to happen due to the fact that a change was made – not the result of the actual change. Aka, it wasn’t the fact that the light was brighter, but the fact that the light had been changed.
  • 85. @michelejkiss However… @michelejkiss However, the study has been the subject of much criticism, including:
  • 86. @michelejkiss@michelejkiss “Glorified anecdote” Once you've got the anecdote, you can throw away the data. Dr. Richard Nisbett
  • 88. @michelejkiss Demand Characteristics @michelejkiss[Martin Orne] One possibility is that the effects were due to “demand characteristics”, and the fact that the employees knew they were being observed. (Martin Orne) (This can go either way – participants may be looking to “please” the researchers, or “correctly” complete the study, or they may seek to sabotage studies)
  • 89. @michelejkiss@michelejkiss[Levitt & List, 2011] But in 2011, Levitt & List conducted a re-analysis of the original data. They actually had microfilm, so they were able to go right to the source. What they found… all of the changes were made on a SUNDAY, as it was the only day the factor was closed. So by…
  • 90. @michelejkiss monday @michelejkiss[Levitt & List, 2011] … Monday, the works were coming in, refreshed from a day off… And were more productive!
  • 92. @michelejkiss@michelejkiss Multiple Explanations @michelejkiss One analysis or set of data may have multiple explanations, and be subject to very different interpretations depending on who is doing the analysis.
  • 93. @michelejkiss People may not agree People may not agree with your conclusions. USE THIS to produce a more comprehensive analysis. Take those viewpoints in to account. Respond to them, disprove those hypotheses.
  • 94. @michelejkiss@michelejkiss Observation may change behaviour @michelejkiss Demand characteristics suggest that observing people may change their behaviour. We need to be aware of this possibility when analyzing behaviour. For example, what data would you get from an anonymous survey, versus one that requires a person’s name and phone number? (Or their employee ID!) How do user testing and a/b testing results differ, based on people’s level of awareness that they are being observed?
  • 96. @michelejkiss@michelejkiss Be aware of your biases @michelejkiss Be aware of your biases, and attempt to control for them. • Are you just using the data to “prove” one particular hypothesis, rather than looking for all the possible explanations? • Are you overgeneralizing to a certain population? • Is your interpretation of the data being influenced by conforming to your company’s norms, and the way things have always been done?
  • 97. @michelejkiss@michelejkiss Work with people’s natural abilities Be aware of the limitations of yourself, and others. Work WITH people’s abilities. Don’t overburden their short term memory. And keep in mind the factors that might be influencing their memory, and their behaviour.
  • 98. @michelejkiss@michelejkiss Get a second opinion @michelejkiss Get someone to keep your biases in check! Have someone else to vet your assumptions. (Especially someone unfamiliar/who hasn’t been involved with your analysis or experiment so far.)
  • 99. @michelejkiss@michelejkiss Our best defense is awareness @michelejkiss The best defense we have is our AWARENESS. While we are analysts, and we are trained to approach problems rationally and methodically, we are human too, and our best analysis comes from being aware of the influences of our humanness on our interpretation of the data.

Editor's Notes

  1. https://www.pexels.com/photo/black-pen-on-white-book-page-159621/
  2. Talk about educational background – unrelated to analytics, law and psychology, totally irrelevant Or is it? Ormond image: https://en.wikipedia.org/wiki/File:Ormond2011.jpg Alternate image of Melb Uni: https://www.google.com/imgres?imgurl=https%3A%2F%2Fteanabroad.org%2Fwp-content%2Fuploads%2F2015%2F12%2FUniMelb-Testimonial-2-1.jpg&imgrefurl=https%3A%2F%2Fteanabroad.org%2Fprograms%2Faustralia%2Fmelbourne%2Funiversity-of-melbourne%2F&docid=MNXCHIfa3_ar5M&tbnid=LXw0mBtgnZOeFM%3A&vet=10ahUKEwjT_fyw9arYAhWJw1QKHSb1BfcQMwhDKAUwBQ..i&w=1280&h=720&bih=1137&biw=2149&q=university%20of%20melbourne&ved=0ahUKEwjT_fyw9arYAhWJw1QKHSb1BfcQMwhDKAUwBQ&iact=mrc&uact=8
  3. Analytics, optimization, etc are all about using quantitative methods to understand PEOPLE, and why we do the stupid stuff we do Today we’re going to talk about some classic studies, and why they matter (and how they help us understand human behaviour) https://pixabay.com/en/a-i-ai-anatomy-2729794/
  4. https://www.pexels.com/photo/background-blur-bokeh-bright-220067/
  5. Those of us working in analytics are pretty familiar with the underlying tenets of data visualization, dashboards, and likely, with Stephen Few.
  6. Stephen Few declared a dashboard to be:
  7. The reason why we need the information to be on one page or screen? The limits of our memory.
  8. In 1956, George A. Miller published one of the most cited papers in psychology, where he spoke of the magic number 7. For our purposes here, Miller found that the amount of information that we can retain our working (or short term) memory is seven, plus or minus two.
  9. However, this doesn’t mean that we can only recall 7 +/- 2 individual letters, or numbers (for example.) The 7 +/- 2 items can be 7 +/- 2 letters, or 7 +/- 2 words, or …
  10. … or 7 +/- 2 song lyrics. We are capable of “chunking” information, so that each piece of information only occupies one “item” within our memory system. https://www.pexels.com/photo/adult-beautiful-blur-casual-374703/
  11. Because, back to Stephen Few’s definition, if you are presenting information, you need to work WITHIN the limits of our perceptual and memory systems. Expecting users to draw connections between two data points six pages apart, or thirty slides ago, is a recipe for failure.
  12. https://www.pexels.com/photo/close-up-of-photo-of-books-327882/
  13. In 1957, Leon Festinger studied a Doomsday cult who believed that aliens would rescue them from a coming flood. Unsurprisingly, no flood (nor aliens) eventuated. However, rather than being proven wrong, the group explained away that they had been “spared”, and clung even more tightly to their beliefs. In their book, When Prophecy Fails, Festinger (et al) said:
  14. This might happen especially if: * Beliefs are deeply held; * Action has been taken, that is difficult to undo; (for example, the Doomsday cult had sold all their possessions) * Believer has social support (the cult continues to reinforce each others’ beliefs) https://unsplash.com/photos/asEF6J0LZ44
  15. Festinger explains this by the theory of cognitive dissonance. This theory tells us that we don’t like the feeling of inconsistency (for example, of our beliefs and our actions.) We seek to reduce this uncomfortable feeling by justifying our beliefs, and avoiding information that might further conflict. For example, smokers who know smoking is bad for them may choose to tell themselves, “It’s not so bad, it’s not as bad as crack cocaine.” We will change our beliefs, to match our behaviour! In the case of the Doomsday cult, the idea that they had sold all their possessions and believed something so asinine is an uncomfortable disconnect, so they choose to believe they were “spared”, which justifies their actions. https://www.flickr.com/photos/emiliano-iko/32383646475/in/photostream/
  16. It also explains why you can never convince your racist uncle of anything.
  17. However, Chanel, Luchini, Massoni, Vergnaud (2010) found that if we are given an opportunity to discuss the evidence and exchange arguments with someone (rather than just reading the evidence and pondering it alone) we are more likely to change our minds, in the face of opposing facts. So… maybe there’s some value in talking to your racist uncle. https://www.pexels.com/photo/beverage-cafe-coffee-coffee-shop-604897/
  18. Why this matters for analysts: Even if your data seems self-evident, if you come in with “breaking news”, that goes against what the business has known, thought, or believed for some time, you may need more data to support your contrary viewpoint. You may also want to allow for plenty of time for discussion, rather than simply sending out your findings, as those discussions are critical to getting buy-in for this new viewpoint. https://www.pexels.com/photo/administration-articles-bank-black-and-white-261949/ \Other option for image: curtain, like “avoid the big reveal” - https://www.pexels.com/photo/people-at-theater-713149/
  19. We know now that “the facts” may not persuade us, even when brought to our attention. However, Confirmation Bias tells us that we intentionally seek out information that continually reinforces our beliefs, rather than searching for all evidence and fully evaluating the possible explanations. PAID https://www.shutterstock.com/image-photo/woman-laptop-working-planning-thinking-concept-303468863?irgwc=1&utm_medium=Affiliate&utm_campaign=Hans%20Braxmeier%20und%20Simon%20Steinberger%20GbR&utm_source=44814&utm_term=
  20. In a 1967 study by Brock & Balloun, subjects listened to several messages, but the recording was staticky. However, the subjects could press a button to clear up the static. They found that people selectively chose to listen to the message that affirmed their existing beliefs. For example, smokers chose to listen more closely when the content disputed a smoking-cancer link.
  21. Wason (1960) conducted a study where participants were presented with a math problem: find the pattern in a series of numbers, such as “2-4-6.” Participants could create three subsequent sets of numbers to “test” their theory, and the researcher would confirm whether these sets followed the pattern or not.
  22. Rather than collecting a list of possible patterns, and using their three “guesses” to prove or disprove each possible pattern, Wason found that participants would come up with a single hypothesis, then seek to prove it. (For example, they might hypothesize that “the pattern is even numbers” and check whether “8-10-12”, “6-8-10” and “20-30-40” correctly matched the pattern.
  23. Their guesses were all multiple versions of the same hypothesis.
  24. When it was confirmed their guesses matched the pattern, they simply stopped. However, the actual pattern was “increasing numbers” – their hypothesis was not correct at all!
  25. Do you do this with your analysis? When you start analyzing data, where do you start? With a hunch, that you seek to prove, then stop your analysis there? (For example, “I think our website traffic is down because our paid search spend decreased.”) Or with multiple hypotheses, which you seek to evaluate one by one?
  26. An alternative – Analysis of Competing Hypotheses Analysis of Competing Hypotheses (or “ACH” was developed in the 1970’s, for use by intelligence analysis. It is an attempt to overcome the natural biases that we all have. The idea is, you come up with multiple hypotheses that could explain the data you are seeing, and seek to evaluate each one in turn. Which ones are more or less likely to be true? What data do you have, and which hypotheses does it support? This attempts to avoid our analysis simply being a self-fulfilling prophecy.
  27. https://pixabay.com/en/zebra-wildlife-africa-animal-1141302/
  28. In 1951, Asch found that we conform to the views of others, even when they are flat-out wrong, surprisingly often! He conducted an experiment where participants were seated in a group of eight others who were “in” on the experiment (“confederates.”) Participants were asked to judge whether a line was most similar in length to three other lines. The task was not particularly “grey area” – there was an obvious right and wrong answer. Each person in the group gave their answer verbally, in turn. The confederates were instructed to give the incorrect answer, and the participant was the sixth of the group to answer.
  29. Asch was surprised to find that 76% of people conformed to others’ (incorrect) conclusions at least once.
  30. Asch was surprised to find that 76% of people conformed to others’ (incorrect) conclusions at least once.
  31. Only 25% never once agreed with the group’s incorrect answers. (The overall conformity rate was 33%.)
  32. https://www.pexels.com/photo/writing-notes-idea-class-7103/
  33. As mentioned previously, if new findings contradict existing beliefs, it may take MORE than just presenting new data.
  34. However, these conformity studies suggest that efforts to do so may be further hampered if you are presenting information to a group. It is less likely that people will stand up for your new findings against the norm of the group. In this case, you may be better to discuss your findings slowly to individuals, and avoid putting people on the spot to agree/disagree within a group setting. https://unsplash.com/photos/WHYHfmBuCB4
  35. Similarly, this argues against jumping straight to a “group brainstorming” session. Once in a group, Asch demonstrated that 76% of us will agree with the group (even if they’re wrong!) so we stand the best chance of getting more varied ideas and minimising “group think” by allowing for individual, uninhibited brainstorming and collection of all ideas first. https://www.pexels.com/photo/people-meeting-workspace-team-7097/
  36. The serial position effect (so named by Ebbinghaus in 1913) finds that we are most likely to recall the FIRST and LAST items in a list, and least likely to recall those in the middle. https://www.pexels.com/photo/athletes-running-on-track-and-field-oval-in-grayscale-photography-34514/
  37. The primacy and recency effects tell us we are most likely to remember the first and last items, and least likely to remember those in the middle.
  38. For example, let’s say you are asked to recall pear, orange, banana, apple and pineapple.
  39. The serial position effect suggests that individuals are more likely to remember pear (the first item; primacy effect) and pineapple (the final item; recency effect) and less likely to remember the items in the middle.
  40. The theory here is that the last item is retained in short term memory …
  41. … while the first item is more likely to have been processed through to long term memory. Also, the last item is being remembered “in context” – the situation in which they’re recalling it is CLOSER to when they learned the last item, than the first.
  42. In 1975, Godden & Baddeley conducted experiments where they had scuba divers memorize information, and found their recall was better when the context in which they were recalling it was the same as when they were remembering it. (For example, they recalled better when they memorized underwater, and recalled underwater, than when they tried to recall on land.) (This is also why we can’t remember what the heck we were thinking until we get up, and go back to the spot we first had the thought. Then bam! It comes to us.) https://www.pexels.com/photo/human-in-black-orange-swimming-suit-in-blue-body-of-water-37530/
  43. The longer the list, the less likely the primacy effect is to apply. (Murdock, Bennet (1962). "Serial Position Effect of Free Recall" (PDF). Journal of Experimental Psychology. 64 (5): 482–488. doi:10.1037/h0045106.)
  44. The recency effect can be reduced, by requiring attention be switched to another task, and then return to the recall. (That way, the information was forced out of the short term memory.)
  45. Asch (1964) found participants told “Steve is smart, diligent, critical, impulsive, and jealous” had a positive evaluation of Steve, https://www.pexels.com/photo/close-up-face-fashion-fine-looking-450212/
  46. whereas participants told “Steve is jealous, impulsive, critical, diligent, and smart” had a negative evaluation of Steve. Even though the adjectives are the exact same – only the order is different! https://www.pexels.com/photo/close-up-face-fashion-fine-looking-450212/
  47. This is related to the recency bias.
  48. The recency bias tells us we make decisions based on the most recent information, rather than on the entire long-term picture (and we may assume that current trends will continue in to the future.) For example, I should sell everything and buy bitcoin. Or, I want the sporty convertible. Last winter wasn’t “that bad” (forget the fact that normally we need four wheel drive to get out of our blacked-out neighborhood, due to terrible snowstorms.)
  49. Why does this matter for analytics? When you present information, your audience is unlikely to remember everything you tell them. So choose wisely. What do you lead with? What do you end with? And what do you prioritize lower, and save for the middle? These findings may also affect the amount of information you provide at one time, and the cadence with which you do so. If you want more retained, you may wish to present smaller amounts of data more slowly, rather than rapid-firing with constant information. For example, rather than presenting twelve different “optimisation opportunities” at once, focusing on one may increase the likelihood that action is taken. This is also an excellent argument against a 50-slide PowerPoint presentation – while you may have mentioned something in it, if it was 22 slides ago, the chance of your audience remembering are slim.
  50. Related to optimization Primacy: The idea that people are familiar with the old experience, so there’s an advantage to the control over the new variation. Novelty: The idea that the new experience is something cool and different, so there is a (short term) life, that won’t sustain over time.
  51. However, sometimes the idea of primacy and recency can be taken too far, and be used to explain results that are simply not driven by them. A/B test results often show dramatic differences in the early days, that seldom continue on. (This is due to the high variance and lower sample size at the start of the test.) However, if the product team has put their sweat and soul in to a feature, it’s easy to ascribe an actual reason to the early results, outside of statistical fluctuation. For example, “Oh, people are just used to the old design, and the variation will perform AMAZINGLY once they get used to it.” However, it’s rare for these early effects to completely flip the other way. So, while the low performance may not continue, it’s also unlikely that the effects will completely reverse. I’ve even seen the “primacy effect” be argued as the reason for the control to perform better (after a reasonable amount of time) in sites that have little to no return visitation! http://www.exp-platform.com/Documents/puzzlingOutcomesInControlledExperiments.pdf
  52. https://pixabay.com/en/halo-sun-circle-atmosphere-rays-2646333/
  53. In 1977, Nisbet and Wilson conducted an experiment with university students. The two students watched a video of the same lecturer deliver the same material, but one group saw a warm and friendly “version” of the lecturer, while the other saw the lecturer present in a cold and distant way. The group who saw the friendly version rated the lecturer as more attractive and likeable. https://pixabay.com/en/conference-public-speaking-2705706/
  54. There are plenty of other examples of this. For example, “physically attractive” students have been found to receive higher grades and/or test scores than “unattractive” students at a variety of ages, including elementary school (Salvia, Algozzine, & Sheare, 1977; Zahr, 1985), high school (Felson, 1980) and college (Singer, 1964.) https://www.pexels.com/photo/girls-on-desk-looking-at-notebook-159823/
  55. Thorndike (1920) found similar effects within the military, where a perception of a subordinate’s intelligence tended to lead to a perception of other positive characteristics such as loyalty or bravery. https://pixabay.com/en/soldiers-military-attention-salute-559761/
  56. Why this matters for analysts: The appearance of your reports/dashboards/analyses, the way you present to a group, your presentation style, even your appearance may affect how others judge your credibility and intelligence.
  57. The Halo Effect can also influence the data you are analysing! It is common with surveys (especially in the case of lengthy surveys) that happy customers will simply respond “10/10” for everything, and unhappy customers will rate “1/10” for everything – even if parts of the experience differed from their overall perception. For example, if a customer had a poor shipping experience, they may extend that negative feeling about the interaction with the brand to all aspects of the interaction – even if only the last part was bad! (And note here: There’s a definite interplay between the Halo Effect and the Recency Effect!)
  58. In 1964 in New York City, a woman name Kitty Genovese was murdered. According to papers, the attack had lasted an hour, and almost forty people witnessed the attack. Yet, no one called the police. Worth noting - later reports suggested this was an exaggeration – that there had been fewer witnesses, and that some had, in fact, called the police. However, this event fascinated psychologists, and triggered several experiments. This became known as the “Bystander Effect”, which proposes that the more bystanders that are present, the less likely it is that an individual will step in and help.
  59. Darley & Latane (1968) manufactured a medical emergency, where one participant was allegedly having an epileptic seizure, and measured how long it took for participants to help. They found that the more participants, the longer it took to respond to the emergency. (This is actually why, if you go to CPR training these days, the FIRST thing you’re instructed to do is to point to a specific person, and tell them “Go call 911.” Because, if you leave it up to an anonymous group, everyone may assume everyone else will call, and no one calls 911!) (DON’T HAVE LICENSE FOR THIS PHOTO)
  60. T
  61. Think about how hard an INDIVIDUAL works in a tug of war with 20 people on one side, versus 2.
  62. Why this matters for analysts: Think about how you present your analyses and recommendations. If you offer them to a large group, without specific responsibility to any individual to act upon them, you decrease the likelihood of any action being taken at all. So when you make a recommendation, be specific. Who should be taking action on this? If your recommendation is a generic “we should do X”, it’s far less likely to happen. Image already bought for MarTech (shutterstock)
  63. In 1999, Simons and Chabris conducted an experiment in awareness at Harvard University. Participants were asked to watch a video of basketball players, where one team was wearing white shirts, and the other team was wearing black shirts. In the video, the white team and black team respectively were passing the ball to each other. Participants were asked to count the number of passes between players of the white team.
  64. (15) Did you see it?? During the video, a man dressed as a gorilla walked into the middle of the court, faced the camera and thumps his chest, then leaves (spending a total of 9 seconds on the screen.) Amazingly? Half of the participants missed the gorilla entirely! Since then, this has been termed “the Invisible Gorilla” experiment. Participants were SO FOCUSED on counting the passes of the white team that they completely missed it. … Selective attention.
  65. Why this matters for analysts: As you are analyzing data, there can be huge, gaping issues that you may not even notice. When we get SO FOCUSED on a particular task (for example, counting passes by the white-shirt players only, or analyzing one subset of our customers) we may overlook something significant at a higher level. Take time before you finalize or present your analysis to think of what other possible explanations, or variables there could be (what could you be missing?) -- or invite a colleague to poke holes in your work.
  66. Experiments have revealed that we tend to believe in a false consensus: that others would respond similarly to the way that we would. For example, Ross, Greene & House (1977) provided participants with a scenario, with two different possible ways of responding. Participants were asked to explain which option THEY would choose, and guess what OTHER PEOPLE would choose. Regardless of which option they actually chose, participants believed that other people would choose the same one. https://unsplash.com/photos/4V1dC_eoCwg
  67. Why this matters for analysts: As you are analyzing data, you are looking at the behaviour of real people. It’s easy to make assumptions about how they will react, or why they did what they did, based on what YOU would do. But our analysis will be far more valuable if we can be aware of those assumptions, and actively seek to understand why our actual customers did these things – without relying on assumptions. https://www.pexels.com/photo/adult-casual-collection-fashion-296881/
  68. There is a related effect here: the Homogeneity of the Outgroup. (Quattrone & Jones, 1980.) In short, we tend to view those who are different to us (the “outgroup”) as all being very similar, while those who are like us (the “ingroup”) are more diverse. For example, all women are chatty, but some men are talkative, some are quiet, some are stoic, some are more emotional, some are cautious, others are more risky… etc. https://pixabay.com/en/football-american-scrimmage-line-557206/
  69. Why this matters for analysts: Similar to the False Consensus Effect, where we may analyse user behaviour assuming everyone thinks as we do, the Homogeneity of the Outgroup suggests that we may OVERSIMPLIFY the behaviour of customers who are different to us, and fail to fully appreciate the nuance of varied behaviour. This may seriously bias our analyses! For example, if we are a large global company, an analysis of customers in another region may be seriously flawed if we are assuming customers in the region are “all the same.” To overcome this tendency, we might consider leveraging local teams or local analysts to conduct or vet such analyses. https://www.pexels.com/photo/black-textile-41949/
  70. The story of the Hawthorne Effect goes as follows - In 1955, Henry Landsberger analyzed several studies conducted between 1924 and 1932 at the Hawthorne Works factory. These studies originally intended to discover whether the level of light (among other changes) within a building changed the productivity of workers. They found that the changes did have an impact, however they were short-term improvements, AND they seemed to happen due to the fact that a change was made – not the result of the actual change. Aka, it wasn’t the fact that the light was brighter, but the fact that the light had been changed.
  71. However, the study has been the subject of much criticism, including:
  72. https://roosterillusionreviews.files.wordpress.com/2013/02/5-helens.jpg
  73. The changes were due to “demand characteristics”, and the fact that they were being observed. (Martin Orne) This can go either way – participants may be looking to “please” the researchers, or “correctly” complete the study, or they may seek to sabotage studies http://www.timewarnermedialab.com/sites/timewarnermedialab/files/styles/900x400_capabilities_slider/public/observation_02.jpg?itok=oN68S2zf
  74. A 2011 re-analysis of the study (using microfilm) suggested that the changes were made on a Sunday (the only day the factory was closed) - so by Monday the workers productivity was refreshed by having a day off. Steven Levitt and John A. List
  75. A 2011 re-analysis of the study (using microfilm) suggested that the changes were made on a Sunday (the only day the factory was closed) - so by Monday the workers productivity was refreshed by having a day off. Steven Levitt and John A. List
  76. One analysis or set of data may have multiple explanations, and be subject to very different interpretations depending on who is doing the analysis.
  77. People may not agree with your conclusions. USE THIS to produce a more comprehensive analysis. Take those viewpoints in to account. Respond to them, disprove those hypotheses.
  78. Demand characteristics suggest that observing people may change their behaviour. We need to be aware of this possibility when analyzing behaviour. For example, what data would you get from an anonymous survey, versus one that requires a person’s name and phone number? How do user testing and a/b testing results differ, based on people’s level of awareness that they are being observed?
  79. So, what should you take away from this? https://pixabay.com/en/question-mark-hand-drawn-solution-2123969/
  80. Be aware of your biases, and attempt to control for them. Are you just using the data to “prove” one particular hypothesis, rather than looking for all the possible explanations? Are you overgeneralizing to a certain population? Is your interpretation of the data being influenced by conforming to your company’s norms, and the way things have always been done? https://www.pexels.com/photo/calm-daylight-evening-grass-267967/
  81. Be aware of the limitations of yourself, and others. Work WITH people’s abilities. Don’t overburden their short term memory. And keep in mind the factors that might be influencing their memory, and their behaviour. https://unsplash.com/photos/gI_s5Rlw63g
  82. Get someone to keep your biases in check! Have someone else to vet your assumptions. (Especially someone unfamiliar/who hasn’t been involved with your analysis or experiment so far.) https://www.pexels.com/photo/advice-advise-advisor-business-7075/
  83. The best defense we have is our AWARENESS. While we are analysts, and we are trained to approach problems rationally and methodically, we are human too, and our best analysis comes from being aware of the influences of our humanness on our interpretation of the data. https://unsplash.com/photos/kqguzgvYrtM
  84. https://unsplash.com/photos/R6xx6fnvPT8