This was a day long workshop I gave at the NZTester 2015 conference. Looking at psychology and cognitive science research and how to apply that to testing.

1. Heuristics, Bias and Critical
Thinking in Testing
Getting to know the most powerful test tool in the world; your brain

2. Quotes
• “There is nothing so useless as
doing efficiently that which
should not be done at all.”
• Peter Drucker
• “Our comforting conviction
that the world makes sense
rests on a secure foundation:
our almost unlimited ability to
ignore our ignorance.”
• Daniel Kahneman

3. Agenda
• What is testing?
• Why do we test?
• What is the value of testing?
• Why do people pay for testing?
• Rocket science
• What are we on about today?
• What are heuristics?
• Why is bias important?
• What is critical thinking
• The Brain Rules
• What are they?
• Why are they important to testing?
• Thinking Fast and Slow
• Systems 1 and 2
• Heuristics
• Bias
• Critical Thinking Skills
• How can I use them?
• Breaking System 1
• Putting it all together

4. Introductions

5. Introduce yourself
What’s your name?
What’s your job?
What is something
people don’t know
about you?
What do you want from
today?

6. Me
Husband and Father
Philosopher and
Psychology Trained
Been a restaurateur,
Tutor, Teacher, Tester,
Leader, Strategist,
Transformer

7. What are we on about today?
How to be a better Test
Analyst by:
Understanding your brain
Understanding your biases
Understanding critical
thinking

8. Lets start thinking now!
Why do we test?
What is the value of
testing?
Why do people pay for
testing?

9. Rocket Science

10. Some quick definitions
All from Wikipedia; for better or worse

11. Heuristic
• A heuristic, is any approach to problem solving, learning, or discovery
that employs a practical methodology not guaranteed to be optimal
or perfect, but sufficient for the immediate goals.
• Where finding an optimal solution is impossible or impractical,
heuristic methods can be used to speed up the process of finding a
satisfactory solution. Heuristics can be mental shortcuts that ease the
cognitive load of making a decision.
• Examples of this method include using a rule of thumb, an educated
guess, an intuitive judgment, stereotyping, profiling, or common
sense.

12. Cognitive Bias
• A cognitive bias is a pattern of deviation in judgment, whereby
inferences about other people and situations may be drawn in an
illogical fashion.
• Individuals create their own "subjective social reality" from their
perception of the input. An individual's construction of social reality,
not the objective input, may dictate their behaviour in the social
world.
• Thus, cognitive biases may sometimes lead to perceptual distortion,
inaccurate judgment, illogical interpretation, or what is broadly called
irrationality.

13. Critical Thinking
• Critical thinking is clear, reasoned thinking involving critique. Its
details vary amongst those who define it.
• According to Beyer (1995), critical thinking means making clear,
reasoned judgments. During the process of critical thinking, ideas
should be reasoned and well thought out/judged.
• The National Council for Excellence in Critical Thinking defines critical
thinking as the intellectually disciplined process of actively and
skilfully conceptualizing, applying, analysing, synthesizing, and/or
evaluating information gathered from, or generated by, observation,
experience, reflection, reasoning, or communication, as a guide to
belief and action.'

14. Cognitive Bias Critical Thinking Heuristic

15. The Brain Rules

17. Exercise and Survival
Exercise boosts brain power The human brain evolved too

18. Attention and Short Term Memory
We don’t pay attention to boring
things Repeat to remember

19. Long Term Memory and Stress
Remember to repeat
Stressed brains don’t learn the same
way

20. Vision and Exploration
Vision trumps all other senses We are powerful and natural explorers.
The desire to explore never
leaves us

21. Hang on! What about Wiring, Sleep, Gender
and Sensory Integration?
Your brain is
unique Sleep well
think well
Men’s and
women’s brains
really are
different
Stimulate multiple
senses to learn well

22. Heuristics and Cognitive Bias

23. An individual has been described by a neighbour
as follows:
“Steve is very shy and withdrawn, invariably
helpful but with little interest in people or the
world of reality. A meek and tidy soul, he has a
need for order and structure, and a passion for
detail.”
Is Steve more likely to be a librarian or a farmer?
Steve is selected at random from a
representative sample:

24. The two systems of your brain
https://www.youtube.com/watch
?v=KyM3d4gQGhM
“Your mind doesn’t stop on what
you don’t know, it just makes the
best of what you do know.”
https://www.youtube.com/watch
?v=uqXVAo7dVRU
“The problem however really
arises when we use system 1
instead of system 2 when system
2 would be the appropriate
system to use, and this leads us to
all kinds of biases and fallacies
that are not optimal.”

25. Easy Hard
Your brain is lazy and likes to do what comes easily

26. It’s easy for your brain to use Heuristics
Not this This

27. An example heuristic and bias - Priming
Priming is an implicit memory effect in which exposure
to one stimulus influences the response to another
stimulus.
https://en.wikipedia.org/wiki/Priming_%28psychology
%29

28. Where Heuristics lead to faulty thinking we
are Biased
Bananas Vomit

29. Cognitive Ease
Related
Experience
Clear Display
Primed Idea
Good Mood
Cognitive Ease
Feels Good
Feels Familiar
Feels True
Feels Effortless

30. Mood effects the working of system 1
“When we are
uncomfortable and
unhappy we lose touch
with our intuition.”
Daniel Kahneman
“When in a good mood
people become more intuitive
and more creative…also less
vigilant and more prone to
logical errors.”
Daniel Kahneman

31. Critical Thinking
Making the best of Systems 1 and 2 and using the Brain Rules to your advantage
How to be a better tester

32. Test the Rocket Science
You have 5 minutes
Decide on as many tests
as possible
Compare your tests with a
partner

33. What tests would you run?

34. What is a Critical Thinker?
Based on what we
know how credible is
this?
What if we
have to
decide and
we don’t
have all the
facts

35. Critical Thinking Tools
Understand Fallacies
•http://www.informationisbeautiful.net/visualizations/rhetological-
fallacies/
Use analytic tools
•http://www.softwaretestingmentor.com/category/test-design-
techniques/
•http://www.umich.edu/~elements/5e/probsolv/strategy/ctskills.htm
Ask more questions
•http://changingminds.org/techniques/questioning/questioning.htm
“Once in school, children’s natural
tendency to learn by questioning
mysteriously evaporates. On that
first day of school, the adult
becomes questioner, while the
student becomes the answerer”
-James Cooper

36. Core Critical Thinking Skills

37. Becoming a better Test
Professional
What does all this mean?

38. What do you think?

39. Some ideas
Combining vision and exploration for
powerful test representation
Visual
input is
best
We are
natural
explorers
Visual analysis
and
representations
of tests (and
plans)
De-stress yourself and make it interesting
and find your work more stimulating
Make it
interesting
Remove
Stress
Cognitively
Engaged
Testers

40. Some more ideas
Both/And not Either/Or
Up Front
Test
Analysis
On the spot
exploration
Powerful and
comprehensive
test
programme
A good environment, regular hours and
good sleep
Change
your
space
Regular
Hours
Stimulated,
energetic
testers

41. Even more ideas
Where would you like to work, how can
you make where you do work more like
where you would like to work?
Engaging physical
space
Regular hours
Visual representation
of tests
Combining analytic and exploratory
approaches
• Test• Explore
• Analyse• Review
Act Plan
DoCheck

42. (Yet) more ideas
Work with people’s natural circadian
rhythms
Test Analysis
in the
morning –
make it fun
Execution in the
afternoon – food
and competition
Planning
book ends
the day
Structure your teams to have variety of
attitudes/approaches/strengths
Inspirer Questioner Analyst Designer Completer
Limit hierarchy to the minimum, horizontal
equality facilitates problem solving, debate and
innovation.

43. Bibliography and References
• Thinking Fast and Slow – Daniel Kahneman
• https://www.youtube.com/watch?v=KyM3d4gQGhM
• The Brain Rules – John Medina
• http://www.brainrules.net/
• Index Cards by Jessica Hagy
• http://thisisindexed.com/
• David McCandless’ Information is Beautiful
• http://www.informationisbeautiful.net/visualizations/rhetological-fallacies/
• Check out StrengthsFinder and Belbin Team Roles
• http://gallupstrengthscenter.com
• http://www.belbin.com/

44. Supplementary Information

45. Another view of Critical Thinking
https://tamimunzani.files.wordpress.com/2010/05/thinking_skills_lr1.jpg