1. What Counts as (Useful) Evidence?
Lisa Farlow
Oct 25, 2014

2. Overview
• Some definitions
• What makes for a good hypothesis?
• What counts as useful evidence?
• What makes for a good test?

3. What Counts as Knowledge?
• Start with a belief/hunch/hypothesis/feeling
• Gather evidence to justify that belief
• Knowledge is justified true belief (Plato!)

4. Types of Evidence
• Any observation about the world can be
evidence
• Before gathering evidence, your hypothesis
has an initial likelihood of being true
– Some evidence increases the likelihood
– Some evidence decreases the likelihood
– Some evidence is irrelevant to the hypothesis

5. HYPOTHESES

6. Start with a Hypothesis
• A good hypothesis is one in which…
– You can find evidence to support/deny it
(testable)
– It is possible to find evidence that proves the
hypothesis wrong (falsifiable)

7. Hypotheses that aren’t testable
• I will have twice as many feelings in December
as in November (not measurable)
• The spell I cast caused my favourite sports
team to win, but it can only be used once (not
repeatable)
• An afterlife exists (not observable)

8. Hypotheses that aren’t falsifiable
• A teapot orbits the Sun somewhere in space
between Earth and Mars
• There is a ghost in the basement of my parents’
house
• I know a lot of people think the new feature on
our site is clunky, but some people love it and
find it extremely useful
“Absence of evidence is not evidence of absence.”
(Carl Sagan)

9. Some Good Hypotheses
• People located in Alberta spend more time on
our site than people located in BC
• Some people struggle to fill out the PDFs on
our site
• On average, people will be able to find our
contact info more quickly if we title the tab
“Give us a Ring-a-Ding” rather than “Contact
Us” (though it is a bad idea, it is a good
hypothesis)

10. WAYS OF TALKING ABOUT EVIDENCE

11. Ways of Talking About Evidence
• All, Some, None
• All, Only, All & Only
• Necessary, Sufficient, Necessary &Sufficient

12. All or None
• All Xs are Y
• Ex: All swans are white
– Confirm: Examine every single swan that has ever
existed in all of time and space and observe that it
is white
– Deny: Find a single swan that is not white
• Same for All Xs are not Y (i.e, No Xs are Y, a
“none” statements)

13. Some (“there exists at least one”)
• Some Xs are Y
• Ex: Some dogs are able to talk
– Confirm: Find a single dog that is able to talk
– Deny: Examine every single dog and determine if
he can talk
• Same for some Xs are not Y (Not all Xs are Y)

14. Open and Closed Worlds
• Open world of data
– You cannot confirm “All” statements or deny “Some”
statements
– Ex: Analytics search key words. “Nobody found your
site by searching for ‘worst burgers in Edmonton’
• Closed world of data
– It’s possible to examine every piece of data
– You can confirm “All” statements and deny “Some”
statements
– Include closed-world caveats
• “Of those we interviewed….”
• “Of those who responded to the survey…”

15. All, Only, All & Only
• All Xs are Y
– Confirm: Examine every X within the study and
observe that it is Y
– Deny: Find a single X that is not Y

16. All, Only, All & Only
• Only Xs are Y
– Confirm: Examine all things that are Y, observe
that they are X
– Deny: Find a non-X that is Y

17. All, Only, All & Only
• All and Only Xs are Y
– Confirm: Examine every X and observe that it is Y
AND examine everything else within the study
and observe that it is not Y
– Deny: Find a single X that is not Y OR find a single
thing that is Y

18. Case Study: All and Only
• You’re running a survey to find out student
attitudes about the cafeteria serving more
vegetables.
• What counts as evidence for/against each
hypothesis?

19. Case Study: All and Only
• Hypothesis: Only female students are more
willing to more spend money on healthier
options
– Confirm: Look through all respondents who said
they were willing to spend more. Determine
whether 100% of them are girls
– Deny: Look through all respondents who said
they were willing to spend more. Find one boy.

20. Case Study: All and Only
• Hypothesis: All and only the grade 6 students
bring healthy snacks from home
– Confirm: Look through all respondents…
– Deny: Look through all respondents …

21. Case Study: All and Only
• Hypothesis: None of the chemistry students
think pizza sauce counts as a vegetable
– Confirm: Look through all respondents…
– Deny: Look through all respondents….

22. Necessary & Sufficient Evidence
• Necessary (If H is true, then E must be
observed)
– Being a woman is a necessary condition to
become a nun
– Submitting your resume is a necessary condition
to getting an interview
– Spending 20+ minutes on the site is necessary for
you to be classified as a happy, high-traffic user

23. Necessary & Sufficient Evidence
• Sufficient (If E is observed, you know H is
true)
– If it’s been raining hard for a while, I know the
ground will be wet
– If you voted for Don Iveson in the last election, I
know you are at least 18 years old
– If you renewed your library books online, I know
you have a a library card

24. Case Study: Necessary & Sufficient
• Hypothesis: An online wizard is a better registration
process for our car-share program than our mail-in PDF

25. What are the necessary conditions for success?

26. Necessary but Insufficient
• You observe: “very few of the graduating class
carry federal or provincial students loans”
• You cannot conclude: “very few students take
out loans to pay for school”
• Nor can you conclude: “very few students
take out federal/provincial loans”

27. Necessary but Insufficient
• You observe: “page abc has more pageviews
than any other page”
• You cannot conclude: “page abc is the page
with the most important content”

28. WHAT MAKES A TEST BAD?

29. What makes a good test?
• Open to gathering evidence that falsifies your
hypothesis
• Repeatable
• Minimal biases

30. Sample Bias
• The sample does not represent the larger
population
– Ex: customer feedback box is likely to be full of
negative comments, because angry people are
more motivated to comment
– Ex: your impression testing showed extremely
high success, but you sent it to stakeholders who
had already seen the wireframes

31. Confirmation Bias
• Your test is set up to lead people to answer a
certain way
– Ex: you ask people what they thought of the meal
you cooked with the options, “tasty,” “very tasty,”
and “extremely tasty”

32. Clever Hans Bias
• A horse who can do math, who was just
responding to cues from his testers
– Common in usability tasks
– Ex: “Hmm, I would try downloading this form, I
guess” “Yes, that’s the right one! Next task…”

33. Anchoring Bias
• In analysis, anchoring is when the researcher
clings to one the first example or piece of
information
• Lessens the impact of gathering more data

34. Hawthorne Effect
• People change when people observed
• Factory workers were more productive when
the lights were turned brighter. But they were
also more productive when the lights were
turned lower.
– Ex: in usability tests, people are less likely to give
up on at ask than in real life

35. QUESTIONS?