Joe Twyman - Making Numbers Count

@JoeTwyman
Making Numbers Count
Joe Twyman
Director and Co-founder

@JoeTwyman
I was wondering if
after all these years
you'd like to meet

@JoeTwyman
They say that
time's supposed
to heal you

@JoeTwyman
But I ain’t done
much healing

@JoeTwyman
Is it Me You’re Looking For?

@JoeTwyman
Why Conduct Research?

@JoeTwyman
A Question of Trust
‘Lies, damned lies
and statistics.’
- Benjamin Disraeli

@JoeTwyman
A Question of Trust

@JoeTwyman
A Question of Trust
‘People can come up with
statistics to prove
anything, Kent.
40% of people know that.’
- Homer Simpson

@JoeTwyman
‘Data that accurately
represents and reports
the opinions of the
population it is trying
to research.’
- Joe Twyman
Data and Results You Can Trust

@JoeTwyman
Can’t Trust Data – Can’t Trust Findings

@JoeTwyman
Things Can Go Wrong
Prostitution and Trafficking in Nine
Countries: An Update on Violence and
Posttraumatic Stress
(Melissa Farley et al, 2003)
Beyond Gender: An Examination of
Exploitation in Sex Work
(Suzanne Jenkins, 2003)

@JoeTwyman
Things Can Go Wrong
Percentage of sex workers wishing to
stop working in the industry within 3
years:
Farley et al 89%
Jenkins 3%

@JoeTwyman
Is it significant?
Is it meaningful?
The Ultimate Questions of Trust

@JoeTwyman
Who is answering
the questions?
Key Questions to Ask

@JoeTwyman
Draw a sample of
respondents from a
population that accurately
represents that population
The General Theory of Sampling

@JoeTwyman
You need an effective
method to select respondents
Why Sampling is So Important

@JoeTwyman
When Sampling Goes Bad

@JoeTwyman
How many people
are answering the
questions?

@JoeTwyman
What it means in theory
The true figure for the entire population is
estimated to fall within +/- X% of the data
at a 95% confidence interval
What it means in practice
Margin of Error = 3%
The true figure for the entire population
will be within +/- 3% of the data 19 out of
every 20 times
Margin of Error

@JoeTwyman
Downing Street E-Petitions Website
‘An own-goal thought up by a prat’
Size Isn’t Everything

@JoeTwyman
1,811,289 ‘respondents’
We the undersigned petition the Prime
Minister to Scrap the planned vehicle
tracking and road pricing policy.
We the undersigned petition the Prime Minister
to Make Jeremy Clarkson Prime Minister.
We the undersigned petition the Prime Minister
to replace the national anthem with ‘Gold’ by
Spandau Ballet.
Size Isn’t Everything

@JoeTwyman
3rd Annual Time 100 Poll
‘The world's most influential person’
Moot
Passive Sampling

@JoeTwyman
Hard to Reach Samples

@JoeTwyman
Who commissioned
the polling?

@JoeTwyman
A Really Bad Idea
LANCASHIRE
HOTPOT CAUSES
CONSTIPATION
HOTSPOT

@JoeTwyman
• ‘The iconic scene in the 1962 Bond movie, in
which a stunning Honey Ryder emerges from the
surf singing to herself, has been voted the most
inspiring bikini moment in history.’
• ‘A survey by diet supplement LIPOBind found
most slimmers used Andress' curve as inspiration
for their beach body diets.’
• ‘It was only after a combination of going to the
gym on a daily basis and taking LIPOBind, Katy
went down from a size 16 to a size 10 over a five
month period.’
Daily Mail (2nd August 2008)
Over the Hill

@JoeTwyman
Odd Similarities
‘Survey says 25% of brides would have cosmetic
surgery.’ - Daily Mirror (July 2008)
‘Truckers and lawyers have topped the list of Britain's
fattest male professions, according to a new survey.’ -
Daily Telegraph (September 2008)
‘British women want a figure like Kelly Brook, a new
study has revealed.’ - Daily Telegraph (December 2008)
‘Half of Britain's women have developed "muffin tops" -
unsightly rolls of fat which bulge over their waistbands,
according to research.’ - Daily Telegraph (April 2009)
‘For more information, visit www.lipobind.com’

@JoeTwyman
Who is asking
the questions?

@JoeTwyman
It’s a Jungle Out There

@JoeTwyman
The General Election 12
Angus Reid
BPIX
ComRes
Gfk NOP
Harris
ICM
Ipsos-MORI
Opinium
Populus
RNB Research
TNS / BMRB
YouGov

@JoeTwyman
Transparency is the Key
• Do they have a postal address, telephone number
and website?
• Can you access full data tables of results?
• Do they fully disclose their methodology?
• Do they have a track record independently
verified against actual outcomes?
• Are they cited in academic literature?
• Are they members of the British Polling Council
and the Market Research Society?
• Do they out-source elements of their operation?

@JoeTwyman
What questions
are they asking?

@JoeTwyman
It’s All About the Phrasing
• Do you happen to have killed your dog?
• As you know, many people have been killing their
dogs these days. Do you happen to have killed
yours?
• Do you know any people who have killed their
dogs?
• Which of the following corresponds most closely
with the current state of your dog?

@JoeTwyman
Currently abortions are only legal up until the
24th week of pregnancy, except for cases of
severe disability. Do you personally believe that
the period during which a legal abortion can be
performed should remain at 24 weeks, or
should it be changed?
46% say they favour a decrease

@JoeTwyman
In the last year for which figures are available,
government statistics show that 52 babies
survived having been born earlier than 24
weeks. At one specialist neo-natal unit in
Britain, 5 out of 7 babies born at 22 weeks
gestation survived. In light of this, do you think
that the upper time limit for abortion should be
kept at 24 weeks, increased, or decreased?
72% say they favour a decrease

@JoeTwyman
Do you agree or disagree with the following
statement:
“Britain should leave the European Union but
maintain close trading links”
55% say they agree
“Britain should remain a full member of the
European Union”
55% say they agree
Agree / Disagree Particularly Bad

@JoeTwyman
The Answer Options Are Important
In your opinion what is the best thing that has
become free?
Interest-free credit or loans
CFC-free deodorant cans
Pensioner’s bus passes
Entry to museums
Willy the whale
Nelson Mandela
Other (please specify below)

@JoeTwyman
The Answer Options Are Important

@JoeTwyman
How are they asking
the questions?

@JoeTwyman
Face-to-Face
Types of Data Collection

@JoeTwyman
Telephone

@JoeTwyman
Online

@JoeTwyman
Self-Completion

@JoeTwyman
Mixed Mode

@JoeTwyman
When are they asking
the questions?

@JoeTwyman
Perfect Timing
When Are They Asking the Questions?

@JoeTwyman
How is the data
being analyzed?

@JoeTwyman
All are equal . . . .
But some are more equal than others
Statistical Weighting

@JoeTwyman
• Target percentage data vitally
important
• Large, independent, national
surveys best
• Without target percentages it is
not possible to weight
• Sample sizes must be
sufficiently large
Target Percentages

@JoeTwyman
How are the results
being reported?

@JoeTwyman
‘8 of 10 Cats Would Buy Whiskers’
A Bad Example of Reporting Results

@JoeTwyman
Linear Extrapolation
-50%
0%
50%
100%
150%
200%
250%
1851
1891
1931
1971
2011
2051
2091
2131
2171
2211
2251
2291
2331
2371
2411
2451
2491
2531
2571
2611
2651
2691
2731
2771
2811
2851
2891
2931
2971
3011
3051
3091
3131
PercentageoftheBritishPopuloationwhoidentifyasMuslim
Year
Linear Projection of Muslim Population of Great Britain (Source: ONS National Census)

@JoeTwyman
Correlation:
Whenever ice cream sales rise, so
do shark attacks
Causation:
The more ice cream you buy the
greater your risk of being attacked
by a shark
Correlation Versus Causation

@JoeTwyman
Correlation:
As vocabulary increases in infancy,
so does appetite
Causation:
Learning new words make you
hungry

@JoeTwyman
Causation:
Help the fight against climate
change by becoming a pirate

@JoeTwyman
• Who commissioned the polling?
• Who is asking the questions?
• What questions are they asking?
• How are they asking the questions?
• When are they asking the questions?
• Who is answering the questions?
• How many people are answering?
• How is the data being analyzed?
• How are the results being reported?
Asking the Right Questions

@JoeTwyman
‘Anything that looks interesting
or unusual is usually wrong.’
Remember Twyman’s Law

@JoeTwyman
joe@deltapoll.co.uk
Twitter: @joetwyman
Instagram: @pollposition
And Finally…

Joe Twyman - Making Numbers Count

Recommended

Recommended

More Related Content

What's hot

What's hot (16)

Similar to Joe Twyman - Making Numbers Count

Similar to Joe Twyman - Making Numbers Count (20)

More from Noisy Little Monkey

More from Noisy Little Monkey (20)

Recently uploaded

Recently uploaded (20)

Joe Twyman - Making Numbers Count

Editor's Notes