The document discusses the problems with observational data collection. It notes that while observational data is more available due to increased processing power, it can lead to spurious correlations, underlying biases, missing the real driver of behaviors, issues with the counterfactual scenario, measurement error, difficulties explaining why behaviors occur, observer effects, and survival bias. The document recommends discussing observational study design with experts, making predictions, identifying the counterfactual, incorporating experiments, exploring multiple causal patterns, and adding explanatory qualitative research to understand underlying reasons for behaviors.

1. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
The Problems with
Observa2onal Data
Ray Poynter
The Future Place

2. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
NewMR 2018 Sponsors
Pla2num
Gold
Silver
Communica2on

3. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Observa2onal Data
• Most big data (e.g. bank records, social media)
• Census or sample (e.g. all phones, sample with an app)
• Objec2ve or subjec2ve (e.g. receipts, ethnography)
• Structured or unstructured (e.g. phone records, images
uploaded to SM)
• Behaviour or mo2va2onal (e.g. loyalty cards, facial
coding)
• Naturally occurring or experiment
• Observa2onal only or with ques2ons (e.g. ad test using
biometrics)

4. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Why the Change?
1. More realisa2on of the
limita2ons of asking
2. More problems with asking – e.g.
response rates and representa2on
3. Observa2onal data more available
4. Increased processing power
5. From samples to census

5. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Problems?

6. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Spurious Correla2ons

7. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Underlying Bias
When HRT was 1st assessed (Nurses
Health Study – large observa2onal study
in the USA), seemed to protect the heart.
Doctors were recommended to prescribe
it more widely.
Women’s Health Ini2a2ve (gold standard
controlled experiment) – suggested it
was slightly bad for the heart.
Why?
Women receiving HRT were
systema2cally healthier and wealthier.

8. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Missing the Real Driver
0
10
20
30
40
50
60
0
10
20
30
40
50
60
70
80
90
Time 1 Time 2 Time 3 Time 4 Time 5 Time 6
SM $ Sales $
Summer
Winter
Winter
Ice Cream Sales

9. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
The Counterfactual
0
10
20
30
40
50
60
0
10
20
30
40
50
60
70
80
90
Time 1 Time 2 Time 3 Time 4 Time 5 Time 6
Summer
Winter
Winter
What would have happened anyway?

10. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Measurement Error
Dana Gruschwitz & Dr. Robert Schönduwe,
ESRA, 2017, Lisbon, Portugal
Long-standing transport study in Germany.
People have been using PAPI and CAPI to capture
journeys – memory based.
Trial with mobiles, to automa2cally capture informa2on.
Less ‘heaping’ of the distances and 2mes J
But 16% fewer journeys (11% less distance, 18% fewer
minutes) were recorded L
Why?
Phone app turned itself off when people’s phone
badery reached 20%

11. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Explaining the Why

12. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Fire Hydrant
Parking Fines
Everywhere in
upper East Side is
likely to get a fine Top 2 hydrants with
fines – over $55K a
year between
them.
Why?

13. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
The Why! Discovered by Qual

14. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Observer Effects
Watching / measuring behaviour can
change behaviour.
UK RAC study of speed cameras, 2013,
found 27% reduc2on in fatal and
serious collisions.
Note, nobody was deliberately crashing,
it was the underlying behaviour that
changed.
hdps://www.racfounda2on.org/media-centre/speed-camera-transparency-data

15. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Survival Bias Presiden2al Commission on
Space Shudle Challenger
Accident
Failures at a range of
temperature – assume temp
does not mader

16. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Presiden2al Commission on
Space Shudle Challenger
Accident
Failures rare at higher temperatures,
but common at lower temperatures –
temperature maders!

17. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Survival Bias
Look at successful companies, see what
they have in common, create
recommenda2ons.
But, aler a few years, many of the
companies were failing.
To measure drivers, you must look at
failure and success.

18. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
U2lising Experiments
Region A
– T1 sales = 100
– T2, TV, sales = 110
– T3, TV & Twider, sales = 130
Region B
– T1, sales 100
– T2, Twider, sales = 110
– T3, TV & Twider, sales = 130
Region C
– T1 sales = 100
– T2, sales = 105
– T3, sales = 110
The counterfactual = some growth would
have happened anyway.

19. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Brent Smart, CMO IAG (Australian insurance company) – ESOMAR APAC May 2018

20. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
The Ethics of Observa2onal Data Collec2on
Mis-selling?

21. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Recommenda2ons
• Talk to experts about designing and interpre2ng observa2onal
data collec2on
– Weigh2ng, matching, Bayesian sta2s2cs etc
• Make predic2ons
• Iden2fy the counter factual
• Build experiments into your data collec2on
• Look for other causal paderns and explana2ons
• Add explanatory research – e.g. qual – understand the why

22. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Thank You!
Follow me
Twider: @RayPoynter
LinkedIn: www.linkedin.com/in/raypoynter

23. The Problems with Observa2onal Data Collec2on
Ray Poynter, The Future Place
The Future of
Data Collection
Q & A
Sue York
NewMR
Ray Poynter
The Future Place