1. NYC 4/22/2016

2. HOW TO BE A SMART DATA CONSUMER
2

3. HOW TO BE A SMART DATA CONSUMER
IT’S NOT THAT EASY…
3

4. HOW TO BE A SMART DATA CONSUMER
MORE REALITY THAN FICTION…
4

5. HOW TO BE A SMART DATA CONSUMER
THREATS TO VALIDITY OF RESULTS
Resource:
http://horan.asu.edu/cook&campbell.htm
From the “Bible”:
5

6. HOW TO BE A SMART DATA CONSUMER
6
INTERNAL VALIDITY?

7. HOW TO BE A SMART DATA CONSUMER
INTERNAL VALIDITY
Given that there is a relationship, is it plausible
there are other explanations for the model?
7

8. HOW TO BE A SMART DATA CONSUMER
THREATS TO INTERNAL…
 History (effects may be due to unforeseen
events)
 Maturation
 Testing (becoming test savvy)
 Instrumentation
 Statistical Regression
 Selection (self or convenient selection)
 Mortality
8

9. HOW TO BE A SMART DATA CONSUMER
THREATS TO INTERNAL…
 Interactions With Selection
 Ambiguity About the Direction of Causal
Inference
 Diffusion or Imitation of Treatments
 Compensatory Equalization of Treatments
(coffee talk)
 Compensatory Rivalry by Respondents'
Receiving Less Desirable Treatments
 Resentful Demoralization of Respondents
Receiving Less Desirable Treatments
9

10. HOW TO BE A SMART DATA CONSUMER
10
EXTERNAL VALIDITY?

11. HOW TO BE A SMART DATA CONSUMER
EXTERNAL VALIDITY
Given that there’s a causal relationship, how likely
is it that the conclusion is generalizable across
people, groups, companies, locations, and time?
11

12. HOW TO BE A SMART DATA CONSUMER
THREATS TO EXTERNAL…
 Interaction of Selection and Treatment
(participants)
 Interaction of Setting and Treatment (places)
 Interaction of History and Treatment
12

13. HOW TO BE A SMART DATA CONSUMER
CONSTRUCT VALIDITY?
13

14. HOW TO BE A SMART DATA CONSUMER
14
CONSTRUCT VALIDITY?

15. HOW TO BE A SMART DATA CONSUMER
CONSTRUCT VALIDITY
Do the relationships in the model actually reflect
the meaning of variables?
15

16. HOW TO BE A SMART DATA CONSUMER
THREATS TO CONSTRUCT…
 Inadequate Preoperational Explication of
Constructs
 Mono-Operation Bias (when the boss asks the
questions)
 Mono-Method Bias
 Hypothesis Guessing within Experimental
Conditions
 Evaluation Apprehension
16

17. HOW TO BE A SMART DATA CONSUMER
THREATS TO CONSTRUCT…
 Experimenter Expectancies (coaching)
 Confounding Constructs and Levels of
Constructs
 Interaction of Different Treatments
 Interaction of Testing and Treatment (attention
time!)
 Restricted Generalizability Across Constructs
17

18. HOW TO BE A SMART DATA CONSUMER
18
STATISTICAL CONCLUSION VALIDITY?

19. HOW TO BE A SMART DATA CONSUMER
19
STATISTICAL CONCLUSION VALIDITY?

20. HOW TO BE A SMART DATA CONSUMER
STATISTICAL CONCLUSION VALIDITY
Are we correctly analyzing the data?
20

21. HOW TO BE A SMART DATA CONSUMER
THREATS TO STATISTICAL…
 Low Statistical Power
 Violated Assumptions of Statistical Tests
 Fishing and the Error Rate Problem (in Kansas
City…)
 The Reliability of Measures
 The Reliability of Treatment Implementation
 Random Irrelevancies in the Experimental
Setting
 Random Heterogeneity of Respondents
21

22. HOW TO BE A SMART DATA CONSUMER
22

23. HOW TO BE A SMART DATA CONSUMER
STEVE LEVY
www.linkedin.com/in/stevenmlevy
www.twitter.com/levyrecruits
www.recruitinginferno.com
levy.steve@gmail.com
23