Understanding data dfm_1_yogi_schulz_2017_05

Corvelle Drives Concepts to Completion
Understanding Data:
What do these numbers mean?
1

Yogi Schulz
Biography
 Partner in Corvelle Consulting
 Information technology related management
consulting
 Microsoft Canada columnist & CBC Radio guest
 PPDM Association board member
 Industry presenter:
– Project World - 6 years
– PMI – SAC - 3 years
– CIPS – many years
– PPDM Association - several years
2

Data Volumes Growing every Year
3

Presentation Outline
 Introduction
 Learning
objectives
 Nine hazards
of data
misinterpretation
 Recommendations
& actions
• Insufficient Domain
Expertise
• Important Variables
Omitted
• Aggregation Obscures Truth
• Inferences are Off Base
• Sources of Variation
Overlooked
• Statistical Significance
Trumps Critical Thinking
• Numerical Analysis Missing
Something
• Correlation Mistaken for
Causation
• Explanation adds Distortion
4

Learning Objectives
 Understand how to accurately interpret data
 Recognize factors
that lead to
misinterpreting data
 Understand actions
to minimize risk of
misinterpreting data
5

Corvelle Drives Concepts to Completion 6
Having all this health data available
is great, but I think I need a degree
in data analytics to sort it all out.

Nine Hazards of
Data Misinterpretation
7

Insufficient Domain Expertise
 Issues:
– Domain experts are not data scientists
– Data scientists are not domain experts
– Imbalance of expertise
 Solutions:
– Is sufficient expertise involved?
– Is the result possible or even likely?
– What experience makes you skeptical?
8
1

Do you have sufficient expertise and
experience to interpret the data?
9
“I know nothing about the subject,
but I’d be happy to give you my
expert opinion!”

Important Variables Omitted
 Issues:
– Too much complexity
– Odd results or strange data
– Too many variables ignored
 Solutions:
– Review the procedures
– Revise the research design
– Narrow the research goal
10
2

Seriously? No worries!
It’s always
something.
I’ve factored in
lift, thrust, drag
and wind speed . . .
Just not
gravity.

Aggregation Obscures Truth
 Issues:
– Story varies by aggregation level
– Aggregation produces surprising relationships
– Pilot results ambiguous
 Solutions:
– Check if low level trends hold up
– Identify potential sources of variation
– Confirm research design
12
Top-level report
Sub-level 1
report
Sub-level 2
report
3

Aggregation to
confirm Trends
13
Use the
SAP database
to aggregate
our findings.
Then use
the survey
database.

Inferences are Off Base
 Issues:
– Misunderstanding group characteristics
– Rose-coloured thinking
– Ideology-based agenda
 Solutions:
– Challenge your research project
– Review the statistical calculations
– Review the work
14
4

When you two have finished
arguing your shaky inferences,
I have actual data!

Sources of Variation Overlooked
 Issues:
– Obvious sources overlooked
– Less obvious sources overlooked
– Hidden sources not considered
 Solutions:
– Search for sources of variation
– Expand data gathering
– Look for unexpected correlations
16
5

Not so subtle impacts
on research outcomes
17
“My diabetic research shows that
test subjects are 98% more likely
to take their diabetic pills
when the pills are covered in chocolate!”

Statistical Significance
Trumps Critical Thinking
 Issues:
– Lack of critical thinking
– Over-emphasizing statistical significance
– Assumptions about big data
 Solutions:
– Use statistical significance as a screen
– Review hypothesis
– Develop a persuasive story
18
6

Statistical Significance
19
2.5%
Significant
effect
2.5%
Significant
effect
95%
Non-significant
effect

If you torture the data
long enough, they will confess.

Numerical Analysis
Missing Something
 Issues:
– Superficial numerical analysis
– Insufficient analysis expertise
– Bias in analysis
 Solutions:
– Visualize data
– Check for false positives and false negatives
– Verify numerical analysis independently
21
7

New study reveals that
reading too many studies
may cause heart disease.

Correlation Mistaken
for Causation
 Issues:
– Correlation described as causation
– Delusional or misleading correlations
– Weak correlation stretched to strong correlation
– Random correlation positioned as real correlation
 Solutions:
– Create a plausible story
– Use correlation as scientific evidence
– Review the calculations
23
8

Correlation ≠
Causation
24
I used to think
correlation implied
causation.
Then I took a
statistics class.
Now I don’t.
Sounds like the
class helped.
Well, maybe.

Explanation adds Distortion
 Issues:
– Overly complex explanation
– Too much jargon
– Exaggeration of inferences
 Solutions:
– Express results clearly
– Develop simple, attractive charts
– Stick to the supportable inferences
25
9

Dubious Explanation
of Survey Data
Public statement:
A survey of more than 25,000
Albertans reviewing the K-12
school curriculum found
“there exists a strong desire for
the removal of Shakespeare as
a required author.”
26

Dubious Explanation
of Survey Data
Reality:
 25,000 survey respondents
 60 added a comment about Shakespeare
 50 called for removal of Shakespeare from
curriculum
27
Alberta Education is interpreting comments
to mean five out of six Albertans favour
removing Shakespeare from curriculum

Number of article retractions
by cause of retraction
28

Recommendations
 Improve your ability
to interpret data
 Watch out for these
nine hazards of data
misinterpretation
 Ensure your research
is accurate and
defensible
29
• Insufficient Domain Expertise
• Important Variables Omitted
• Aggregation Obscures Truth
• Inferences are Off Base
• Sources of Variation
Overlooked
• Statistical Significance Trumps
Critical Thinking
• Numerical Analysis Missing
Something
• Correlation Mistaken for
Causation
• Explanation adds Distortion

Questions &
Discussion
30
Can you help us
understand data
better?Please
fill out
evaluation
form

Understanding Data:
What do these numbers mean?
Corvelle Consulting
300, 400 - 5 Ave. S. W.
Calgary, Alberta T2P 0L6
Phone: (403) 860-5348
E-mail: YogiSchulz@corvelle.com
Web: www.corvelle.com
Yogi Schulz
Partner of Corvelle Consulting
Information technology related
management consulting
Microsoft Canada columnist
& CBC Radio host
Industry presenter
Former PPDM Association
board member
31

Plausible Inferences for
Ranges of p Values
32
1.00
.10
.05
.01
.001
0.0
p > 0.10
0.05 < p < 0.10
0.01 < p < 0.05
0.001 < p < 0.01
0.0 < p < 0.001
Not significant
Marginally significant
Fairly significant
Strongly significant
Definitely significant
Values of p Plausible inference

Questionable
Analysis Goals
33

Bibliography
 A data analyst needs these 5 skills
– https://www.viva-viva.ca/index.php/news-events/114-a-data-analyst-needs-these-5-skills
 Analyzing, Interpreting and Reporting Basic Research Results
– http://managementhelp.org/businessresearch/analysis.htm
 Analyzing Data and Communicating Results
– http://strengtheningnonprofits.org/resources/e-learning/online/analyzingdata/
 Believe It Or Not, Most Published Research Findings Are Probably False
– Simon Oxenham
– http://bigthink.com/neurobonkers/believe-it-or-not-most-published-research-findings-are-
probably-false
 Big Data Solutions for Healthcare
– Stanislas Odinot, April 11, 2013
– https://www.slideshare.net/LarryCover/big-data-solutions-for-healthcare
 Big Data in Healthcare: Separating The Hype From The Reality
– Jared Crapo, Health Catalyst
– https://www.healthcatalyst.com/healthcare-big-data-realities
34

Bibliography
 Confirmationist and falsificationist paradigms of science
– Andrew, 5 September 2014
– http://andrewgelman.com/2014/09/05/confirmationist-falsificationist-paradigms-science/
 Correlation, causation and coincidence
– 05 November 2015
– http://behindlabdoors.com/correlation-causation-and-coincidence/
 Correlation does not imply causation
– https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation
 Correlation does not imply causation – except when it does
– The Original Skeptical Raptor, August 9, 2015
– http://www.skepticalraptor.com/skepticalraptorblog.php/correlation-does-not-imply-causation-except-
when-it-does/
 Data Analysis, Interpretation and Presentation
– http://www.uio.no/studier/emner/matnat/ifi/INF4260/h10/undervisningsmateriale/DataAnalysis.pdf
 Data Analysis and Interpretation
– Anne E. Egger, Ph.D., Anthony Carpi, Ph.D.
– http://www.visionlearning.com/en/library/Process-of-Science/49/Data-Analysis-and-Interpretation/154
 Data analysis and presentation
– http://www.statcan.gc.ca/pub/12-539-x/2009001/analysis-analyse-eng.htm
35

Bibliography
 Data Collection and Interpretation
– The Gale Group Inc.
– http://www.encyclopedia.com/education/news-wires-white-papers-and-books/data-
collection-and-interpretation
 Dataviz: Making Smarter, More Persuasive Data Visualizations
– Scott Berinato, March 30, 2016
– https://hbr.org/webinar/2016/05/dataviz-making-smarter-more-persuasive-data-
visualizations
 Data-driven decision-making process
– http://www.txprofdev.org/apps/datadecisions/node/52.html
 Describing and Interpreting Data
– www.uh.edu/~tech132/sln12.doc
 Descriptive Statistics and Interpreting Statistics
– http://www.statisticssolutions.com/descriptive-statistics-and-interpreting-statistics/
 Economics methods in Cochrane systematic reviews of health promotion and
public health related interventions
– Ian Shemilt, 15 November 2006
– https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-6-55
36

Bibliography
 Excel errors and science papers
– Spreadsheets are playing havoc with scientists
– Economist, September 7, 2016
– http://www.economist.com/blogs/graphicdetail/2016/09/daily-chart-3
 How science goes wrong
– Scientific research has changed the world. Now it needs to change itself
– Economist, October 21 2013
– http://www.economist.com/news/leaders/21588069-scientific-research-has-changed-world-now-it-
needs-change-itself-how-science-goes-wrong
 HealthToon: Taking down chronic diseases with advanced analytics
– Oliver Clark, April 17, 2015
– http://www.ibmbigdatahub.com/blog/healthtoon-taking-down-chronic-diseases-advanced-analytics
 Hilarious Graphs Prove That Correlation Isn’t Causation
– https://www.fastcodesign.com/3030529/hilarious-graphs-prove-that-correlation-isnt-causation
– http://www.tylervigen.com/spurious-correlations
 Interpretation of Data: The Basics
– Tania, May 30, 2014
– https://blog.udemy.com/interpretation-of-data/
37

Bibliography
 Interpreting and Presenting Data
– http://www.deq.state.or.us/lab/wqm/docs/InterpretingandPresentingData.pdf
 Misconduct, not error, is the source of most retracted papers
– Ashutosh Jogalekar, October 2, 2012
– https://blogs.scientificamerican.com/the-curious-wavefunction/misconduct-and-not-error-is-
the-source-of-most-retracted-papers/
 9 Causes Of Data Misinterpretation
– Lisa Morgan, 17 July 2015
– http://www.informationweek.com/big-data/big-data-analytics/9-causes-of-data-
misinterpretation/d/d-id/1321338
 Presentation, Analysis and Interpretation of Data
– https://www.slideshare.net/31mikaella/presentation-analysis-and-interpretation-of-data
 Retraction Watch
– http://retractionwatch.com/
 Sample Data Interpretation Questions
– http://www.psychometric-success.com/faq/faq-sample-data-interpretation-questions.htm
 Science and Engineering Practice of Analyzing and Interpreting Data
– http://ngss.nsta.org/Practices.aspx?id=4
38

Bibliography
 Statistical significance and its part in science downfalls
– Imagine if there were a simple single statistical measure everybody could use with any set of data and it
would reliably separate true from false
– Hilda Bastian, November 11, 2013
– https://blogs.scientificamerican.com/absolutely-maybe/statistical-significance-and-its-part-in-science-
downfalls/
 Statistical Significance for CRO: 6 Things You Need to Know
– Tom Capper, May 08, 2014
– https://www.distilled.net/resources/statistical-significance-for-cro-6-things-you-need-to-know/
 Statistics Done Wrong
– The woefully complete guide
– Alex Reinhart
– https://www.statisticsdonewrong.com/
 Student Competencies & Requirements in Health Economics
– Cumming School of Medicine, 2017
– https://cumming.ucalgary.ca/gse/files/gse/competencies_health_economics_2017.pdf
 Summary and discussion of: “Why Most Published Research Findings Are False”
– Statistics Journal Club, 36-825
– Dallas Card and Shashank Srivastava, December 10, 2014
– http://www.stat.cmu.edu/~ryantibs/journalclub/ioannidis.pdf
39

Bibliography
 Trouble at the lab
– Scientists like to think of science as self-correcting. To an alarming degree, it is not
– Economist, October 18 2013
– http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-
alarming-degree-it-not-trouble
 Understanding the Growth, Value of Healthcare Big Data Analytics
– HealthITAnalytics, July 30, 2015
– http://healthitanalytics.com/news/understanding-the-growth-value-of-healthcare-big-data-
analytics
 Watson Analytics
– https://www.ibm.com/analytics/watson-analytics/us-en/What’s Significant?
– Patrick Barlow, University of Tennessee
– https://www.slideshare.net/pbbarlow1/whats-significant-hypothesis-testing-effect-size-
confidence-intervals-the-pvalue-fallacy
 Why Most Published Research Findings Are False
– John P. A. Ioannidis, PLoS Med 2(8): e124
– http://faculty.dbmi.pitt.edu/day/Bioinf2118/Bioinf-2118-2013/Ioannidis-
journal.pmed.0020124.pdf
40

Understanding data dfm_1_yogi_schulz_2017_05

Recommended

Recommended

More Related Content

Similar to Understanding data dfm_1_yogi_schulz_2017_05

Similar to Understanding data dfm_1_yogi_schulz_2017_05 (20)

Recently uploaded

Recently uploaded (20)

Understanding data dfm_1_yogi_schulz_2017_05

Editor's Notes