This document discusses several key aspects of research and statistics: 1. It emphasizes that reliable evidence generally comes from multiple studies and research teams, and the totality of evidence matters most. 2. Statistics are useful for determining whether differences or associations are likely due to chance or represent real effects, but numbers can be easily manipulated. 3. Several types of biases and errors can influence research, decision making, memory, and social judgments. Rigorous methodology is important to produce high quality, accurate research and publications.

1. Brian Wells

2. The lessons herein are useful for QI and
research
It is useful to know criticisms of the literature,
what makes a good study v. a bad study, and
that there are variations in quality of
publication
Thinking about the process will help spur action
and ideas for future projects

3. “Truth in science can be defined as the working
hypothesis best suited to open the way to the
next better one.”—Konrad Lorenz, Austria
Reliable evidence generally comes from several
studies and from several teams of researchers,
and that what matters is the totality of the
evidence.
The PLoS Medicine Editors. Minimizing Mistakes and Embracing Uncertainty. PLoS Med. 2005 Aug; 2(8): e272.

4. Scientific information is an economic commodity, and that
scientific journals are a medium for its dissemination and
exchange
It shares the goal of transferring the commodity (knowledge)
from its producers (scientists) to its consumers (other scientists,
administrators, physicians, patients, and funding agencies).
The function of this system has major consequences.
Idealists may be offended that research be compared to
widgets, but realists will acknowledge that journals generate
revenue; publications are critical in drug development and
marketing and to attract venture capital; and publishing defines
successful scientific careers.
Economic modelling of science may yield important insights
Neal S Young, John P. A Ioannidis, and Omar Al-Ubaydli. Why Current Publication Practices May Distort Science.
PLoS Med. 2008 Oct; 5(10): e201.

6. “There are lies, damned lies, and statistics”
-- Disraeli

7. “Fact”: "If you look at the results of Obamacare,
what you see is emergency room visits are up
over 50 percent.“ – Carly Fiorina on Sunday,
August 9th, 2015 in an interview on CNN's
"State of the Union”
Reality: Actually, surveys did not count the
number of visits but rather the numbers of
doctors saying visits are up. And indeed, over
50% of ER docs say they feel they’re seeing
more patients. But this does not necessarily
mean visits themselves are up over 50%, a
number which was not counted.

8. “Fact”: "97 percent of the work that Planned
Parenthood does is about mammograms and
preventative health." - Martin O'Malley on
Sunday, July 26th, 2015 in a New Hampshire
interview
Reality: 3% of Planned Parenthood’s activity are
abortions, leaving 97% non-abortion. However,
PP does not perform mammograms, and the
97% leaves out other services. So while this
statistic sounds true, it actually leaves out a
number of things such as STI/STD testing and
treatment and adoption referrals.

9. Science is clean, right? Unfortunately, no.
Dr. John Darsee, the author of more than 100 publications,
fabricated research for over 14 years before he was caught.
A Harvard researcher, it was initially thought it was a simple
mistake from a young researcher.
Dr. Darsee was able to fabricate research and slip it by editors
using misleading statistics and “big names” on papers, such as
Eugene Braunwald.
He evaded the “triple safety net” that guards against chicanery:
peer review, referee system, in which scientific journals send a
manuscript out for review to judge whether it merits
publication, and replication.
It was ''the extraordinary difficulty of detecting fabrication by a
clever individual.'' – Eugene Braunwald

10. Examples of Decision-making, Belief, and Behavioral Biases
Ambiguity effect (avoid options with missing information)
Backfire effect (disconfirming evidence strengthens belief)
Empathy gap (underestimate the influence or strength of
feelings)
Focusing effect (placing too much importance on one
aspect)
IKEA effect (placing disproportionately high value on
something because you worked on it)
Ostrich effect (ignoring a situation)
Reactive devaluation (ignoring viewpoints because they
originated from an adversary)
Semmelweis reflex (rejecting evidence that contradicts a
paradigm)

11. Examples of Social Biases
Halo effect (traits “spill over)
Worse-than-average effect (believe ourselves to be worse
than others at tasks)
Projection bias (assuming that others share one’s thoughts
or values)
Examples of Memory Error Biases
Bizarreness effect (bizarre material is remembers better
than common material)
False memory (a form of misattribution where imagination
is mistaken for a memory)
Illusory correlation (inaccurately remembering a
relationship between two events)

12. Understand the Process
What goes into research?
How is it analyzed?
How is it published?

13. Statistics. But why use statistics?
Anecdotal evidence is unreliable!
“Why does the phone ring when I’m in the
shower?”
“Why does it rain after I wash my car?”
“Why do patients have issues when Dr. X comes on
service?”
“He/she is a black/white cloud because X happens
when he/she is on.”
Statistics provides us with the way to tell the
difference between chance and real effects

14. Mean – arithmetic average = Σ(x) / n
Median – the “halfway” point
Mode – the most common answer
Range – Overall difference between the highest
and lowest scores
Variance – average difference from the mean

15. Data is always “signal and noise”
The signal is what we’re trying to measure
The noise is the error is our measurement
It is extremely important to choose the right
test and measurement. The wrong one may be
more likely to fail to find a difference when
there is one, or find a difference where one
does not exist.

16. The “Normal Curve”

17. Bimodal Distributions
Skewed Distributions
Uniform Distributions

18. “I want to know if there’s a difference.”
One Sample
Parametric
One Sample t-test
Non-parametric
Wilcoxon Signed Rank Test
More than one sample
Nominal
Chi-Squared Test
Ordinal or above
One Independent Variable
Data Repeated, Independent, or Mixed?
• … And so on…

19. Real effect No effect
Find difference Correct decision False positive (Type I)
Don’t find difference False negative (Type II) Correct decision
Decision Table

20. P value is the probability that a particular set of
data was gained by chance alone
Statistical significance is typically set at the α =
0.05 level, but this is entirely arbitrary. It can
easily be α = 0.10, α = 0.01, or α = 0.000000001.
However, in nearly every research project, a
chance of the data being associated due to
chance alone being less than 5% is considered
convincing

21. Positive Correlation (0 < R < 1)
Negative Correlation (-1 < R < 0)
No Correlation (R = 0)

24. Exaggerated presentations of accurate data
Vs.

25. Percent of US income going to federal taxes

26. The quality of the project or publication
depends on honest and accurate representation
of data
Analysis with statistics is useful for finding
associations likely not due to chance
Numbers are not absolute, and they can be
easily manipulated
Always scrutinize data carefully, and draw your
own conclusions.