This document proposes redefining how practical significance in quantitative research is determined for leadership decision making. It argues that solely relying on statistical significance and effect sizes can be misleading. Instead, it recommends evaluating the actual performance of the experimental group to see if it meets target goals. Guidelines are provided for EdD students to critically analyze research findings based on practical significance rather than just reporting researcher conclusions. The goal is to make quantitative research more applicable to leadership practice and EdD work.

1. by
Stanley Pogrow,
Professor of Leadership and Equity
San Francisco State University
stanpogrow@att.net
CPED 15
Rethinking the Teaching and Definition of the Practical Significance
of Quantitative Research for Leadership Decision-Making

2. Abstracted from the forthcoming book…
This book will be available online this July from
http://www.ncpeapublications.org, then look under
NCPEA Press (BOOKS)

3. The focus of this quant methods book is on how to
assess and apply evidence for leadership decision-
making
The statistical criteria taught in quantitative
methods courses to determine the importance
of research findings for informing leadership
decision-making are misleading

4. The first mistake is teaching that a finding of
statistical significance (p<.001 ***) means that the
intervention is effective and leaders should adopt it
Criterion #1
Statistical
Significance
Tells you whether a difference between
groups or a relationship was “significant”
Alas: All “statistical significance” tells you is that
the difference between the groups or a
relationship was probably something other
than nothing—this is hardly a finding of
“significant” importance for practice.

5. The second mistake is in interpreting whether Effect Sizes
favoring experimental groups demonstrates that the
benefits of the new intervention has “practical significance”
Criterion #2 Is the Effect Size (i.e., magnitude) of
difference between groups at least .2?
Alas Cohen’s description of an effect size of .2
means that the differences are “difficult to
detect“—this is not a criterion that should be
the basis for leadership practice

6. When leaders do not follow such research findings they are
inappropriately criticized by the academic community for not
utilizing the latest research.
The current statistical criteria used by statisticians and
researchers to determine that a practice is “evidence based”
are of little to no value for actually informing practice!

7. • The amount of relative difference between groups
considered important (ES=.2) is too small
• Relative difference comparisons can be misleading
Consider the following scenario on the next slide
There are two problems with the current statistical definition
of practical significance in terms of using Effect Size (ES):

8. Shivering couple in the Midwest planning their January
dream getaway vacation
• Wife: I cannotwait for our vacationin January.Let’s go somewherewarm.
• Husband: Definitely.
• Wife: Whereshouldwego?
• Husband: I justread that Greenlandis warmer thanAntarcticain January.
• Wife: That soundsgreat.
• Husband: Evenbetter, due to climatewarming Greenlandwillbe warmer thisyear than
last.Plus,ithas 27,394milesof coastline,soit willbeno problemfindingbeachesfor
sunbathing.
• Wife: That’s great. It will bewonderfulto go somewherewhere we canleaveourwinter
clothesbehind.

9. Why is this couple’s evidence-based decision so bad? What
are the lessons from this scenario?
• Relativecomparisonscan make a terribleoptionappearto be desirable
• Arelative comparisoncan make an experimentalgroup lookgood when infact the
studentsdidterribly
• Thisisparticularlytrue when the research weights/adjuststhe resultsfor the experimental
group upwards (e.g., usingan Analysisof Covariance)
• Sucha researchstrategy hasbeen employed overa 25 year periodin top research
journalsto make the widely-usedprogram, “SuccessforAll”,appearto be successful
when infact the experimentalstudentsdidterribly
• The BOTTOM LINE isthat educatorscannotrely on relativedata by themselvesto male
decisions—theyneed keypiecesof absolute/actualdata on actualresults.Thisisthe basis
of the recommendedredefinitionof “PracticalSignificance”that follows

10. ARedefinition of “Practical Significance” for Improving
Practice
• STEP1: Collectdata on howyour school(s)isdoing.
• STEP2: Set a target improvementgoal.
• STEP3: Determinethe actualperformanceof the experimental group(only) inthe
researchby lookingat the unweighted/unadjusted Meanand StandardDeviation
(Median?)for howthe experimentalgroup performed
• STEP4: Didthe unweighted/unadjustedperformanceof the experimental groupmeet the
target outcomefor your school(s)?
• STEP5: DECISION—Iftheexperimentalgroupperformed at thedesiredtargetedlevel,
giveserious considerationto adoptingthe newpractice/approach.
• If the experimentalgroup’sperformancedid not meet thiscriterion,you are probablybetter
off ignoringthe researchand seekinga differentalternative—(regardlessof how largethe
ES statistic inthe reported researchis or how many asterisksthere are for p< *****)

11. This definition of “practical significance” means thatanyone
can examine any research study and determine on their
own whether the results have practical importance for
leadership decision-making

12. But suppose you cannot find the unweighted/unadjusted
Mean for the performance of the experimental group in the
research?
The table on the next slide suggests basic cutoffs for the
value of ES in order to consider that a relative comparison
may have “potential practical significance”
• STEP1: Curse journaleditorswho do not thinkit is importantto let readers knowhowthe
experimental studentsactuallydid
• STEP2: Reluctantlyrely on “potentialpracticalsignificance”by interpretingthe ESrelative
outcomesto seeifthere were BIG relative differencesbetweenthegroups

13. Suggested Minimum ES/SDU for Considering Experimental Research to
Have Potential Practical Significance by Grade Level andType of
Research
These recommended cutoffs were formed by synthesizing a variety of recommendations
in the literature as well as a variety of results from meta-analyses

14. Short
Term/Proximal
Gains &
Researcher
Developed
Test
Amount of
Variation The
Relationship
Can Predict
(r2, R2)
r .39 15%
R .39 15%
Stan’s suggested minimum r and R for indicating that
Relationships (correlations and regressions) between
variables have potential practical significance
An important exception is that a small r/R can be masking
an important non-linear relationship: e.g., the non-linear
relationship between “stress” and “performance”.

15. Using these techniquesany EdD student or leader can access and critique the
findings of any quantitative research in even the most mathematically complex
articles in the most prestigious journals by ignoring the esoteric parts and simply
searching for the simple few key pieces of data and criteria previously discussed.
Providing the capability for non-mathematicians to easily judge the applicability of
quantitative research for leadership decision-making has major implications for
leadership practice and EdD dissertations.
BOTTOM LINE

16. Do not rely on published reports on how effective a given intervention is: Check out
the unweighted Means (practical significance) and ES, r or R (potential practical
significance) yourself.
Practicalsignificance is more compelling thanpotential practical significance.
Potential practical significance only means that you should look at it more closely.
There are three conditions when you may want to consider an intervention with a
low ES that does not meet the standard forpotential practical significance:
Implications for Leadership Decision-Making
1. Philosophic rationale, e.g., enhancement of opportunity
2. Poor implementation in the study which creates the possibility that
with a better and more intensive application of the same approach
will produce better results
3. Want to combine this intervention with another and create synergy

17. Implications for the dissertation literature review
The table on the next slide suggests some language that students can
use to characterize the practical importance of research findings.
• Ratherthan merely reportingthe conclusions ofthe researchers,EdD studentsshould
criticallyanalyzethe findingsin accordanceof whether the findings demonstrate“practical
significance”or “potentialpracticalsignificance”
• Studentsshouldnot use the term “significant”in describing researchconclusions

18. How to Critically Report Quantitative Research Outcomes in Your Literature
Review
Finding of Study Practical
Importance
Implications
for Practice
Lit Review Characterization
Study reports a positive
finding/relationship, but no data
given on the amount of the
relationship
None None There was no evidence
provided to support the
conclusion.
Study reports a statistically
significant, but low, ES, with no
absolute finding for how the
experimental group did.
Very Weak None The results were statistically
significant but the differences
between groups (ES=.17) was
too small to have potential
practical significance.
Study reports a positive
relationship that is statistically
significant, but with a low value
of r or R.
Very Weak None The results were statistically
significant but the value of r
(.27) was too small to be of
any potential practical
significance.
Reports a relative finding that is
statistically significant, with a
large Effect Size or R
2
, but no
absolute finding for how the
experimental group did.
Moderate Potential
implications
for practice
There was a strong positive
effect (e.g., ES = .56, or
R= .49) but there were no data
on how well the experimental
students actually did.
Reports a relative finding that is
statistically significant, but with a
large Effect Size or R2
, and an
absolute finding for how the
experimental group actually did.
Potentially strong
depending on how
well the
experimental
group did.
Potential
major
implications
for practice.
The ES of .51 indicated that
the results had potential of
practical significance, and the
experimental students ended
up doing …

19. — Reform the teaching of quantitative methods courses to make them
more accessible to, and authentic for, EdD students and faculty
— Provide suggested guidelines for applied, scientifically rigorous, EdD
dissertations that provide the potential for improving practice
The discussed methods are just one of the ways that my
forthcoming book seeks to:
The latter is built upon some of the emerging conceptions of the
scientific process that are more relevant to EdD work.
For more info contact NCPEA press
– or -
stanpogrow@att.net