Reviewing research reports

1. GEDU 6170 Reviewing
Saad Chahine, PhD
May 15, 2014

2. Initial Questions
• Why should I be interested in this article?
– Relationship to my current work/research
• Does this article state its problems clearly and succinctly?
– Difference between difficult (high level of ability) and unclear (poorly
written)
• Is the literature that is cited relevant, important, and current?
– Seminal articles & recent work
• Do the findings in this article address the issues that it raises?
– Find the links to the RQs/Hypotheses
(Shank & Brown, 2007)

3. Initial Questions
• Are the measurements and findings reliable?
– Checks of consistency
• Are the measurements and findings valid?
– Checks for conceptual meanings and interpretations
• Are the statistical tests overly sensitive, so that they might be
“Finding” results that are not really there?
– E.g., Correlations are very sensitive to sample size…they will often be
significant. You need to check the actual r value!
• Are the statistical tests overly conservative, so that they might
be missing results that are really there?
– E.g., Aggregated data to the school level. The grouping averages
individual student’s variation.
(Shank & Brown, 2007)

4. Detailed Questions
• How generalizable are the results and findings?
– Be carful of absolute statements with small sample sizes…
• Generalizability
• Representativeness
• Imbalance
• Qualitative?
• How realistic are the results and findings?
– Do you think this could really happen in real life?
• Ecological validity
• Independent variables
• Dependent variables
• Interferences & collinearity
• Hawthorn Effect (Qq)
• Novelty
• Experimenter effect
• Issues with pre/post designs (Shank & Brown, 2007)

5. Detailed Questions
• Does the article fulfill its potential? (More focused on Qual)
– Competiveness: Proving a point
– Appropriation: Denying personal beliefs
– Rigidity: Method applied needed to be modified
– Superficiality: Do you think you would get the same results in different
settings?
– Sentimentality: Empathy or Manipulation
– Narcissism: Sometimes its too much about he researcher
– Timidity: What did they leave out and why?
(Shank & Brown 2007)

6. Can you answer these questions?
• How can an understanding of the literature help me improve
my skills as a personal reviewer?
• Do I have enough content knowledge to understand this
article? Do I need more content background? Where and how
can I get it?
• Do I have the technical skills I need in order to understand this
article?
(Shank & Brown, 2007)

7. Activity
• In groups examine the reviews and address
the following:
– Elements of the review that were really done well
– Elements that are missing from the review
– Does the review help you decide whether to read
the book?
– What else would you have liked to have seen in
the review?