Critical appraisal of a journal article

1. Critical Appraisal of a
Journal Article
Dr. Sahil Kumar

2. What is a Journal Club?
An educational meeting in which a group of
individuals read, evaluate and discuss current
articles from the biomedical literature.
Successful Journal Clubs include:
A well constructed clinical question
Searching for evidence
A critical appraisal

3. Critical appraisal: Definition
A systematic process used to identify the
strengths and weaknesses of a research
article to assess the usefulness and
validity of research findings.
Why is it important?

4. Despite the plethora of documents
available to guide the process,
no 'gold-standard' instrument for critical
appraisal exists.

5. Assessing the Relevance & Validity of a
Research Article
1) Is the study question relevant?
2) Does the study add anything new?
3) What type of research question is being asked?
4) Was the study design appropriate for the research question?
5) Did the methods address the most important sources of bias?
6) Was the study performed according to the original protocol?
7) Does the study test a stated hypothesis?
8) Were the statistical analyses performed correctly?
9) Do the data justify the conclusions?
10) Are there any conflicts of interest?

6. 1.) Is the Study's Research
Question Relevant?
• Important topic clinically
• Adds to what is already known about that subject.

7. 2.) Does the Study Add Anything New?
• New ideas and knowledge are developed on the
basis of previous work.
• Substantial new contribution to knowledge are a
relative rarity.
• However, incremental advance can also be of
value.

8. 3.) What Type of Research
Question is being asked?
• Most fundamental task of critical appraisal is to
identify the specific research question that
an article addresses.
• A well-developed research question usually
identifies these components:
• P –Patient population
• I – Intervention
• C – Comparator
• O - Outcomes

9. 4.) Was the Study Design Appropriate
for the Research Question?
• Questions About Effectiveness of Treatment.
• Meta-analyses of well-conducted RCTs and
individual RCTs provide the most robust
evidence followed by nonrandomized
controlled trials, cohort studies, case-control
studies may be done based on feasibility.

11. 4.) Was the Study Design Appropriate for
the Research Question?
• Questions About the Frequency of Events.
• Observational studies, rather than RCTs, are
the most appropriate study design here.

12. 5.) Did the Study Methods Address
the Key Potential Sources of Bias?
• Bias can be attributed to chance or to the study
methods.
• Systematic biases arise from the way in which
the study is conducted, be it how study
participants were selected, how data was
collected, or through the researchers' analysis
or interpretation.

13. Critical Appraisal Skills Programme
(CASP)
Program coordinators have developed
separate checklists for the appraisal of
systematic reviews, RCTs, cohort studies,
case-control studies, diagnostic test studies,
economic evaluations and qualitative
research that each comprise 10 questions.

15. Systematic Reviews and Meta-
analyses
Risk of bias:
• Selection of studies and the quality of
these primary sources.
• Citation bias, citation or non-citation of
research findings, depending on the nature
and direction of the results.
• Publication bias outcome of study
influences the decision to publish
• Language bias

16. • QUORUM (Quality of Reporting of Meta-Analyses)
Provides a comprehensive framework for
assessments of the quality of reporting in meta-
analyses and systematic reviews.
• AMSTAR - Assessment tool for appraisal of
systematic reviews, CASP checklist
• MOOSE (Meta-Analysis Of Observational Studies
in Epidemiology) guidelines – Non-RCTs

17. Randomized Controlled Trials
• Bias – Patient allocation
• Random allocation of participants should ensure
that treatment groups are equivalent in terms
of known and unknown confounding factors.
• Computer generated allocation, are preferable to
less concealed methods, such as use of color-
coded forms.

18. CONSORT
• Consolidated Standards of Reporting Trials
• Comprehensive tool to assess the standard of
reporting in randomized trials.

19.  Randomization?
 Blinding?
 Were outcomes assessed objectively?
 Were all participants accounted for in final analysis?
 Was it per protocol or ITT?

20. Cohort Studies & Non-RCTs
• Bias - high potential for selection bias and
confounding factors.
• Patient's or clinician's preferences, referral
patterns, current treatment paradigms or local
policy.
• STROBE (Strengthening the Reporting of
Observational Studies in Epidemiology)

21. Case control Studies
• Bias – Selection of appropriate control
• ‘Recall bias'
• STROBE
Cross-sectional Studies
• “Snapshot”
• STROBE

22. 6.) Was the Study Performed in
Line with the Original Protocol?
• Deviations from the planned protocol can affect
the validity of a study.
• Common deviations -
 Failure to recruit planned no. of participants
 Changes to the inclusion and exclusion criteria
 Variation in treatments or interventions
 Changes to employed techniques/technologies
 Changes to the duration of follow-up

23. 7.) Does the Study Test a Stated
Hypothesis?
• Hypothesis - what the investigators expect the
study to find.
• Null hypothesis - p (probability) value. 95% CI.
• Crucial to identify study hypothesis a priori.
• Check that all data relevant to the stated study
objectives have been reported, no omission.
• Noninferiority/ Equivalence trial – noninferiority
margin has to be pre-specified.

24. 8.) Were the Statistical Analyses
Performed Correctly?
• All articles should include a segment within
'Methods' that explains the tools used in the
statistical analysis and their rationale.
• Patients who are lost in follow-up and missing
data should be clearly identified in the 'Results'
section.
• The CONSORT flowchart enables the flow of
participants to be clearly identified.

25. 9.) Do the Data Justify the
Conclusions?
• Sometimes, overemphasis is placed on
statistically significant findings that have
differences too small to be of clinical value.
• Might dismiss large and potentially important
differences b/w non-statistically significant
groups, often because sample sizes were small.
• Generalizability?

26. 10.)Are There any Conflicts of
Interest?
• COI occur when personal factors have the
potential to influence professional roles or
responsibilities.
• Check for a declaration/ open disclosure about
the source of funding for the study.
• If a potential COI has been identified, how this
conflict was managed.
• Journals now routinely require authors to declare
any potential COI when an article is submitted.

27. CONCLUSIONS
• Critical appraisal is a process through which the
strengths and weaknesses of a research study
can be identified.
• Most important component of critical appraisal
is careful assessment of the study design.
• Evaluation of the statistical methods used,
interpretation of the findings and potential
conflicts of interest are also essential.
• Clinicians identify the most relevant, high-
quality studies available to guide their clinical
practice.

33. A study by Block J et al.

34. THANK YOU