Critical appraisal
Paula Funnell
p.a.funnell@qmul.ac.uk
Faculty Liaison Librarian
(Medicine and Dentistry)
Introduction to
Outline
Introduction to critical appraisal
Appraising quantitative research
Levels of evidence
Statistics
Group appraisal
...
Introduction
Evidence based healthcare
“When we intervene in the lives of
others we should do so on the basis
of the best evidence avai...
Steps of evidence based healthcare
1. The patient presents with a clinical
problem
2. Formulate a focused research questio...
Why critically appraise?
To weigh up how valid and useful
the research will be
Why critically appraise?
• In order to keep up to date,
clinicians would have to read
17 articles a day, 365 days a
year
•...
How it works
• Involves answering a short
questionnaire
• We use the CASP questionnaires at
http://www.casp-uk.net/
• The ...
Quantitative research
What is quantitative research?
• Research on something that can be
accurately measured
• Tends to use large, statistically...
Systematic reviews
• A review of all the literature on a topic
• Systematically identified
• Appraised
• Summarised
Question 1
Did the review ask a clearly
focused question?
PICO
A focused clinical question:
P
I
C
O
Population / patient / problem
Intervention
Comparison
Outcome
Question 2
Did the authors look for the right
type of papers?
Levels of evidence (therapy)
Systematic reviews
RCTs
Cohort studies
Case control studies
Expert opinion
1a
1b
2
3
4
5
Case series
http://www.cebm.net/i...
Randomised controlled trials
Outcome
Outcome
new treatment
control treatment
Question 3
Do you think the important
relevant studies were included?
Publication bias
Papers with “interesting” results are more
likely to be:
• Submitted and accepted for publication
• Publi...
Question 4
Did the review’s authors do
enough to assess the quality of
the included studies?
RCT checklist
Quality assessment
• Were the inclusion/exclusion criteria
clear?
• Did they use any sort of scoring
system?
• Were the st...
Question 5
If the results of the review have
been combined, was it reasonable
to do so?
Heterogeneity
• Are the studies similar in terms of:
– Population
– Intervention
– Outcome
– Methodology
• Are the results...
Question 6
What are the overall results of the
review?
How are the results presented?
• Number needed to treat (NNT)
• Odds Ratio
• Relative risk
• Mean difference
Odds and risk
Odds of winning
You versus the rest
Risk of winning
You versus all the
runners
10 horses running,
you bet on...
Odds and risk
Odds of winning
You versus the rest
Risk of winning
You versus all the
runners
10 horses running,
you bet on...
Forest plots
more than 1less than 1 1
Forest plots
more than 1less than 1 1
Line of no effect
Forest plots
more than 1less than 1 1
Line of no effect
Forest plots
more than 1less than 1 1
Line of no effect
Best estimate
Forest plots
more than 1less than 1 1
Line of no effect
Best estimate
Confidence interval
Forest plots
more than 1less than 1 1
Line of no effect
Best estimate
Confidence interval
Forest plots
more than 1less than 1 1
Line of no effect
Best estimate
Pooled result
Confidence interval
Question 7
How precise are the results?
Confidence intervals
Practical exercise
P-value
Could the result have
occurred by chance?
p = 0.001 (1 in 1000)
p = 0.2 (1 in 5)
A p-value of less than 0.05 (1 in...
Question 8
Can the results be applied to the
local population?
Application
• Difficult to answer if you don‟t have
your own population
• Need to rely on experts
• Think about whether th...
Question 9
Were all the important outcomes
considered?
Outcomes
• Do the authors address all the
outcomes they set out to study?
• Are the outcomes considered from
different vie...
Question 10
Are the benefits worth the harms
and the costs?
Benefit v harms and costs
• How significant are the
benefits?
• Is there any discussion of
possible side effects etc.?
• A...
Summary
Validity
Is it
trustworthy?
Results
What does
it say?
Relevance
Will it help?
Qualitative research
What is qualitative research?
• Concerned with the world of human
experience from the view of the
participants
• Natural s...
Components of qualitative research
Research question
Sampling process
Data collection
Data analysis
Question 1
Was there a clear statement of
the aims of the research?
The research question
• Describes why the research is being
carried out
• Qualitative research addresses the
questions
• A...
Question 2
Is a qualitative methodology
appropriate?
Why qualitative research?
• Increases knowledge in an area that is
poorly understood
• Challenges assumptions and practice...
Question 3
Was the research design
appropriate to address the aims of
the research?
Research methodologies
phenomenology
field
research
grounded
theory
action
research
ethnography
Question 4
Was the recruitment strategy
appropriate to the aims of the
research?
Sampling
• Sample size can be determined by
– Data analysis
– Time constraints
• What population does the sample refer
to?...
Sampling
Methods of sampling:
• Convenience
• Purposive
• Quota
• Snowballing
Question 5
Was the data collected in a way
that addressed the research
issue?
Data collection
• Are the methods the most suitable for
the research question?
• Are the methods explicit?
• Is the venue ...
Question 6
Has the relationship between
researcher and participants been
adequately considered?
Reflexivity and researcher bias
VenueArea being
studied
Types of interview
questions asked
Meaning given
to data
Question 7
Have ethical issues been taken
into consideration?
Ethics
Need to consider:
• Consent
• Confidentiality
• Professional responsibility
• Reporting
• Ethics committee approval
Question 8
Was the data analysis sufficiently
rigorous?
Data analysis
• Description of analysis
• Clarity of approach
• Use of all the data
• Potential for bias
Question 9
Is there a clear statement of
findings?
The findings
• Are the findings explicit?
• Are they credible?
– respondent validation
– quality of argument
– would anoth...
Triangulation
questionnaire professionals
Question 10
How valuable is the research?
Value of the research
Consider your appraisal of the paper in
terms of:
• Credibility
• Dependability
• Transferability
• ...
Questions to ask
Running a critical
appraisal workshop
Before the session
• Who is the group?
• Is it qualitative, quantitative, or both?
• Know what equipment and facilities ar...
Structuring the session
• Dependent on the group and time frame
• Presentation
• Encourage audience participation
• Use in...
Choosing an article
• For a mixed group choose a fairly generic
article
• For a discipline specific group:
– Do you want t...
Any
Questions?
The end!
Paula Funnell
Faculty Liaison Librarian (Medicine & Dentistry)
Queen Mary University of London
p.a...
Upcoming SlideShare
Loading in...5
×

Critical appraisal

639

Published on

Critical appraisal workshop delivered to East of England Health Libraries Network, 1st April 2014

Published in: Health & Medicine
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
639
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
33
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Critical appraisal

  1. 1. Critical appraisal Paula Funnell p.a.funnell@qmul.ac.uk Faculty Liaison Librarian (Medicine and Dentistry) Introduction to
  2. 2. Outline Introduction to critical appraisal Appraising quantitative research Levels of evidence Statistics Group appraisal Appraising qualitative research Qualitative methodologies Group appraisal Running a critical appraisal workshop
  3. 3. Introduction
  4. 4. Evidence based healthcare “When we intervene in the lives of others we should do so on the basis of the best evidence available regarding the likely consequences of that intervention” G Macdonald, 1998 Macdonald, G. (1998). Promoting Evidence-Based Practice in Child Protection. Clinical Child Psychology and Psychiatry, 3 (1), 71–85.
  5. 5. Steps of evidence based healthcare 1. The patient presents with a clinical problem 2. Formulate a focused research question 3. Search for the best evidence 4. Critically appraise the evidence you find 5. Consider the evidence in the light of your expertise and decide whether to apply it or not
  6. 6. Why critically appraise? To weigh up how valid and useful the research will be
  7. 7. Why critically appraise? • In order to keep up to date, clinicians would have to read 17 articles a day, 365 days a year • Research is of variable quality • Only an estimated 1% is judged clinically relevant • Need to find the 1%
  8. 8. How it works • Involves answering a short questionnaire • We use the CASP questionnaires at http://www.casp-uk.net/ • The questionnaires were devised by clinicians for clinicians • Remember to focus on appraisal not criticism
  9. 9. Quantitative research
  10. 10. What is quantitative research? • Research on something that can be accurately measured • Tends to use large, statistically representative samples • Uses statistical methods to analyse data • Represents findings with numbers • Attempts to eliminate bias
  11. 11. Systematic reviews • A review of all the literature on a topic • Systematically identified • Appraised • Summarised
  12. 12. Question 1 Did the review ask a clearly focused question?
  13. 13. PICO A focused clinical question: P I C O Population / patient / problem Intervention Comparison Outcome
  14. 14. Question 2 Did the authors look for the right type of papers?
  15. 15. Levels of evidence (therapy)
  16. 16. Systematic reviews RCTs Cohort studies Case control studies Expert opinion 1a 1b 2 3 4 5 Case series http://www.cebm.net/index.aspx?o=1025 Levels of evidence (therapy)
  17. 17. Randomised controlled trials Outcome Outcome new treatment control treatment
  18. 18. Question 3 Do you think the important relevant studies were included?
  19. 19. Publication bias Papers with “interesting” results are more likely to be: • Submitted and accepted for publication • Published in a major journal • Published in English • Quoted by authors • Quoted in newspapers
  20. 20. Question 4 Did the review’s authors do enough to assess the quality of the included studies?
  21. 21. RCT checklist
  22. 22. Quality assessment • Were the inclusion/exclusion criteria clear? • Did they use any sort of scoring system? • Were the studies for inclusion assessed by more than one reviewer?
  23. 23. Question 5 If the results of the review have been combined, was it reasonable to do so?
  24. 24. Heterogeneity • Are the studies similar in terms of: – Population – Intervention – Outcome – Methodology • Are the results similar from study to study? • Were any tests for heterogeneity carried out? • Are there any discussions around heterogeneity?
  25. 25. Question 6 What are the overall results of the review?
  26. 26. How are the results presented? • Number needed to treat (NNT) • Odds Ratio • Relative risk • Mean difference
  27. 27. Odds and risk Odds of winning You versus the rest Risk of winning You versus all the runners 10 horses running, you bet on 1 horse 1:9 1:10
  28. 28. Odds and risk Odds of winning You versus the rest Risk of winning You versus all the runners 10 horses running, you bet on 4 horses 4:6 4:10
  29. 29. Forest plots more than 1less than 1 1
  30. 30. Forest plots more than 1less than 1 1 Line of no effect
  31. 31. Forest plots more than 1less than 1 1 Line of no effect
  32. 32. Forest plots more than 1less than 1 1 Line of no effect Best estimate
  33. 33. Forest plots more than 1less than 1 1 Line of no effect Best estimate Confidence interval
  34. 34. Forest plots more than 1less than 1 1 Line of no effect Best estimate Confidence interval
  35. 35. Forest plots more than 1less than 1 1 Line of no effect Best estimate Pooled result Confidence interval
  36. 36. Question 7 How precise are the results?
  37. 37. Confidence intervals Practical exercise
  38. 38. P-value Could the result have occurred by chance? p = 0.001 (1 in 1000) p = 0.2 (1 in 5) A p-value of less than 0.05 (1 in 20) is considered to be statistically significant
  39. 39. Question 8 Can the results be applied to the local population?
  40. 40. Application • Difficult to answer if you don‟t have your own population • Need to rely on experts • Think about whether the study could be easy replicated with another population
  41. 41. Question 9 Were all the important outcomes considered?
  42. 42. Outcomes • Do the authors address all the outcomes they set out to study? • Are the outcomes considered from different viewpoints e.g. patient, practitioners, policy makers?
  43. 43. Question 10 Are the benefits worth the harms and the costs?
  44. 44. Benefit v harms and costs • How significant are the benefits? • Is there any discussion of possible side effects etc.? • Are financial costs mentioned?
  45. 45. Summary Validity Is it trustworthy? Results What does it say? Relevance Will it help?
  46. 46. Qualitative research
  47. 47. What is qualitative research? • Concerned with the world of human experience from the view of the participants • Natural settings are used • A study is „led‟ by the subjects‟ experiences • Qualitative research is rigorous and systematic • Results are not usually generalisable or totally replicable
  48. 48. Components of qualitative research Research question Sampling process Data collection Data analysis
  49. 49. Question 1 Was there a clear statement of the aims of the research?
  50. 50. The research question • Describes why the research is being carried out • Qualitative research addresses the questions • Are the aims of the research clear?
  51. 51. Question 2 Is a qualitative methodology appropriate?
  52. 52. Why qualitative research? • Increases knowledge in an area that is poorly understood • Challenges assumptions and practices • Acts as a precursor to quantitative research • Generates new ideas
  53. 53. Question 3 Was the research design appropriate to address the aims of the research?
  54. 54. Research methodologies phenomenology field research grounded theory action research ethnography
  55. 55. Question 4 Was the recruitment strategy appropriate to the aims of the research?
  56. 56. Sampling • Sample size can be determined by – Data analysis – Time constraints • What population does the sample refer to? • How was the sample selected?
  57. 57. Sampling Methods of sampling: • Convenience • Purposive • Quota • Snowballing
  58. 58. Question 5 Was the data collected in a way that addressed the research issue?
  59. 59. Data collection • Are the methods the most suitable for the research question? • Are the methods explicit? • Is the venue clearly described?
  60. 60. Question 6 Has the relationship between researcher and participants been adequately considered?
  61. 61. Reflexivity and researcher bias VenueArea being studied Types of interview questions asked Meaning given to data
  62. 62. Question 7 Have ethical issues been taken into consideration?
  63. 63. Ethics Need to consider: • Consent • Confidentiality • Professional responsibility • Reporting • Ethics committee approval
  64. 64. Question 8 Was the data analysis sufficiently rigorous?
  65. 65. Data analysis • Description of analysis • Clarity of approach • Use of all the data • Potential for bias
  66. 66. Question 9 Is there a clear statement of findings?
  67. 67. The findings • Are the findings explicit? • Are they credible? – respondent validation – quality of argument – would another researcher make a similar interpretation? – are alternative interpretations explored?
  68. 68. Triangulation questionnaire professionals
  69. 69. Question 10 How valuable is the research?
  70. 70. Value of the research Consider your appraisal of the paper in terms of: • Credibility • Dependability • Transferability • Confirmability How useful is the paper to you?
  71. 71. Questions to ask
  72. 72. Running a critical appraisal workshop
  73. 73. Before the session • Who is the group? • Is it qualitative, quantitative, or both? • Know what equipment and facilities are available • Ask participants to read the article in advance
  74. 74. Structuring the session • Dependent on the group and time frame • Presentation • Encourage audience participation • Use interactive activities • Practical critical appraisal in groups • Feedback
  75. 75. Choosing an article • For a mixed group choose a fairly generic article • For a discipline specific group: – Do you want to use a discipline specific article? – Ask for topic ideas – Not a good idea to let them choose the articles • Choose studies with both strengths and weaknesses • Make sure it‟s easily available to all the participants
  76. 76. Any Questions? The end! Paula Funnell Faculty Liaison Librarian (Medicine & Dentistry) Queen Mary University of London p.a.funnell@qmul.ac.uk
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×