SlideShare a Scribd company logo
Quality assessment in SLR
Systematic Literature Review Workshop (SLRW)
Dr. Jingjing Lin
Assistant Professor
Centre of IT-based Education
Toyohashi University of Technology
lin.jingjing.qc@tut.jp
Don't judge a book by its cover. (English idiom)
Do not take published studies at face value. (Boland, Cherry, and Dickson, 2017, p.109)
Which side are you on?
• It is published in
Nature! It must be a
good quality study. I
have no doubt of it.
• It is published in Nature! But
the sample is rather small
than usual, I doubt its
generalizability.
(Boland, Cherry, and Dickson, 2017)
Why you
need QA in
your SLR?
Develop a greater understanding of your studies
and others’ results.
Distinguish between good-quality and poor-
quality studies.
More likely to draw meaningful conclusions from
the data.
Offer you opportunity to acquire critical
appraisal skill.
Validated SLR checklist often includes it to
evaluate SLR’s quality.
(Boland, Cherry, and Dickson, 2017)
What is quality?
Quality of individual studies Quality of your SLR
The degree to which a study employs measures to
minimize error and bias in its design, conduct, and
analysis. (Khan et al., 2003, p.39)
To check against validated SLR checklist tools
(Boland, Cherry, and Dickson, 2017)
“This is a good quality study”.
Translated into:
• “I am confident that this study’s design, conduct and analysis are
robust to provide results that are credible, trustworthy and
generalizable, and are highly likely to be a true representation of the
results of the tested intervention, phenomenon, or exposure.”
(Boland, Cherry, and Dickson, 2017)
When to do QA
Before data extraction During data extraction After data extraction
• When you intend to exclude
poor quality studies in your SLR
NA • You will be blind towards the
quality of individual studies
when extracting data, and your
report is likely to be biased.
• Your greater familiarity with
pooled studies can help you
answer QA questions.
(Boland, Cherry, and Dickson, 2017)
Main elements of QA
Element/Bias type What is it Significance
Selection bias Is the sample representative of the population? Generalizability, transferability
Allocation bias How participants get assigned to treatment group? Any human that
influences or interferes such allocation?
Types of design influence such allocation strategy. The stronger the design,
the less bias from allocation to treatment.
Performance bias Is anyone aware of the treatment or made blinded?
(participants, intervention providers, study investigators)
Can such awareness or blindness cause bias in the study?
Detection bias Is anyone aware of the treatment or made blinded?
(people who measure the outcome)
Attrition bias Proportion of participants who stopped the treatment either by self
dropout or withdrawal by the study.
Weaken the generalizability, give insight to the compliance rates, treatment
groups inequality due to participant size change
Reporting bias Are all outcomes stated to be measured actually reported despite its being
beneficial/favorable or not to the authors? Are some results measured
post hoc for the sake of enhancing favorable outcomes?
Reasons that lead to the failure of reporting all stated outcomes need to be
provided, the treatment appears more favorable than it really is
Confounders Participants’ characteristics are similar across all treatments?
(gender, age, health status, social status, etc.)
Participants should be equally balanced otherwise there is a risk that results
are biased in favor of one group than the other.
Concurrent or subsequent intervention Was any other treatment received by participants beside the study
intervention?
Participants should be treated in the same way to reduce the risk of other
effects, which weakens the study intervention’s effect.
Analysis Were the data for all participants included in final analysis even whose
withdrew ones?
If there are data missing for a number of participants and these data are not
considered, published results will not properly reflect the results of the
study.
Funding bias Who funded the study? Funders may want your study to favor positive outcomes instead of negative
outcomes.
QA tools ask questions about bias. Bias types are many. Assessment of
risk of bias covers at least six bias as colored in red in the table below
according to EUnetHTA (2015)
(Boland, Cherry, and Dickson, 2017)
Selection bias
Is the sample representative of the population?
Duckling image from: https://pixabay.com/illustrations/ptak-brzydkie-kacz%C4%85tko-bajki-2468019/
Treatment Group: Ducks are fed with only company A’s
duck food
Control Group: Ducks are fed with usual farm-food
Allocation bias
How participants get assigned to treatment group?
Image by Clker-Free-Vector-Images from Pixabay
Image by ChaminaGallery from Pixabay
“I am eating healthy food.
I will run more to get fit!”
“I need to give special
care to this fat duck
because the
investigators said it is
the most important
duck.”
“Need to feed this fat
duck from treatment
group better.”
Performance bias
Is anyone aware of the treatment or made blinded? (participants, intervention providers, study investigators)
The Hawthorne Effect/Observer Effect
Image by Clker-Free-Vector-Images from Pixabay
“This fat duck’s performance is indeed better than other ducks!”
Detection bias
Is anyone aware of the treatment or made blinded? (people who measure the outcome)
Attrition bias
Proportion of participants who stopped the treatment either by self dropout or withdrawal by the study.
Reporting bias
Are all outcomes stated to be measured actually reported despite its being beneficial/favorable or not to the
authors? Are some results measured post hoc for the sake of enhancing favorable outcomes?
Control Group: Ducks are fed with usual
farm-food
Confounders
Participants’ characteristics are similar across all treatments? (gender, age, health status, social status, etc.)
Treatment group
Control group
OR
QA steps in SLR
1. Note the design(s) of your included studies.
2. Identify the type(s) of QA tool(s) to suit your review.
3. Choose the appropriate QA tool(s).
4. Carry out QA using the appropriate tool(s).
5. Tabulate and summarize results of your QA.
6. Think about how your QA results might impact on the conclusions
and recommendations of your SLR.
(Boland, Cherry, and Dickson, 2017)
It is the study design that guide your choice of
QA tool, not the review topic area.
(Boland, Cherry, and Dickson, 2017, p.112)
Strength Level Design Randomization Control
High Level 1 RCT Yes Yes
Meta-analysis of level 1 studies with homogeneous results No No
Level 2 Prospective cohort study (comparative, therapeutic) No Yes
Meta-analysis of level 1 or level 2 studies with heterogeneous
results
No No
Level 3 Retrospective cohort study No Yes
Case control study No Yes
Meta analysis of level 3 studies No No
Level 4 Case series No No
Low Level 5 Case report No No
Expert opinion No No
Personal observation No No
The strength of the experimental design is therefore largely reliant on 4 factors: Randomization, Control Groups, Sample Size, and Generalizability.
Different study designs: Levels of evidence
https://www.hydroassoc.org/research-101-levels-of-evidence-in-hydrocephalus-clinical-research-studies/
RCT Randomized Controlled Trial
https://images.app.goo.gl/DsEvBuQTPJt5XZFD8
Non-randomized studies (NRS)
e.g., Each participant is assigned to a treatment group based on the previous participant’s assignment.
https://images.app.goo.gl/chfsWhrDCLU63Mnn8
Cohort study (prospective comparative study or retrospective cohort study)
https://images.app.goo.gl/jwsJcGd4zUeste2QA https://images.app.goo.gl/zyZ9R2WqETHFbGXC8
Case-control
A group of participants with a
particular conditions are
matched for age and other
characteristics with a control
group of participants who do
not have the conditions.
https://images.app.goo.gl/qpDvU3mfhchGXx6a8
Case series A person or (series of people) who has been given a similar
treatment is followed for a specific time period.
https://images.app.goo.gl/YKr9n7YZ56FN7zwP6
Cross-sectional Data are collected from a number of people or other sources
(e.g., databases) at one point in time.
https://images.app.goo.gl/rVqGvD9yGgPuZ5gDA
Levels of
evidence
https://images.app.goo.gl/47CKR2gHipdRq1qS6
QA steps in SLR
1. Note the design(s) of your included studies.
2. Identify the type(s) of QA tool(s) to suit your review.
3. Choose the appropriate QA tool(s).
4. Carry out QA using the appropriate tool(s).
5. Tabulate and summarize results of your QA.
6. Think about how your QA results might impact on the conclusions
and recommendations of your SLR.
(Boland, Cherry, and Dickson, 2017)
QA tools
• ROBINS-I tool or previously known as ACROBAT-NRSI (A Cochrane Risk of Bias
Assessment Tool)
• the Berger/ISPOR questionnaire
• the Cowley checklist
• the Downs-Black checklist
• EPHPP (Effective Public Health Practice Project Quality Assessment Tool)
• the GRACE checklist (Good ReseArch for Comparative Effectiveness)
• MINORS (Methodological Index for Non-randomised Studies)
• NOS (Newcastle-Ottawa Scale)
• the Reisch-Tyson checklist
• RoBANS (Risk of Bias Assessment Tool for Non-randomised Studies)
• TFCPS (Task Force on Community Preventive Services)
(EUnetHTA, 2015)
QA tools Items Year Applicable study designs
ROBINS-I tool or previously known as ACROBAT-NRSI (A Cochrane Risk of Bias
Assessment Tool)
22-29 2014 NRS (incl. cohort studies + case-control studies),
RCT
Berger/ISPOR questionnaire 33
Cowley checklist 13 1995 NRS, RCT, uncontrolled case series
Downs-Black checklist 27 1998 NRS, RCT
EPHPP (Effective Public Health Practice Project Quality Assessment Tool)
GRACE checklist (Good ReseArch for Comparative Effectiveness) 11
MINORS (Methodological Index for Non-randomised Studies) 12 NRS, RCT
NOS (Newcastle-Ottawa Scale) 8 NRS
Reisch-Tyson checklist 57 1989 Any study design
RoBANS (Risk of Bias Assessment Tool for Non-randomised Studies) 8 2013 NRS (not to non-comparative studies)
TFCPS (Task Force on Community Preventive Services) 26+23 NRS, RCT
Thomas 21 (no date) Any study design
Zaza et al. 22 2000 Any study design
Critical appraisal skills program (CASP) 2013 NRS, RCT, SLR
Centre for reviews and dissemination guidance (health care studies) x 2009 SLR guideline, guides to criteria important in the
assessment of studies
Joanna Briggs Institute (health care studies) x 2014 SLR guideline, , a set of appraisal tools for a range
of study designs
Social care institute for excellence (SCIE) x 2010 SLR guideline, minimum generic criteria for
assessing quality of primary research.
(EUnetHTA, 2015; Boland, Cherry, & Dickson, 2017)
Bias domains
coverage by
tool
(EUnetHTA, 2015)
QA steps in SLR
1. Note the design(s) of your included studies.
2. Identify the type(s) of QA tool(s) to suit your review.
3. Choose the appropriate QA tool(s).
4. Carry out QA using the appropriate tool(s).
5. Tabulate and summarize results of your QA.
6. Think about how your QA results might impact on the conclusions
and recommendations of your SLR.
(Boland, Cherry, and Dickson, 2017)
Peterson, Welch, Losos, & Tugwell, (2011)
Peterson, Welch, Losos, & Tugwell, (2011)
References
• Boland, A., Cherry, G., & Dickson, R. (Eds.). (2017). Doing a
systematic review: A student's guide. Sage.
• EUnetHTA. (2015). WP: Internal validity of non-randomised
studies (nrs) on interventions, 2015.
• Peterson, J., Welch, V., Losos, M., & Tugwell, P. J. O. O. H. R. I.
(2011). The Newcastle-Ottawa scale (NOS) for assessing the
quality of nonrandomised studies in meta-analyses. Ottawa:
Ottawa Hospital Research Institute.

More Related Content

What's hot

Confounding and interaction seminar
Confounding and interaction seminarConfounding and interaction seminar
Confounding and interaction seminar
arundhati garud
 
Presentation on validity and reliability in research methods
Presentation on validity and reliability in research methodsPresentation on validity and reliability in research methods
Presentation on validity and reliability in research methods
Mehwish Iqbal
 
Spotlight Webinar: ROBINS-I
Spotlight Webinar: ROBINS-I Spotlight Webinar: ROBINS-I
General Introduction to Health research (Basic)
General Introduction to Health research (Basic)General Introduction to Health research (Basic)
General Introduction to Health research (Basic)
Public Health Update
 
Randomized controlled trial: Going for the Gold
Randomized controlled trial: Going for the GoldRandomized controlled trial: Going for the Gold
Randomized controlled trial: Going for the Gold
Gaurav Kamboj
 
Introduction to scoping reviews
Introduction to scoping reviewsIntroduction to scoping reviews
Introduction to scoping reviews
Rizwan S A
 
What is a Mann Whitney U?
What is a Mann Whitney U?What is a Mann Whitney U?
What is a Mann Whitney U?
Ken Plummer
 
Hierarchies of Evidence 2017
Hierarchies of Evidence 2017Hierarchies of Evidence 2017
Hierarchies of Evidence 2017
Lucia Ravi
 
How to Conduct a Literature Review
How to Conduct a Literature ReviewHow to Conduct a Literature Review
How to Conduct a Literature ReviewRobin Featherstone
 
Theoretical and conceptual framework
Theoretical and conceptual frameworkTheoretical and conceptual framework
Theoretical and conceptual framework
Kenny Cheah
 
Meta analysis ppt
Meta analysis pptMeta analysis ppt
Meta analysis ppt
SKVA
 
Systematic Reviews: Context & Methodology for Librarians
Systematic Reviews: Context & Methodology for LibrariansSystematic Reviews: Context & Methodology for Librarians
Systematic Reviews: Context & Methodology for Librarians
University of Michigan Taubman Health Sciences Library
 
Critical Appraisal of systematic review and meta analysis articles
Critical Appraisal of systematic review and meta analysis articlesCritical Appraisal of systematic review and meta analysis articles
Critical Appraisal of systematic review and meta analysis articles
Dr. Majdi Al Jasim
 
Systematic review and meta analaysis course - part 2
Systematic review and meta analaysis course - part 2Systematic review and meta analaysis course - part 2
Systematic review and meta analaysis course - part 2
Ahmed Negida
 
Systematic Literature Review and Meta-Analysis (1).pptx
Systematic Literature Review and Meta-Analysis (1).pptxSystematic Literature Review and Meta-Analysis (1).pptx
Systematic Literature Review and Meta-Analysis (1).pptx
EDCEILIYAHDENTALCARE
 
PRISMA
PRISMAPRISMA
Systematic review and meta analysis
Systematic review and meta analysisSystematic review and meta analysis
Systematic review and meta analysis
umaisashraf
 
Implementation Research-What is it?
Implementation Research-What is it?Implementation Research-What is it?
Implementation Research-What is it?
CORE Group
 
Reporting an ANCOVA
Reporting an ANCOVAReporting an ANCOVA
Reporting an ANCOVA
Ken Plummer
 

What's hot (20)

Health research methodology
Health research methodologyHealth research methodology
Health research methodology
 
Confounding and interaction seminar
Confounding and interaction seminarConfounding and interaction seminar
Confounding and interaction seminar
 
Presentation on validity and reliability in research methods
Presentation on validity and reliability in research methodsPresentation on validity and reliability in research methods
Presentation on validity and reliability in research methods
 
Spotlight Webinar: ROBINS-I
Spotlight Webinar: ROBINS-I Spotlight Webinar: ROBINS-I
Spotlight Webinar: ROBINS-I
 
General Introduction to Health research (Basic)
General Introduction to Health research (Basic)General Introduction to Health research (Basic)
General Introduction to Health research (Basic)
 
Randomized controlled trial: Going for the Gold
Randomized controlled trial: Going for the GoldRandomized controlled trial: Going for the Gold
Randomized controlled trial: Going for the Gold
 
Introduction to scoping reviews
Introduction to scoping reviewsIntroduction to scoping reviews
Introduction to scoping reviews
 
What is a Mann Whitney U?
What is a Mann Whitney U?What is a Mann Whitney U?
What is a Mann Whitney U?
 
Hierarchies of Evidence 2017
Hierarchies of Evidence 2017Hierarchies of Evidence 2017
Hierarchies of Evidence 2017
 
How to Conduct a Literature Review
How to Conduct a Literature ReviewHow to Conduct a Literature Review
How to Conduct a Literature Review
 
Theoretical and conceptual framework
Theoretical and conceptual frameworkTheoretical and conceptual framework
Theoretical and conceptual framework
 
Meta analysis ppt
Meta analysis pptMeta analysis ppt
Meta analysis ppt
 
Systematic Reviews: Context & Methodology for Librarians
Systematic Reviews: Context & Methodology for LibrariansSystematic Reviews: Context & Methodology for Librarians
Systematic Reviews: Context & Methodology for Librarians
 
Critical Appraisal of systematic review and meta analysis articles
Critical Appraisal of systematic review and meta analysis articlesCritical Appraisal of systematic review and meta analysis articles
Critical Appraisal of systematic review and meta analysis articles
 
Systematic review and meta analaysis course - part 2
Systematic review and meta analaysis course - part 2Systematic review and meta analaysis course - part 2
Systematic review and meta analaysis course - part 2
 
Systematic Literature Review and Meta-Analysis (1).pptx
Systematic Literature Review and Meta-Analysis (1).pptxSystematic Literature Review and Meta-Analysis (1).pptx
Systematic Literature Review and Meta-Analysis (1).pptx
 
PRISMA
PRISMAPRISMA
PRISMA
 
Systematic review and meta analysis
Systematic review and meta analysisSystematic review and meta analysis
Systematic review and meta analysis
 
Implementation Research-What is it?
Implementation Research-What is it?Implementation Research-What is it?
Implementation Research-What is it?
 
Reporting an ANCOVA
Reporting an ANCOVAReporting an ANCOVA
Reporting an ANCOVA
 

Similar to Quality assessment in systematic literature review

CASP Randomised Controlled Trial Standard Checklist 11 que.docx
CASP Randomised Controlled Trial Standard Checklist  11 que.docxCASP Randomised Controlled Trial Standard Checklist  11 que.docx
CASP Randomised Controlled Trial Standard Checklist 11 que.docx
bartholomeocoombs
 
Introduction to Systematic Reviews
Introduction to Systematic ReviewsIntroduction to Systematic Reviews
Introduction to Systematic Reviews
Laura Koltutsky
 
Research design
Research designResearch design
Research design
Abhijeet Birari
 
How to conduct a systematic review
How to conduct a systematic reviewHow to conduct a systematic review
How to conduct a systematic review
DrNidhiPruthiShukla
 
Critical Appraisal Paper
Critical Appraisal PaperCritical Appraisal Paper
Critical Appraisal Paper
Pay To Write Papers High Point
 
Critical appraisal 2012 updated
Critical appraisal 2012 updatedCritical appraisal 2012 updated
Critical appraisal 2012 updatedsanghagirl
 
Methodology
MethodologyMethodology
Methodology
Nelyn Joy Castillon
 
Casp rct checklist_pdf
Casp rct checklist_pdfCasp rct checklist_pdf
Casp rct checklist_pdf
GiziKesehatanUGM2017
 
JC SEBMA Prognosis Appraisal Template V1
JC SEBMA Prognosis Appraisal Template V1JC SEBMA Prognosis Appraisal Template V1
JC SEBMA Prognosis Appraisal Template V1
Imad Hassan
 
K7 - Critical Appraisal.pdf
K7 - Critical Appraisal.pdfK7 - Critical Appraisal.pdf
K7 - Critical Appraisal.pdf
JeslynTengkawan1
 
Introduction to Evidence Based Dentistry
Introduction to Evidence Based DentistryIntroduction to Evidence Based Dentistry
Introduction to Evidence Based Dentistry
Rasha Adel
 
Outcome Measures Issues Opportunities in Healthcare Organizations.docx
Outcome Measures Issues Opportunities in Healthcare Organizations.docxOutcome Measures Issues Opportunities in Healthcare Organizations.docx
Outcome Measures Issues Opportunities in Healthcare Organizations.docx
sdfghj21
 
EBM Systematic Review Appraisal Template V1
EBM Systematic Review Appraisal Template V1EBM Systematic Review Appraisal Template V1
EBM Systematic Review Appraisal Template V1
Imad Hassan
 
Risk of Bias_StaR Child Health Summit_07May12
Risk of Bias_StaR Child Health Summit_07May12Risk of Bias_StaR Child Health Summit_07May12
Risk of Bias_StaR Child Health Summit_07May12michele_hamm
 
Developing a Systematic Review Protocol
Developing a Systematic Review ProtocolDeveloping a Systematic Review Protocol
Developing a Systematic Review Protocol
ACSRM
 
When to Select Observational Studies as Evidence for Comparative Effectivenes...
When to Select Observational Studies as Evidence for Comparative Effectivenes...When to Select Observational Studies as Evidence for Comparative Effectivenes...
When to Select Observational Studies as Evidence for Comparative Effectivenes...Effective Health Care Program
 
Systematic Reviews Class 4c
Systematic Reviews Class 4cSystematic Reviews Class 4c
Systematic Reviews Class 4cguestf5d7ac
 
EBM Therapy Appraisal Template F1
EBM Therapy Appraisal Template F1EBM Therapy Appraisal Template F1
EBM Therapy Appraisal Template F1
Imad Hassan
 
Health Evidence™ Critical Appraisal Tool for Economic Evaluations (Sample Ans...
Health Evidence™ Critical Appraisal Tool for Economic Evaluations (Sample Ans...Health Evidence™ Critical Appraisal Tool for Economic Evaluations (Sample Ans...
Health Evidence™ Critical Appraisal Tool for Economic Evaluations (Sample Ans...
The National Collaborating Centre for Methods and Tools
 

Similar to Quality assessment in systematic literature review (20)

CASP Randomised Controlled Trial Standard Checklist 11 que.docx
CASP Randomised Controlled Trial Standard Checklist  11 que.docxCASP Randomised Controlled Trial Standard Checklist  11 que.docx
CASP Randomised Controlled Trial Standard Checklist 11 que.docx
 
Introduction to Systematic Reviews
Introduction to Systematic ReviewsIntroduction to Systematic Reviews
Introduction to Systematic Reviews
 
Research design
Research designResearch design
Research design
 
How to conduct a systematic review
How to conduct a systematic reviewHow to conduct a systematic review
How to conduct a systematic review
 
Critical Appraisal Paper
Critical Appraisal PaperCritical Appraisal Paper
Critical Appraisal Paper
 
Critical appraisal 2012 updated
Critical appraisal 2012 updatedCritical appraisal 2012 updated
Critical appraisal 2012 updated
 
Methodology
MethodologyMethodology
Methodology
 
Casp rct checklist_pdf
Casp rct checklist_pdfCasp rct checklist_pdf
Casp rct checklist_pdf
 
JC SEBMA Prognosis Appraisal Template V1
JC SEBMA Prognosis Appraisal Template V1JC SEBMA Prognosis Appraisal Template V1
JC SEBMA Prognosis Appraisal Template V1
 
K7 - Critical Appraisal.pdf
K7 - Critical Appraisal.pdfK7 - Critical Appraisal.pdf
K7 - Critical Appraisal.pdf
 
Study Eligibility Criteria
Study Eligibility CriteriaStudy Eligibility Criteria
Study Eligibility Criteria
 
Introduction to Evidence Based Dentistry
Introduction to Evidence Based DentistryIntroduction to Evidence Based Dentistry
Introduction to Evidence Based Dentistry
 
Outcome Measures Issues Opportunities in Healthcare Organizations.docx
Outcome Measures Issues Opportunities in Healthcare Organizations.docxOutcome Measures Issues Opportunities in Healthcare Organizations.docx
Outcome Measures Issues Opportunities in Healthcare Organizations.docx
 
EBM Systematic Review Appraisal Template V1
EBM Systematic Review Appraisal Template V1EBM Systematic Review Appraisal Template V1
EBM Systematic Review Appraisal Template V1
 
Risk of Bias_StaR Child Health Summit_07May12
Risk of Bias_StaR Child Health Summit_07May12Risk of Bias_StaR Child Health Summit_07May12
Risk of Bias_StaR Child Health Summit_07May12
 
Developing a Systematic Review Protocol
Developing a Systematic Review ProtocolDeveloping a Systematic Review Protocol
Developing a Systematic Review Protocol
 
When to Select Observational Studies as Evidence for Comparative Effectivenes...
When to Select Observational Studies as Evidence for Comparative Effectivenes...When to Select Observational Studies as Evidence for Comparative Effectivenes...
When to Select Observational Studies as Evidence for Comparative Effectivenes...
 
Systematic Reviews Class 4c
Systematic Reviews Class 4cSystematic Reviews Class 4c
Systematic Reviews Class 4c
 
EBM Therapy Appraisal Template F1
EBM Therapy Appraisal Template F1EBM Therapy Appraisal Template F1
EBM Therapy Appraisal Template F1
 
Health Evidence™ Critical Appraisal Tool for Economic Evaluations (Sample Ans...
Health Evidence™ Critical Appraisal Tool for Economic Evaluations (Sample Ans...Health Evidence™ Critical Appraisal Tool for Economic Evaluations (Sample Ans...
Health Evidence™ Critical Appraisal Tool for Economic Evaluations (Sample Ans...
 

More from Jingjing Lin

Connect curriculum to reality via creation-based learning: The pre-experiment...
Connect curriculum to reality via creation-based learning: The pre-experiment...Connect curriculum to reality via creation-based learning: The pre-experiment...
Connect curriculum to reality via creation-based learning: The pre-experiment...
Jingjing Lin
 
Mahara Training in 90 Minutes
Mahara Training in 90 MinutesMahara Training in 90 Minutes
Mahara Training in 90 Minutes
Jingjing Lin
 
Hosting Online Journal Club Events on a Global Scale using Moodle: A Report f...
Hosting Online Journal Club Events on a Global Scale using Moodle: A Report f...Hosting Online Journal Club Events on a Global Scale using Moodle: A Report f...
Hosting Online Journal Club Events on a Global Scale using Moodle: A Report f...
Jingjing Lin
 
使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区
使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区
使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区
Jingjing Lin
 
Use Online Journal Club Events To Train The Educational Research Literacy: Re...
Use Online Journal Club Events To Train The Educational Research Literacy: Re...Use Online Journal Club Events To Train The Educational Research Literacy: Re...
Use Online Journal Club Events To Train The Educational Research Literacy: Re...
Jingjing Lin
 
Measuring and Evaluating Educational Research Literacy in Higher Education: A...
Measuring and Evaluating Educational Research Literacy in Higher Education: A...Measuring and Evaluating Educational Research Literacy in Higher Education: A...
Measuring and Evaluating Educational Research Literacy in Higher Education: A...
Jingjing Lin
 
Use Online Journal Club Events For Better Scholarly Reading Habits And Skills...
Use Online Journal Club Events For Better Scholarly Reading Habits And Skills...Use Online Journal Club Events For Better Scholarly Reading Habits And Skills...
Use Online Journal Club Events For Better Scholarly Reading Habits And Skills...
Jingjing Lin
 
The CLEAR Framework to Implement Active Learning in STEM Education
The CLEAR Framework to Implement Active Learning in STEM EducationThe CLEAR Framework to Implement Active Learning in STEM Education
The CLEAR Framework to Implement Active Learning in STEM Education
Jingjing Lin
 
Crafting Hackerspaces with Moodle and Mahara: The Potential of Creation based...
Crafting Hackerspaces with Moodle and Mahara: The Potential of Creation based...Crafting Hackerspaces with Moodle and Mahara: The Potential of Creation based...
Crafting Hackerspaces with Moodle and Mahara: The Potential of Creation based...
Jingjing Lin
 
E-learning in Entrepreneurship Education: A systematic literature review
E-learning in Entrepreneurship Education: A systematic literature reviewE-learning in Entrepreneurship Education: A systematic literature review
E-learning in Entrepreneurship Education: A systematic literature review
Jingjing Lin
 
Mahara in a nutshell
Mahara in a nutshellMahara in a nutshell
Mahara in a nutshell
Jingjing Lin
 
Entrepreneurship,education, and films
Entrepreneurship,education, and filmsEntrepreneurship,education, and films
Entrepreneurship,education, and films
Jingjing Lin
 
The Delphi Method and its Contribution to Decision-Making
The Delphi Method and its Contribution to Decision-MakingThe Delphi Method and its Contribution to Decision-Making
The Delphi Method and its Contribution to Decision-Making
Jingjing Lin
 
What it means to do a Ph.D.?
What it means to do a Ph.D.?What it means to do a Ph.D.?
What it means to do a Ph.D.?
Jingjing Lin
 
China Travel Service Industry Development Report 2017
China Travel Service Industry Development Report 2017China Travel Service Industry Development Report 2017
China Travel Service Industry Development Report 2017
Jingjing Lin
 
MOOCs in tourism and hospitality
MOOCs in tourism and hospitalityMOOCs in tourism and hospitality
MOOCs in tourism and hospitality
Jingjing Lin
 
Moocs in tourism and hospitality
Moocs in tourism and hospitalityMoocs in tourism and hospitality
Moocs in tourism and hospitality
Jingjing Lin
 
Define massive open online course: results from systematic review of 84 publi...
Define massive open online course: results from systematic review of 84 publi...Define massive open online course: results from systematic review of 84 publi...
Define massive open online course: results from systematic review of 84 publi...
Jingjing Lin
 
Backlink Manifestations
Backlink ManifestationsBacklink Manifestations
Backlink Manifestations
Jingjing Lin
 
Academic tourism and hospitality MOOCs: A review (2008-2015)
Academic tourism and hospitality MOOCs: A review (2008-2015)Academic tourism and hospitality MOOCs: A review (2008-2015)
Academic tourism and hospitality MOOCs: A review (2008-2015)
Jingjing Lin
 

More from Jingjing Lin (20)

Connect curriculum to reality via creation-based learning: The pre-experiment...
Connect curriculum to reality via creation-based learning: The pre-experiment...Connect curriculum to reality via creation-based learning: The pre-experiment...
Connect curriculum to reality via creation-based learning: The pre-experiment...
 
Mahara Training in 90 Minutes
Mahara Training in 90 MinutesMahara Training in 90 Minutes
Mahara Training in 90 Minutes
 
Hosting Online Journal Club Events on a Global Scale using Moodle: A Report f...
Hosting Online Journal Club Events on a Global Scale using Moodle: A Report f...Hosting Online Journal Club Events on a Global Scale using Moodle: A Report f...
Hosting Online Journal Club Events on a Global Scale using Moodle: A Report f...
 
使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区
使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区
使用在线期刊俱乐部和人工 智能工具培训教育研究素养: 走向新的培训模式和构建在 线实践社区
 
Use Online Journal Club Events To Train The Educational Research Literacy: Re...
Use Online Journal Club Events To Train The Educational Research Literacy: Re...Use Online Journal Club Events To Train The Educational Research Literacy: Re...
Use Online Journal Club Events To Train The Educational Research Literacy: Re...
 
Measuring and Evaluating Educational Research Literacy in Higher Education: A...
Measuring and Evaluating Educational Research Literacy in Higher Education: A...Measuring and Evaluating Educational Research Literacy in Higher Education: A...
Measuring and Evaluating Educational Research Literacy in Higher Education: A...
 
Use Online Journal Club Events For Better Scholarly Reading Habits And Skills...
Use Online Journal Club Events For Better Scholarly Reading Habits And Skills...Use Online Journal Club Events For Better Scholarly Reading Habits And Skills...
Use Online Journal Club Events For Better Scholarly Reading Habits And Skills...
 
The CLEAR Framework to Implement Active Learning in STEM Education
The CLEAR Framework to Implement Active Learning in STEM EducationThe CLEAR Framework to Implement Active Learning in STEM Education
The CLEAR Framework to Implement Active Learning in STEM Education
 
Crafting Hackerspaces with Moodle and Mahara: The Potential of Creation based...
Crafting Hackerspaces with Moodle and Mahara: The Potential of Creation based...Crafting Hackerspaces with Moodle and Mahara: The Potential of Creation based...
Crafting Hackerspaces with Moodle and Mahara: The Potential of Creation based...
 
E-learning in Entrepreneurship Education: A systematic literature review
E-learning in Entrepreneurship Education: A systematic literature reviewE-learning in Entrepreneurship Education: A systematic literature review
E-learning in Entrepreneurship Education: A systematic literature review
 
Mahara in a nutshell
Mahara in a nutshellMahara in a nutshell
Mahara in a nutshell
 
Entrepreneurship,education, and films
Entrepreneurship,education, and filmsEntrepreneurship,education, and films
Entrepreneurship,education, and films
 
The Delphi Method and its Contribution to Decision-Making
The Delphi Method and its Contribution to Decision-MakingThe Delphi Method and its Contribution to Decision-Making
The Delphi Method and its Contribution to Decision-Making
 
What it means to do a Ph.D.?
What it means to do a Ph.D.?What it means to do a Ph.D.?
What it means to do a Ph.D.?
 
China Travel Service Industry Development Report 2017
China Travel Service Industry Development Report 2017China Travel Service Industry Development Report 2017
China Travel Service Industry Development Report 2017
 
MOOCs in tourism and hospitality
MOOCs in tourism and hospitalityMOOCs in tourism and hospitality
MOOCs in tourism and hospitality
 
Moocs in tourism and hospitality
Moocs in tourism and hospitalityMoocs in tourism and hospitality
Moocs in tourism and hospitality
 
Define massive open online course: results from systematic review of 84 publi...
Define massive open online course: results from systematic review of 84 publi...Define massive open online course: results from systematic review of 84 publi...
Define massive open online course: results from systematic review of 84 publi...
 
Backlink Manifestations
Backlink ManifestationsBacklink Manifestations
Backlink Manifestations
 
Academic tourism and hospitality MOOCs: A review (2008-2015)
Academic tourism and hospitality MOOCs: A review (2008-2015)Academic tourism and hospitality MOOCs: A review (2008-2015)
Academic tourism and hospitality MOOCs: A review (2008-2015)
 

Recently uploaded

Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptxStudents, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
EduSkills OECD
 
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
AzmatAli747758
 
Introduction to Quality Improvement Essentials
Introduction to Quality Improvement EssentialsIntroduction to Quality Improvement Essentials
Introduction to Quality Improvement Essentials
Excellence Foundation for South Sudan
 
Chapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptxChapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptx
Mohd Adib Abd Muin, Senior Lecturer at Universiti Utara Malaysia
 
Additional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdfAdditional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdf
joachimlavalley1
 
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
EugeneSaldivar
 
Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......
Ashokrao Mane college of Pharmacy Peth-Vadgaon
 
The Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdfThe Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdf
kaushalkr1407
 
Fish and Chips - have they had their chips
Fish and Chips - have they had their chipsFish and Chips - have they had their chips
Fish and Chips - have they had their chips
GeoBlogs
 
Model Attribute Check Company Auto Property
Model Attribute  Check Company Auto PropertyModel Attribute  Check Company Auto Property
Model Attribute Check Company Auto Property
Celine George
 
Supporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptxSupporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptx
Jisc
 
How to Make a Field invisible in Odoo 17
How to Make a Field invisible in Odoo 17How to Make a Field invisible in Odoo 17
How to Make a Field invisible in Odoo 17
Celine George
 
Language Across the Curriculm LAC B.Ed.
Language Across the  Curriculm LAC B.Ed.Language Across the  Curriculm LAC B.Ed.
Language Across the Curriculm LAC B.Ed.
Atul Kumar Singh
 
The Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official PublicationThe Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official Publication
Delapenabediema
 
How to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS ModuleHow to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS Module
Celine George
 
GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...
GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...
GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...
Nguyen Thanh Tu Collection
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
siemaillard
 
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdfUnit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Thiyagu K
 
Sectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdfSectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdf
Vivekanand Anglo Vedic Academy
 
Thesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.pptThesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.ppt
EverAndrsGuerraGuerr
 

Recently uploaded (20)

Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptxStudents, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptx
 
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...Cambridge International AS  A Level Biology Coursebook - EBook (MaryFosbery J...
Cambridge International AS A Level Biology Coursebook - EBook (MaryFosbery J...
 
Introduction to Quality Improvement Essentials
Introduction to Quality Improvement EssentialsIntroduction to Quality Improvement Essentials
Introduction to Quality Improvement Essentials
 
Chapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptxChapter 3 - Islamic Banking Products and Services.pptx
Chapter 3 - Islamic Banking Products and Services.pptx
 
Additional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdfAdditional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdf
 
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
 
Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......Ethnobotany and Ethnopharmacology ......
Ethnobotany and Ethnopharmacology ......
 
The Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdfThe Roman Empire A Historical Colossus.pdf
The Roman Empire A Historical Colossus.pdf
 
Fish and Chips - have they had their chips
Fish and Chips - have they had their chipsFish and Chips - have they had their chips
Fish and Chips - have they had their chips
 
Model Attribute Check Company Auto Property
Model Attribute  Check Company Auto PropertyModel Attribute  Check Company Auto Property
Model Attribute Check Company Auto Property
 
Supporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptxSupporting (UKRI) OA monographs at Salford.pptx
Supporting (UKRI) OA monographs at Salford.pptx
 
How to Make a Field invisible in Odoo 17
How to Make a Field invisible in Odoo 17How to Make a Field invisible in Odoo 17
How to Make a Field invisible in Odoo 17
 
Language Across the Curriculm LAC B.Ed.
Language Across the  Curriculm LAC B.Ed.Language Across the  Curriculm LAC B.Ed.
Language Across the Curriculm LAC B.Ed.
 
The Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official PublicationThe Challenger.pdf DNHS Official Publication
The Challenger.pdf DNHS Official Publication
 
How to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS ModuleHow to Split Bills in the Odoo 17 POS Module
How to Split Bills in the Odoo 17 POS Module
 
GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...
GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...
GIÁO ÁN DẠY THÊM (KẾ HOẠCH BÀI BUỔI 2) - TIẾNG ANH 8 GLOBAL SUCCESS (2 CỘT) N...
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdfUnit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdf
 
Sectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdfSectors of the Indian Economy - Class 10 Study Notes pdf
Sectors of the Indian Economy - Class 10 Study Notes pdf
 
Thesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.pptThesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.ppt
 

Quality assessment in systematic literature review

  • 1. Quality assessment in SLR Systematic Literature Review Workshop (SLRW) Dr. Jingjing Lin Assistant Professor Centre of IT-based Education Toyohashi University of Technology lin.jingjing.qc@tut.jp
  • 2. Don't judge a book by its cover. (English idiom) Do not take published studies at face value. (Boland, Cherry, and Dickson, 2017, p.109)
  • 3. Which side are you on? • It is published in Nature! It must be a good quality study. I have no doubt of it. • It is published in Nature! But the sample is rather small than usual, I doubt its generalizability. (Boland, Cherry, and Dickson, 2017)
  • 4. Why you need QA in your SLR? Develop a greater understanding of your studies and others’ results. Distinguish between good-quality and poor- quality studies. More likely to draw meaningful conclusions from the data. Offer you opportunity to acquire critical appraisal skill. Validated SLR checklist often includes it to evaluate SLR’s quality. (Boland, Cherry, and Dickson, 2017)
  • 5. What is quality? Quality of individual studies Quality of your SLR The degree to which a study employs measures to minimize error and bias in its design, conduct, and analysis. (Khan et al., 2003, p.39) To check against validated SLR checklist tools (Boland, Cherry, and Dickson, 2017)
  • 6. “This is a good quality study”. Translated into: • “I am confident that this study’s design, conduct and analysis are robust to provide results that are credible, trustworthy and generalizable, and are highly likely to be a true representation of the results of the tested intervention, phenomenon, or exposure.” (Boland, Cherry, and Dickson, 2017)
  • 7. When to do QA Before data extraction During data extraction After data extraction • When you intend to exclude poor quality studies in your SLR NA • You will be blind towards the quality of individual studies when extracting data, and your report is likely to be biased. • Your greater familiarity with pooled studies can help you answer QA questions. (Boland, Cherry, and Dickson, 2017)
  • 8. Main elements of QA Element/Bias type What is it Significance Selection bias Is the sample representative of the population? Generalizability, transferability Allocation bias How participants get assigned to treatment group? Any human that influences or interferes such allocation? Types of design influence such allocation strategy. The stronger the design, the less bias from allocation to treatment. Performance bias Is anyone aware of the treatment or made blinded? (participants, intervention providers, study investigators) Can such awareness or blindness cause bias in the study? Detection bias Is anyone aware of the treatment or made blinded? (people who measure the outcome) Attrition bias Proportion of participants who stopped the treatment either by self dropout or withdrawal by the study. Weaken the generalizability, give insight to the compliance rates, treatment groups inequality due to participant size change Reporting bias Are all outcomes stated to be measured actually reported despite its being beneficial/favorable or not to the authors? Are some results measured post hoc for the sake of enhancing favorable outcomes? Reasons that lead to the failure of reporting all stated outcomes need to be provided, the treatment appears more favorable than it really is Confounders Participants’ characteristics are similar across all treatments? (gender, age, health status, social status, etc.) Participants should be equally balanced otherwise there is a risk that results are biased in favor of one group than the other. Concurrent or subsequent intervention Was any other treatment received by participants beside the study intervention? Participants should be treated in the same way to reduce the risk of other effects, which weakens the study intervention’s effect. Analysis Were the data for all participants included in final analysis even whose withdrew ones? If there are data missing for a number of participants and these data are not considered, published results will not properly reflect the results of the study. Funding bias Who funded the study? Funders may want your study to favor positive outcomes instead of negative outcomes. QA tools ask questions about bias. Bias types are many. Assessment of risk of bias covers at least six bias as colored in red in the table below according to EUnetHTA (2015) (Boland, Cherry, and Dickson, 2017)
  • 9. Selection bias Is the sample representative of the population? Duckling image from: https://pixabay.com/illustrations/ptak-brzydkie-kacz%C4%85tko-bajki-2468019/
  • 10. Treatment Group: Ducks are fed with only company A’s duck food Control Group: Ducks are fed with usual farm-food Allocation bias How participants get assigned to treatment group?
  • 11. Image by Clker-Free-Vector-Images from Pixabay Image by ChaminaGallery from Pixabay “I am eating healthy food. I will run more to get fit!” “I need to give special care to this fat duck because the investigators said it is the most important duck.” “Need to feed this fat duck from treatment group better.” Performance bias Is anyone aware of the treatment or made blinded? (participants, intervention providers, study investigators) The Hawthorne Effect/Observer Effect
  • 12. Image by Clker-Free-Vector-Images from Pixabay “This fat duck’s performance is indeed better than other ducks!” Detection bias Is anyone aware of the treatment or made blinded? (people who measure the outcome)
  • 13. Attrition bias Proportion of participants who stopped the treatment either by self dropout or withdrawal by the study.
  • 14. Reporting bias Are all outcomes stated to be measured actually reported despite its being beneficial/favorable or not to the authors? Are some results measured post hoc for the sake of enhancing favorable outcomes? Control Group: Ducks are fed with usual farm-food
  • 15. Confounders Participants’ characteristics are similar across all treatments? (gender, age, health status, social status, etc.) Treatment group Control group OR
  • 16. QA steps in SLR 1. Note the design(s) of your included studies. 2. Identify the type(s) of QA tool(s) to suit your review. 3. Choose the appropriate QA tool(s). 4. Carry out QA using the appropriate tool(s). 5. Tabulate and summarize results of your QA. 6. Think about how your QA results might impact on the conclusions and recommendations of your SLR. (Boland, Cherry, and Dickson, 2017)
  • 17. It is the study design that guide your choice of QA tool, not the review topic area. (Boland, Cherry, and Dickson, 2017, p.112)
  • 18. Strength Level Design Randomization Control High Level 1 RCT Yes Yes Meta-analysis of level 1 studies with homogeneous results No No Level 2 Prospective cohort study (comparative, therapeutic) No Yes Meta-analysis of level 1 or level 2 studies with heterogeneous results No No Level 3 Retrospective cohort study No Yes Case control study No Yes Meta analysis of level 3 studies No No Level 4 Case series No No Low Level 5 Case report No No Expert opinion No No Personal observation No No The strength of the experimental design is therefore largely reliant on 4 factors: Randomization, Control Groups, Sample Size, and Generalizability. Different study designs: Levels of evidence https://www.hydroassoc.org/research-101-levels-of-evidence-in-hydrocephalus-clinical-research-studies/
  • 19. RCT Randomized Controlled Trial https://images.app.goo.gl/DsEvBuQTPJt5XZFD8
  • 20. Non-randomized studies (NRS) e.g., Each participant is assigned to a treatment group based on the previous participant’s assignment. https://images.app.goo.gl/chfsWhrDCLU63Mnn8
  • 21. Cohort study (prospective comparative study or retrospective cohort study) https://images.app.goo.gl/jwsJcGd4zUeste2QA https://images.app.goo.gl/zyZ9R2WqETHFbGXC8
  • 22. Case-control A group of participants with a particular conditions are matched for age and other characteristics with a control group of participants who do not have the conditions. https://images.app.goo.gl/qpDvU3mfhchGXx6a8
  • 23. Case series A person or (series of people) who has been given a similar treatment is followed for a specific time period. https://images.app.goo.gl/YKr9n7YZ56FN7zwP6
  • 24. Cross-sectional Data are collected from a number of people or other sources (e.g., databases) at one point in time. https://images.app.goo.gl/rVqGvD9yGgPuZ5gDA
  • 26. QA steps in SLR 1. Note the design(s) of your included studies. 2. Identify the type(s) of QA tool(s) to suit your review. 3. Choose the appropriate QA tool(s). 4. Carry out QA using the appropriate tool(s). 5. Tabulate and summarize results of your QA. 6. Think about how your QA results might impact on the conclusions and recommendations of your SLR. (Boland, Cherry, and Dickson, 2017)
  • 27. QA tools • ROBINS-I tool or previously known as ACROBAT-NRSI (A Cochrane Risk of Bias Assessment Tool) • the Berger/ISPOR questionnaire • the Cowley checklist • the Downs-Black checklist • EPHPP (Effective Public Health Practice Project Quality Assessment Tool) • the GRACE checklist (Good ReseArch for Comparative Effectiveness) • MINORS (Methodological Index for Non-randomised Studies) • NOS (Newcastle-Ottawa Scale) • the Reisch-Tyson checklist • RoBANS (Risk of Bias Assessment Tool for Non-randomised Studies) • TFCPS (Task Force on Community Preventive Services) (EUnetHTA, 2015)
  • 28. QA tools Items Year Applicable study designs ROBINS-I tool or previously known as ACROBAT-NRSI (A Cochrane Risk of Bias Assessment Tool) 22-29 2014 NRS (incl. cohort studies + case-control studies), RCT Berger/ISPOR questionnaire 33 Cowley checklist 13 1995 NRS, RCT, uncontrolled case series Downs-Black checklist 27 1998 NRS, RCT EPHPP (Effective Public Health Practice Project Quality Assessment Tool) GRACE checklist (Good ReseArch for Comparative Effectiveness) 11 MINORS (Methodological Index for Non-randomised Studies) 12 NRS, RCT NOS (Newcastle-Ottawa Scale) 8 NRS Reisch-Tyson checklist 57 1989 Any study design RoBANS (Risk of Bias Assessment Tool for Non-randomised Studies) 8 2013 NRS (not to non-comparative studies) TFCPS (Task Force on Community Preventive Services) 26+23 NRS, RCT Thomas 21 (no date) Any study design Zaza et al. 22 2000 Any study design Critical appraisal skills program (CASP) 2013 NRS, RCT, SLR Centre for reviews and dissemination guidance (health care studies) x 2009 SLR guideline, guides to criteria important in the assessment of studies Joanna Briggs Institute (health care studies) x 2014 SLR guideline, , a set of appraisal tools for a range of study designs Social care institute for excellence (SCIE) x 2010 SLR guideline, minimum generic criteria for assessing quality of primary research. (EUnetHTA, 2015; Boland, Cherry, & Dickson, 2017)
  • 30. QA steps in SLR 1. Note the design(s) of your included studies. 2. Identify the type(s) of QA tool(s) to suit your review. 3. Choose the appropriate QA tool(s). 4. Carry out QA using the appropriate tool(s). 5. Tabulate and summarize results of your QA. 6. Think about how your QA results might impact on the conclusions and recommendations of your SLR. (Boland, Cherry, and Dickson, 2017)
  • 31. Peterson, Welch, Losos, & Tugwell, (2011)
  • 32. Peterson, Welch, Losos, & Tugwell, (2011)
  • 33. References • Boland, A., Cherry, G., & Dickson, R. (Eds.). (2017). Doing a systematic review: A student's guide. Sage. • EUnetHTA. (2015). WP: Internal validity of non-randomised studies (nrs) on interventions, 2015. • Peterson, J., Welch, V., Losos, M., & Tugwell, P. J. O. O. H. R. I. (2011). The Newcastle-Ottawa scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa: Ottawa Hospital Research Institute.