SlideShare a Scribd company logo
A Workshop on the Basics of Systematic Review &  Meta-Analysis  Philip C. Abrami, Robert M. Bernard C. Anne Wade, Evgueni Borokhovski, Rana Tamim, Gretchen Lowerison & Mike Surkes Centre for the Study of Learning and Performance  and CanKnow Concordia University
02/25/11 What is a Systematic Review? ,[object Object],[object Object],[object Object]
What is Meta-Analysis? ,[object Object],[object Object],02/25/11
02/25/11
Purpose:  Explaining Variability in Effect Size 02/25/11 Effect Sizes Study Features Shared Variability Unique Variability Unique Variability Prediction
10 Steps in Planning and Conducting a Systematic Review/Meta-Analysis 02/25/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
02/25/11
02/25/11 1. Determine the research question The “big question” that guides the research. It usually involves asking about the difference between two conditions (i.e., usually treatment and control) or the relationship between two measures. 10 Steps in a Meta-Analysis
Questions the Researcher  Should Ask ,[object Object],[object Object],[object Object],[object Object],[object Object],02/25/11
Example:  Critical Thinking 02/25/11 Research Question:   What  instructional interventions,  to what extent, and under what particular circumstances, impact on the development and effective use of learner’s  critical thinking skills   and dispositions?
02/25/11 2.  Develop terms and definitions related to the question This  helps refine the research question and inform the search strategies. 10 Steps in a Meta-Analysis
02/25/11 3.  Develop a search strategy for the identification of relevant studies This involves the planning/implementation of search and retrieval for primary studies  (e.g., electronic databases, branching). 10 Steps in a Meta-Analysis
02/25/11 Information Retrieval:  A Continuous Process ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
02/25/11 Preliminary Searches Reference Sources: Purpose :  To obtain definitions for the terms; creativity, critical thinking, decision making, divergent thinking, intelligence; problem solving, reasoning, thinking. Sources: Bailin, S. (1998).  Critical Thinking: Philosophical Issues . [CD-ROM] Education: The Complete Encyclopedia. Elsevier Science, Ltd.  Barrow, R., & Milburn, G. (1990).  A critical dictionary of educational concepts: An appraisal of selected ideas and issues in educational theory and practice  (2 nd  ed.) .  Hertfordshire, UK: Harvester Wheatsheaf Colman (2001). Dictionary of Psychology (complete reference to be obtained) Corsini, R. J. (1999).  The dictionary of psychology . Philadelphia, PA: Brunner/Mazel Dejnoka, E. L., & Kapel, D. E. (1991).  American educator’s encyclopedia . Westport, CT: Greenwood Press. ……  (see handout)
02/25/11 Main Searches: Decisions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Examples of Databases ,[object Object],[object Object],[object Object],[object Object],02/25/11
Example: Critical Thinking ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],02/25/11
02/25/11 Main Searches: More Decisions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Main Searches:  Yet More Decisions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],02/25/11
Example: ERIC ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],02/25/11
Documenting Your Searches 02/25/11 Example from our Codebook: ERIC   (Date: September 21, 2003; AW) Purpose :  To retrieve the first set of abstracts to be reviewed by team according to the current inclusive/exclusion criteria.  Result :  Hit rate of  514/1520 Source code:  ERIC1 Searches and records below from: The ERIC Database (1966-2003, June) #5  #3 and #4 (1520 records) #4  DTC = 142 or DTC = 143 or control group* (322893  records) #3  #1 or #2 (7718 records) #2  critical thinking in DE,ID (7562 records) #1  thinking skills in DE and critical thinking (1269 records)
Next Steps ,[object Object],[object Object],02/25/11
02/25/11 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Main Searches:  Yet Still More Decisions
02/25/11 Information Retrieval:  Wrap Up ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
02/25/11 10 Steps in a Meta-Analysis 4. Establish criteria for inclusion  and exclusion of studies   These are the criteria that guide the search for literature and ultimately determine what studies  are in and out of the review.
Inclusion/Exclusion: Questions ,[object Object],[object Object],[object Object],[object Object],[object Object],02/25/11
02/25/11 10 Steps in a Meta-Analysis 5.  Select studies based on  abstract review This is the initial decision as to what  studies will be retrieved as full-text documents.
02/25/11 10 Steps in a Meta-Analysis 6. Select studies based on  full-text review This is the second decision as to what studies  will be included in the review.
02/25/11 10 Steps in a Meta-Analysis 7. Extract effect sizes Effect sizes extraction involves converting descriptive or other statistical information contained in studies into a standard metric by which studies can be compared.
What is an Effect size? ,[object Object],[object Object],02/25/11
Characteristics of  Effect Sizes ,[object Object],[object Object],[object Object],[object Object],02/25/11
Effect Size Extraction ,[object Object],[object Object],[object Object],[object Object],02/25/11
02/25/11
02/25/11
Example of  ES  Extraction with Descriptive Statistics 02/25/11 Study reports: Treatment mean = 42.8 Control Mean = 32.5 Treatment  SD  = 8.6 Control  SD  = 7.4 n  = 26 n  = 31 Procedure:   Calculate  SD pooled   Calculate  d  and  g
Extracting Effect Sizes in the  Absence of Descriptive Statistics ,[object Object],[object Object],[object Object],02/25/11
Examples of Alternative Methods of  ES  Extraction   02/25/11 •  Study Reports:  t  (63) = 2.56,  p  < .05 •  Study Reports:  F  (1, 63) = 2.56,  p  < .05 Convert  F  to  t  and apply the above equation:
Zero Effect Size 02/25/11 ES  = 0.00 Control  Condition Treatment  Condition Overlapping Distributions
Moderate Effect Size 02/25/11 Control  Condition Treatment  Condition ES  = 0.40
Large Effect Size 02/25/11 Control  Condition Treatment  Condition ES  = 0.85
Mean and Variability 02/25/11 Variability ES+ Note:  Results from Bernard, Abrami, Lou, et al. (2004)  RER
02/25/11 10 Steps in a Meta-Analysis 8.  Develop a codebook Study feature coding involves describing the relevant characteristics for each study (e.g., research methodology, publication source).The codebook details the study feature categories and their levels.
Examining Study Features ,[object Object],[object Object],[object Object],[object Object],02/25/11
Examples of Study Features ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],02/25/11
10 Steps in a Meta-Analysis 02/25/11 9. Code studies for study features Coding study features is perhaps the most time-consuming and onerous aspect of conducting a meta-analysis.  However, it is arguably the most important step because it provides the possibility for explaining variability in effect sizes.
02/25/11 10 Steps in a Meta-Analysis 10: Analysis and interpretation Analysis involves invoking a range of standard statistical tests to examine average effect sizes, variability and the relationship between study features and effect size. Interpretation is drawing conclusion from these analyses.
Questions:  Statistical Analysis ,[object Object],[object Object],[object Object],[object Object],02/25/11
Homogeneity vs. Heterogeneity of Effect Size ,[object Object],[object Object],02/25/11
02/25/11
02/25/11 Statistics in Comprehensive  Meta-Analysis™ Comprehensive Meta-Analysis 2.0 is a trademark of BioStat® Interpretation:   Moderate  ES  for all outcomes ( g+  = 0.34) in favor of the intervention condition. Homogeneity of ES is violated.  Q-value  is significant (i.e., there is too much variability for  g+  to represent a true average in the population).
Examining the Study  Feature “Type of Research Design” 02/25/11 g + = +0.34 Overall Effect Pre-Post  Designs Post-Only  Designs Quasi-Exp.  Designs
Tests of Levels of “Type of Research Design” 02/25/11 Interpretation:   Small to   Moderate  ESs  for all categories in favor of the intervention condition. Homogeneity of ES is violated.  Q-value  is significant for all categories (i.e., type of research design does not explain enough variability to reach homogeneity.
Sensitivity Analysis ,[object Object],[object Object],[object Object],02/25/11
Selected References ,[object Object],[object Object],[object Object],[object Object],02/25/11

More Related Content

What's hot

Systematic review and meta analaysis course - part 2
Systematic review and meta analaysis course - part 2Systematic review and meta analaysis course - part 2
Systematic review and meta analaysis course - part 2
Ahmed Negida
 
Descriptive Analysis in Statistics
Descriptive Analysis in StatisticsDescriptive Analysis in Statistics
Descriptive Analysis in Statistics
Azmi Mohd Tamil
 
Seminaar on meta analysis
Seminaar on meta analysisSeminaar on meta analysis
Seminaar on meta analysisPreeti Rai
 
Statistical analysis using spss
Statistical analysis using spssStatistical analysis using spss
Statistical analysis using spssjpcagphil
 
Study design in research
Study design in  research Study design in  research
Study design in research Kusum Gaur
 
Parametric and non parametric test
Parametric and non parametric testParametric and non parametric test
Parametric and non parametric test
Ajay Malpani
 
The mann whitney u test
The mann whitney u testThe mann whitney u test
The mann whitney u test
Regent University
 
Basics of Systematic Review and Meta-analysis: Part 2
Basics of Systematic Review and Meta-analysis: Part 2Basics of Systematic Review and Meta-analysis: Part 2
Basics of Systematic Review and Meta-analysis: Part 2
Rizwan S A
 
Systematic Reviews: the process, quantitative, qualitative and mixed methods ...
Systematic Reviews: the process, quantitative, qualitative and mixed methods ...Systematic Reviews: the process, quantitative, qualitative and mixed methods ...
Systematic Reviews: the process, quantitative, qualitative and mixed methods ...
healthlibaust2012
 
Anova post hoc
Anova post hocAnova post hoc
Anova post hoc
Nouran Hamza, MSc, PgDPH
 
Research design new ppt
Research design new pptResearch design new ppt
Research design new ppt
Rekha Marbate
 
Meta analysis techniques in epidemiology
Meta analysis techniques in epidemiologyMeta analysis techniques in epidemiology
Meta analysis techniques in epidemiology
Bhoj Raj Singh
 
Descriptive statistics
Descriptive statisticsDescriptive statistics
Descriptive statistics
Sarfraz Ahmad
 
Introduction to Systematic Reviews
Introduction to Systematic ReviewsIntroduction to Systematic Reviews
Introduction to Systematic Reviews
Laura Koltutsky
 
Randomized Controlled Trials (RCTs)
Randomized Controlled Trials (RCTs)Randomized Controlled Trials (RCTs)
Randomized Controlled Trials (RCTs)
Amit Agrawal
 
Study designs
Study designsStudy designs
Study designs
Parth Vachhani
 
Introduction to meta analysis
Introduction to meta analysisIntroduction to meta analysis
Introduction to meta analysis
Santam Chakraborty
 
Hypothesis Formulation
Hypothesis FormulationHypothesis Formulation
Hypothesis Formulation
Sarang Bhola
 

What's hot (20)

Systematic review and meta analaysis course - part 2
Systematic review and meta analaysis course - part 2Systematic review and meta analaysis course - part 2
Systematic review and meta analaysis course - part 2
 
Descriptive Analysis in Statistics
Descriptive Analysis in StatisticsDescriptive Analysis in Statistics
Descriptive Analysis in Statistics
 
Seminaar on meta analysis
Seminaar on meta analysisSeminaar on meta analysis
Seminaar on meta analysis
 
Statistical analysis using spss
Statistical analysis using spssStatistical analysis using spss
Statistical analysis using spss
 
Study design in research
Study design in  research Study design in  research
Study design in research
 
Parametric and non parametric test
Parametric and non parametric testParametric and non parametric test
Parametric and non parametric test
 
The mann whitney u test
The mann whitney u testThe mann whitney u test
The mann whitney u test
 
Basics of Systematic Review and Meta-analysis: Part 2
Basics of Systematic Review and Meta-analysis: Part 2Basics of Systematic Review and Meta-analysis: Part 2
Basics of Systematic Review and Meta-analysis: Part 2
 
Systematic Reviews: the process, quantitative, qualitative and mixed methods ...
Systematic Reviews: the process, quantitative, qualitative and mixed methods ...Systematic Reviews: the process, quantitative, qualitative and mixed methods ...
Systematic Reviews: the process, quantitative, qualitative and mixed methods ...
 
Systematic review
Systematic reviewSystematic review
Systematic review
 
Anova post hoc
Anova post hocAnova post hoc
Anova post hoc
 
Research design new ppt
Research design new pptResearch design new ppt
Research design new ppt
 
Meta analysis techniques in epidemiology
Meta analysis techniques in epidemiologyMeta analysis techniques in epidemiology
Meta analysis techniques in epidemiology
 
Descriptive statistics
Descriptive statisticsDescriptive statistics
Descriptive statistics
 
Systematic review
Systematic reviewSystematic review
Systematic review
 
Introduction to Systematic Reviews
Introduction to Systematic ReviewsIntroduction to Systematic Reviews
Introduction to Systematic Reviews
 
Randomized Controlled Trials (RCTs)
Randomized Controlled Trials (RCTs)Randomized Controlled Trials (RCTs)
Randomized Controlled Trials (RCTs)
 
Study designs
Study designsStudy designs
Study designs
 
Introduction to meta analysis
Introduction to meta analysisIntroduction to meta analysis
Introduction to meta analysis
 
Hypothesis Formulation
Hypothesis FormulationHypothesis Formulation
Hypothesis Formulation
 

Similar to meta analysis

Research methods l1 3
Research methods l1 3Research methods l1 3
Research methods l1 3
Kegesa Danvas A.
 
Research Writing Methodology
Research Writing MethodologyResearch Writing Methodology
Research Writing Methodology
Aiden Yeh
 
research process in nursing nursing process.ppsx
research process in nursing  nursing process.ppsxresearch process in nursing  nursing process.ppsx
research process in nursing nursing process.ppsx
lovedhaliwal1
 
Part 1 research and evaluation edited
Part 1 research and evaluation editedPart 1 research and evaluation edited
Part 1 research and evaluation edited
YISMAW MENGGISTU
 
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
gerardkortney
 
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
odiliagilby
 
Research seminar lecture_10_analysing_qualitative_data
Research seminar lecture_10_analysing_qualitative_dataResearch seminar lecture_10_analysing_qualitative_data
Research seminar lecture_10_analysing_qualitative_data
Daria Bogdanova
 
Systematic literature review technique.pptx
Systematic literature review technique.pptxSystematic literature review technique.pptx
Systematic literature review technique.pptx
TANMAY DAS GUPTA
 
Information Skills For Researchers V3
Information Skills For Researchers V3Information Skills For Researchers V3
Information Skills For Researchers V3Jacqueline Thomas
 
Lit Reviews for the Health Sciences
Lit Reviews for the Health SciencesLit Reviews for the Health Sciences
Lit Reviews for the Health Sciences
Robin Featherstone
 
1Week 5Critiquing Research Articles to Prepare an Annotated B.docx
1Week 5Critiquing Research Articles to Prepare an Annotated B.docx1Week 5Critiquing Research Articles to Prepare an Annotated B.docx
1Week 5Critiquing Research Articles to Prepare an Annotated B.docx
felicidaddinwoodie
 
business-research.ppt
business-research.pptbusiness-research.ppt
business-research.ppt
KaneezElahi
 
Statement of problem_Z.pdf
Statement of problem_Z.pdfStatement of problem_Z.pdf
Statement of problem_Z.pdf
jiregnaetichadako
 
Systematic Review
Systematic ReviewSystematic Review
Systematic Review
pvhead123
 
Module 3.2Review the Prospectus Template, Dissertation Proposa.docx
Module 3.2Review the Prospectus Template, Dissertation Proposa.docxModule 3.2Review the Prospectus Template, Dissertation Proposa.docx
Module 3.2Review the Prospectus Template, Dissertation Proposa.docx
raju957290
 
writing a research proposal.pdf
writing a research proposal.pdfwriting a research proposal.pdf
writing a research proposal.pdf
mshalimatulsaadiah
 
John Lavis | Making research work for decision makers: international perspect...
John Lavis | Making research work for decision makers: international perspect...John Lavis | Making research work for decision makers: international perspect...
John Lavis | Making research work for decision makers: international perspect...
Sax Institute
 

Similar to meta analysis (20)

Research methods l1 3
Research methods l1 3Research methods l1 3
Research methods l1 3
 
Research Writing Methodology
Research Writing MethodologyResearch Writing Methodology
Research Writing Methodology
 
research process in nursing nursing process.ppsx
research process in nursing  nursing process.ppsxresearch process in nursing  nursing process.ppsx
research process in nursing nursing process.ppsx
 
Part 1 research and evaluation edited
Part 1 research and evaluation editedPart 1 research and evaluation edited
Part 1 research and evaluation edited
 
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
 
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
· Assignment List· Week 7 - Philosophical EssayWeek 7 - Philos.docx
 
Research seminar lecture_10_analysing_qualitative_data
Research seminar lecture_10_analysing_qualitative_dataResearch seminar lecture_10_analysing_qualitative_data
Research seminar lecture_10_analysing_qualitative_data
 
Systematic literature review technique.pptx
Systematic literature review technique.pptxSystematic literature review technique.pptx
Systematic literature review technique.pptx
 
Information Skills For Researchers V3
Information Skills For Researchers V3Information Skills For Researchers V3
Information Skills For Researchers V3
 
Lit Reviews for the Health Sciences
Lit Reviews for the Health SciencesLit Reviews for the Health Sciences
Lit Reviews for the Health Sciences
 
Research question
Research questionResearch question
Research question
 
1Week 5Critiquing Research Articles to Prepare an Annotated B.docx
1Week 5Critiquing Research Articles to Prepare an Annotated B.docx1Week 5Critiquing Research Articles to Prepare an Annotated B.docx
1Week 5Critiquing Research Articles to Prepare an Annotated B.docx
 
business-research.ppt
business-research.pptbusiness-research.ppt
business-research.ppt
 
Statement of problem_Z.pdf
Statement of problem_Z.pdfStatement of problem_Z.pdf
Statement of problem_Z.pdf
 
Systematic Review
Systematic ReviewSystematic Review
Systematic Review
 
2 purpose of business research, inductive & deductive approaches
2 purpose of business research, inductive & deductive approaches2 purpose of business research, inductive & deductive approaches
2 purpose of business research, inductive & deductive approaches
 
Module 3.2Review the Prospectus Template, Dissertation Proposa.docx
Module 3.2Review the Prospectus Template, Dissertation Proposa.docxModule 3.2Review the Prospectus Template, Dissertation Proposa.docx
Module 3.2Review the Prospectus Template, Dissertation Proposa.docx
 
writing a research proposal.pdf
writing a research proposal.pdfwriting a research proposal.pdf
writing a research proposal.pdf
 
SOC2002 Lecture 2
SOC2002 Lecture 2SOC2002 Lecture 2
SOC2002 Lecture 2
 
John Lavis | Making research work for decision makers: international perspect...
John Lavis | Making research work for decision makers: international perspect...John Lavis | Making research work for decision makers: international perspect...
John Lavis | Making research work for decision makers: international perspect...
 

Recently uploaded

2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...
Sandy Millin
 
The geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideasThe geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideas
GeoBlogs
 
Unit 8 - Information and Communication Technology (Paper I).pdf
Unit 8 - Information and Communication Technology (Paper I).pdfUnit 8 - Information and Communication Technology (Paper I).pdf
Unit 8 - Information and Communication Technology (Paper I).pdf
Thiyagu K
 
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
MysoreMuleSoftMeetup
 
Biological Screening of Herbal Drugs in detailed.
Biological Screening of Herbal Drugs in detailed.Biological Screening of Herbal Drugs in detailed.
Biological Screening of Herbal Drugs in detailed.
Ashokrao Mane college of Pharmacy Peth-Vadgaon
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
siemaillard
 
Guidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th SemesterGuidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th Semester
Atul Kumar Singh
 
Instructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptxInstructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptx
Jheel Barad
 
special B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdfspecial B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdf
Special education needs
 
Additional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdfAdditional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdf
joachimlavalley1
 
Language Across the Curriculm LAC B.Ed.
Language Across the  Curriculm LAC B.Ed.Language Across the  Curriculm LAC B.Ed.
Language Across the Curriculm LAC B.Ed.
Atul Kumar Singh
 
The French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free downloadThe French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free download
Vivekanand Anglo Vedic Academy
 
Model Attribute Check Company Auto Property
Model Attribute  Check Company Auto PropertyModel Attribute  Check Company Auto Property
Model Attribute Check Company Auto Property
Celine George
 
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXXPhrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
MIRIAMSALINAS13
 
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
EugeneSaldivar
 
Acetabularia Information For Class 9 .docx
Acetabularia Information For Class 9  .docxAcetabularia Information For Class 9  .docx
Acetabularia Information For Class 9 .docx
vaibhavrinwa19
 
Thesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.pptThesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.ppt
EverAndrsGuerraGuerr
 
The Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptxThe Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptx
DhatriParmar
 
Overview on Edible Vaccine: Pros & Cons with Mechanism
Overview on Edible Vaccine: Pros & Cons with MechanismOverview on Edible Vaccine: Pros & Cons with Mechanism
Overview on Edible Vaccine: Pros & Cons with Mechanism
DeeptiGupta154
 
How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...
Jisc
 

Recently uploaded (20)

2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...2024.06.01 Introducing a competency framework for languag learning materials ...
2024.06.01 Introducing a competency framework for languag learning materials ...
 
The geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideasThe geography of Taylor Swift - some ideas
The geography of Taylor Swift - some ideas
 
Unit 8 - Information and Communication Technology (Paper I).pdf
Unit 8 - Information and Communication Technology (Paper I).pdfUnit 8 - Information and Communication Technology (Paper I).pdf
Unit 8 - Information and Communication Technology (Paper I).pdf
 
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
Mule 4.6 & Java 17 Upgrade | MuleSoft Mysore Meetup #46
 
Biological Screening of Herbal Drugs in detailed.
Biological Screening of Herbal Drugs in detailed.Biological Screening of Herbal Drugs in detailed.
Biological Screening of Herbal Drugs in detailed.
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 
Guidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th SemesterGuidance_and_Counselling.pdf B.Ed. 4th Semester
Guidance_and_Counselling.pdf B.Ed. 4th Semester
 
Instructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptxInstructions for Submissions thorugh G- Classroom.pptx
Instructions for Submissions thorugh G- Classroom.pptx
 
special B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdfspecial B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdf
 
Additional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdfAdditional Benefits for Employee Website.pdf
Additional Benefits for Employee Website.pdf
 
Language Across the Curriculm LAC B.Ed.
Language Across the  Curriculm LAC B.Ed.Language Across the  Curriculm LAC B.Ed.
Language Across the Curriculm LAC B.Ed.
 
The French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free downloadThe French Revolution Class 9 Study Material pdf free download
The French Revolution Class 9 Study Material pdf free download
 
Model Attribute Check Company Auto Property
Model Attribute  Check Company Auto PropertyModel Attribute  Check Company Auto Property
Model Attribute Check Company Auto Property
 
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXXPhrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
Phrasal Verbs.XXXXXXXXXXXXXXXXXXXXXXXXXX
 
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
 
Acetabularia Information For Class 9 .docx
Acetabularia Information For Class 9  .docxAcetabularia Information For Class 9  .docx
Acetabularia Information For Class 9 .docx
 
Thesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.pptThesis Statement for students diagnonsed withADHD.ppt
Thesis Statement for students diagnonsed withADHD.ppt
 
The Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptxThe Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptx
 
Overview on Edible Vaccine: Pros & Cons with Mechanism
Overview on Edible Vaccine: Pros & Cons with MechanismOverview on Edible Vaccine: Pros & Cons with Mechanism
Overview on Edible Vaccine: Pros & Cons with Mechanism
 
How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...
 

meta analysis

  • 1. A Workshop on the Basics of Systematic Review & Meta-Analysis Philip C. Abrami, Robert M. Bernard C. Anne Wade, Evgueni Borokhovski, Rana Tamim, Gretchen Lowerison & Mike Surkes Centre for the Study of Learning and Performance and CanKnow Concordia University
  • 2.
  • 3.
  • 5. Purpose: Explaining Variability in Effect Size 02/25/11 Effect Sizes Study Features Shared Variability Unique Variability Unique Variability Prediction
  • 6.
  • 8. 02/25/11 1. Determine the research question The “big question” that guides the research. It usually involves asking about the difference between two conditions (i.e., usually treatment and control) or the relationship between two measures. 10 Steps in a Meta-Analysis
  • 9.
  • 10. Example: Critical Thinking 02/25/11 Research Question: What instructional interventions, to what extent, and under what particular circumstances, impact on the development and effective use of learner’s critical thinking skills and dispositions?
  • 11. 02/25/11 2. Develop terms and definitions related to the question This helps refine the research question and inform the search strategies. 10 Steps in a Meta-Analysis
  • 12. 02/25/11 3. Develop a search strategy for the identification of relevant studies This involves the planning/implementation of search and retrieval for primary studies (e.g., electronic databases, branching). 10 Steps in a Meta-Analysis
  • 13.
  • 14. 02/25/11 Preliminary Searches Reference Sources: Purpose : To obtain definitions for the terms; creativity, critical thinking, decision making, divergent thinking, intelligence; problem solving, reasoning, thinking. Sources: Bailin, S. (1998). Critical Thinking: Philosophical Issues . [CD-ROM] Education: The Complete Encyclopedia. Elsevier Science, Ltd. Barrow, R., & Milburn, G. (1990). A critical dictionary of educational concepts: An appraisal of selected ideas and issues in educational theory and practice (2 nd ed.) . Hertfordshire, UK: Harvester Wheatsheaf Colman (2001). Dictionary of Psychology (complete reference to be obtained) Corsini, R. J. (1999). The dictionary of psychology . Philadelphia, PA: Brunner/Mazel Dejnoka, E. L., & Kapel, D. E. (1991). American educator’s encyclopedia . Westport, CT: Greenwood Press. …… (see handout)
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21. Documenting Your Searches 02/25/11 Example from our Codebook: ERIC (Date: September 21, 2003; AW) Purpose : To retrieve the first set of abstracts to be reviewed by team according to the current inclusive/exclusion criteria. Result : Hit rate of 514/1520 Source code: ERIC1 Searches and records below from: The ERIC Database (1966-2003, June) #5 #3 and #4 (1520 records) #4 DTC = 142 or DTC = 143 or control group* (322893 records) #3 #1 or #2 (7718 records) #2 critical thinking in DE,ID (7562 records) #1 thinking skills in DE and critical thinking (1269 records)
  • 22.
  • 23.
  • 24.
  • 25. 02/25/11 10 Steps in a Meta-Analysis 4. Establish criteria for inclusion and exclusion of studies These are the criteria that guide the search for literature and ultimately determine what studies are in and out of the review.
  • 26.
  • 27. 02/25/11 10 Steps in a Meta-Analysis 5. Select studies based on abstract review This is the initial decision as to what studies will be retrieved as full-text documents.
  • 28. 02/25/11 10 Steps in a Meta-Analysis 6. Select studies based on full-text review This is the second decision as to what studies will be included in the review.
  • 29. 02/25/11 10 Steps in a Meta-Analysis 7. Extract effect sizes Effect sizes extraction involves converting descriptive or other statistical information contained in studies into a standard metric by which studies can be compared.
  • 30.
  • 31.
  • 32.
  • 35. Example of ES Extraction with Descriptive Statistics 02/25/11 Study reports: Treatment mean = 42.8 Control Mean = 32.5 Treatment SD = 8.6 Control SD = 7.4 n = 26 n = 31 Procedure: Calculate SD pooled Calculate d and g
  • 36.
  • 37. Examples of Alternative Methods of ES Extraction 02/25/11 • Study Reports: t (63) = 2.56, p < .05 • Study Reports: F (1, 63) = 2.56, p < .05 Convert F to t and apply the above equation:
  • 38. Zero Effect Size 02/25/11 ES = 0.00 Control Condition Treatment Condition Overlapping Distributions
  • 39. Moderate Effect Size 02/25/11 Control Condition Treatment Condition ES = 0.40
  • 40. Large Effect Size 02/25/11 Control Condition Treatment Condition ES = 0.85
  • 41. Mean and Variability 02/25/11 Variability ES+ Note: Results from Bernard, Abrami, Lou, et al. (2004) RER
  • 42. 02/25/11 10 Steps in a Meta-Analysis 8. Develop a codebook Study feature coding involves describing the relevant characteristics for each study (e.g., research methodology, publication source).The codebook details the study feature categories and their levels.
  • 43.
  • 44.
  • 45. 10 Steps in a Meta-Analysis 02/25/11 9. Code studies for study features Coding study features is perhaps the most time-consuming and onerous aspect of conducting a meta-analysis. However, it is arguably the most important step because it provides the possibility for explaining variability in effect sizes.
  • 46. 02/25/11 10 Steps in a Meta-Analysis 10: Analysis and interpretation Analysis involves invoking a range of standard statistical tests to examine average effect sizes, variability and the relationship between study features and effect size. Interpretation is drawing conclusion from these analyses.
  • 47.
  • 48.
  • 50. 02/25/11 Statistics in Comprehensive Meta-Analysis™ Comprehensive Meta-Analysis 2.0 is a trademark of BioStat® Interpretation: Moderate ES for all outcomes ( g+ = 0.34) in favor of the intervention condition. Homogeneity of ES is violated. Q-value is significant (i.e., there is too much variability for g+ to represent a true average in the population).
  • 51. Examining the Study Feature “Type of Research Design” 02/25/11 g + = +0.34 Overall Effect Pre-Post Designs Post-Only Designs Quasi-Exp. Designs
  • 52. Tests of Levels of “Type of Research Design” 02/25/11 Interpretation: Small to Moderate ESs for all categories in favor of the intervention condition. Homogeneity of ES is violated. Q-value is significant for all categories (i.e., type of research design does not explain enough variability to reach homogeneity.
  • 53.
  • 54.