SlideShare a Scribd company logo
1 of 42
Download to read offline
A 8–Year Review and Lessons Learned from Federal Education Evaluations: 2002-2009 Prepared by: Margaret Cahalan Sharon Stout Prepared for: AEA Meeting  November 13 2009 Draft not for Quotation or Citation:  All  views and data presented are solely those of the authors and do not reflect any review or authorization from the US Department of Education
A Review of IES/NCEE & OPEPD/PPSS Research Reports ,[object Object],[object Object],[object Object]
Part I:  Overview of Studies of Two ED-Offices ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Published Reports from IESCEE and OPEPD/PPSS: 2002-2009  117 ---Evaluation Reports IES/NCEE:  Primarily experimental design 27 reports (2003 to 2009) OPEPD/PPSS:  Primarily Implementation Studies 90 reports (2002-2009) 15 Studies (study with multiple reports counted once) 50 Studies (study with multiple reports counted once)
IES/NCEE Studies: 2003-2009 27 reports– of 15 studies (Largely Practice Based) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
PPSS—50 studies (Largely Program Based) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Distribution of Studies By School Level  ,[object Object],[object Object]
Topic Summary: IES/NCEE
Topics for OPEPD/PPSS Elementary-Secondary Studies
Topics/Programs of Studies with Reports Published  in Secondary-Postsecondary by PPSS: 2002-2009
Major Method IES/NCEE: 15 Studies
OPEPD/PPSS Major Methods: 50 Studies
Summary of Effects IES/NCEE: 15 Studies
Effects Summary 23 PPSS Studies that report comparative effects or outcomes
15 IES/NCEE Studies Topics Effects Summary Reading 6 studies (one 9 th  grade positive; 3 small positive on decoding not comprehension;  2 not detected; findings that practices were different but not outcomes; the one study with positive involved more time on reading—9 th  grade study) DC Voucher 1 study --First two years no effects; third year began to see some effects for reading not for math; 26 percent did not take opportunity Mentoring 1 study-- not overall positive effects--some positive for some subgroups—60 percent of controls also had mentoring;  Reading and Math Software 1 study ( not detected positive overall ) Teacher—Reading PD, Induction, Alternative Certification 3 studies (not detected overall on student outcomes; some on teacher practice for reading) 4 Math Curricula 1-study 2 curriculum’s had positive effects—relative to each other Afterschool 2 studies—1--not effects on academics or behavior;  second study—positive on math not reading–48 hours more instruction
Observations— 15 IES/NCEE Studies ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
K-12 PPSS Studies Topics  Effects Summary Title 1-general & Funding Proficiency increased; however most states will not meet goals by 2013-14; Over 1100 schools in need of improvement by 2006 and  most were in advanced stages of corrective action; did not show changes in time on subjects; poorest schools get less per capita; federal more targeted than state and local Teacher Over 90 percent of teachers met highly qualified; however those in  high poverty have less experience and  less likely to have degree in subject teaching and less professional development than  less poverty schools;  Supplemental ED Services (SES) Increase in number of schools eligible; however participation rate is stable--about 17 percent taking advantage ; Those who participated were more likely to show growth than eligible non-participants in sub-study of districts—selection effects issues;  Waiver studies—mixed results—higher proportion eligible participating when district  service—in two large districts do better in two small did not do better Choice/Voluntary  School Choice About 1 percent of students participating; Mixed results—some show no difference in outcomes when compared with those not participating; one that  students did better.  Most districts kept choice within district and most did not go to high performing schools when transferred out of low performing schools Reading First More time on reading and adopted practices; limited evidence 3rd and 4th  graders  increased reading more than those not in programs; regression discontinuity
K-12 PPSS Studies  (continued) Topics  Effects Summary Charter Facilities study found loans were  given to schools that served more needy population  than other charter schools in areas; most would not have qualified for private loans-too high risk Descriptive study on characteristics focus on accountability Comprehensive School Reform No effects for those who got grants; evidence that those who implemented  practices and longer term  had increase in growth especially  in math Small Learning Communities and class size  Primarily implementation study; performance reports before and after for schools found increase in and decrease in violence incidence; 9 th  grade indicators and drop out rate;  plains for college; no change in test Even Start Random assignment—no difference detected on major child or parent outcomes; counter factual issues Technology Meta-analysis found significant effects for on-line courses; small number of studies; extensive reports on status of technology use access in schools and class rooms.  Documents growth of use computer and  internet and also less use in poorer schools. Use generally not related to test score differences Teaching History & Libraries History professional development---most who took were experienced teachers; increased knowledge.  Library grants increased materials and associated with higher  than expected growth in test scores
Observations— 50 PPSS Studies ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Part 2:  Example of Issues Faced in Using Evaluations for PART and Budget PPSS Postsecondary College Access and Retention Studies Case History
 
Rough Estimates of Percent of Eligible Population Served
PPSS  College Access and Postsecondary Studies Topics  Effects Summary and Policy History Student Support Services (SSS) Propensity matching found positive impact for participation in SSS supported services whether funded by the SSS project or some other sources on college retention and completion Moderately effective- 2003 Talent Search Propensity matching found positive impacts in 3 states using longitudinal data bases on high school completion, financial aid and postsecondary entrance Moderately effective-2005 GEAR Up Matched comparison of 7 th  graders in original cohort when re-surveyed in 8 th  grade found increases in parent knowledge and aspirations and student knowledge.  Did not find impact on academic grades or course taking.  Found impact on course taking for some sub-groups. Adequate- 2003 Upward Bound Math/ Science Propensity matching found positive impacts on college entrance, 4-year degree completion, planning to major in math or science and majoring in social science; majoring in math or natural science not statistically significant although positive direction—small n’s
Upward Bound Ineffective—PART-2002 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
National Evaluation of Upward Bound: 1992-2003-04 ,[object Object],[object Object],[object Object],[object Object]
UB Study Strong Interaction with Policy History ,[object Object],[object Object],[object Object]
Policy History (cont) ,[object Object],[object Object],[object Object]
HEOA:2008  & TRIO FY10   Budget ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
ED PPSS Staff QA Re-examination of Sample Design and Re-Analyses ,[object Object],[object Object],[object Object]
Basic Assumptions of Random Assignment Studies – ,[object Object],[object Object],[object Object],[object Object]
Extreme unequal weighting and serious representation issues ,[object Object],[object Object],[object Object]
Bias in Favor of Control Group: Imbalance in Project 69
Almost Balance in 66 other Projects
Uncontrolled for Bias in Favor of Control Group in Overall Sample with project 69 included
Impact Estimates from Two Stage Instrumental Variables Regression for Percent Obtaining a BA in +6 years  based on UB Random Assignment Evaluation Note:  All estimates significant at the .01 level or higher. Estimates based on 66 of 67 projects in sample representing 74 percent of UB at the time of the study. One project removed due to introducing bias into estimates and representational issues.
Estimated tax impact of UB compared to estimated cost based on average length of participation Increased taxes collected are 4.9 times higher than per participant cost of program
Part 3: Lessons Learned ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Current Models in Evaluation Profession ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GEAR UP Next Generation Studies ,[object Object],[object Object],[object Object],[object Object],[object Object]
Process of Evaluations for Improvement
UB Descriptive Follow up Findings ,[object Object],[object Object],[object Object],[object Object],[object Object]
UB Degree or Credential Attainment Impact ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

More Related Content

What's hot

Read to Achieve Report
Read to Achieve ReportRead to Achieve Report
Read to Achieve ReportEducationNC
 
Preparing Teachers: Building Evidence for Sound Policy
Preparing Teachers: Building Evidence for Sound PolicyPreparing Teachers: Building Evidence for Sound Policy
Preparing Teachers: Building Evidence for Sound PolicyPioneer One
 
Comparative and Non-Comparative
Comparative and Non-ComparativeComparative and Non-Comparative
Comparative and Non-Comparativeu068692
 
Oral Defense presentation
Oral Defense presentationOral Defense presentation
Oral Defense presentationDwayne Squires
 
Building an Evaluation Model for Online HBSE Social Work Courses
Building an Evaluation Model for Online HBSE Social Work CoursesBuilding an Evaluation Model for Online HBSE Social Work Courses
Building an Evaluation Model for Online HBSE Social Work CoursesJimmy Young
 
Comparative and non-comparative evaluation in educational technology
Comparative and non-comparative evaluation in educational technologyComparative and non-comparative evaluation in educational technology
Comparative and non-comparative evaluation in educational technologysara Al-thihli
 
Study on students dropouts in advanced certificate in pre school education pr...
Study on students dropouts in advanced certificate in pre school education pr...Study on students dropouts in advanced certificate in pre school education pr...
Study on students dropouts in advanced certificate in pre school education pr...The Open University of Sri Lanka
 
Comparartive and non-Comparative study
Comparartive and non-Comparative studyComparartive and non-Comparative study
Comparartive and non-Comparative studyKhadeeja Al-Shidhani
 
School to Prison Pipeline-ACLU
School to Prison Pipeline-ACLUSchool to Prison Pipeline-ACLU
School to Prison Pipeline-ACLUNikki Parekh
 
2013, year 4 leaders eval report
2013, year 4 leaders eval report2013, year 4 leaders eval report
2013, year 4 leaders eval reportLouise Smyth
 
ECSU NERCHE study Sept 21 2011
ECSU NERCHE study Sept 21 2011ECSU NERCHE study Sept 21 2011
ECSU NERCHE study Sept 21 2011ecsuaaup
 
Students achievement differences
Students achievement differencesStudents achievement differences
Students achievement differencesNabin Bhattarai
 
Standardized Information Literacy Assessment
Standardized Information Literacy AssessmentStandardized Information Literacy Assessment
Standardized Information Literacy AssessmentNicoleBranch
 
Capston Presentation - Rainey Institute
Capston Presentation - Rainey InstituteCapston Presentation - Rainey Institute
Capston Presentation - Rainey InstituteAllison Silverman, MPH
 
Using Market Research to Improve Student Experiences
Using Market Research to Improve Student ExperiencesUsing Market Research to Improve Student Experiences
Using Market Research to Improve Student ExperiencesCass Erbs
 
Using Common Assessment Data to Predict High Stakes Performance- An Efficien...
Using Common Assessment Data to Predict High Stakes Performance-  An Efficien...Using Common Assessment Data to Predict High Stakes Performance-  An Efficien...
Using Common Assessment Data to Predict High Stakes Performance- An Efficien...Bethany Silver
 

What's hot (20)

Read to Achieve Report
Read to Achieve ReportRead to Achieve Report
Read to Achieve Report
 
Preparing Teachers: Building Evidence for Sound Policy
Preparing Teachers: Building Evidence for Sound PolicyPreparing Teachers: Building Evidence for Sound Policy
Preparing Teachers: Building Evidence for Sound Policy
 
Comparative and Non-Comparative
Comparative and Non-ComparativeComparative and Non-Comparative
Comparative and Non-Comparative
 
Oral Defense presentation
Oral Defense presentationOral Defense presentation
Oral Defense presentation
 
Building an Evaluation Model for Online HBSE Social Work Courses
Building an Evaluation Model for Online HBSE Social Work CoursesBuilding an Evaluation Model for Online HBSE Social Work Courses
Building an Evaluation Model for Online HBSE Social Work Courses
 
Comparative and non-comparative evaluation in educational technology
Comparative and non-comparative evaluation in educational technologyComparative and non-comparative evaluation in educational technology
Comparative and non-comparative evaluation in educational technology
 
Study on students dropouts in advanced certificate in pre school education pr...
Study on students dropouts in advanced certificate in pre school education pr...Study on students dropouts in advanced certificate in pre school education pr...
Study on students dropouts in advanced certificate in pre school education pr...
 
Phase I: Overview
Phase I: OverviewPhase I: Overview
Phase I: Overview
 
Comparartive and non-Comparative study
Comparartive and non-Comparative studyComparartive and non-Comparative study
Comparartive and non-Comparative study
 
School to Prison Pipeline-ACLU
School to Prison Pipeline-ACLUSchool to Prison Pipeline-ACLU
School to Prison Pipeline-ACLU
 
2013, year 4 leaders eval report
2013, year 4 leaders eval report2013, year 4 leaders eval report
2013, year 4 leaders eval report
 
ECSU NERCHE study Sept 21 2011
ECSU NERCHE study Sept 21 2011ECSU NERCHE study Sept 21 2011
ECSU NERCHE study Sept 21 2011
 
Students achievement differences
Students achievement differencesStudents achievement differences
Students achievement differences
 
Standardized Information Literacy Assessment
Standardized Information Literacy AssessmentStandardized Information Literacy Assessment
Standardized Information Literacy Assessment
 
Capston Presentation - Rainey Institute
Capston Presentation - Rainey InstituteCapston Presentation - Rainey Institute
Capston Presentation - Rainey Institute
 
Presentation module 6
Presentation module 6Presentation module 6
Presentation module 6
 
Using Market Research to Improve Student Experiences
Using Market Research to Improve Student ExperiencesUsing Market Research to Improve Student Experiences
Using Market Research to Improve Student Experiences
 
Jennings Final Presenation
Jennings Final PresenationJennings Final Presenation
Jennings Final Presenation
 
Using Common Assessment Data to Predict High Stakes Performance- An Efficien...
Using Common Assessment Data to Predict High Stakes Performance-  An Efficien...Using Common Assessment Data to Predict High Stakes Performance-  An Efficien...
Using Common Assessment Data to Predict High Stakes Performance- An Efficien...
 
Action research workshop
Action research workshopAction research workshop
Action research workshop
 

Viewers also liked

Springhill Lake Garden, Mar 25, 2012
Springhill Lake Garden, Mar 25, 2012Springhill Lake Garden, Mar 25, 2012
Springhill Lake Garden, Mar 25, 2012CHEARS
 
One Circle Diet for Three Sisters Demonstration Gardens
One Circle Diet for Three Sisters Demonstration GardensOne Circle Diet for Three Sisters Demonstration Gardens
One Circle Diet for Three Sisters Demonstration GardensCHEARS
 
Stormwater and rain barrels
Stormwater and rain barrelsStormwater and rain barrels
Stormwater and rain barrelsCHEARS
 
Gcan 2-9-11
Gcan 2-9-11Gcan 2-9-11
Gcan 2-9-11CHEARS
 
The Art of Teaching: Chestory Virtual Archive
The Art of Teaching: Chestory Virtual ArchiveThe Art of Teaching: Chestory Virtual Archive
The Art of Teaching: Chestory Virtual ArchiveCHEARS
 
Forest Fragmentation: The States of Our Forests & Prospects for Conservation
Forest Fragmentation: The States of Our Forests & Prospects for ConservationForest Fragmentation: The States of Our Forests & Prospects for Conservation
Forest Fragmentation: The States of Our Forests & Prospects for ConservationCHEARS
 
Rain Barrels: Harvesting the Heavens Saving Streams
Rain Barrels:  Harvesting the Heavens Saving StreamsRain Barrels:  Harvesting the Heavens Saving Streams
Rain Barrels: Harvesting the Heavens Saving StreamsCHEARS
 
MLK Day of Service Tree Protection Workday at Buddy Attick Park
MLK Day of Service Tree Protection Workday at Buddy Attick ParkMLK Day of Service Tree Protection Workday at Buddy Attick Park
MLK Day of Service Tree Protection Workday at Buddy Attick ParkCHEARS
 

Viewers also liked (8)

Springhill Lake Garden, Mar 25, 2012
Springhill Lake Garden, Mar 25, 2012Springhill Lake Garden, Mar 25, 2012
Springhill Lake Garden, Mar 25, 2012
 
One Circle Diet for Three Sisters Demonstration Gardens
One Circle Diet for Three Sisters Demonstration GardensOne Circle Diet for Three Sisters Demonstration Gardens
One Circle Diet for Three Sisters Demonstration Gardens
 
Stormwater and rain barrels
Stormwater and rain barrelsStormwater and rain barrels
Stormwater and rain barrels
 
Gcan 2-9-11
Gcan 2-9-11Gcan 2-9-11
Gcan 2-9-11
 
The Art of Teaching: Chestory Virtual Archive
The Art of Teaching: Chestory Virtual ArchiveThe Art of Teaching: Chestory Virtual Archive
The Art of Teaching: Chestory Virtual Archive
 
Forest Fragmentation: The States of Our Forests & Prospects for Conservation
Forest Fragmentation: The States of Our Forests & Prospects for ConservationForest Fragmentation: The States of Our Forests & Prospects for Conservation
Forest Fragmentation: The States of Our Forests & Prospects for Conservation
 
Rain Barrels: Harvesting the Heavens Saving Streams
Rain Barrels:  Harvesting the Heavens Saving StreamsRain Barrels:  Harvesting the Heavens Saving Streams
Rain Barrels: Harvesting the Heavens Saving Streams
 
MLK Day of Service Tree Protection Workday at Buddy Attick Park
MLK Day of Service Tree Protection Workday at Buddy Attick ParkMLK Day of Service Tree Protection Workday at Buddy Attick Park
MLK Day of Service Tree Protection Workday at Buddy Attick Park
 

Similar to A 8–Year Review and Lessons Learned from Federal Education Evaluations: 2002-2009 by M Cahalan and S Stout

Ceppe march 2011 final (Susanna Loeb)
Ceppe march 2011 final (Susanna Loeb)Ceppe march 2011 final (Susanna Loeb)
Ceppe march 2011 final (Susanna Loeb)Ceppe Chile
 
Assessment and Examination.ppt
Assessment and Examination.pptAssessment and Examination.ppt
Assessment and Examination.pptNasirMahmood976516
 
Hudsonville Evaulation Logic Design
Hudsonville Evaulation Logic DesignHudsonville Evaulation Logic Design
Hudsonville Evaulation Logic DesignRobert Leneway
 
Hudsonville PS Evaluation Design
Hudsonville PS Evaluation DesignHudsonville PS Evaluation Design
Hudsonville PS Evaluation DesignRobert Leneway
 
Topic 10 Issues and Concerns Related to Assessment in Malaysia
Topic 10 Issues and Concerns Related to Assessment in MalaysiaTopic 10 Issues and Concerns Related to Assessment in Malaysia
Topic 10 Issues and Concerns Related to Assessment in MalaysiaYee Bee Choo
 
The Effectiveness Of Academic Standards
The Effectiveness Of Academic StandardsThe Effectiveness Of Academic Standards
The Effectiveness Of Academic Standardsnoblex1
 
Innovations in assessment: an investigation into the role of blended learning...
Innovations in assessment: an investigation into the role of blended learning...Innovations in assessment: an investigation into the role of blended learning...
Innovations in assessment: an investigation into the role of blended learning...UK Centre for Legal Education
 
Designing an evaluation of a tertiary preparatory program sounds
Designing an evaluation of a tertiary preparatory program soundsDesigning an evaluation of a tertiary preparatory program sounds
Designing an evaluation of a tertiary preparatory program soundsphysrcd
 
Designing an evaluation of a tertiary preparatory program sounds
Designing an evaluation of a tertiary preparatory program soundsDesigning an evaluation of a tertiary preparatory program sounds
Designing an evaluation of a tertiary preparatory program soundsphysrcd
 
Steve Vitto Response to Intervention (RTI)
Steve Vitto Response to Intervention (RTI)Steve Vitto Response to Intervention (RTI)
Steve Vitto Response to Intervention (RTI)Steve Vitto
 
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009Steve Vitto
 
Non-Cognitive Testing
Non-Cognitive TestingNon-Cognitive Testing
Non-Cognitive Testingjwilliams77
 
Research presentation
Research presentationResearch presentation
Research presentationdavis1mj
 
Dr. Teresa Ann Hughes, PhD Dissertation Defense, Dr. William Allan Kritsonis,...
Dr. Teresa Ann Hughes, PhD Dissertation Defense, Dr. William Allan Kritsonis,...Dr. Teresa Ann Hughes, PhD Dissertation Defense, Dr. William Allan Kritsonis,...
Dr. Teresa Ann Hughes, PhD Dissertation Defense, Dr. William Allan Kritsonis,...William Kritsonis
 
Teresa Ann Hughes, Dissertation Defense, Dr. William Kritsonis, Dissertation ...
Teresa Ann Hughes, Dissertation Defense, Dr. William Kritsonis, Dissertation ...Teresa Ann Hughes, Dissertation Defense, Dr. William Kritsonis, Dissertation ...
Teresa Ann Hughes, Dissertation Defense, Dr. William Kritsonis, Dissertation ...William Kritsonis
 

Similar to A 8–Year Review and Lessons Learned from Federal Education Evaluations: 2002-2009 by M Cahalan and S Stout (20)

Ceppe march 2011 final (Susanna Loeb)
Ceppe march 2011 final (Susanna Loeb)Ceppe march 2011 final (Susanna Loeb)
Ceppe march 2011 final (Susanna Loeb)
 
Oerc june 2014 final ppt combined
Oerc june 2014 final ppt combinedOerc june 2014 final ppt combined
Oerc june 2014 final ppt combined
 
EdD Handoutv2
EdD Handoutv2EdD Handoutv2
EdD Handoutv2
 
Assessment and Examination.ppt
Assessment and Examination.pptAssessment and Examination.ppt
Assessment and Examination.ppt
 
Interview presentation.pptx
Interview presentation.pptxInterview presentation.pptx
Interview presentation.pptx
 
Hudsonville Evaulation Logic Design
Hudsonville Evaulation Logic DesignHudsonville Evaulation Logic Design
Hudsonville Evaulation Logic Design
 
Hudsonville PS Evaluation Design
Hudsonville PS Evaluation DesignHudsonville PS Evaluation Design
Hudsonville PS Evaluation Design
 
Topic 10 Issues and Concerns Related to Assessment in Malaysia
Topic 10 Issues and Concerns Related to Assessment in MalaysiaTopic 10 Issues and Concerns Related to Assessment in Malaysia
Topic 10 Issues and Concerns Related to Assessment in Malaysia
 
The Effectiveness Of Academic Standards
The Effectiveness Of Academic StandardsThe Effectiveness Of Academic Standards
The Effectiveness Of Academic Standards
 
Innovations in assessment: an investigation into the role of blended learning...
Innovations in assessment: an investigation into the role of blended learning...Innovations in assessment: an investigation into the role of blended learning...
Innovations in assessment: an investigation into the role of blended learning...
 
Impact of et
Impact of etImpact of et
Impact of et
 
Designing an evaluation of a tertiary preparatory program sounds
Designing an evaluation of a tertiary preparatory program soundsDesigning an evaluation of a tertiary preparatory program sounds
Designing an evaluation of a tertiary preparatory program sounds
 
Designing an evaluation of a tertiary preparatory program sounds
Designing an evaluation of a tertiary preparatory program soundsDesigning an evaluation of a tertiary preparatory program sounds
Designing an evaluation of a tertiary preparatory program sounds
 
Petri PSC Findings
Petri PSC FindingsPetri PSC Findings
Petri PSC Findings
 
Steve Vitto Response to Intervention (RTI)
Steve Vitto Response to Intervention (RTI)Steve Vitto Response to Intervention (RTI)
Steve Vitto Response to Intervention (RTI)
 
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009
 
Non-Cognitive Testing
Non-Cognitive TestingNon-Cognitive Testing
Non-Cognitive Testing
 
Research presentation
Research presentationResearch presentation
Research presentation
 
Dr. Teresa Ann Hughes, PhD Dissertation Defense, Dr. William Allan Kritsonis,...
Dr. Teresa Ann Hughes, PhD Dissertation Defense, Dr. William Allan Kritsonis,...Dr. Teresa Ann Hughes, PhD Dissertation Defense, Dr. William Allan Kritsonis,...
Dr. Teresa Ann Hughes, PhD Dissertation Defense, Dr. William Allan Kritsonis,...
 
Teresa Ann Hughes, Dissertation Defense, Dr. William Kritsonis, Dissertation ...
Teresa Ann Hughes, Dissertation Defense, Dr. William Kritsonis, Dissertation ...Teresa Ann Hughes, Dissertation Defense, Dr. William Kritsonis, Dissertation ...
Teresa Ann Hughes, Dissertation Defense, Dr. William Kritsonis, Dissertation ...
 

More from CHEARS

Errors Found in National Evaluation of UpwardBound- Postive Re-Analysis Results
Errors Found in National Evaluation of UpwardBound- Postive Re-Analysis ResultsErrors Found in National Evaluation of UpwardBound- Postive Re-Analysis Results
Errors Found in National Evaluation of UpwardBound- Postive Re-Analysis ResultsCHEARS
 
A Permaculture Design for a York PA Old Farmstead
A Permaculture Design for a York PA Old FarmsteadA Permaculture Design for a York PA Old Farmstead
A Permaculture Design for a York PA Old FarmsteadCHEARS
 
Eleanor Roosevelt High School Career Day presenation
Eleanor Roosevelt High School Career Day presenationEleanor Roosevelt High School Career Day presenation
Eleanor Roosevelt High School Career Day presenationCHEARS
 
Stormwater Solutions: Rain Barrels and More!
Stormwater Solutions: Rain Barrels and More!Stormwater Solutions: Rain Barrels and More!
Stormwater Solutions: Rain Barrels and More!CHEARS
 
Prince George's County Raincheck Rebate Program
Prince George's County Raincheck Rebate ProgramPrince George's County Raincheck Rebate Program
Prince George's County Raincheck Rebate ProgramCHEARS
 
Forest Pests
Forest PestsForest Pests
Forest PestsCHEARS
 
Emerald Ash Borer
Emerald Ash BorerEmerald Ash Borer
Emerald Ash BorerCHEARS
 
Invasive Species
Invasive SpeciesInvasive Species
Invasive SpeciesCHEARS
 
Seeing the Forest and the Trees
Seeing the Forest and the TreesSeeing the Forest and the Trees
Seeing the Forest and the TreesCHEARS
 
Tim Culbreth: Tree ID Presentation
Tim Culbreth: Tree ID PresentationTim Culbreth: Tree ID Presentation
Tim Culbreth: Tree ID PresentationCHEARS
 
Replenishing Ecosystems: Forest Gardening
Replenishing Ecosystems: Forest GardeningReplenishing Ecosystems: Forest Gardening
Replenishing Ecosystems: Forest GardeningCHEARS
 
The Value of Trees
The Value of TreesThe Value of Trees
The Value of TreesCHEARS
 
Water and Forests
Water and ForestsWater and Forests
Water and ForestsCHEARS
 
April 6th Presentation: Greenbelt Forest Stewards
April 6th Presentation: Greenbelt Forest StewardsApril 6th Presentation: Greenbelt Forest Stewards
April 6th Presentation: Greenbelt Forest StewardsCHEARS
 
Greenbelt Food Forest Phase II Photo Journal
Greenbelt Food Forest Phase II Photo JournalGreenbelt Food Forest Phase II Photo Journal
Greenbelt Food Forest Phase II Photo JournalCHEARS
 
Population and Climate Change: Are They Related?
Population and Climate Change: Are They Related?Population and Climate Change: Are They Related?
Population and Climate Change: Are They Related?CHEARS
 
Permaculture Design for Wild Meadows Farm
Permaculture Design for Wild Meadows FarmPermaculture Design for Wild Meadows Farm
Permaculture Design for Wild Meadows FarmCHEARS
 
2012 PASA Conference
2012 PASA Conference2012 PASA Conference
2012 PASA ConferenceCHEARS
 
Greenbelt Food Forest: Phase I Photo Journal
Greenbelt Food Forest: Phase I Photo JournalGreenbelt Food Forest: Phase I Photo Journal
Greenbelt Food Forest: Phase I Photo JournalCHEARS
 
Pesticide Report of Green Aces (Greenbelt Committee on Enviromental Sustaina...
Pesticide Report of  Green Aces (Greenbelt Committee on Enviromental Sustaina...Pesticide Report of  Green Aces (Greenbelt Committee on Enviromental Sustaina...
Pesticide Report of Green Aces (Greenbelt Committee on Enviromental Sustaina...CHEARS
 

More from CHEARS (20)

Errors Found in National Evaluation of UpwardBound- Postive Re-Analysis Results
Errors Found in National Evaluation of UpwardBound- Postive Re-Analysis ResultsErrors Found in National Evaluation of UpwardBound- Postive Re-Analysis Results
Errors Found in National Evaluation of UpwardBound- Postive Re-Analysis Results
 
A Permaculture Design for a York PA Old Farmstead
A Permaculture Design for a York PA Old FarmsteadA Permaculture Design for a York PA Old Farmstead
A Permaculture Design for a York PA Old Farmstead
 
Eleanor Roosevelt High School Career Day presenation
Eleanor Roosevelt High School Career Day presenationEleanor Roosevelt High School Career Day presenation
Eleanor Roosevelt High School Career Day presenation
 
Stormwater Solutions: Rain Barrels and More!
Stormwater Solutions: Rain Barrels and More!Stormwater Solutions: Rain Barrels and More!
Stormwater Solutions: Rain Barrels and More!
 
Prince George's County Raincheck Rebate Program
Prince George's County Raincheck Rebate ProgramPrince George's County Raincheck Rebate Program
Prince George's County Raincheck Rebate Program
 
Forest Pests
Forest PestsForest Pests
Forest Pests
 
Emerald Ash Borer
Emerald Ash BorerEmerald Ash Borer
Emerald Ash Borer
 
Invasive Species
Invasive SpeciesInvasive Species
Invasive Species
 
Seeing the Forest and the Trees
Seeing the Forest and the TreesSeeing the Forest and the Trees
Seeing the Forest and the Trees
 
Tim Culbreth: Tree ID Presentation
Tim Culbreth: Tree ID PresentationTim Culbreth: Tree ID Presentation
Tim Culbreth: Tree ID Presentation
 
Replenishing Ecosystems: Forest Gardening
Replenishing Ecosystems: Forest GardeningReplenishing Ecosystems: Forest Gardening
Replenishing Ecosystems: Forest Gardening
 
The Value of Trees
The Value of TreesThe Value of Trees
The Value of Trees
 
Water and Forests
Water and ForestsWater and Forests
Water and Forests
 
April 6th Presentation: Greenbelt Forest Stewards
April 6th Presentation: Greenbelt Forest StewardsApril 6th Presentation: Greenbelt Forest Stewards
April 6th Presentation: Greenbelt Forest Stewards
 
Greenbelt Food Forest Phase II Photo Journal
Greenbelt Food Forest Phase II Photo JournalGreenbelt Food Forest Phase II Photo Journal
Greenbelt Food Forest Phase II Photo Journal
 
Population and Climate Change: Are They Related?
Population and Climate Change: Are They Related?Population and Climate Change: Are They Related?
Population and Climate Change: Are They Related?
 
Permaculture Design for Wild Meadows Farm
Permaculture Design for Wild Meadows FarmPermaculture Design for Wild Meadows Farm
Permaculture Design for Wild Meadows Farm
 
2012 PASA Conference
2012 PASA Conference2012 PASA Conference
2012 PASA Conference
 
Greenbelt Food Forest: Phase I Photo Journal
Greenbelt Food Forest: Phase I Photo JournalGreenbelt Food Forest: Phase I Photo Journal
Greenbelt Food Forest: Phase I Photo Journal
 
Pesticide Report of Green Aces (Greenbelt Committee on Enviromental Sustaina...
Pesticide Report of  Green Aces (Greenbelt Committee on Enviromental Sustaina...Pesticide Report of  Green Aces (Greenbelt Committee on Enviromental Sustaina...
Pesticide Report of Green Aces (Greenbelt Committee on Enviromental Sustaina...
 

Recently uploaded

Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxMYDA ANGELICA SUAN
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.EnglishCEIPdeSigeiro
 
Philosophy of Education and Educational Philosophy
Philosophy of Education  and Educational PhilosophyPhilosophy of Education  and Educational Philosophy
Philosophy of Education and Educational PhilosophyShuvankar Madhu
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxDr. Asif Anas
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsEugene Lysak
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICESayali Powar
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17Celine George
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxKatherine Villaluna
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfMohonDas
 
UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfMohonDas
 
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfMaximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfTechSoup
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxiammrhaywood
 
M-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxM-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxDr. Santhosh Kumar. N
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxSaurabhParmar42
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptxraviapr7
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?TechSoup
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesCeline George
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptxmary850239
 

Recently uploaded (20)

Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptx
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.
 
Philosophy of Education and Educational Philosophy
Philosophy of Education  and Educational PhilosophyPhilosophy of Education  and Educational Philosophy
Philosophy of Education and Educational Philosophy
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptx
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George Wells
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICE
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptx
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdf
 
UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdf
 
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfMaximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
 
M-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxM-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptx
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptx
 
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdfPersonal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?
 
How to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 SalesHow to Manage Cross-Selling in Odoo 17 Sales
How to Manage Cross-Selling in Odoo 17 Sales
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptx
 

A 8–Year Review and Lessons Learned from Federal Education Evaluations: 2002-2009 by M Cahalan and S Stout

  • 1. A 8–Year Review and Lessons Learned from Federal Education Evaluations: 2002-2009 Prepared by: Margaret Cahalan Sharon Stout Prepared for: AEA Meeting November 13 2009 Draft not for Quotation or Citation: All views and data presented are solely those of the authors and do not reflect any review or authorization from the US Department of Education
  • 2.
  • 3.
  • 4. Published Reports from IESCEE and OPEPD/PPSS: 2002-2009 117 ---Evaluation Reports IES/NCEE: Primarily experimental design 27 reports (2003 to 2009) OPEPD/PPSS: Primarily Implementation Studies 90 reports (2002-2009) 15 Studies (study with multiple reports counted once) 50 Studies (study with multiple reports counted once)
  • 5.
  • 6.
  • 7.
  • 9. Topics for OPEPD/PPSS Elementary-Secondary Studies
  • 10. Topics/Programs of Studies with Reports Published in Secondary-Postsecondary by PPSS: 2002-2009
  • 13. Summary of Effects IES/NCEE: 15 Studies
  • 14. Effects Summary 23 PPSS Studies that report comparative effects or outcomes
  • 15. 15 IES/NCEE Studies Topics Effects Summary Reading 6 studies (one 9 th grade positive; 3 small positive on decoding not comprehension; 2 not detected; findings that practices were different but not outcomes; the one study with positive involved more time on reading—9 th grade study) DC Voucher 1 study --First two years no effects; third year began to see some effects for reading not for math; 26 percent did not take opportunity Mentoring 1 study-- not overall positive effects--some positive for some subgroups—60 percent of controls also had mentoring; Reading and Math Software 1 study ( not detected positive overall ) Teacher—Reading PD, Induction, Alternative Certification 3 studies (not detected overall on student outcomes; some on teacher practice for reading) 4 Math Curricula 1-study 2 curriculum’s had positive effects—relative to each other Afterschool 2 studies—1--not effects on academics or behavior; second study—positive on math not reading–48 hours more instruction
  • 16.
  • 17. K-12 PPSS Studies Topics Effects Summary Title 1-general & Funding Proficiency increased; however most states will not meet goals by 2013-14; Over 1100 schools in need of improvement by 2006 and most were in advanced stages of corrective action; did not show changes in time on subjects; poorest schools get less per capita; federal more targeted than state and local Teacher Over 90 percent of teachers met highly qualified; however those in high poverty have less experience and less likely to have degree in subject teaching and less professional development than less poverty schools; Supplemental ED Services (SES) Increase in number of schools eligible; however participation rate is stable--about 17 percent taking advantage ; Those who participated were more likely to show growth than eligible non-participants in sub-study of districts—selection effects issues; Waiver studies—mixed results—higher proportion eligible participating when district service—in two large districts do better in two small did not do better Choice/Voluntary School Choice About 1 percent of students participating; Mixed results—some show no difference in outcomes when compared with those not participating; one that students did better. Most districts kept choice within district and most did not go to high performing schools when transferred out of low performing schools Reading First More time on reading and adopted practices; limited evidence 3rd and 4th graders increased reading more than those not in programs; regression discontinuity
  • 18. K-12 PPSS Studies (continued) Topics Effects Summary Charter Facilities study found loans were given to schools that served more needy population than other charter schools in areas; most would not have qualified for private loans-too high risk Descriptive study on characteristics focus on accountability Comprehensive School Reform No effects for those who got grants; evidence that those who implemented practices and longer term had increase in growth especially in math Small Learning Communities and class size Primarily implementation study; performance reports before and after for schools found increase in and decrease in violence incidence; 9 th grade indicators and drop out rate; plains for college; no change in test Even Start Random assignment—no difference detected on major child or parent outcomes; counter factual issues Technology Meta-analysis found significant effects for on-line courses; small number of studies; extensive reports on status of technology use access in schools and class rooms. Documents growth of use computer and internet and also less use in poorer schools. Use generally not related to test score differences Teaching History & Libraries History professional development---most who took were experienced teachers; increased knowledge. Library grants increased materials and associated with higher than expected growth in test scores
  • 19.
  • 20. Part 2: Example of Issues Faced in Using Evaluations for PART and Budget PPSS Postsecondary College Access and Retention Studies Case History
  • 21.  
  • 22. Rough Estimates of Percent of Eligible Population Served
  • 23. PPSS College Access and Postsecondary Studies Topics Effects Summary and Policy History Student Support Services (SSS) Propensity matching found positive impact for participation in SSS supported services whether funded by the SSS project or some other sources on college retention and completion Moderately effective- 2003 Talent Search Propensity matching found positive impacts in 3 states using longitudinal data bases on high school completion, financial aid and postsecondary entrance Moderately effective-2005 GEAR Up Matched comparison of 7 th graders in original cohort when re-surveyed in 8 th grade found increases in parent knowledge and aspirations and student knowledge. Did not find impact on academic grades or course taking. Found impact on course taking for some sub-groups. Adequate- 2003 Upward Bound Math/ Science Propensity matching found positive impacts on college entrance, 4-year degree completion, planning to major in math or science and majoring in social science; majoring in math or natural science not statistically significant although positive direction—small n’s
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
  • 32. Bias in Favor of Control Group: Imbalance in Project 69
  • 33. Almost Balance in 66 other Projects
  • 34. Uncontrolled for Bias in Favor of Control Group in Overall Sample with project 69 included
  • 35. Impact Estimates from Two Stage Instrumental Variables Regression for Percent Obtaining a BA in +6 years based on UB Random Assignment Evaluation Note: All estimates significant at the .01 level or higher. Estimates based on 66 of 67 projects in sample representing 74 percent of UB at the time of the study. One project removed due to introducing bias into estimates and representational issues.
  • 36. Estimated tax impact of UB compared to estimated cost based on average length of participation Increased taxes collected are 4.9 times higher than per participant cost of program
  • 37.
  • 38.
  • 39.
  • 40. Process of Evaluations for Improvement
  • 41.
  • 42.