SlideShare a Scribd company logo
1 of 32
District Program Evaluation to
Improve RTI/MTSS
Clarisse Haxton, Ph.D.
Program Evaluation Coordinator
Palo Alto Unified School District
CERA Conference 2017
Agenda
• PAUSD District and RTI Overview
• RTI Evaluation Overview
• Findings
• RTI Monitoring Tool (data)
• Implementation progress and challenges
• Student outcomes
• Recommendations and next steps
PAUSD District and RTI Overview
PAUSD District and RTI Overview
• High-performing overall; substantial
within-district achievement gaps
• RTI/MTSS
• History of RTI work in elementary
• Recognized need for district system
• Recommendation for evaluation in MATD
Committee report and district Equity
Plan
• Program Evaluation Coordinator role
Bufum, Matos, & Weber (2012) 5
RTI Definition from NCRTI
“Response to intervention integrates assessment and
intervention within a multi-level prevention system to maximize
student achievement and to reduce behavioral problems. With
RTI, schools use data to identify students at risk for poor
learning outcomes, monitor student progress, provide
evidence-based interventions and adjust the intensity and
nature of those interventions depending on a student’s
responsiveness, and identify students with learning disabilities
or other disabilities… This system includes three levels of
intensity or three levels of prevention, which represent a
continuum of supports. Many schools use more than one
intervention within a given level of prevention.”
Palo Alto Unified School District 6National Center on Response to Intervention (2010)
RTI/MTSS “Rules of Thumb”
• Tier 1: classroom instruction (“best first instruction”)
provided to all students; serves 80 to 85% of
students
• Tier 2: interventions are supplementary instruction
and support on targeted areas of student need;
serves approximately 10-15% of students
• Tier 3: interventions are more intensive and
individualized, supplementary to core classroom
instruction; serve approximately 5 to 10% of
students
7
PAUSD Elementary RTI
Evaluation Overview
Evaluation Purpose
• Provide an inventory of Tier 2 interventions:
– # students are being served
– student characteristics
– Tier 2 interventions are being used
• Describe:
– screening and identification practices
– intervention implementation
– progress monitoring practices
– RTI student outcomes
• Make recommendations for continuous
improvement
Evaluation Questions
• Who’s being served?
• How are they identified?
• What are they getting?
• How are we monitoring progress?
• What progress are students making?
• What’s working well?
• What are challenges?
• What can we do better?
PAUSD Elementary RTI
Evaluation Findings
Lesson 1: Need data
• Problem of Practice
⁻ Lacked system to identify and track
students in Tier 2 intervention
⁻ Need to answer “who” and “what”
before you can answer “How are we
doing?”
• Solution: Created a RTI Monitoring Tool
(Google sheet) across all elementary
schools
RTI Monitoring Tool
(Google Sheets)
• Students who received RTI
• Classroom teacher and grade level
• RTI subject (math, reading/literacy)
• Intervention program name(s)
• Intervention teacher(s)
• Assessment scores that made a student eligible for
RTI
• Beginning of intervention (BOI) assessment score
• Progress monitoring score (at least 1)
• End of intervention (EOI) assessment score
RTI Monitoring Tool
(Google Sheets)
• 2016-17 RTI Monitoring Tool
Who are we serving and
what are they getting?
Grade Level
% of RTI
Students in
Math
Intervention
% of RTI
Students in
Reading
Intervention
% of RTI
Students in
Math &
Reading
Intervention
% of RTI
Students with
Unspecified
Intervention
Total # of
Students in
RTI
K 4.7% (5) 100% (106) 4.7% (5) 3.7% (4) 106
1 21.1% (31) 93.2% (137) 17% (25) 8.8% (13) 147
2 42.1% (64) 61.8% (94) 15.1% (23) 3.3% (5) 152
3 58.9% (96) 63.2% (103) 25.1% (41) 4.3% (7) 163
4 64.8% (114) 42.6% (75) 11.4% (20) 4% (7) 176
5 55% (72) 48.9% (64) 14.5% (19) 8.4% (11) 131
Total 43.6% 66.1% 15.2% 5.4% 875
n=382 n=579 n=133 n=47
Number and Percent of Students in Tier 2 Interventions
in 2016-17, by Grade Level and Subject
Source: District RTI Data. Note: 382 + 579 = 961 duplicated RTI students – 133 = 828 total RTI students in
reading or math. 828 + 47 = 875 total RTI students. “Unspecified” interventions include “progress
monitoring”, “small group,” and other titles that did not distinguish them as math or reading RTI.
Who are we serving?
Background Characteristics of RTI Students
Source: District RTI Data. Note: HUR_Race includes Black, Hispanic, Native
Hawaiian/Pacific Islander, and Alaska Native students.
Student Characteristics
% of RTI
Students
% K-5 in
PAUSD
Historically Underrepresented Race/Ethnicity (HUR_Race) 47.2% 18.7%
Socioeconomically Disadvantaged (SED) 40.6% 12.4%
Voluntary Transfer Program (VTP) 21.8% 6.5%
Historically Underrepresented (HUR_Race + SED + VTP) 51.3% 21.7%
English Learner (EL) 37.8% 18.9%
Special Education (IEP) 11.7% 8.6%
Male 53.3% 52.1%
Total 875 5224
Lesson 2: Need (somewhat)
standardized RTI eligibility criteria
• Variability in student performance among RTI
students
– Below “PAUSD average” versus at or below
benchmark/standard
– Raised important discussions about Tier 1 and Tier 2
– Has now led to more common eligibility criteria
• RTI students’ CAASPP performance
– Among fourth and fifth grade RTI students, 11 to 37 percent of
Met or Exceeded standard on their prior year (2015-16) CAASPP
assessment, in the subject they received Tier 2 intervention.
Changing systems while systems
are always changing
•RTI reading students’ BOY reading performance
– Messy because we were transitioning to new assessment
– In fall 2016, 17 percent of students in Tier 2 reading
intervention performed at or above the district’s BOY reading
benchmark level on BAS and 46 percent performed at or above
the district’s BOY benchmark on DRA.
•Math RTI students’ BOY math performance
– Exploring how to best use NWEA MAP data in Year 1
– Less than 1 percent of RTI math students in grades 3 to 5 scored
at or above the PAUSD average on the BOY NWEA MAP
assessment.
– However, over 40 percent of students in RTI math intervention
scored above the national average for their grade level.
Lesson 3: Implementation progress;
system challenges remain
• Remaining challenges
– Finding qualified staff to teach after school
– Student transportation
– Dosage and implementing interventions with
fidelity
– Setting targeted goals for students and targeting
RTI instruction in groups with varying needs
– Progress monitoring
– Improving student achievement and closing
achievement gaps
Lesson 4: Time + Targeted
Instruction = Learning
• Need to focus on targeting instruction
• Need to improve outcomes
–Among K-5 RTI students who began the
year below benchmark, 44 percent in math
and 56 percent in reading “closed the gap”
(got closer to benchmark) from BOY to EOY
–There were no consistent patterns or
significant differences in outcomes by
student characteristics, program, or staff
type.
Lesson 5: REA-Elem-Site
partnership has moved this work
• Research, Evaluation, and Assessment
(REA) Department
• Elementary Education Dept
• Elementary site principals
• … and now, some interest and
exploration in secondary schools
Palo Alto Unified School District
PAUSD Elementary RTI Evaluation
Recommendations and Next Steps
Recommendations (2016-17) and
Next Steps (2017-18)
• Time + Targeted Instruction = Learning
1. Continue to use and improve the RTI
Monitoring Tool
–Fixed format template
–Focusing more on measurable goals and
progress monitoring
–Trying to focus on fixed intervention rounds
with pauses between to examine the data
and adjust students and goals
Recommendations (2016-17) and
Next Steps (2017-18)
2. Set common expectation of prioritizing
students below benchmark or standard
(and generally, not in special education)
–Use state and local assessment data to
improve screening and identification
–Data available in DataZone, Google Sheets,
and NWEA
Recommendations (2016-17) and
Next Steps (2017-18)
3. Provide support for continuous
improvement of interventions
- TOSAs mapped standards to Bridges
Intervention volumes
- Bridges Intervention training
Recommendations (2016-17) and
Next Steps (2017-18)
4. Create a multi-tiered system of supports
(MTSS) that includes all staff PK-12
•Supporting an RTI/IST pilot at two sites,
connecting Tier 1 and 2 practices and data
through linked Google Forms and coaching
•Integrating RTI into LCAP and Equity Plan
•Supporting new high school RTI teams
•Conducting special education program
evaluation
Internal District Program Evaluation–
Breaking Down Department Silos
Internal District Program Evaluation–
Breaking Down Department Silos
Internal District Program Evaluation–
Breaking Down Department Silos
Internal District Program Evaluation
• Cost effective
⁻ $150K+ for a major research firm to do a
similar study
⁻ Previously, $70K research contractor
• Cost beneficial
⁻ We spend $130K + site expenditures on
elementary RTI. Important to know whether
the $ makes a difference to inform resource
allocation decisions.
Internal District Program Evaluation
• Research-Department partnerships
within the district can deepen and
move the work
⁻ Benefit of district context and relationships
⁻ Ability to regularly visit schools, observe, talk to
people
⁻ Strategically link programs and evaluations
together
⁻ Help the Board and administrators make
data-driven decisions
Questions?
• Thanks!
Palo Alto Unified School District

More Related Content

What's hot

Sse workshop 2 spring 2014
Sse workshop 2 spring 2014Sse workshop 2 spring 2014
Sse workshop 2 spring 2014Martin Brown
 
Sse pp 11 november 2013 (1)
Sse pp   11 november 2013 (1)Sse pp   11 november 2013 (1)
Sse pp 11 november 2013 (1)Martin Brown
 
response to intervention.ppt
response to intervention.pptresponse to intervention.ppt
response to intervention.pptJanet Van Heck
 
Response to Intervention: What is it?
Response to Intervention: What is it?Response to Intervention: What is it?
Response to Intervention: What is it?paulaulloa671
 
Handout 3 SSE case study school (self-evaluation report: literacy)
Handout 3 SSE case study school (self-evaluation report: literacy)Handout 3 SSE case study school (self-evaluation report: literacy)
Handout 3 SSE case study school (self-evaluation report: literacy)Martin Brown
 
Franklin Public Schools: MCAS Update 2014
Franklin Public Schools: MCAS Update 2014Franklin Public Schools: MCAS Update 2014
Franklin Public Schools: MCAS Update 2014Franklin Matters
 
Handout 2: SSE case study: The SSE journey for our case study school
Handout 2: SSE case study: The SSE journey for our case study schoolHandout 2: SSE case study: The SSE journey for our case study school
Handout 2: SSE case study: The SSE journey for our case study schoolMartin Brown
 
Franklin Public Schools: MCAS Presentation 2017
Franklin Public Schools: MCAS Presentation 2017Franklin Public Schools: MCAS Presentation 2017
Franklin Public Schools: MCAS Presentation 2017Franklin Matters
 
Leveraging data to improve feedback processes: what counts in the journey fro...
Leveraging data to improve feedback processes: what counts in the journey fro...Leveraging data to improve feedback processes: what counts in the journey fro...
Leveraging data to improve feedback processes: what counts in the journey fro...Rita Ndagire Kizito
 
RtI Best Practices for Henderson Elementary Scenario 3
RtI Best Practices for Henderson Elementary Scenario 3RtI Best Practices for Henderson Elementary Scenario 3
RtI Best Practices for Henderson Elementary Scenario 3Heidi Veal
 
MEDLINE search skill retention
MEDLINE search skill retentionMEDLINE search skill retention
MEDLINE search skill retentionjustmelissa
 
Response To Intervention
Response To InterventionResponse To Intervention
Response To InterventionPaul Schumann
 
Stakeholder feedback data
Stakeholder feedback dataStakeholder feedback data
Stakeholder feedback datalyacosta09
 
Teachers’ satisfaction of Assessment Process of Competency Based Curriculum...
Teachers’ satisfaction   of Assessment Process of Competency Based Curriculum...Teachers’ satisfaction   of Assessment Process of Competency Based Curriculum...
Teachers’ satisfaction of Assessment Process of Competency Based Curriculum...The Open University of Sri Lanka
 
1.vina serevina ahmad rampiki
1.vina serevina ahmad rampiki1.vina serevina ahmad rampiki
1.vina serevina ahmad rampikivinaserevina
 
Dr. Jennifer T. Butcher, PhD Dissertation Defense, Dr. William Allan Kritsoni...
Dr. Jennifer T. Butcher, PhD Dissertation Defense, Dr. William Allan Kritsoni...Dr. Jennifer T. Butcher, PhD Dissertation Defense, Dr. William Allan Kritsoni...
Dr. Jennifer T. Butcher, PhD Dissertation Defense, Dr. William Allan Kritsoni...William Kritsonis
 
Handout 6: SSE case study school (sample targets)
Handout 6: SSE case study school (sample targets)Handout 6: SSE case study school (sample targets)
Handout 6: SSE case study school (sample targets)Martin Brown
 
Response to intervention
Response to interventionResponse to intervention
Response to interventionSusanful
 
Evaluation in swaziland schools
Evaluation in swaziland schoolsEvaluation in swaziland schools
Evaluation in swaziland schoolsMabuza Moses
 

What's hot (20)

Sse workshop 2 spring 2014
Sse workshop 2 spring 2014Sse workshop 2 spring 2014
Sse workshop 2 spring 2014
 
Sse pp 11 november 2013 (1)
Sse pp   11 november 2013 (1)Sse pp   11 november 2013 (1)
Sse pp 11 november 2013 (1)
 
response to intervention.ppt
response to intervention.pptresponse to intervention.ppt
response to intervention.ppt
 
Response to Intervention: What is it?
Response to Intervention: What is it?Response to Intervention: What is it?
Response to Intervention: What is it?
 
Handout 3 SSE case study school (self-evaluation report: literacy)
Handout 3 SSE case study school (self-evaluation report: literacy)Handout 3 SSE case study school (self-evaluation report: literacy)
Handout 3 SSE case study school (self-evaluation report: literacy)
 
Franklin Public Schools: MCAS Update 2014
Franklin Public Schools: MCAS Update 2014Franklin Public Schools: MCAS Update 2014
Franklin Public Schools: MCAS Update 2014
 
Handout 2: SSE case study: The SSE journey for our case study school
Handout 2: SSE case study: The SSE journey for our case study schoolHandout 2: SSE case study: The SSE journey for our case study school
Handout 2: SSE case study: The SSE journey for our case study school
 
IRA poster presentation
IRA poster presentationIRA poster presentation
IRA poster presentation
 
Franklin Public Schools: MCAS Presentation 2017
Franklin Public Schools: MCAS Presentation 2017Franklin Public Schools: MCAS Presentation 2017
Franklin Public Schools: MCAS Presentation 2017
 
Leveraging data to improve feedback processes: what counts in the journey fro...
Leveraging data to improve feedback processes: what counts in the journey fro...Leveraging data to improve feedback processes: what counts in the journey fro...
Leveraging data to improve feedback processes: what counts in the journey fro...
 
RtI Best Practices for Henderson Elementary Scenario 3
RtI Best Practices for Henderson Elementary Scenario 3RtI Best Practices for Henderson Elementary Scenario 3
RtI Best Practices for Henderson Elementary Scenario 3
 
MEDLINE search skill retention
MEDLINE search skill retentionMEDLINE search skill retention
MEDLINE search skill retention
 
Response To Intervention
Response To InterventionResponse To Intervention
Response To Intervention
 
Stakeholder feedback data
Stakeholder feedback dataStakeholder feedback data
Stakeholder feedback data
 
Teachers’ satisfaction of Assessment Process of Competency Based Curriculum...
Teachers’ satisfaction   of Assessment Process of Competency Based Curriculum...Teachers’ satisfaction   of Assessment Process of Competency Based Curriculum...
Teachers’ satisfaction of Assessment Process of Competency Based Curriculum...
 
1.vina serevina ahmad rampiki
1.vina serevina ahmad rampiki1.vina serevina ahmad rampiki
1.vina serevina ahmad rampiki
 
Dr. Jennifer T. Butcher, PhD Dissertation Defense, Dr. William Allan Kritsoni...
Dr. Jennifer T. Butcher, PhD Dissertation Defense, Dr. William Allan Kritsoni...Dr. Jennifer T. Butcher, PhD Dissertation Defense, Dr. William Allan Kritsoni...
Dr. Jennifer T. Butcher, PhD Dissertation Defense, Dr. William Allan Kritsoni...
 
Handout 6: SSE case study school (sample targets)
Handout 6: SSE case study school (sample targets)Handout 6: SSE case study school (sample targets)
Handout 6: SSE case study school (sample targets)
 
Response to intervention
Response to interventionResponse to intervention
Response to intervention
 
Evaluation in swaziland schools
Evaluation in swaziland schoolsEvaluation in swaziland schools
Evaluation in swaziland schools
 

Similar to CERA 17: District Program Evaluation to Improve RTI/MTSS

Project from Start to Finish
Project from Start to FinishProject from Start to Finish
Project from Start to Finishclearsateam
 
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009Steve Vitto
 
Steve Vitto Response to Intervention (RTI)
Steve Vitto Response to Intervention (RTI)Steve Vitto Response to Intervention (RTI)
Steve Vitto Response to Intervention (RTI)Steve Vitto
 
Response To Intervention (RTI) presentation
Response To Intervention (RTI) presentationResponse To Intervention (RTI) presentation
Response To Intervention (RTI) presentationRobert Kulanda
 
DOMAIN-3-STRAND-3.4-LEARNER-ACHIEVEMENT-AND-OTHER-PERFORMANCE-INDICATORS.pdf
DOMAIN-3-STRAND-3.4-LEARNER-ACHIEVEMENT-AND-OTHER-PERFORMANCE-INDICATORS.pdfDOMAIN-3-STRAND-3.4-LEARNER-ACHIEVEMENT-AND-OTHER-PERFORMANCE-INDICATORS.pdf
DOMAIN-3-STRAND-3.4-LEARNER-ACHIEVEMENT-AND-OTHER-PERFORMANCE-INDICATORS.pdfriamagdosa18
 
Field Response To Intervention Mo SW-PBS SI 2008
Field Response To Intervention Mo SW-PBS SI 2008Field Response To Intervention Mo SW-PBS SI 2008
Field Response To Intervention Mo SW-PBS SI 2008Nanci Johnson
 
Rt i training module.ppt nelson
Rt i training module.ppt nelsonRt i training module.ppt nelson
Rt i training module.ppt nelsonJackie Stone
 
training-workshop-HOTS-ppt-deligero (1).pptx
training-workshop-HOTS-ppt-deligero (1).pptxtraining-workshop-HOTS-ppt-deligero (1).pptx
training-workshop-HOTS-ppt-deligero (1).pptxlucilapasus
 
Aligning Goals and Evaluations MEMSPA2014
Aligning Goals and Evaluations MEMSPA2014Aligning Goals and Evaluations MEMSPA2014
Aligning Goals and Evaluations MEMSPA2014sikojp
 
Psy kirby wehr
Psy kirby wehrPsy kirby wehr
Psy kirby wehrJulie Wise
 
School improvement plan
School improvement planSchool improvement plan
School improvement plantjune1
 
A multi tiered approach to instruction presentation
A multi tiered approach to instruction presentationA multi tiered approach to instruction presentation
A multi tiered approach to instruction presentationbwfranger
 
Connecting evidence based instructional practices to rti
Connecting evidence based instructional practices to rtiConnecting evidence based instructional practices to rti
Connecting evidence based instructional practices to rtiEast Central ISD
 
Gifted And Talented Program Overview For Charmys Naz Class
Gifted And Talented Program Overview For Charmys Naz ClassGifted And Talented Program Overview For Charmys Naz Class
Gifted And Talented Program Overview For Charmys Naz ClassEd Donnelly
 
Fasp pd skills & beliefs
Fasp pd skills & beliefsFasp pd skills & beliefs
Fasp pd skills & beliefsyeolhuh
 
Key assessement project 2
Key assessement project 2Key assessement project 2
Key assessement project 2cynnmoore
 

Similar to CERA 17: District Program Evaluation to Improve RTI/MTSS (20)

Jennings Final Presenation
Jennings Final PresenationJennings Final Presenation
Jennings Final Presenation
 
Project from Start to Finish
Project from Start to FinishProject from Start to Finish
Project from Start to Finish
 
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009
Steve Vitto Response to Intvervention (RTI) in School-wide Behavior Support 2009
 
Steve Vitto Response to Intervention (RTI)
Steve Vitto Response to Intervention (RTI)Steve Vitto Response to Intervention (RTI)
Steve Vitto Response to Intervention (RTI)
 
Response To Intervention (RTI) presentation
Response To Intervention (RTI) presentationResponse To Intervention (RTI) presentation
Response To Intervention (RTI) presentation
 
Rti ppt
Rti pptRti ppt
Rti ppt
 
NASPA AnP 2014
NASPA AnP 2014NASPA AnP 2014
NASPA AnP 2014
 
DOMAIN-3-STRAND-3.4-LEARNER-ACHIEVEMENT-AND-OTHER-PERFORMANCE-INDICATORS.pdf
DOMAIN-3-STRAND-3.4-LEARNER-ACHIEVEMENT-AND-OTHER-PERFORMANCE-INDICATORS.pdfDOMAIN-3-STRAND-3.4-LEARNER-ACHIEVEMENT-AND-OTHER-PERFORMANCE-INDICATORS.pdf
DOMAIN-3-STRAND-3.4-LEARNER-ACHIEVEMENT-AND-OTHER-PERFORMANCE-INDICATORS.pdf
 
Field Response To Intervention Mo SW-PBS SI 2008
Field Response To Intervention Mo SW-PBS SI 2008Field Response To Intervention Mo SW-PBS SI 2008
Field Response To Intervention Mo SW-PBS SI 2008
 
Rt i training module.ppt nelson
Rt i training module.ppt nelsonRt i training module.ppt nelson
Rt i training module.ppt nelson
 
training-workshop-HOTS-ppt-deligero (1).pptx
training-workshop-HOTS-ppt-deligero (1).pptxtraining-workshop-HOTS-ppt-deligero (1).pptx
training-workshop-HOTS-ppt-deligero (1).pptx
 
Aligning Goals and Evaluations MEMSPA2014
Aligning Goals and Evaluations MEMSPA2014Aligning Goals and Evaluations MEMSPA2014
Aligning Goals and Evaluations MEMSPA2014
 
Psy kirby wehr
Psy kirby wehrPsy kirby wehr
Psy kirby wehr
 
School improvement plan
School improvement planSchool improvement plan
School improvement plan
 
A multi tiered approach to instruction presentation
A multi tiered approach to instruction presentationA multi tiered approach to instruction presentation
A multi tiered approach to instruction presentation
 
Monitoring
MonitoringMonitoring
Monitoring
 
Connecting evidence based instructional practices to rti
Connecting evidence based instructional practices to rtiConnecting evidence based instructional practices to rti
Connecting evidence based instructional practices to rti
 
Gifted And Talented Program Overview For Charmys Naz Class
Gifted And Talented Program Overview For Charmys Naz ClassGifted And Talented Program Overview For Charmys Naz Class
Gifted And Talented Program Overview For Charmys Naz Class
 
Fasp pd skills & beliefs
Fasp pd skills & beliefsFasp pd skills & beliefs
Fasp pd skills & beliefs
 
Key assessement project 2
Key assessement project 2Key assessement project 2
Key assessement project 2
 

More from Christopher Kolar

Building Institutional Research Capacity in a K-12 Unified District
Building Institutional Research Capacity in a K-12 Unified DistrictBuilding Institutional Research Capacity in a K-12 Unified District
Building Institutional Research Capacity in a K-12 Unified DistrictChristopher Kolar
 
NAGC 2014: Revisiting a selective STEM high school admissions policy: 7 years...
NAGC 2014: Revisiting a selective STEM high school admissions policy: 7 years...NAGC 2014: Revisiting a selective STEM high school admissions policy: 7 years...
NAGC 2014: Revisiting a selective STEM high school admissions policy: 7 years...Christopher Kolar
 
IERC 2014: Retention of Talented STEM Students in the Illinois Higher Educati...
IERC 2014: Retention of Talented STEM Students in the Illinois Higher Educati...IERC 2014: Retention of Talented STEM Students in the Illinois Higher Educati...
IERC 2014: Retention of Talented STEM Students in the Illinois Higher Educati...Christopher Kolar
 
2014 Consortium Advancement "Huddle"
2014 Consortium Advancement "Huddle"2014 Consortium Advancement "Huddle"
2014 Consortium Advancement "Huddle"Christopher Kolar
 
Wallace Symposium 2014: STEM school response to the data
Wallace Symposium 2014: STEM school response to the dataWallace Symposium 2014: STEM school response to the data
Wallace Symposium 2014: STEM school response to the dataChristopher Kolar
 
Integrating research data into school policy plans
Integrating research data into school policy plansIntegrating research data into school policy plans
Integrating research data into school policy plansChristopher Kolar
 

More from Christopher Kolar (7)

Building Institutional Research Capacity in a K-12 Unified District
Building Institutional Research Capacity in a K-12 Unified DistrictBuilding Institutional Research Capacity in a K-12 Unified District
Building Institutional Research Capacity in a K-12 Unified District
 
2015 PAUSD strategic survey
2015 PAUSD strategic survey2015 PAUSD strategic survey
2015 PAUSD strategic survey
 
NAGC 2014: Revisiting a selective STEM high school admissions policy: 7 years...
NAGC 2014: Revisiting a selective STEM high school admissions policy: 7 years...NAGC 2014: Revisiting a selective STEM high school admissions policy: 7 years...
NAGC 2014: Revisiting a selective STEM high school admissions policy: 7 years...
 
IERC 2014: Retention of Talented STEM Students in the Illinois Higher Educati...
IERC 2014: Retention of Talented STEM Students in the Illinois Higher Educati...IERC 2014: Retention of Talented STEM Students in the Illinois Higher Educati...
IERC 2014: Retention of Talented STEM Students in the Illinois Higher Educati...
 
2014 Consortium Advancement "Huddle"
2014 Consortium Advancement "Huddle"2014 Consortium Advancement "Huddle"
2014 Consortium Advancement "Huddle"
 
Wallace Symposium 2014: STEM school response to the data
Wallace Symposium 2014: STEM school response to the dataWallace Symposium 2014: STEM school response to the data
Wallace Symposium 2014: STEM school response to the data
 
Integrating research data into school policy plans
Integrating research data into school policy plansIntegrating research data into school policy plans
Integrating research data into school policy plans
 

Recently uploaded

OSCM Unit 2_Operations Processes & Systems
OSCM Unit 2_Operations Processes & SystemsOSCM Unit 2_Operations Processes & Systems
OSCM Unit 2_Operations Processes & SystemsSandeep D Chaudhary
 
SURVEY I created for uni project research
SURVEY I created for uni project researchSURVEY I created for uni project research
SURVEY I created for uni project researchCaitlinCummins3
 
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjj
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjjStl Algorithms in C++ jjjjjjjjjjjjjjjjjj
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjjMohammed Sikander
 
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading RoomSternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading RoomSean M. Fox
 
Andreas Schleicher presents at the launch of What does child empowerment mean...
Andreas Schleicher presents at the launch of What does child empowerment mean...Andreas Schleicher presents at the launch of What does child empowerment mean...
Andreas Schleicher presents at the launch of What does child empowerment mean...EduSkills OECD
 
e-Sealing at EADTU by Kamakshi Rajagopal
e-Sealing at EADTU by Kamakshi Rajagopale-Sealing at EADTU by Kamakshi Rajagopal
e-Sealing at EADTU by Kamakshi RajagopalEADTU
 
24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...
24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...
24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...Nguyen Thanh Tu Collection
 
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUMDEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUMELOISARIVERA8
 
Improved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio AppImproved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio AppCeline George
 
8 Tips for Effective Working Capital Management
8 Tips for Effective Working Capital Management8 Tips for Effective Working Capital Management
8 Tips for Effective Working Capital ManagementMBA Assignment Experts
 
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...EADTU
 
How To Create Editable Tree View in Odoo 17
How To Create Editable Tree View in Odoo 17How To Create Editable Tree View in Odoo 17
How To Create Editable Tree View in Odoo 17Celine George
 
The Story of Village Palampur Class 9 Free Study Material PDF
The Story of Village Palampur Class 9 Free Study Material PDFThe Story of Village Palampur Class 9 Free Study Material PDF
The Story of Village Palampur Class 9 Free Study Material PDFVivekanand Anglo Vedic Academy
 
When Quality Assurance Meets Innovation in Higher Education - Report launch w...
When Quality Assurance Meets Innovation in Higher Education - Report launch w...When Quality Assurance Meets Innovation in Higher Education - Report launch w...
When Quality Assurance Meets Innovation in Higher Education - Report launch w...Gary Wood
 
How to Send Pro Forma Invoice to Your Customers in Odoo 17
How to Send Pro Forma Invoice to Your Customers in Odoo 17How to Send Pro Forma Invoice to Your Customers in Odoo 17
How to Send Pro Forma Invoice to Your Customers in Odoo 17Celine George
 
male presentation...pdf.................
male presentation...pdf.................male presentation...pdf.................
male presentation...pdf.................MirzaAbrarBaig5
 
Trauma-Informed Leadership - Five Practical Principles
Trauma-Informed Leadership - Five Practical PrinciplesTrauma-Informed Leadership - Five Practical Principles
Trauma-Informed Leadership - Five Practical PrinciplesPooky Knightsmith
 

Recently uploaded (20)

Supporting Newcomer Multilingual Learners
Supporting Newcomer  Multilingual LearnersSupporting Newcomer  Multilingual Learners
Supporting Newcomer Multilingual Learners
 
OSCM Unit 2_Operations Processes & Systems
OSCM Unit 2_Operations Processes & SystemsOSCM Unit 2_Operations Processes & Systems
OSCM Unit 2_Operations Processes & Systems
 
SURVEY I created for uni project research
SURVEY I created for uni project researchSURVEY I created for uni project research
SURVEY I created for uni project research
 
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjj
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjjStl Algorithms in C++ jjjjjjjjjjjjjjjjjj
Stl Algorithms in C++ jjjjjjjjjjjjjjjjjj
 
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading RoomSternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
 
Andreas Schleicher presents at the launch of What does child empowerment mean...
Andreas Schleicher presents at the launch of What does child empowerment mean...Andreas Schleicher presents at the launch of What does child empowerment mean...
Andreas Schleicher presents at the launch of What does child empowerment mean...
 
e-Sealing at EADTU by Kamakshi Rajagopal
e-Sealing at EADTU by Kamakshi Rajagopale-Sealing at EADTU by Kamakshi Rajagopal
e-Sealing at EADTU by Kamakshi Rajagopal
 
24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...
24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...
24 ĐỀ THAM KHẢO KÌ THI TUYỂN SINH VÀO LỚP 10 MÔN TIẾNG ANH SỞ GIÁO DỤC HẢI DƯ...
 
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUMDEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
 
Improved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio AppImproved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio App
 
8 Tips for Effective Working Capital Management
8 Tips for Effective Working Capital Management8 Tips for Effective Working Capital Management
8 Tips for Effective Working Capital Management
 
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...
Transparency, Recognition and the role of eSealing - Ildiko Mazar and Koen No...
 
How To Create Editable Tree View in Odoo 17
How To Create Editable Tree View in Odoo 17How To Create Editable Tree View in Odoo 17
How To Create Editable Tree View in Odoo 17
 
Mattingly "AI and Prompt Design: LLMs with NER"
Mattingly "AI and Prompt Design: LLMs with NER"Mattingly "AI and Prompt Design: LLMs with NER"
Mattingly "AI and Prompt Design: LLMs with NER"
 
The Story of Village Palampur Class 9 Free Study Material PDF
The Story of Village Palampur Class 9 Free Study Material PDFThe Story of Village Palampur Class 9 Free Study Material PDF
The Story of Village Palampur Class 9 Free Study Material PDF
 
When Quality Assurance Meets Innovation in Higher Education - Report launch w...
When Quality Assurance Meets Innovation in Higher Education - Report launch w...When Quality Assurance Meets Innovation in Higher Education - Report launch w...
When Quality Assurance Meets Innovation in Higher Education - Report launch w...
 
Mattingly "AI & Prompt Design: Named Entity Recognition"
Mattingly "AI & Prompt Design: Named Entity Recognition"Mattingly "AI & Prompt Design: Named Entity Recognition"
Mattingly "AI & Prompt Design: Named Entity Recognition"
 
How to Send Pro Forma Invoice to Your Customers in Odoo 17
How to Send Pro Forma Invoice to Your Customers in Odoo 17How to Send Pro Forma Invoice to Your Customers in Odoo 17
How to Send Pro Forma Invoice to Your Customers in Odoo 17
 
male presentation...pdf.................
male presentation...pdf.................male presentation...pdf.................
male presentation...pdf.................
 
Trauma-Informed Leadership - Five Practical Principles
Trauma-Informed Leadership - Five Practical PrinciplesTrauma-Informed Leadership - Five Practical Principles
Trauma-Informed Leadership - Five Practical Principles
 

CERA 17: District Program Evaluation to Improve RTI/MTSS

  • 1. District Program Evaluation to Improve RTI/MTSS Clarisse Haxton, Ph.D. Program Evaluation Coordinator Palo Alto Unified School District CERA Conference 2017
  • 2. Agenda • PAUSD District and RTI Overview • RTI Evaluation Overview • Findings • RTI Monitoring Tool (data) • Implementation progress and challenges • Student outcomes • Recommendations and next steps
  • 3. PAUSD District and RTI Overview
  • 4. PAUSD District and RTI Overview • High-performing overall; substantial within-district achievement gaps • RTI/MTSS • History of RTI work in elementary • Recognized need for district system • Recommendation for evaluation in MATD Committee report and district Equity Plan • Program Evaluation Coordinator role
  • 5. Bufum, Matos, & Weber (2012) 5
  • 6. RTI Definition from NCRTI “Response to intervention integrates assessment and intervention within a multi-level prevention system to maximize student achievement and to reduce behavioral problems. With RTI, schools use data to identify students at risk for poor learning outcomes, monitor student progress, provide evidence-based interventions and adjust the intensity and nature of those interventions depending on a student’s responsiveness, and identify students with learning disabilities or other disabilities… This system includes three levels of intensity or three levels of prevention, which represent a continuum of supports. Many schools use more than one intervention within a given level of prevention.” Palo Alto Unified School District 6National Center on Response to Intervention (2010)
  • 7. RTI/MTSS “Rules of Thumb” • Tier 1: classroom instruction (“best first instruction”) provided to all students; serves 80 to 85% of students • Tier 2: interventions are supplementary instruction and support on targeted areas of student need; serves approximately 10-15% of students • Tier 3: interventions are more intensive and individualized, supplementary to core classroom instruction; serve approximately 5 to 10% of students 7
  • 9. Evaluation Purpose • Provide an inventory of Tier 2 interventions: – # students are being served – student characteristics – Tier 2 interventions are being used • Describe: – screening and identification practices – intervention implementation – progress monitoring practices – RTI student outcomes • Make recommendations for continuous improvement
  • 10. Evaluation Questions • Who’s being served? • How are they identified? • What are they getting? • How are we monitoring progress? • What progress are students making? • What’s working well? • What are challenges? • What can we do better?
  • 12. Lesson 1: Need data • Problem of Practice ⁻ Lacked system to identify and track students in Tier 2 intervention ⁻ Need to answer “who” and “what” before you can answer “How are we doing?” • Solution: Created a RTI Monitoring Tool (Google sheet) across all elementary schools
  • 13. RTI Monitoring Tool (Google Sheets) • Students who received RTI • Classroom teacher and grade level • RTI subject (math, reading/literacy) • Intervention program name(s) • Intervention teacher(s) • Assessment scores that made a student eligible for RTI • Beginning of intervention (BOI) assessment score • Progress monitoring score (at least 1) • End of intervention (EOI) assessment score
  • 14. RTI Monitoring Tool (Google Sheets) • 2016-17 RTI Monitoring Tool
  • 15. Who are we serving and what are they getting? Grade Level % of RTI Students in Math Intervention % of RTI Students in Reading Intervention % of RTI Students in Math & Reading Intervention % of RTI Students with Unspecified Intervention Total # of Students in RTI K 4.7% (5) 100% (106) 4.7% (5) 3.7% (4) 106 1 21.1% (31) 93.2% (137) 17% (25) 8.8% (13) 147 2 42.1% (64) 61.8% (94) 15.1% (23) 3.3% (5) 152 3 58.9% (96) 63.2% (103) 25.1% (41) 4.3% (7) 163 4 64.8% (114) 42.6% (75) 11.4% (20) 4% (7) 176 5 55% (72) 48.9% (64) 14.5% (19) 8.4% (11) 131 Total 43.6% 66.1% 15.2% 5.4% 875 n=382 n=579 n=133 n=47 Number and Percent of Students in Tier 2 Interventions in 2016-17, by Grade Level and Subject Source: District RTI Data. Note: 382 + 579 = 961 duplicated RTI students – 133 = 828 total RTI students in reading or math. 828 + 47 = 875 total RTI students. “Unspecified” interventions include “progress monitoring”, “small group,” and other titles that did not distinguish them as math or reading RTI.
  • 16. Who are we serving? Background Characteristics of RTI Students Source: District RTI Data. Note: HUR_Race includes Black, Hispanic, Native Hawaiian/Pacific Islander, and Alaska Native students. Student Characteristics % of RTI Students % K-5 in PAUSD Historically Underrepresented Race/Ethnicity (HUR_Race) 47.2% 18.7% Socioeconomically Disadvantaged (SED) 40.6% 12.4% Voluntary Transfer Program (VTP) 21.8% 6.5% Historically Underrepresented (HUR_Race + SED + VTP) 51.3% 21.7% English Learner (EL) 37.8% 18.9% Special Education (IEP) 11.7% 8.6% Male 53.3% 52.1% Total 875 5224
  • 17. Lesson 2: Need (somewhat) standardized RTI eligibility criteria • Variability in student performance among RTI students – Below “PAUSD average” versus at or below benchmark/standard – Raised important discussions about Tier 1 and Tier 2 – Has now led to more common eligibility criteria • RTI students’ CAASPP performance – Among fourth and fifth grade RTI students, 11 to 37 percent of Met or Exceeded standard on their prior year (2015-16) CAASPP assessment, in the subject they received Tier 2 intervention.
  • 18. Changing systems while systems are always changing •RTI reading students’ BOY reading performance – Messy because we were transitioning to new assessment – In fall 2016, 17 percent of students in Tier 2 reading intervention performed at or above the district’s BOY reading benchmark level on BAS and 46 percent performed at or above the district’s BOY benchmark on DRA. •Math RTI students’ BOY math performance – Exploring how to best use NWEA MAP data in Year 1 – Less than 1 percent of RTI math students in grades 3 to 5 scored at or above the PAUSD average on the BOY NWEA MAP assessment. – However, over 40 percent of students in RTI math intervention scored above the national average for their grade level.
  • 19. Lesson 3: Implementation progress; system challenges remain • Remaining challenges – Finding qualified staff to teach after school – Student transportation – Dosage and implementing interventions with fidelity – Setting targeted goals for students and targeting RTI instruction in groups with varying needs – Progress monitoring – Improving student achievement and closing achievement gaps
  • 20. Lesson 4: Time + Targeted Instruction = Learning • Need to focus on targeting instruction • Need to improve outcomes –Among K-5 RTI students who began the year below benchmark, 44 percent in math and 56 percent in reading “closed the gap” (got closer to benchmark) from BOY to EOY –There were no consistent patterns or significant differences in outcomes by student characteristics, program, or staff type.
  • 21. Lesson 5: REA-Elem-Site partnership has moved this work • Research, Evaluation, and Assessment (REA) Department • Elementary Education Dept • Elementary site principals • … and now, some interest and exploration in secondary schools Palo Alto Unified School District
  • 22. PAUSD Elementary RTI Evaluation Recommendations and Next Steps
  • 23. Recommendations (2016-17) and Next Steps (2017-18) • Time + Targeted Instruction = Learning 1. Continue to use and improve the RTI Monitoring Tool –Fixed format template –Focusing more on measurable goals and progress monitoring –Trying to focus on fixed intervention rounds with pauses between to examine the data and adjust students and goals
  • 24. Recommendations (2016-17) and Next Steps (2017-18) 2. Set common expectation of prioritizing students below benchmark or standard (and generally, not in special education) –Use state and local assessment data to improve screening and identification –Data available in DataZone, Google Sheets, and NWEA
  • 25. Recommendations (2016-17) and Next Steps (2017-18) 3. Provide support for continuous improvement of interventions - TOSAs mapped standards to Bridges Intervention volumes - Bridges Intervention training
  • 26. Recommendations (2016-17) and Next Steps (2017-18) 4. Create a multi-tiered system of supports (MTSS) that includes all staff PK-12 •Supporting an RTI/IST pilot at two sites, connecting Tier 1 and 2 practices and data through linked Google Forms and coaching •Integrating RTI into LCAP and Equity Plan •Supporting new high school RTI teams •Conducting special education program evaluation
  • 27. Internal District Program Evaluation– Breaking Down Department Silos
  • 28. Internal District Program Evaluation– Breaking Down Department Silos
  • 29. Internal District Program Evaluation– Breaking Down Department Silos
  • 30. Internal District Program Evaluation • Cost effective ⁻ $150K+ for a major research firm to do a similar study ⁻ Previously, $70K research contractor • Cost beneficial ⁻ We spend $130K + site expenditures on elementary RTI. Important to know whether the $ makes a difference to inform resource allocation decisions.
  • 31. Internal District Program Evaluation • Research-Department partnerships within the district can deepen and move the work ⁻ Benefit of district context and relationships ⁻ Ability to regularly visit schools, observe, talk to people ⁻ Strategically link programs and evaluations together ⁻ Help the Board and administrators make data-driven decisions
  • 32. Questions? • Thanks! Palo Alto Unified School District