The document summarizes the findings of a program evaluation conducted by the Palo Alto Unified School District (PAUSD) to evaluate and improve their Response to Intervention (RTI) and Multi-Tiered System of Support (MTSS) programs. Some key findings included: (1) the need for better data collection tools to track students receiving RTI services, (2) variability in eligibility criteria that has led to more standardized criteria, and (3) implementation challenges around staffing, transportation, and intervention fidelity. Recommendations focused on continuing to improve data tools, prioritizing students below benchmarks, supporting intervention quality, and expanding MTSS district-wide with the goal of improving student outcomes.

1. District Program Evaluation to
Improve RTI/MTSS
Clarisse Haxton, Ph.D.
Program Evaluation Coordinator
Palo Alto Unified School District
CERA Conference 2017

2. Agenda
• PAUSD District and RTI Overview
• RTI Evaluation Overview
• Findings
• RTI Monitoring Tool (data)
• Implementation progress and challenges
• Student outcomes
• Recommendations and next steps

3. PAUSD District and RTI Overview

4. PAUSD District and RTI Overview
• High-performing overall; substantial
within-district achievement gaps
• RTI/MTSS
• History of RTI work in elementary
• Recognized need for district system
• Recommendation for evaluation in MATD
Committee report and district Equity
Plan
• Program Evaluation Coordinator role

5. Bufum, Matos, & Weber (2012) 5

6. RTI Definition from NCRTI
“Response to intervention integrates assessment and
intervention within a multi-level prevention system to maximize
student achievement and to reduce behavioral problems. With
RTI, schools use data to identify students at risk for poor
learning outcomes, monitor student progress, provide
evidence-based interventions and adjust the intensity and
nature of those interventions depending on a student’s
responsiveness, and identify students with learning disabilities
or other disabilities… This system includes three levels of
intensity or three levels of prevention, which represent a
continuum of supports. Many schools use more than one
intervention within a given level of prevention.”
Palo Alto Unified School District 6National Center on Response to Intervention (2010)

7. RTI/MTSS “Rules of Thumb”
• Tier 1: classroom instruction (“best first instruction”)
provided to all students; serves 80 to 85% of
students
• Tier 2: interventions are supplementary instruction
and support on targeted areas of student need;
serves approximately 10-15% of students
• Tier 3: interventions are more intensive and
individualized, supplementary to core classroom
instruction; serve approximately 5 to 10% of
students
7

8. PAUSD Elementary RTI
Evaluation Overview

9. Evaluation Purpose
• Provide an inventory of Tier 2 interventions:
– # students are being served
– student characteristics
– Tier 2 interventions are being used
• Describe:
– screening and identification practices
– intervention implementation
– progress monitoring practices
– RTI student outcomes
• Make recommendations for continuous
improvement

10. Evaluation Questions
• Who’s being served?
• How are they identified?
• What are they getting?
• How are we monitoring progress?
• What progress are students making?
• What’s working well?
• What are challenges?
• What can we do better?

11. PAUSD Elementary RTI
Evaluation Findings

12. Lesson 1: Need data
• Problem of Practice
⁻ Lacked system to identify and track
students in Tier 2 intervention
⁻ Need to answer “who” and “what”
before you can answer “How are we
doing?”
• Solution: Created a RTI Monitoring Tool
(Google sheet) across all elementary
schools

13. RTI Monitoring Tool
(Google Sheets)
• Students who received RTI
• Classroom teacher and grade level
• RTI subject (math, reading/literacy)
• Intervention program name(s)
• Intervention teacher(s)
• Assessment scores that made a student eligible for
RTI
• Beginning of intervention (BOI) assessment score
• Progress monitoring score (at least 1)
• End of intervention (EOI) assessment score

14. RTI Monitoring Tool
(Google Sheets)
• 2016-17 RTI Monitoring Tool

15. Who are we serving and
what are they getting?
Grade Level
% of RTI
Students in
Math
Intervention
% of RTI
Students in
Reading
Intervention
% of RTI
Students in
Math &
Reading
Intervention
% of RTI
Students with
Unspecified
Intervention
Total # of
Students in
RTI
K 4.7% (5) 100% (106) 4.7% (5) 3.7% (4) 106
1 21.1% (31) 93.2% (137) 17% (25) 8.8% (13) 147
2 42.1% (64) 61.8% (94) 15.1% (23) 3.3% (5) 152
3 58.9% (96) 63.2% (103) 25.1% (41) 4.3% (7) 163
4 64.8% (114) 42.6% (75) 11.4% (20) 4% (7) 176
5 55% (72) 48.9% (64) 14.5% (19) 8.4% (11) 131
Total 43.6% 66.1% 15.2% 5.4% 875
n=382 n=579 n=133 n=47
Number and Percent of Students in Tier 2 Interventions
in 2016-17, by Grade Level and Subject
Source: District RTI Data. Note: 382 + 579 = 961 duplicated RTI students – 133 = 828 total RTI students in
reading or math. 828 + 47 = 875 total RTI students. “Unspecified” interventions include “progress
monitoring”, “small group,” and other titles that did not distinguish them as math or reading RTI.

16. Who are we serving?
Background Characteristics of RTI Students
Source: District RTI Data. Note: HUR_Race includes Black, Hispanic, Native
Hawaiian/Pacific Islander, and Alaska Native students.
Student Characteristics
% of RTI
Students
% K-5 in
PAUSD
Historically Underrepresented Race/Ethnicity (HUR_Race) 47.2% 18.7%
Socioeconomically Disadvantaged (SED) 40.6% 12.4%
Voluntary Transfer Program (VTP) 21.8% 6.5%
Historically Underrepresented (HUR_Race + SED + VTP) 51.3% 21.7%
English Learner (EL) 37.8% 18.9%
Special Education (IEP) 11.7% 8.6%
Male 53.3% 52.1%
Total 875 5224

17. Lesson 2: Need (somewhat)
standardized RTI eligibility criteria
• Variability in student performance among RTI
students
– Below “PAUSD average” versus at or below
benchmark/standard
– Raised important discussions about Tier 1 and Tier 2
– Has now led to more common eligibility criteria
• RTI students’ CAASPP performance
– Among fourth and fifth grade RTI students, 11 to 37 percent of
Met or Exceeded standard on their prior year (2015-16) CAASPP
assessment, in the subject they received Tier 2 intervention.

18. Changing systems while systems
are always changing
•RTI reading students’ BOY reading performance
– Messy because we were transitioning to new assessment
– In fall 2016, 17 percent of students in Tier 2 reading
intervention performed at or above the district’s BOY reading
benchmark level on BAS and 46 percent performed at or above
the district’s BOY benchmark on DRA.
•Math RTI students’ BOY math performance
– Exploring how to best use NWEA MAP data in Year 1
– Less than 1 percent of RTI math students in grades 3 to 5 scored
at or above the PAUSD average on the BOY NWEA MAP
assessment.
– However, over 40 percent of students in RTI math intervention
scored above the national average for their grade level.

19. Lesson 3: Implementation progress;
system challenges remain
• Remaining challenges
– Finding qualified staff to teach after school
– Student transportation
– Dosage and implementing interventions with
fidelity
– Setting targeted goals for students and targeting
RTI instruction in groups with varying needs
– Progress monitoring
– Improving student achievement and closing
achievement gaps

20. Lesson 4: Time + Targeted
Instruction = Learning
• Need to focus on targeting instruction
• Need to improve outcomes
–Among K-5 RTI students who began the
year below benchmark, 44 percent in math
and 56 percent in reading “closed the gap”
(got closer to benchmark) from BOY to EOY
–There were no consistent patterns or
significant differences in outcomes by
student characteristics, program, or staff
type.

21. Lesson 5: REA-Elem-Site
partnership has moved this work
• Research, Evaluation, and Assessment
(REA) Department
• Elementary Education Dept
• Elementary site principals
• … and now, some interest and
exploration in secondary schools
Palo Alto Unified School District

22. PAUSD Elementary RTI Evaluation
Recommendations and Next Steps

23. Recommendations (2016-17) and
Next Steps (2017-18)
• Time + Targeted Instruction = Learning
1. Continue to use and improve the RTI
Monitoring Tool
–Fixed format template
–Focusing more on measurable goals and
progress monitoring
–Trying to focus on fixed intervention rounds
with pauses between to examine the data
and adjust students and goals

24. Recommendations (2016-17) and
Next Steps (2017-18)
2. Set common expectation of prioritizing
students below benchmark or standard
(and generally, not in special education)
–Use state and local assessment data to
improve screening and identification
–Data available in DataZone, Google Sheets,
and NWEA

25. Recommendations (2016-17) and
Next Steps (2017-18)
3. Provide support for continuous
improvement of interventions
- TOSAs mapped standards to Bridges
Intervention volumes
- Bridges Intervention training

26. Recommendations (2016-17) and
Next Steps (2017-18)
4. Create a multi-tiered system of supports
(MTSS) that includes all staff PK-12
•Supporting an RTI/IST pilot at two sites,
connecting Tier 1 and 2 practices and data
through linked Google Forms and coaching
•Integrating RTI into LCAP and Equity Plan
•Supporting new high school RTI teams
•Conducting special education program
evaluation

27. Internal District Program Evaluation–
Breaking Down Department Silos

28. Internal District Program Evaluation–
Breaking Down Department Silos

29. Internal District Program Evaluation–
Breaking Down Department Silos

30. Internal District Program Evaluation
• Cost effective
⁻ $150K+ for a major research firm to do a
similar study
⁻ Previously, $70K research contractor
• Cost beneficial
⁻ We spend $130K + site expenditures on
elementary RTI. Important to know whether
the $ makes a difference to inform resource
allocation decisions.

31. Internal District Program Evaluation
• Research-Department partnerships
within the district can deepen and
move the work
⁻ Benefit of district context and relationships
⁻ Ability to regularly visit schools, observe, talk to
people
⁻ Strategically link programs and evaluations
together
⁻ Help the Board and administrators make
data-driven decisions

32. Questions?
• Thanks!
Palo Alto Unified School District