In higher education, Institutional Research (IR) offices function to audit the academic output of the institution, evaluate program efficacy, and monitor student success. Effective institutional research supports the understanding, planning, and operation of programs informed by a recognition that different functions of an institution are interrelated and dependent. This session will outline practices by the Department of Research, Evaluation, and Assessment in the Palo Alto Unified School District – a division designed and staffed using an IR model.

1. Building Institutional Research
Capacity in a K-12 Unified District
Christopher Kolar, Clarisse Haxton, &
Janine Penney
Research, Evaluation, and Assessment (REA) Department
Palo Alto Unified School District
CERA Conference 2017

2. Institutional Research Approach
IR Offices in higher education
• Audit institutional academic output
• Evaluate program efficiency
• Monitor student success
• Support program planning and operations,
recognizing interdependency across the institution
• Often oversee traditional IT roles by being both
brokers of information, as well as the strategic
and operational connection

3. The Five Functions
3

4. The five functions in practice
1. Identify measures and
concepts
2. Collect and store data
3. Restructure and
analyze facts
4. Deliver and report
information
5. Use and influence
knowledge
1. Literacy, summative
common core goals
2. Deploy BAS assessment to
measure student progress
in new reading curriculum
3. Use data warehouse to
identify students who
need supports
4. Track student progress in
RTI/evaluation memo
5. Make course corrections
in partnership with
curriculum leaders
4

5. Benefits of IR approach in PK12
district research offices
• Collaboration across departments
• Shift from “to” to “with”
• Internal/external lens
• Research rigor, focus on action steps
for continuous improvement
It is about building capacity!It is about building capacity!

6. REA supports the work of all
district departments
1. District common formative
assessments
2. Data dashboard system
3. Surveys
4. Systematizing and improving
district practices
5. Program evaluation
Newfunctions:notIT
Newfunctions:notIT

7. District Common Formative
Assessments

8. Entering the data –
Common formative assessments
Problems of Practice
– Limitations of SIS
– Data-entry “outside” of an assessment system
– Introducing new system to teachers
Solution
– Google Sheets for data-entry
– Rolled out through screencasts, sample templates,
meetings
Partnership and training teachersPartnership and training teachers

9. Multiple modes of
training and support
• Introduced to principals during drop-in office
hours
• Principals shared during staff meeting
• All info housed on PAUSD Google Site
– Screencast (video)
– Sample/practice template (Google Sheet)
– Sample Templates with Instructional overlay (pdf)
9
IR is capacity buildingIR is capacity building

10. Entering the data –
Directions for teachers
10
• ADD VIDEO

11. 11
EnteringAssessment Resultsinto Google Sheets– SAMPLEK1 BRIDGESBASELINETEMPLATE
PAUSD Department of Research, Evaluation, and Assessment 091917
HELPFULHINTS:
• Kinder and First Grade math templates follow the SAME format, with different items.
• If unable to view more than a few rows of students, adjust settings: hold COMMAND key (MAC) or Control key (PC) and press +or -
• Top two rows are frozen, scroll up or down to locate classroom.
• First five columns are frozen, scroll back and forth as needed to view appropriate assessment fields.
• Google sheet auto-saves. This process eliminates need to download templates and email results.
• A new Google Sheet will be provide to each school site each term (BOY/fall, MOY/winter, EOY/spring).
Click tab to view appropriate sheet for entering scores
ALL math results are entered using drop-down menus or by typing result manually.
If drop-down menu freezes, click outside of cell, then click back inside cell.
Teachers requested additional details in the item header. Note the following:
• Interview Items and Written Items
• Item number, abbreviated description of item, total point value, standard assessed
Notes about these two columns:
• The total score auto-sums
interview and written items
as results are entered.
• “Level of Proficiency” defined
in the Bridges Baseline
Assessment (see cell for
definition).

12. District common formative
assessment overview
12
GL Assessment Data Entry
K – 5 F&P BAS (Reading)*
TC (Writing)
Bridges (Math)*
Google Sheet*
2 – 6 NWEA NWEA system
3 – 5 IAB Interim Assessment Reporting
system with item analysis*
6 – 8 Writing Scores remain at site
9 – 12
*New in 2017-18
**Gr 2 in Year 1; Gr 3-5 in Year 2; Gr 6 Pilot

13. System to support ALL students
13

14. Summative vs. Formative
Assessment OF vs. FOR
• Assessments OF learning
determine how much has been
learned at a particular point in
time (status) in order to report
to others.
• Assessments FOR learning
provide feedback that is used
to modify teaching and
learning activities during
instruction.
14

15. California Standards for the Teaching Profession
Standard 5: Assessing Students for Learning
15

16. What is NWEA MAP?
Measures of Academic Progress (MAP) assessments:
●Growth Math K-2 (grade 2)
●Growth Math 2-5 (grade 3-5)
●Growth 6+ (grade 6)
An individualized assessment
experience which adapts to each
student's learning level.
•Normed & highly reliable
•Common Core Aligned
•Computer Adaptive
•Results available in 24 hours
16

17. Using the data - Common math
formative assessments
• NWEA (2 – 6):
• Year 1: Focus on administration/formative aspect
• Year 2: Refine year one processes; using data to
inform grouping and differentiated instruction;
monitor progress of RTI students;
Responding to questions as they emergeResponding to questions as they emerge
to make the data more useful to staff.to make the data more useful to staff.

18. This view is “online” with interactive features.
Clicking on “Mathematics” produces breakdown by math strand.
18
NWEA class report by RIT

19. Student example of
NWEA’s SBAC prediction
Palo Alto Unified School District 19

20. NWEA’s prediction accuracy
SBAC
prediction
Correct Actual
Percent
Correct
4 1179 1497 79%
3 360 641 56%
2 234 359 65%
1 139 183 76%
PAUSD Actual
Achievement
PAUSD Predicted
Achievement

21. NWEA’s prediction accuracy
by grade
21

22. Data Dashboard/Warehouse
System

23. DataZone
• Had to take ownership of data quality
and management
• Make clear: it is a tool for teaching and
learning
Palo Alto Unified School District

24. Surveys

25. (Many, Many) Surveys
• Regular, district-wide feedback (n=6 in 2016-17)
– e.g., Course Evaluation Survey, Strategic Plan Survey,
PARC (School Climate) Survey
• Department-specific (n=5)
– e.g., Family Engagement Specialist Survey, EL Survey,
VTP Survey, HR Benefits Survey
• Program evaluation (n=3)
– e.g., Full-Day Kindergarten
• Board-initiated topics (n=3)
– Weighted GPA, Renaming Schools, Calendar
• Site-specific (n=3)

26. (Many, Many) Surveys
• We advise and collaborate with departments
on survey design
• We use the Qualtrics platform to administer
district-wide surveys and do basic descriptive
reporting
• This helps us to:
– Improve survey quality
– Manage participant burden
– Provide findings to staff quickly
– Starting to… plan surveys to inform the LCAP

27. Systematizing and Improving
District Processes

28. Systematizing and improving
district practices
• Example: Curriculum adoption processes
–Ensure representativeness of pilot
committee
–Help structure pilot process to ensure
representativeness of units taught
–Co-design curriculum evaluation rubrics
–Design and facilitate the final evaluation
and voting process

29. Curriculum adoption process–
The vote
• Randomly assigned teachers to voting six
groups, specialists in one group,
community members in one group
• Committee members took “Three
Essential Questions” survey and REA
analyzed and presented the results
• Then, they voted using the following
process…

30. Curriculum adoption process–
The vote
• Each voting group will have 4 points to
allocate to the three piloted curricula, using
the following rules:
– All four points must be used.
– No points indicates “Do not recommend.”
– 1 point indicates “Recommend, with moderate
reservations.”
– 2 points indicates “Recommend, with minor
reservations.”
– 3 points indicates “Strongly recommend.”

31. Curriculum adoption process–
The vote
The EMCPC is a volunteer committee that is comprised of at
least one member from each elementary school and each
grade level (PK-5), but membership is not equal across sites or
grade levels. The CAO, elementary math TOSAs, and the REA
Department devised the “four-point system” and randomly
assigned teacher committee members to voting groups to:
•honor the individual contributions of every math committee
member,
•mitigate any potential bias in the EMCPC composition, and
•come to consensus as a committee on the curriculum
recommendation that will work best for all PAUSD students,
teachers, and families.

32. Curriculum adoption process–
The vote
• The final EMCPC discussion and voting process
is intended to move members from an
individual pilot experience to a district
decision. We do not want individuals to leave
the final EMCPC meeting feeling that their
favorite curriculum “won” or “lost,” but that,
given the varying experiences, perspectives,
and needs of our students, staff, and families,
the EMCPC came to a decision together that
will best serve our entire district.

33. Systematizing and improving
district practices
• Example: Vendor RFP and selection
process
–Co-design RFP and proposal evaluation
rubric
–Design and facilitate the proposal
evaluation process
–Ensure representativeness of review
committee

34. Systematizing and improving district
practices through program evaluation
• Example: Elementary RTI
–Identify and describe who is participating
and selection criteria (RTI Monitoring Tool)
–Describe the various interventions and
structures being used
• Identify promising practices and challenges
–Examine student outcomes
–Make recommendations to guide system
improvement

35. Program Evaluation

36. Program evaluation
2016-17
• Elementary RTI
• Read 180
• Responsive
Classroom
• Full-day
kindergarten
• Algebra pilot
2017-18
• Special education
• AVID
• School-Connect
• (Algebra)

37. Questions?
• Thanks!