1. The Student Learning Objective (SLO) Process is comprised of three (3) components: Design, Build, and Review. Student Learning Objectives provide a
valid assessment of teacher effectiveness through student performance outcomes based on standards.
Welcome to Training Module 3: “Reviewing”
Concept
Within the Review Module, the trainer will guide participants to implement a multifaceted, three-level review of Student Learning Objectives, including
identified performance measures.
Key Points for Trainers
1.
2.
3.
Explain in macro-terms the “big picture” of how SLOs are but one component of a much larger teacher effectiveness system.
a.
As required by Act 82, an Educator Effectiveness Rating Tool must contain measures based on teacher observation and practice and
multiple measures of student performance. Student Performance data will include the following: Building Level Data (15%),
Teacher Specific Data (15%), and Elective Data (20% for teachers with eligible PVAAS data, 35% for teachers without eligible PVAAS
data).
b.
This Educator Effectiveness Rating Tool functions as a framework for the evaluation and summative process for classroom teachers.
Clarify that the Review Module will guide participants through a three-level quality assurance review of the created Student Learning Objective
and the identified Performance Measures. The Review phase helps ensure that the operational and technical expectations of the process are
met or exceeded.
Ensure participants understand that this phase is the quality assurance component, focusing on issues that range from completing the SLO
Process Template correctly to examining the validity evidence of a vendor-made assessment/performance measure.
Learning Activity
___________________________________________________________________________________________________________________________
Technical Notes
“Structure”
1.
Concept – “What is this slide telling the audience?”
2.
Key Points – “What/Where are the details ‘needed for teaching’?”
3.
Learning Activity – “How can the participant’s learning be enhanced?” (This item will not be populated for every slide.)
Module 3-SLO Reviewing
1

2. Student Learning Objective
Concept
As you work through the Review Process, you will need to access the following materials:
1) Handout #4 – Quality Assurance Checklist-SLO (found in SLO/Review/Guides)
2) Handout #5 – SLO Procedural “Cheat Sheet” (found in SLO/Review/Guides)
3) Template #6 – SLO Coherency Rubric (found in SLO/Review/Templates)
4) Performance Measure Rubric for Teachers (found in ALS/Review/Stuff)
5) Refinement Control Checklist (on-demand Performance Measures only, found in SLO/Review/Stuff)
Key Points for Trainers
1.
2.
3.
Assist the participants in conceptualizing that the SLO Process Template is organized into two major
components: SLO Goals and Performance Indicators on page 1 of the template, and Performance
Measures and Expectations on page 2.
Clarify that the comprehensive evaluation of a new performance measure cannot be fully completed
until after the measure is administered and data is collected. The current tools focus on the
development and administration of the measures, with additional training being developed for the
“post-administration” review. For vendor-made measures, there should be sufficient evidence on the
inferences of scores, reliability, etc. that also includes how the performance measure was developed.
Stress to participants that the Review process includes a quality assurance check on each
Performance Measure selected to ensure that they are aligned with the content standards, free of
bias and sensitive content, and developmentally appropriate.
Learning Activity
1.
Have participants go to www.pdesas.org to retrieve the five items mentioned above from the SAS
Portal /Homeroom learning portal. Also make sure they have the completed SLO Process Template
(Template #4) found in SLO/Build/Templates.
Module 3-SLO Reviewing
2

3. Process Components
Concept
This slide reviews the parts associated with each phase of the SLO Process. This Module
will focus on the Review Phase of the SLOs.
Key Points for Trainers
1. It will be important for each educator to have all the completed templates from their
work with Module 1: Design and Module 2: Build. That would include the following
templates:
a. Template #1: Goal Statement (found in SLO/Design/Templates)
b. Template #2: Targeted Standards (found in SLO/Design/Templates)
c. Template #3: SLO Blueprint (found in SLO/Design/Templates)
d. Template #4: SLO Process Template (found in SLO/Build/Templates)
e. Template #5: Performance Task Framework (found in SLO/Build/Templates)
2. A key point for teachers is that the Review phase is iterative, meaning that quality
assurance criteria are designed to inform corrective action rather than simply assign
a score. Likewise, the performance measure quality requires at least one full
administration to determine basic psychometric characteristics of newly developed
assessments. Validity evidence builds upon successful iterations of the assessment
“lifecycle.”
Learning Activity
Module 3-SLO Reviewing
3

4. Session Objectives
Concept
The three-tier quality assurance review process of each SLO includes the following:
• Completeness
• Using the “Quality Assurance Checklist” (Handout #4) and the SLO Procedural “Cheat Sheet”
(Handout #5), educators will determine the completeness of the SLO Process Template
(Template #4).
• Comprehensiveness
• Using the Performance Measure Rubric (found in ALS/Review/Stuff) and the SLO Procedural
“Cheat Sheet” (Handout #5), educators will evaluate the quality of each teacher-made
performance measure.
• Coherency
• Using the SLO Coherency Rubric (found in SLO/Review/Templates) and the SLO Procedural
“Cheat Sheet” (Handout #5), educators will evaluate the entire SLO Process.
Key Points for Trainers
1.
2.
Emphasize that all performance measures are evaluated in terms of high quality and rigor. The
Performance Measure Rubric for Teachers (found in ALS/Review/Stuff) will guide educators through
each phase of the build-out; Design, Build, and Review.
Reinforce the 3 C’s concepts (Completeness, Comprehensiveness, Coherency) so the participants
understand the reason each aspect (“C”) must be considered before statements regarding “high
quality” SLO can be made and supported.
Learning Activity
1.
Have the participants spend 5-10 minutes reviewing and discussing in their small groups the different
tools and how they are used in evaluating an SLO. After this exploration, have the members select a
model and “test drive” Handout #4 (Quality Assurance Checklist-SLO).
Module 3-SLO Reviewing
4

5. Task Structure
Concept
The first step in the Review Process is to check for the completeness of the SLO Process Template. This
step ensures that the SLO Process Template is complete. While reviewing, any incomplete, incorrect, or
missing element will be highlighted and potential corrective actions will be identified. The “Help Desk”
(found in SLO/Build/Stuff) provides detailed information on filling in the template.
The second step in the Review Process is to check the comprehensiveness of the performance measures
selected and/or developed. Educators will be guided through a process to review each aspect of the
Performance Measure: Design, Build, Review (checking for quality). The Performance Measure RubricTeacher (found in ALS/Review Stuff) and Quality Assurance Screening Tool are helpful. For vendor-made
assessments, a more detailed rubric is available.
The final step in the Review Process is to check the coherency of the SLO Process. Coherency is about
examining the “parts” of a teacher’s SLO and verifying the alignment to the Big Idea. The SLO Coherency
Rubric (found in SLO/Review/Templates) focuses on examining how the components fit into a cogent
process for applying evidence of student achievement to teacher evaluation.
Key Points for Trainers
1.
2.
Clarify that when evaluating the entire SLO Process (coherency) educators will be checking to ensure
that everything is aligned with the Goal Statement and Targeted Content Standards. Handout #5 SLO
Procedural “Cheat Sheet” and the SLO Coherency Rubric will aid educators through the process and
identify any issues that will need to be addressed.
Emphasis that completing the form correctly is a relatively straight forward task and the evaluation of
a performance measure’s technical data is much more complex and time consuming. It
(Comprehensiveness) should be viewed as a two stage process for both the SLO and the performance
measures within the SLO: before administration and after administration.
Learning Activity
Module 3-SLO Reviewing
5

6. Design Coherency
Concept
This figure is designed to help visualize how the “parts” fit into the “whole.”
Key Points for Trainers
1. Explain that this figure is designed to help visualize how the “parts” fit into the “whole.” It is
essential for the audience to fully understand how only a coherent SLO approach should be
used within the greater Educator Effectiveness System.
• Goal Statements are the “big idea” upon which the SLO is based. These goals must
be based upon Pennsylvania standards and provide a rationale for choosing a
particular standard which addresses the important student learning.
• Performance Indicators are the expected levels of achievement for students in the
SLO population.
• Performance Measures are the various tools/assessments which will be used to
measure student achievement toward a specific goal.
2. A quality discussion can emerge from this slide by having the participants identify validity
threat associated with student samples, assessment quality, and indicator rigor. It is
essential to the “Elective” rating’s validity characteristics that systematic errors are not
introduced into the SLO process at either the component level or in how the components
are aligned to the SLO Goal Statement.
Learning Activity
Module 3-SLO Reviewing
6

7. Helpful Tools
Concept
As noted previously, many resources are available for educators to use during the Review process. They
include the following.
1. Handout #4 – Quality Assurance Checklist-SLO (found in SLO/Review/Guides)
2. Handout #5 – SLO Procedural “Cheat Sheet” (found in SLO/Review/Guides)
3. Template #6 – SLO Coherency Rubric (found in SLO/Review/Templates)
4. Performance Measure Rubric for Teachers (found in ALS/Review/Stuff)
5. Refinement Control Checklist (on-demand Performance Measures only; found in SLO/Review/Stuff)
All training documents and materials can be found on the SAS Portal and RIA’s Homeroom learning portal.
Go to http://www.pdesas.org and you will see a link to RIA’s Homeroom portal as well.
Key Points for Trainers
1.
Differentiate how tools can assist in the building task:
a. Guides have materials such as handouts, rules of thumb, model SLOs, etc. that reinforce
content presented in the videos. In this module, models are provided to guide new users in
what a completed SLO, including developed and/or selected performance measures.
b. Templates are used to complete each phase of the process. The key template is SLO Process
Template 10.0 (available as a Word document-Template #4- and as an online tool in
SLO/Build/Templates); however, Template #5 is provided (found in SLO/Build/Templates)
for performance tasks that may be culminating events over multiple days (e.g., student
projects). Other development templates are found within the Assessment Literacy
Series/Review.
Learning Activity
1.
Have participants retrieve and review the Handouts and Templates associated with the Review
Process.
Module 3-SLO Reviewing
7

8. Final Deliverables
Upon completion of Step 3: Review; educators will have a completed SLO Process Template that identifies the
rigorous, high-quality performance measures as well as any area that was identified through the process needing
improvement.
Key Points for Trainers
1.
2.
3.
Articulate the expected deliverables for teacher-made performance measures:
a. Purpose Statement
b. Targeted Standards
c. Specifications and Blueprints
d. Scoring Keys and Rubrics
e. Operational Forms (tests/tasks)
f. Administrative Guidelines
Expected deliverable for vendor-made performance measures:
a. This deliverable will depend on the measure adopted by the school and/or district (i.e., Dibels, AIMS
Math, etc.); however, the vendor will often have written evidence and data supporting the
measurement claims. Educators should simply reference this information, thus demonstrating that the
selection of the measure meets the intended need of the educator (i.e., to measure the targeted
content standards within the “Big Idea”).
Using the Performance Measure Rubric for Teachers (found in ALS/Review/Stuff), the QA Screening Tool (found
in ALS/Review/Stuff) and the Quality Assurance Checklist (Handout #10, found in ALS/Review/Guides),
participants will be able to identify/document any shortcoming found and determine corrective actions needed.
Learning Activity
1.
Probe the audience for their understanding of what is meant by a “rigorous performance measure” using the
below set of questions:
a. To what degree does the performance measure align with targeted standards?
b. What is the cognitive demand of the tasks/items, both individually and in totality? (Refer to Depth of
Knowledge Chart found in ALS/Build/Guides and DoK Chart II found in ALS/Build/Stuff).
c. Is the performance measure’s reading level and vocabulary age and grade appropriate?
d. Are the targeted skills appropriate?
e. Are the items/tasks clear, concise, and understandable?
f. Are the items/tasks free of bias?
g. Are the items/tasks sensitive to different cultures, religions, ethnic and socio-economic groups?
Module 3-SLO Reviewing
8

9. Generic Review Workflow
Concept
This slide provides a general overview of the workflow for the Review Process.
“Work flow” within this context means the step-by-step Review process needed to ensure high-quality SLOs have been created.
Key Points for Trainers
1.
2.
3.
4.
5.
6.
Step 1: Packet Inventory –
a.
Checking to make sure that all required documents are included in the packet for review.
i.
Template #4 – SLO Process Template (found in SLO/Build/Templates)
ii.
Template #5 – Performance Task Framework (for each Performance Indicator; found in SLO/Build/Templates)
iii.
Teacher-developed Performance Measures, specifications/blueprints, scoring sheets, administrative guides
Step 2: Review for Completeness – Using the SLO Procedural “Cheat Sheet” (Handout #5; found in SLO/Review/Guides) and the Quality Assurance Checklist-SLO (Handout #4; found in SLO/Review/Guides),
participants will:
a.
Review each section of the SLO Process Template to make sure it has been completed correctly.
b.
Highlight any missing or incomplete elements.
c.
Circle any incorrect element and outline the discrepancy.
d.
Recommend corrective actions.
Step 3: Evaluate SLO Template for Comprehensiveness – Using the Performance Measure Rubric for Teachers (found in ALS/Review/Stuff), participants will evaluate the quality of each teacher-made
performance measure. This rubric is comprised of 18 technical aspects of quality assessment organized into three (3) strands; Strand 1: Design, Strand 2: Build, Strand 3: Review. Participants will rate each
element and list supporting technical evidence, provide further explanations and details where needed, and make any recommended actions for shortcomings.
a.
Strand 1: Design – has five (5) elements
b.
Strand 2: Build – has six (6) elements
c.
Strand 3: Review – has seven (7) elements
Step 4: Evaluate Performance Measure Quality – Another step in assuring performance measure quality is to use the following:
a.
QA Screening Tool (found in ALS/Review/Stuff)
a.
Part I – Screening
i.
Ensures that all deliverables are included for the performance measure.
b.
Part II – Form/Item Rigor
i.
Ensures that the measure is developmentally appropriate.
ii.
Ensures that items are aligned with targeted content standards.
iii.
Ensures items have been screened for sensitivity, bias, and fairness.
c.
Part III – Quality
i.
Ensures that specifications/blueprints are aligned with operational forms.
ii.
Administrative guidelines are clear and standardized.
iii.
Accommodation guidelines are provided/referenced.
iv.
Scoring guidelines are standardized.
v.
Scoring rubrics provide detailed scoring information.
b.
Quality Assurance Checklist (Handout #10; found in ALS/Review/Guides))
i.
Provides a review for specifications and blueprints, scoring key and rubrics, operational forms, and administrative guides to ensure that performance measures meet
the criteria for high-quality assessments.
Step 5: Determine Rating/Score Coherency – Using the SLO Coherency Rubric (found in SLO/Review/Templates), participants will review the entire SLO Process. This rubric has several technical
requirements divided into three (3) Strands: Design, Technical, and Quality. Through this review process, participants will check for alignment among all the elements (goal statement, targeted content
standards, performance indicators, performance measures), identify any issues, and make recommendations for corrective actions.
a.
Strand 1: Design – has five (5) elements
b.
Strand 2: Technical – has six (6) elements
c.
Strand 3: Quality – has four (4) elements
Step 6: Identify Corrective Actions – There are several places where corrective actions can be identified and documented throughout this process. The following documents can be used for this purpose:
a.
SLO Procedural “Cheat Sheet” (Handout #5)
b.
SLO Coherency Rubric (Template #6)
c.
Quality Assurance Checklist-SLO (Handout #4)
d.
Performance Measure Rubric for Teachers
e.
Quality Assurance Checklist (Handout #10)
f.
QA Screening Tool
Learning Activity
Module 3-SLO Reviewing
9

10. Modified (with calibration) Review Workflow
Concept
The Train-the-Trainer approach allows Pennsylvania to “calibrate” additional trainers to ensure consistency throughout the state. It
provides an opportunity for district professional development staff, school administrators, and lead teachers to thoroughly
understand the SLO Process and ensure that it is being implemented in classrooms across the state with fidelity.
Key Points for Trainers
1.
2.
3.
4.
5.
Once educators have completed the Review procedure and if they have identified any corrective actions, it is imperative that a
process on how and when those identified corrective actions will occur is determined.
Dimension 1: Template for Completeness
a.
Apply a quality assurance checklist to the developed materials to ensure all documents meet the requirements.
Dimension 2: Template for Comprehensiveness
a.
Take the Performance Measure Rubric for Teachers (found in ALS/Review/Stuff) and evaluate the quality of each
teacher-made performance.
b. Reviewing forms involves a content review; a review for bias, sensitivity, fairness, and alignment.
c.
Content review:
•
Determine if each item/task clearly aligns to the targeted content standards.
•
Evaluate all items for accuracy.
•
Evaluate if items are age-appropriate.
•
Ensure that each item and answer options are clear and understandable.
d. Item Bias:
•
Bias-free items/tasks provide an equal opportunity for all students to demonstrate their knowledge and
skills.
e. Fairness and Sensitivity:
•
Ensure items are sensitive to different cultures, religions, ethnic and socio-economic groups, and
disabilities.
•
Ensure there is a balance of gender roles.
•
Ensure that positive language, situations, and images are used for performance measures.
•
Ensure that items avoid text that may elicit strong emotions in specific groups of students and prevent
those students from accurately demonstrating their skills and knowledge.
Dimension 3: Rating/Score Coherency
a.
Using the SLO Coherency Rubric, the entire SLO Process is evaluated to determine whether the technical
requirements have been met.
Dimension 4: Performance Measure Quality
a.
Applying a quality assurance checklist to the developed performance measures to ensure all documents meet the
basic principles of quality assessment.
Learning Activity
Module 3-SLO Reviewing
10

11. Calibration Procedures
Concept
The Calibration Procedures increase standardization among SLOs while strengthening
SLO review teams’ technical skills. Participants should have the skills and knowledge
related to the basic principles of quality assessment practices.
Quality Assurance is a critical step in the SLO Process. It is important that procedures are
in place to ensure that SLOs and Performance Measures are selected/developed
properly. Quality Assurance procedures provide a way for reviewers to do that.
Key Points for Trainers
1. Participants will review each packet through four (4) different lenses (dimensions). It
is important to identify any missing or incomplete areas and develop
recommendations for corrective actions. If needed, seek clarifications from
facilitators. Misinterpretations of the SLO criteria or any element throughout the
process can lead to unintended consequences (i.e., developmentally inappropriate
performance measures being used with students is both unfair to the students and
to the teachers who will be using the student outcome data to determine their
Elective rating on the Teacher Effectiveness Rating Tool).
Learning Activity
Module 3-SLO Reviewing
11

12. Summary
Concept
During Module 3 participants reviewed, refined, and corrected the SLOs and the
Performance Measures associated with each Performance Indicator. At this point,
teachers should have all completed templates that were required throughout the entire
SLO Process.
Key Points for Trainers
1. This professional development provided participants with a better understanding of
several quality assurance practices used in the SLO Process.
2. As noted previously, the SLO process was created to meet the need of Act 82, which
requires PA to have an Educator Effectiveness Rating Tool that contains measures
based on teacher observation and practice as well as multiple measures of student
performance. SLO is the process that develops an Elective Rating through evidence of
student achievement.
3. The SLO Process also assists educators in developing/selecting fair, valid, high-quality
measures to evaluate student achievement.
Learning Activity
Module 3-SLO Reviewing
12