1. STUDENT LEARNING OBJECTIVES (SLOs)
Annotated Notes
M3-Reviewing
Slide No.
Notes
Slide 1
The Student Learning Objective (SLO) Process is comprised of three (3) components:
Design, Build, and Review. Student Learning Objectives provide a valid assessment of
teacher effectiveness through student performance outcomes based on standards.
Welcome to Training Module 3: “Reviewing”
Concept
Within the Review Module, the trainer will guide participants to implement a
multifaceted, three-level review of Student Learning Objectives, including identified
performance measures.
Key Points for Trainers
1. Explain in macro-terms the “big picture” of how SLOs are but one component of a
much larger teacher effectiveness system.
a. As required by Act 82, an Educator Effectiveness Rating Tool must contain
measures based on teacher observation and practice and multiple
measures of student performance. Student Performance data will include
the following: Building Level Data (15%), Teacher Specific Data (15%), and
Elective Data (20% for teachers with eligible PVAAS data, 35% for teachers
without eligible PVAAS data).
b. This Educator Effectiveness Rating Tool functions as a framework for the
evaluation and summative process for classroom teachers.
2. Clarify that the Review Module will guide participants through a three-level quality
assurance review of the created Student Learning Objective and the identified
Performance Measures. The Review phase helps ensure that the operational and
technical expectations of the process are met or exceeded.
3. Ensure participants understand that this phase is the quality assurance component,
focusing on issues that range from completing the SLO Process Template correctly
to examining the validity evidence of a vendor-made assessment/performance
measure.
Learning Activity
_______________________________________________________________________
Technical Notes
“Structure”
PDE SLO Annotated Notes-M3 Reviewing
1

2. Slide No.
Notes
1. Concept – “What is this slide telling the audience?”
2. Key Points – “What/Where are the details ‘needed for teaching’?”
3. Learning Activity – “How can the participant’s learning be enhanced?” (This item
will not be populated for every slide.)
Slide 2
Student Learning Objective
Concept
As you work through the Review Process, you will need to access the following
materials:
1) Handout #4 – Quality Assurance Checklist-SLO (found in SLO/Review/Guides)
2) Handout #5 – SLO Procedural “Cheat Sheet” (found in SLO/Review/Guides)
3) Template #6 – SLO Coherency Rubric (found in SLO/Review/Templates)
4) Performance Measure Rubric for Teachers (found in ALS/Review/Stuff)
5) Refinement Control Checklist (on-demand Performance Measures only, found in
SLO/Review/Stuff)
Key Points for Trainers
1. Assist the participants in conceptualizing that the SLO Process Template is
organized into two major components: SLO Goals and Performance Indicators on
page 1 of the template, and Performance Measures and Expectations on page 2.
2. Clarify that the comprehensive evaluation of a new performance measure cannot
be fully completed until after the measure is administered and data is collected.
The current tools focus on the development and administration of the measures,
with additional training being developed for the “post-administration” review. For
vendor-made measures, there should be sufficient evidence on the inferences of
scores, reliability, etc. that also includes how the performance measure was
developed.
3. Stress to participants that the Review process includes a quality assurance check on
each Performance Measure selected to ensure that they are aligned with the
content standards, free of bias and sensitive content, and developmentally
appropriate.
Learning Activity
1. Have participants go to www.pdesas.org to retrieve the five items mentioned
above from the SAS Portal /Homeroom learning portal. Also make sure they have
the completed SLO Process Template (Template #4) found in SLO/Build/Templates.
PDE SLO Annotated Notes-M3 Reviewing
2

3. Slide No.
Notes
Slide 3
Process Components
Concept
This slide reviews the parts associated with each phase of the SLO Process. This Module
will focus on the Review Phase of the SLOs.
Key Points for Trainers
1. It will be important for each educator to have all the completed templates from
their work with Module 1: Design and Module 2: Build. That would include the
following templates:
a. Template #1: Goal Statement (found in SLO/Design/Templates)
b. Template #2: Targeted Standards (found in SLO/Design/Templates)
c. Template #3: SLO Blueprint (found in SLO/Design/Templates)
d. Template #4: SLO Process Template (found in SLO/Build/Templates)
e. Template #5: Performance Task Framework (found in SLO/Build/Templates)
2. A key point for teachers is that the Review phase is iterative, meaning that quality
assurance criteria are designed to inform corrective action rather than simply assign
a score. Likewise, the performance measure quality requires at least one full
administration to determine basic psychometric characteristics of newly developed
assessments. Validity evidence builds upon successful iterations of the assessment
“lifecycle.”
Learning Activity
Slide 4
Session Objectives
Concept
The three-tier quality assurance review process of each SLO includes the following:
• Completeness
• Using the “Quality Assurance Checklist” (Handout #4) and the SLO
Procedural “Cheat Sheet” (Handout #5), educators will determine the
completeness of the SLO Process Template (Template #4).
• Comprehensiveness
• Using the Performance Measure Rubric (found in ALS/Review/Stuff) and the
SLO Procedural “Cheat Sheet” (Handout #5), educators will evaluate the
quality of each teacher-made performance measure.
• Coherency
• Using the SLO Coherency Rubric (found in SLO/Review/Templates) and the
SLO Procedural “Cheat Sheet” (Handout #5), educators will evaluate the
entire SLO Process.
PDE SLO Annotated Notes-M3 Reviewing
3

4. Slide No.
Notes
Key Points for Trainers
1. Emphasize that all performance measures are evaluated in terms of high quality
and rigor. The Performance Measure Rubric for Teachers (found in
ALS/Review/Stuff) will guide educators through each phase of the build-out; Design,
Build, and Review.
2. Reinforce the 3 C’s concepts (Completeness, Comprehensiveness, Coherency) so
the participants understand the reason each aspect (“C”) must be considered
before statements regarding “high quality” SLO can be made and supported.
Learning Activity
1. Have the participants spend 5-10 minutes reviewing and discussing in their small
groups the different tools and how they are used in evaluating an SLO. After this
exploration, have the members select a model and “test drive” Handout #4 (Quality
Assurance Checklist-SLO).
Slide 5
Task Structure
Concept
The first step in the Review Process is to check for the completeness of the SLO Process
Template. This step ensures that the SLO Process Template is complete. While
reviewing, any incomplete, incorrect, or missing element will be highlighted and
potential corrective actions will be identified. The “Help Desk” (found in
SLO/Build/Stuff) provides detailed information on filling in the template.
The second step in the Review Process is to check the comprehensiveness of the
performance measures selected and/or developed. Educators will be guided through a
process to review each aspect of the Performance Measure: Design, Build, Review
(checking for quality). The Performance Measure Rubric-Teacher (found in ALS/Review
Stuff) and Quality Assurance Screening Tool are helpful. For vendor-made assessments,
a more detailed rubric is available.
The final step in the Review Process is to check the coherency of the SLO Process.
Coherency is about examining the “parts” of a teacher’s SLO and verifying the alignment
to the Big Idea. The SLO Coherency Rubric (found in SLO/Review/Templates) focuses on
examining how the components fit into a cogent process for applying evidence of
student achievement to teacher evaluation.
Key Points for Trainers
1. Clarify that when evaluating the entire SLO Process (coherency) educators will be
checking to ensure that everything is aligned with the Goal Statement and Targeted
Content Standards. Handout #5 SLO Procedural “Cheat Sheet” and the SLO
PDE SLO Annotated Notes-M3 Reviewing
4

5. Slide No.
Notes
Coherency Rubric will aid educators through the process and identify any issues
that will need to be addressed.
2. Emphasis that completing the form correctly is a relatively straight forward task and
the evaluation of a performance measure’s technical data is much more complex
and time consuming. It (Comprehensiveness) should be viewed as a two stage
process for both the SLO and the performance measures within the SLO: before
administration and after administration.
Learning Activity
Slide 6
Design Coherency
Concept
This figure is designed to help visualize how the “parts” fit into the “whole.”
Key Points for Trainers
1. Explain that this figure is designed to help visualize how the “parts” fit into the
“whole.” It is essential for the audience to fully understand how only a coherent
SLO approach should be used within the greater Educator Effectiveness System.
• Goal Statements are the “big idea” upon which the SLO is based. These
goals must be based upon Pennsylvania standards and provide a rationale
for choosing a particular standard which addresses the important student
learning.
• Performance Indicators are the expected levels of achievement for
students in the SLO population.
• Performance Measures are the various tools/assessments which will be
used to measure student achievement toward a specific goal.
2. A quality discussion can emerge from this slide by having the participants identify
validity threat associated with student samples, assessment quality, and indicator
rigor. It is essential to the “Elective” rating’s validity characteristics that systematic
errors are not introduced into the SLO process at either the component level or in
how the components are aligned to the SLO Goal Statement.
Learning Activity
PDE SLO Annotated Notes-M3 Reviewing
5

6. Slide No.
Notes
Slide 7
Helpful Tools
Concept
As noted previously, many resources are available for educators to use during the
Review process. They include the following.
1. Handout #4 – Quality Assurance Checklist-SLO (found in SLO/Review/Guides)
2. Handout #5 – SLO Procedural “Cheat Sheet” (found in SLO/Review/Guides)
3. Template #6 – SLO Coherency Rubric (found in SLO/Review/Templates)
4. Performance Measure Rubric for Teachers (found in ALS/Review/Stuff)
5. Refinement Control Checklist (on-demand Performance Measures only; found in
SLO/Review/Stuff)
All training documents and materials can be found on the SAS Portal and RIA’s
Homeroom learning portal. Go to http://www.pdesas.org and you will see a link to
RIA’s Homeroom portal as well.
Key Points for Trainers
1. Differentiate how tools can assist in the building task:
a. Guides have materials such as handouts, rules of thumb, model SLOs, etc. that
reinforce content presented in the videos. In this module, models are provided
to guide new users in what a completed SLO, including developed and/or
selected performance measures.
b. Templates are used to complete each phase of the process. The key template
is SLO Process Template 10.0 (available as a Word document-Template #4- and
as an online tool in SLO/Build/Templates); however, Template #5 is provided
(found in SLO/Build/Templates) for performance tasks that may be culminating
events over multiple days (e.g., student projects). Other development
templates are found within the Assessment Literacy Series/Review.
Learning Activity
1. Have participants retrieve and review the Handouts and Templates associated with
the Review Process.
PDE SLO Annotated Notes-M3 Reviewing
6

7. Slide No.
Notes
Slide 8
Final Deliverables
Upon completion of Step 3: Review; educators will have a completed SLO Process
Template that identifies the rigorous, high-quality performance measures as well as any
area that was identified through the process needing improvement.
Key Points for Trainers
1. Articulate the expected deliverables for teacher-made performance measures:
a. Purpose Statement
b. Targeted Standards
c. Specifications and Blueprints
d. Scoring Keys and Rubrics
e. Operational Forms (tests/tasks)
f. Administrative Guidelines
2. Expected deliverable for vendor-made performance measures:
a. This deliverable will depend on the measure adopted by the school and/or
district (i.e., Dibels, AIMS Math, etc.); however, the vendor will often have
written evidence and data supporting the measurement claims. Educators
should simply reference this information, thus demonstrating that the
selection of the measure meets the intended need of the educator (i.e., to
measure the targeted content standards within the “Big Idea”).
3. Using the Performance Measure Rubric for Teachers (found in ALS/Review/Stuff),
the QA Screening Tool (found in ALS/Review/Stuff) and the Quality Assurance
Checklist (Handout #10, found in ALS/Review/Guides), participants will be able to
identify/document any shortcoming found and determine corrective actions
needed.
Learning Activity
1. Probe the audience for their understanding of what is meant by a “rigorous
performance measure” using the below set of questions:
a. To what degree does the performance measure align with targeted
standards?
b. What is the cognitive demand of the tasks/items, both individually and in
totality? (Refer to Depth of Knowledge Chart found in ALS/Build/Guides and
DoK Chart II found in ALS/Build/Stuff).
c. Is the performance measure’s reading level and vocabulary age and grade
appropriate?
d. Are the targeted skills appropriate?
e. Are the items/tasks clear, concise, and understandable?
f. Are the items/tasks free of bias?
g. Are the items/tasks sensitive to different cultures, religions, ethnic and
socio-economic groups?
PDE SLO Annotated Notes-M3 Reviewing
7

8. Slide No.
Notes
Slide 9
Generic Review Workflow
Concept
This slide provides a general overview of the workflow for the Review Process.
“Work flow” within this context means the step-by-step Review process needed to
ensure high-quality SLOs have been created.
Key Points for Trainers
1. Step 1: Packet Inventory –
a. Checking to make sure that all required documents are included in the
packet for review.
i.
Template #4 – SLO Process Template (found in
SLO/Build/Templates)
ii.
Template #5 – Performance Task Framework (for each Performance
Indicator; found in SLO/Build/Templates)
iii.
Teacher-developed Performance Measures,
specifications/blueprints, scoring sheets, administrative guides
2. Step 2: Review for Completeness – Using the SLO Procedural “Cheat Sheet”
(Handout #5; found in SLO/Review/Guides) and the Quality Assurance Checklist-SLO
(Handout #4; found in SLO/Review/Guides), participants will:
a. Review each section of the SLO Process Template to make sure it has been
completed correctly.
b. Highlight any missing or incomplete elements.
c. Circle any incorrect element and outline the discrepancy.
d. Recommend corrective actions.
3. Step 3: Evaluate SLO Template for Comprehensiveness – Using the Performance
Measure Rubric for Teachers (found in ALS/Review/Stuff), participants will evaluate
the quality of each teacher-made performance measure. This rubric is comprised of
18 technical aspects of quality assessment organized into three (3) strands; Strand
1: Design, Strand 2: Build, Strand 3: Review. Participants will rate each element and
list supporting technical evidence, provide further explanations and details where
needed, and make any recommended actions for shortcomings.
a. Strand 1: Design – has five (5) elements
b. Strand 2: Build – has six (6) elements
c. Strand 3: Review – has seven (7) elements
4. Step 4: Evaluate Performance Measure Quality – Another step in assuring
performance measure quality is to use the following:
a. QA Screening Tool (found in ALS/Review/Stuff)
a. Part I – Screening
i.
Ensures that all deliverables are included for the
performance measure.
b. Part II – Form/Item Rigor
i.
Ensures that the measure is developmentally appropriate.
ii.
Ensures that items are aligned with targeted content
PDE SLO Annotated Notes-M3 Reviewing
8

9. Slide No.
Notes
standards.
Ensures items have been screened for sensitivity, bias, and
fairness.
c. Part III – Quality
i.
Ensures that specifications/blueprints are aligned with
operational forms.
ii.
Administrative guidelines are clear and standardized.
iii.
Accommodation guidelines are provided/referenced.
iv.
Scoring guidelines are standardized.
v.
Scoring rubrics provide detailed scoring information.
b. Quality Assurance Checklist (Handout #10; found in ALS/Review/Guides))
i.
Provides a review for specifications and blueprints, scoring key and
rubrics, operational forms, and administrative guides to ensure
that performance measures meet the criteria for high-quality
assessments.
5. Step 5: Determine Rating/Score Coherency – Using the SLO Coherency Rubric
(found in SLO/Review/Templates), participants will review the entire SLO Process.
This rubric has several technical requirements divided into three (3) Strands:
Design, Technical, and Quality. Through this review process, participants will check
for alignment among all the elements (goal statement, targeted content standards,
performance indicators, performance measures), identify any issues, and make
recommendations for corrective actions.
a. Strand 1: Design – has five (5) elements
b. Strand 2: Technical – has six (6) elements
c. Strand 3: Quality – has four (4) elements
6. Step 6: Identify Corrective Actions – There are several places where corrective
actions can be identified and documented throughout this process. The following
documents can be used for this purpose:
a. SLO Procedural “Cheat Sheet” (Handout #5)
b. SLO Coherency Rubric (Template #6)
c. Quality Assurance Checklist-SLO (Handout #4)
d. Performance Measure Rubric for Teachers
e. Quality Assurance Checklist (Handout #10)
f. QA Screening Tool
iii.
Learning Activity
Slide 10
Modified (with calibration) Review Workflow
Concept
The Train-the-Trainer approach allows Pennsylvania to “calibrate” additional trainers to
ensure consistency throughout the state. It provides an opportunity for district
professional development staff, school administrators, and lead teachers to thoroughly
understand the SLO Process and ensure that it is being implemented in classrooms
across the state with fidelity.
PDE SLO Annotated Notes-M3 Reviewing
9

10. Slide No.
Notes
Key Points for Trainers
1. Once educators have completed the Review procedure and if they have identified
any corrective actions, it is imperative that a process on how and when those
identified corrective actions will occur is determined.
2. Dimension 1: Template for Completeness
a. Apply a quality assurance checklist to the developed materials to ensure all
documents meet the requirements.
3. Dimension 2: Template for Comprehensiveness
a. Take the Performance Measure Rubric for Teachers (found in
ALS/Review/Stuff) and evaluate the quality of each teacher-made
performance.
b. Reviewing forms involves a content review; a review for bias, sensitivity,
fairness, and alignment.
c. Content review:
• Determine if each item/task clearly aligns to the targeted content
standards.
• Evaluate all items for accuracy.
• Evaluate if items are age-appropriate.
• Ensure that each item and answer options are clear and
understandable.
d. Item Bias:
• Bias-free items/tasks provide an equal opportunity for all students
to demonstrate their knowledge and skills.
e. Fairness and Sensitivity:
• Ensure items are sensitive to different cultures, religions, ethnic
and socio-economic groups, and disabilities.
• Ensure there is a balance of gender roles.
• Ensure that positive language, situations, and images are used for
performance measures.
• Ensure that items avoid text that may elicit strong emotions in
specific groups of students and prevent those students from
accurately demonstrating their skills and knowledge.
4. Dimension 3: Rating/Score Coherency
a. Using the SLO Coherency Rubric, the entire SLO Process is evaluated to
determine whether the technical requirements have been met.
5. Dimension 4: Performance Measure Quality
a. Applying a quality assurance checklist to the developed performance
measures to ensure all documents meet the basic principles of quality
assessment.
Learning Activity
PDE SLO Annotated Notes-M3 Reviewing
10

11. Slide No.
Notes
Slide 11
Calibration Procedures
Concept
The Calibration Procedures increase standardization among SLOs while strengthening
SLO review teams’ technical skills. Participants should have the skills and knowledge
related to the basic principles of quality assessment practices.
Quality Assurance is a critical step in the SLO Process. It is important that procedures
are in place to ensure that SLOs and Performance Measures are selected/developed
properly. Quality Assurance procedures provide a way for reviewers to do that.
Key Points for Trainers
1. Participants will review each packet through four (4) different lenses (dimensions).
It is important to identify any missing or incomplete areas and develop
recommendations for corrective actions. If needed, seek clarifications from
facilitators. Misinterpretations of the SLO criteria or any element throughout the
process can lead to unintended consequences (i.e., developmentally inappropriate
performance measures being used with students is both unfair to the students and
to the teachers who will be using the student outcome data to determine their
Elective rating on the Teacher Effectiveness Rating Tool).
Learning Activity
Slide 12
Summary
Concept
During Module 3 participants reviewed, refined, and corrected the SLOs and the
Performance Measures associated with each Performance Indicator. At this point,
teachers should have all completed templates that were required throughout the entire
SLO Process.
Key Points for Trainers
1. This professional development provided participants with a better understanding of
several quality assurance practices used in the SLO Process.
2. As noted previously, the SLO process was created to meet the need of Act 82, which
requires PA to have an Educator Effectiveness Rating Tool that contains measures
based on teacher observation and practice as well as multiple measures of student
performance. SLO is the process that develops an Elective Rating through evidence
of student achievement.
3. The SLO Process also assists educators in developing/selecting fair, valid, highquality measures to evaluate student achievement.
Learning Activity
PDE SLO Annotated Notes-M3 Reviewing
11