1. Inter-Rater Reliability in
Teacher Evaluation

2. Welcome!
• Jenny Froehle
– ZCS District Lead Administrator for Rise
– Secondary Evaluator of 80 Faculty
• Matt Walter
– Primary Evaluator of 20 HS Teachers
– 1of 5 Evaluators for 90 HS Faculty
• Using RISE Model
– 345Certified Faculty Evaluated
– 21 Evaluators

3. Assumptions
• This Year is Different Than the Past!
– New Tool/Rubric for Most
– More Evaluations
– Higher Stakes ($)
• Learning as we go
– New Laws
– New Job Dynamics
• Applies to Any Evaluation Model

4. Why Is This Important?
• Consistent Message About What Your
District Values Most
– Evaluations Lead to Conversations
– Great Opportunities for PD
• Clarity of Expectations
• Fairness to Teachers
– They Will Ask…And They Should!

5. Consider This…
Highly Effective Effective Improvement
Necessary
Ineffective
Students are able
to answer higher-
level questions
with meaningful
responses
Teacher frequently
develops higher-
level
understanding
through effective
questioning
Some questions
used may not be
effective in
developing higher-
level
understanding (too
complex or
confusing)
Teacher may not
use questioning as
an effective tool to
increase
understanding.
Students only
show a surface
understanding of
concepts.
RISE Competency 2.6

6. Does she get an “E”?

7. Turn & Discuss Video
Highly Effective Effective Improvement
Necessary
Ineffective
Students are able
to answer higher-
level questions
with meaningful
responses
Teacher frequently
develops higher-
level
understanding
through effective
questioning
Some questions
used may not be
effective in
developing higher-
level
understanding (too
complex or
confusing)
Teacher may not
use questioning as
an effective tool to
increase
understanding.
Students only
show a surface
understanding of
concepts.

8. Legal Implications
• IC 20-28-11.5-8 (LAW)
State board actions; model plan; approval of plan by teachers
Sec. 8. (a) To implement this chapter, the state board shall do the
following:
(1) Before January 31, 2012, adopt rules under IC 4-22-2 that establish
(D) an acceptable standard for training evaluators.
• 511 lAC 10-6-3 Evaluator training (RULE)
– Authority: IC 20-19-2-8; IC 20-28-11.5-8
– Affected: IC 20-28; IC 20-31-4; IC 20-35-5; IC 20-37-1-1; IC 36-1-7

9. The Rule Says:
Any individual responsible for collecting evidence
toward summative evaluations must be provided with
training on how to collect and analyze evidence. Such
training may include, but is not limited to, training
incorporated into professional development programs,
supervisor-led training, or virtual training. In developing
training programs, a school corporation must
incorporate mechanisms to assess evaluators'
improvement in collecting and using evidence.

10. The Rule Says:
Any individual responsible for summative evaluations
shall be provided with training on how to evaluate
evidence and how to make a final summative judgment.
Such training may include, but is not limited to, training
incorporated into professional development programs,
supervisor-led training, or virtual training. In developing
training programs, a school corporation must incorporate
mechanisms to assess evaluators' competence in
collecting and using evidence.

11. Beyond Legal Compliance
• Our Team Has Grown By:
– Operating as a PLC of leaders
– Having conversations about what we really
want to see in our classrooms
• What did we used to talk about?
– Taking command of our own PD
• We are now training ourselves
• Our daily practice is the focus of our meetings

12. What We Have Done
• CIESC Training (RISE)
– Mostly Procedural
– Limited in Creating a Shared Understanding
• Cheat Sheets
– Simplified Rubric Language, focused on “E”
– Clarification of Rubric Wording
• Analyzed Real Scripts & Comments

13. What We Have Done
• Video Analysis of our own teachers
– Mini-Rubric
– Focused on Specific Indicators
– Effective vs. Improvement Necessary
– To Mark or Not to Mark?
– Data Capture and Discussion via Google
Form
• Let’s Try One!

14. Data Capture
• Google Form
– Go to www.zionsvilleeagles.org
– Find the “April 25-26 Conference” Link at the
bottom of the page.
– Enter your ratings into the form
– Live data and instant feedback
– https://docs.google.com/forms/d/1RFuXsMrHe
Re8aiBhPr8uiW4O6zKJ2bF2jjkOmLI2Eto/vie
wform

15. Rate Him on the Mini-Rubric

16. Data Tracking and Analysis
HE E IN I UNM Total Date Desc % Marked % UNM% Pos % Neg
2.3.2 15 1 0 1 17
26-
FebKG 94.1% 5.9% 93.8% 6.3%
2.3.3 1 8 2 0 6 17
18-
Dec
HS
Math 64.7% 35.3% 81.8% 18.2%
2.3.3 2 7 4 2 4 1914-JanHS Art 78.9% 21.1% 60.0% 40.0%
2.3.3 16 1 0 0 17
26-
FebKG 100.0% 0.0% 94.1% 5.9%

17. Challenges
• Ambiguous Rubric Wording
• Everyone Brings a Unique Viewpoint
• Video vs. Real Person
– Not the same feel as a classroom visit
– No coaching relationship
• Mini-Rubric limits choices
• Building Consensus on Details

18. What’s Next?
• Norming Summative Ratings
• Administrators observing across buildings
– Seeing the same teachers
– Discussion afterward
• Engaging teachers in observations
• Talking about “What does that look like?”

19. Questions & Discussion
Thank you!