In many training programs, successful completion relies on scores provided by evaluators. For effective scoring, evaluators must be consistent with their efforts. You can measure the consistency of their scoring practices with inter-rater reliability. In this session, explore a method that determines consistency in evaluator practices, the implications of inconsistent evaluator scoring and ways to improve inconsistent evaluator practices. Speaker: Kendra Lester, JetBlue Airways

1. Kendra Lester | Instructional Designer
MEASUREMENT
OF EVALUATOR
CONSISTENCY

2. EVALUATION ACTIVITY

4. EVALUATION ACTIVITY DEBRIEF

5. SESSION OVERVIEW
Review evaluator consistency case study
Share lessons learned from case study
Recommend ways to improve evaluator consistency

6. FLIGHT ATTENDANT TRAINING ELEMENTS

7. INTER-RATER RELIABILITY
“…consistency with which different
examiners produce similar ratings in
judging the same abilities or
characteristics in the same target
person or object…”
Psychology
Today
Wikipedia “…degree of agreement among
raters…”

8. CONSTRUCT VS. CALCULATION
SAME
MEASURE
COHEN’S KAPPA

9. CASE STUDY: DEVELOPMENT
Consult
evaluators
Draft
scenarios
Review
scenarios

10. CASE STUDY: IMPLEMENTATION
Read
scenarios
Evaluate
scenarios
Debrief
ratings

11. CASE STUDY: SCENARIO EXCERPT
Flight Attendant #3 starts with Jumpseat C, opening
the under-seat compartment and explaining that she
is checking for the presence of the MRT and life
vest. The flight attendant also ensures that the
copper wire is intact on the Halon but doesn’t seem
to look very closely at the pressure gauge. The flight
attendant touches the flashlight strap and says that
the flashlight should blink every few seconds…

12. Scenario #1 Evaluator Guide
CASE STUDY: EVALUATION TOOLS
Scenario #1 Evaluator Form

13. CASE STUDY: SAMPLE RATINGS
5
9
4 44
5 5
0
1
2
3
4
5
6
7
8
9
10
n03qtmsv1 k07qtmsv1 k08qtmsv1 k06qtmsv1
CONDUCT PRE-FLIGHT
CHECKS
EMERGENCY EQUIPMENT
FAMILIARIZATION
EMERGENCY
EQUIPMENT LOCATIONS
OPERATOR POLICIES
AND PROCEDURES
Satisfactory Score
Unsatisfactory Score

14. LESSONS LEARNED

15. RECOMMENDATIONS

16. QUESTIONS/COMMENTS

17. THANK YOU!