More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (16)
Similar to Feasibility of Using RATE with Teacher Candidates
Similar to Feasibility of Using RATE with Teacher Candidates (20)
More from Liz Fogarty
More from Liz Fogarty (20)
Recently uploaded
Recently uploaded (20)
Feasibility of Using RATE with Teacher Candidates
- 1. 2016 ANN BULLOCK, KRISTEN CUTHRELL, ELIZABETH FOGARTY, JOY STAPLETON EAST CAROLINA UNIVERSITY C O L L E G E O F E D U C A T I O N D E P A R T M E N T O F E L E M E N T A R Y E D U C A T I O N A N D M I D D L E G R A D E S E D U C A T I O N Feasibility of Using RATE with Teacher Candidates
- 2. • Recommends approximately 700 initial license candidates a year • 54% ELEM • Largest producer of teachers in NC • Highest employment rate in NC
- 3. Context Importance of assessment system with multiple measures of effectiveness Our home-grown assessments found to be invalid or unreliable measures of performance (Henry et al., 2013) Portfolio assessment Dispositions Observation Instrument
- 4. Rapid Assessment of Teacher Effectiveness RATE is a validated observation instrument in use with practicing teachers Draws on existing teacher effectiveness constructs Observe classroom lesson videos; Results in less expensive teacher observation Focuses on direct instruction RATE design has 6 items, each item is assessed on a 3 point scale (Gargani and Strong,2014)
- 5. Methods Proof of concept study
- 6. Research Questions RQ 1: Does the RATE observation instrument’s reliability and validity hold when used with pre- service teachers? RQ2: Is the RATE observation instrument a feasible instrument to be used by large teacher education programs? RQ3: Do you have to use two raters in order for the rater scores to remain reliable and valid?
- 7. Procedures - Videos Preservice Teachers: ELEM candidates submit full lesson videos to an electronic portfolio system during final semester of internship identify their subject, topic, and 20 consecutive minutes of direct instruction to be viewed 179 videos were used for scoring in this study
- 8. Procedures- Training March 2015 Strong and Gargani held an orientation session with 38 faculty, university supervisors, instructional coaches and COE administrators Focus was on introducing faculty to the RATE instrument May 2015 Day 1: 38 participants received training on the RATE instrument Day 2: participants scored videos
- 9. Procedures - Scoring 8 30 minute sessions were held on Day 2 Within each 30 minute session a participant would score the 20 minute video individually then meet with the other person who scored the video. Scores were compared and reconciled 150/179 videos were scored twice using both rater pairs and solo raters At a later date- RATE personnel evaluated all videos for comparison purposes.
- 10. Comparing ECU Faculty independent scores to RATE independent scores
- 11. Paired Raters Arriving at Same Scores
- 12. Independent VS Reconciled Sum Scores
- 13. Conclusions RQ 1: Discrepancies in paired scores exist but rater reliability was higher than in initial RATE validation study Biggest discrepancy: clarity and flow RQ 2: Instrument is shorter and less complex than existing instrument However, having two individuals rate one video seems prohibitive in a large program. RQ 3: To maintain reliability and validity two raters are needed.
- 14. Next Steps Pilot RATE instrument with preservice teachers in live observations (spring 2016) Determine predictive nature of RATE with preservice teacher observation Explore relationship between RATE and CLASS instruments Explore relationship and possible predictive nature of RATE with edTPA
- 15. References Cuthrell, K., Stapleton, J., Bullock, A., Lys, D., Smith, J., and Fogarty, E. (2014). Mapping the journey of reform and assessment for an elementary education teacher preparation program. Journal of Curriculum and Instruction, 8(1). Gargani, J., & Strong, M. A. (2014). Can we identify a successful teacher better, faster, and cheaper? Evidence for innovating teacher observation systems. Journal of Teacher Education. Pre-print copies available online. Good, T., & Lavigne, A. (2015). Response to "Rating Teachers Cheaper, Faster, and Better: Not So Fast": It's about evidence. Journal of Teacher Education, 0: 0022487115587110v1- 22487115587110 Henry, G. T., Campbell, S. L., Thompson, C. L., Patriarca, L. A., Luterbach, K .J., Lys, D. B., & Covington, V. M. (2013). The predictive validity of measures of teacher candidate programs and performance: Toward an evidence-based approach to teacher preparation. Journal of Teacher Education 64, 439-453. Sawchuk, S. (2011). University of Michigan project scales up "high leverage" teaching practices. Education Week. Retrieved from the www on August 31, 2012 at http://blogs.edweek.org/edweek/teacherbeat/2011/10/high_leverage_teaching_practices.html Strong, M., Gargani, J., & Hacifazlioğlu, Ö. (2011). Do we know a successful teacher when we see one? Experiments in the identification of effective teacher. Journal of Teacher Education, 62(4), 367-382. Strong, M. (2011). The Highly Qualified Teacher: What is Teacher Quality and How Do We Measure It? New York: Teachers College Press.
- 16. KRISTEN CUTHRELL: CUTHRELLMA@ECU.EDU JOY STAPLETON: STAPLETONJ@ECU.EDU AERA 2016 EAST CAROLINA UNIVERSITY Feasibility of Using RATE with Teacher Candidates
Editor's Notes
- COE Innovations were borne out of a 9 million dollar TQP grant and allowed for the evaluation partnership with EPIC mentioned earlier. The goals of the
- The lack of ability to make reliable assumptions about our candidates based on available data drove both program leadership and faculty to look for a more reliable method for assessing the readiness of teacher candidates edTPA, Dispositions form that is enacted 3 times during the program, by multiple evaluators Follow up conversations with university supervisors revealed that they were reluctant to score a teacher candidate down in an area on the instrument because they were trying to be supportive, thus not using the observation instrument as an evaluation tool but rather a developmental tool with teacher candidates. The use of the observation instrument for different purposes led to grade documentation issues with teacher candidates. For example, many times at the end of the semester the “scores” on the observation instrument would not show any deficits but the supervisor would want to give the teacher candidate a lower grade. Without evidence of reduced performance from the observation instrument, giving the intern a lower grade was not always possible. This data-driven knowledge led us to investigate other reliable and valid observation instruments.
- RATE design consists of six items relating to the lesson objective, instructional delivery mechanisms, teacher questioning strategies, clarity of presentation of concepts, time on task, and level of student understanding. Each item is assessed on a 3-point scale, and evaluators are encouraged to jot down notes to support their scores. Widely accepted frameworks include Danielson (2011) Framework for Teaching; Teaching Works Framework (Sawchuck, 2011), the Gates Foundation Measures of Effective Teaching (MET) project (Danielson, 2010); and Marzano’s Classroom Instruction that Works (Marzano, Pickering, and Pollock, 2009).
- In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
- In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
- As a program requirement, teacher candidates submitted a final teaching video at the end of their internship semester. Full lesson videos were submitted to Taskstream, an electronic portfolio platform. Upon submission, candidates were asked to identify their subject, topic, and 20 consecutive minutes of direct instruction captured in the full lesson video. These videos were the subject of this research project. In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos. In March 2015, Strong and Gargani provided an orientation session on the RATE instrument for 38 ECU faculty (across all content areas), university supervisors, school district instructional coaches, and college of education administration faculty. This session focused on introducing faculty to the RATE instrument and how it may be helpful for teacher candidates. Strong and Gargani returned in May 2015. On the first day, thirty-eight university faculty (across all content areas) participated in eight hours of orientation and training on the RATE instrument. In the second day, each participant scored 8 videos in 30 minute time segments (a four hour time period plus breaks). The 30 minute timeframe included viewing the video clip, recording the scores on the RATE instrument, then conferring with a partner who viewed the same video. During the conferring, scores were reconciled. Each of the 150 videos were scored twice using both rater pairs and solo raters. Off site, trained RATE personnel evaluated all videos for comparison purposes.
- In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos. In March 2015, Strong and Gargani provided an orientation session on the RATE instrument for 38 ECU faculty (across all content areas), university supervisors, school district instructional coaches, and college of education administration faculty. This session focused on introducing faculty to the RATE instrument and how it may be helpful for teacher candidates. Strong and Gargani returned in May 2015. On the first day, thirty-eight university faculty (across all content areas) participated in eight hours of orientation and training on the RATE instrument. In the second day, each participant scored 8 videos in 30 minute time segments (a four hour time period plus breaks). The 30 minute timeframe included viewing the video clip, recording the scores on the RATE instrument, then conferring with a partner who viewed the same video. During the conferring, scores were reconciled. Each of the 150 videos were scored twice using both rater pairs and solo raters. Off site, trained RATE personnel evaluated all videos for comparison purposes. In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
- In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos. In March 2015, Strong and Gargani provided an orientation session on the RATE instrument for 38 ECU faculty (across all content areas), university supervisors, school district instructional coaches, and college of education administration faculty. This session focused on introducing faculty to the RATE instrument and how it may be helpful for teacher candidates. Strong and Gargani returned in May 2015. On the first day, thirty-eight university faculty (across all content areas) participated in eight hours of orientation and training on the RATE instrument. In the second day, each participant scored 8 videos in 30 minute time segments (a four hour time period plus breaks). The 30 minute timeframe included viewing the video clip, recording the scores on the RATE instrument, then conferring with a partner who viewed the same video. During the conferring, scores were reconciled. Each of the 150 videos were scored twice using both rater pairs and solo raters. Off site, trained RATE personnel evaluated all videos for comparison purposes. In total, 179 university videos were used for scoring. Of the 179, 150 were scored by both members of a pair. Of the 150 videos scored by two raters, only 70 videos were watched with no flags. Flags consisted of video issues including the video being too long, too short, the rater knowing the teacher in the video, and/or audio issues. For the purposes of this study, some of the analyses used all 150 videos while others used 70 videos.
- ECU faculty tended to rate teacher candidates higher on the lesson objective and student thinking portion of the videos. They tended to score teacher candidates lower on checking for understanding, clarity of delivery, flow of instruction, and time on task. While there was good agreement among raters in most areas, flow of instruction seemed to show the biggest discrepancy. The cause of this discrepancy is not immediately apparent and warrants further examination.
- It appears that discussing ratings with another person brings the extreme raters closer back to the norm. The closer the score was to the norm, the less it moved. The closer it was to the norm, the more it moved. This finding especially holds for ECU raters. RATE team scores were less variable across the rater.
- Open circle is independent score Close is revised sum scores
- Even though discrepancies existed, rater reliability was higher in this study than in the RATE’s (VS7) validation study and the MET study conducted by the Gates Family Foundation. The biggest discrepancies occurred in the ECU faculty’s rating of clarity and flow of the lesson. Further exploration into the faculty’s interpretation of these categories is needed before that discrepancy can be explained. The instrument itself is shorter and less complex than the current version of the instrument and requires only a 20 minute observation. One concern coming into the proof of concept study was that two raters were needed for each video. Given the large number of individuals within the program, having two individuals rate each person seems prohibitive. However, data indicate that having two raters for each video tends to hold weakly justified extreme scores in check. Since the raters are trained to do each session in a 30 minute time period, the process may actually end up taking the same amount of time, if not less time. Continuing research is being conducted on observation practices that are effective and whether RATE is an observation instrument that can best support candidate growth in challenging clinical experiences. Being able to use the RATE instrument in live setting with teacher candidates would help teacher education programs have a reliable and valid outcome measure of their teachers. This measure in addition to other reliable outcomes measures, such as, the edTPA, would strengthen programs’ ability to predict the success of their graduates.