Aspects of Appraisal - Getting it Right
•Download as PPTX, PDF•
0 likes•305 views
- Teacher appraisal and feedback can lead to moderate or large changes in teaching practices, confidence, and job satisfaction for teachers. - Common methods for teacher appraisal include classroom observations, student surveys, test scores, self-assessments, and parent feedback. - Effective teacher appraisal systems set clear expectations, require accurate observations, average feedback over multiple lessons, and track reliability at the system level. - Value-added measures of teacher quality based on student test scores have limitations and may not consistently reflect teacher quality over time or account for external factors.
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (13)
Similar to Aspects of Appraisal - Getting it Right
Similar to Aspects of Appraisal - Getting it Right (20)
Aspects of Appraisal - Getting it Right
- 1. Setting the context The Importance of Getting Teacher Appraisal Right
- 2. 2 An excellent teacher can raise test scores by as much as 50% compared to the average teacher
- 3. Average: 62% A moderate or large change in their teaching practices after they received feedback 3Source: OECD, 2013
- 4. Average: 71% A moderate or large change in their confidence as a teacher after they received feedback 4Source: OECD, 2013
- 5. Average: 63% A moderately or large change in their job satisfaction 5Source: OECD, 2013
- 6. Average : 47% Feedback is provided to teachers based on a thorough assessment of their teaching 6Source: OECD, 2013
- 7. Average: 51% Teacher appraisal and feedback are largely done to fulfil administrative requirements 7Source: OECD, 2013
- 8. 8 Average: 31% If a teacher is consistently underperforming, he/she would be dismissed Source: OECD, 2013
- 9. Average : 38% The best performing teachers in this school receive the greatest recognition 9Source: OECD, 2013
- 10. Planning and Preparation Classroom Environment Instruction Professional Responsibilities Teaching 10Source: Danielson, Framework for Teaching (1996, 2007) A consistent and objective set of measures
- 11. Teacher’s professional development Contribution to school development Links to external partners or the community Contribution to extra- curricular activities Professional Responsibilities 11 Additional responsibilities
- 12. Frequency of appraisal for performance management 12 0 5 10 15 20 25 30 35 40 45 Never <1 every 2 years Once every 2 years Once per year Twice or more per year Frequency of formal appraisal by the school principal (%) 0 5 10 15 20 25 30 35 40 45 Never <1 every 2 years Once every 2 years Once per year Twice or more per year Frequency of formal appraisal by the school management team (%)
- 13. 13 76% Content knowledge 79% Student Surveys 95% Test Scores 95% Classroom observation 89% Parent feedback 81% Self-Assessment Commonly used information sources
- 16. Appraisal Methods Classroom Observation Objective Setting and Interviews Teacher Self- Appraisal 16
- 19. Appraisal Methods Classroom Observation Objective Setting and Interviews Teacher Self- Appraisal Teacher’s Portfolio Pupil and Parent Surveys Gains in Student Results 19
- 20. Appraisal Methods Classroom Observation Objective Setting and Interviews Teacher Self- Appraisal Teacher’s Portfolio Pupil and Parent Surveys Gains in Student Results 20 Correlatio n: 0.24 95% source: MET (2012)
- 21. Improving Classroom Observation 1. Set clear expectations 2. Require observers to demonstrate accuracy 3. Average over multiple lessons 4. Track system-level reliability 5. Regularly verify link to student achievement gains Source: Measures of Effective Teaching (2012)21
- 22. Appraisal Methods Classroom Observation Objective Setting and Interviews Teacher Self- Appraisal Teacher’s Portfolio Pupil and Parent Surveys Gains in Student Results 22 Correlatio n: 0.37 79% source: MET (2012)
- 23. Appraisal Methods Classroom Observation Objective Setting and Interviews Teacher Self- Appraisal Teacher’s Portfolio Pupil and Parent Surveys Gains in Student Results 23 Correlatio n: 0.69 95% source: MET (2012)
- 24. 24 Assumptions needed: • Teachers should be unaffected by their working environment • Allocation of pupils needs to be random • Value added is equal regardless of the baseline • Teachers are equally effective with all pupils Is VA a truly unbiased measure of teacher quality?
- 25. Is VA a consistent measure over time? Source: Based on 941 teachers, Koedel and Betts (2007) 25
- 26. 26 Percentage increase in the chance of an average teacher being misjudged as exceptionally good or bad when using a single year measure 25% Source: Schochet and Chiang (2010) Is VA a consistent measure over time?
- 27. 27 If VA test scores placed a teacher in the bottom 3% then there was an 80% probability they would be classified in the bottom quartile on all measures Is VA an accurate reflection of teacher quality?
- 28. 28 Conclusions
Editor's Notes
- In my presentation this morning I am going to look at why it is important to get teacher appraisal right, I will look at how appraisal is currently perceived by, I will examine the common components of appraisal systems teachers throughout the world and then explore in detail how effective some of those components are in judging the quality of teaching. I will conclude by stating why appraisal is a key element of school improvement.
- It is generally accepted that you cannot have a good school without good teaching – and therefore addressing the quality of teaching should be at the centre of any school improvement programme. The recent availability of good quality data has enabled the link between good teaching and gains in test scores. One such study showed that an excellent teacher raises maths scores by 20% and when adjusted for poorer students the gains were as much as 50% compared to the average teacher. For a school to ensure it has as many of these excellent teachers as possible it must be able to identify areas for improvement for good teachers to enable them to become excellent. This is where appraisal systems come into play and why it is important to get teacher appraisal right. For an appraisal system to be effective it must be accurate and trusted by the staff as a fair and objective system so that they will look to improve the areas of weakness identified by appraisal. Appraisal is not just used for this formative type of appraisal but also summative appraisal which can impact on pay and progression. This reinforces the need for objectivity and consistency. So let’s look at the current state of play and find out what teachers think of the appraisal systems in place at the moment by looking at the OECD, TALIS reports which survey teachers and schools throughout the world.
- When asked if teachers changed their teaching practices after appraisal an encouraging 62% of teachers surveyed reported a moderate or large change in their teaching practices after they received feedback – this is evidence that appraisal can have a direct impact on teaching in a school. The lowest impact was reported by Finland at just 38% whilst the highest impact was reported by Malaysia at 95%.
- 71% of teachers surveyed reported a moderate or large change in their confidence as a teacher after receiving feedback and…
- … 63% reported an increase in their job satisfaction. When the morale of the teaching profession is so often cited in the media as a cause for concern it is important to see how appraisal and feedback can help improve confidence and job satisfaction whilst also bringing about a change in teaching practice. However, it is not all good news…..
- When we look at the perceived quality of the appraisal systems used throughout the world then the majority of teachers surveyed think the systems used are not rigorous enough. The results were somewhat skewed by the responses of the nine or so countries at the far right of this graph. Taking Finland as an example only 17% of Finnish teachers felt that feedback was based on a thorough assessment of their teaching and perhaps as a result only 38% of Finnish teachers reported changing their teaching practice based on their appraisal – the lowest score in the study.
- Disappointingly over half of teachers surveyed agreed with the statement that appraisal and feedback was largely done to fulfil administrative requirements. Again looking at Finland we can see that 62% of teachers felt that appraisal was simply a tick box exercise.
- This cynicism was supported by the result that only 31% of respondents thought that a consistently underperforming teacher would be dismissed, in Finland only 16% of teachers felt that an underperforming colleague would be dismissed.
- And only 25% of Finnish teachers felt that the best performing teachers in their schools received the recognition they deserved, compared to an average of 38%. The obvious conclusion from these results is that even if an appraisal system is good schools need to follow through take action on the results or teachers will lose faith in the system. It is clear from these surveys that current appraisal systems require improvement. In Finland especially which was once the top performing education system in PISA league tables, teachers appear to have little faith in their appraisal systems. Finland has slipped in its position in the PISA tables and perhaps addressing the area of appraisal may help them climb back towards the top. I will now take some time to look at the frameworks for appraisal, the frequency of appraisal and the most commonly used components of appraisal systems throughout the world. I will then then examine some of the research relating to these components to see how accurate they are as a reflection of teaching quality and how the measures can be improved.
- According to a recent survey 22 of the 28 countries surveyed reported having national policy frameworks for teacher appraisal. The countries where no state mandated framework was in place often devolved responsibility to the school or board level. The challenge for countries with de-centralised systems is to hold schools accountable for implementing effective teacher appraisal systems. A fair and robust teacher appraisal system needs reference standards against which teachers are evaluated. Without a consistent set of standards an appraiser will be open to bias. There is an additional requirement to have a shared understanding of what is considered to be accomplished teaching – this can happen at the national, district or even school level. Danielson’s framework for teaching groups teachers’ responsibilities into four major areas: planning and preparation; instruction; the classroom environment; and professional responsibilities. Elements of this framework form the basis for many countries’ sets of teaching standards and criteria for assessment. In addition to these core principles different country systems assess aspects such as work efficiency, interpersonal skills, and authority and even in France their level of “radiance”.
- However, teachers’ responsibilities extend beyond the classroom and it is appropriate that they are assessed against additional responsibilities that are less directly related to teaching. For example, in Australia in addition to state teaching standards the teacher code of conduct is also used as a reference. Now let’s look at how frequently schools conduct appraisal.
- There is considerable diversity in the how frequently teachers are appraised for performance management although the majority (65%) are appraised at least once a year by the principle. The frequency of appraisal should be adjusted for teachers at different stages of their career or where intervention is needed for performance issues. A frequent light touch approach to appraisal which is able to adapt to circumstances where there is cause for concern would seem to be the most appropriate approach. Let’s now look at the sources of evidence that schools rely on when conducting appraisal.
- In the OECD survey we can see that test scores and classroom observation appear are almost ubiquitous in appraisal systems throughout the world. Other common sources of evidence reported through the survey were content knowledge, parent feedback, self assessment and student surveys. These numbers do not reveal the weighting given by the school principals to each of these evidence sources nor do they provide any evidence as to which are the more effective at judging teacher quality. I will now spend some time looking at the main appraisal methods used by schools throughout the world and then return to a few of them to test their ability to judge teaching quality by considering the relevant research.
- Most key aspects of teaching are displayed while teachers interact with their students in the classroom. Almost all countries use classroom observations for regular performance management and it is frequently used at the end of the probation period. Schools will also use observation as part of the recruitment process for new staff. In Korea and Israel classroom observation is used both for appraisal for promotion and for their performance-based incentive programme. In Portugal observations are optional as part of regular performance management but required for the awarding of “very good” or “excellent” marks and for certain promotions.
- Most teacher appraisal models require the individual teacher to set performance objectives for a given period of time in agreement with the school leadership team – typically the appraisal then assesses the extent to which the objectives were met. These interviews and meetings are usually carried out on an individual basis and are intended to be reflective discussions between appraisers and teachers. They are used for regular performance management and at the end of registration and probation period. The interview is also used as a formative assessment tool to enable both sides to agree on the professional development goals for the teacher. It may also act as a way to communicate performance issues that need to be addressed. The frequency of objective setting can be as frequent as termly but extend to beyond two years depending on the jurisdiction and the individual school.
- Requesting that the teacher being appraised evaluates his or her own performance is considered an essential part of appraisal by Schleicher in his report for the OECD and through Educate’s experience in schools we wholeheartedly agree with this approach. Self-appraisal encourages teachers to reflect on their own performance and can be used to measure themselves against a set of standards. It is not without its difficulties as our anecdotal evidence suggests that excellent teachers underestimate the quality of their teaching whilst the opposite is true for underperforming teachers. By evaluating performance against a set of standards both parties establish a baseline of performance which allows for changes in performance over time to be measured. Most countries adopt self-appraisal as part of their regular appraisal regime where it is used for performance management purposes.
- Portfolios are used for all types of appraisal and can include lesson plans, teaching materials, samples of students’ work together with examples of marking, comment and feedback, reflection sheets and questionnaires. Portfolios can be used for both summative and formative appraisals – researchers have observed that portfolios that focus on teacher development enhance professional outcomes. School leaders should consider the work involved in creating a portfolio and try and ensure that it dovetails with the teachers own work, e.g. it could include planning work for a lesson the teacher is actually teaching, they should avoid creating additional bureaucracy and paperwork.
- Studies have shown how surveys, especially student surveys, can be especially useful for appraising the quality of teaching, and are used systematically in formal teacher appraisals in countries such as Mexico, New Zealand and the Slovak Republic. Canada uses parental surveys and Korea uses a multi-dimension parent and student survey for regular teacher appraisal. Advocates recommend that the surveys are based on the opinions of pupils built up over the school year in order to make them harder to manipulate.
- Student learning outcomes are the essential measure of success for a school. Despite this there are some countries who judge the majority of their teachers at the highest rating and yet at the aggregate level the performance of their students is unsatisfactory. 65% of teachers polled reported that they considered student test scores to be an important part of the appraisal and feedback they received. Student learning outcomes, including student results in standardised assessments, are an appealing measure to assess teaching performance, since the ultimate goal of teaching is to improve student learning but their use for measuring performance remains contentious not least due to the challenge of isolating the specific contribution that an individual teacher makes to those outcomes. Just some of the other contributory factors are the student’s own innate skills, expectations, motivation and behaviour; the support received at home; the influence of their peer group; their school environment and resources and the structure of the curriculum. In studies Headteachers were found it difficult to adjust or filter for these external factors and tended to be swayed by higher absolute scores rather than the value added by the teacher. Some systems look at student outcomes in a broader sense, for example performance could be measured for groups of students as well as the entire class, or a group of struggling pupils could be identified and specific goals addressed to them. In New York State student learning objectives are used to measure a teacher’s contribution to students’ progress in all subjects. Teachers receive guidance in setting appropriate learning objectives and the measure of that progress constitutes 40% of teacher-evaluation scores whilst other state-approved measures, such as classroom observations, surveys and portfolios, constitute the remaining 60%. I will now take some time to look in depth at three commonly used areas of appraisal – classroom observation, pupil surveys and value added test scores to see how accurate they are as appraisal instruments and how they can be improved.
- Research shows that a teachers’ history such as teaching experience, undergraduate university attended, teacher certification and current tenure are all poor predictors of a person’s teaching ability; in fact one found that a teachers’ history explained less than 8% of teacher quality. So it is no surprise that we look to the classroom to assess the quality of teaching. Research in New York found that even a single lesson observation of a trainee teacher was a significant predictor of later teacher quality and a strong predictor of future pupil test score gains. As the observation period grew longer, the reliability of the observational measures increased. The implication is that subjective evaluations contained meaningful information about a teacher’s effectiveness that is not captured in value-added measures. Further research found that classroom observation scores are strongly linked to gains in pupil test scores for established teachers as well as trainee teachers. However, there are pitfalls– poorly trained observers were more likely to rate teachers highly if that teacher had received high evaluation ratings in the past. In order for observation to be effective the observer must be properly prepared, know what to look for and be able to provide effective feedback and keep subjective opinions to a minimum. Unsurprisingly where observations are made over a long period of time they make it harder to “game” the system and evidence showed that ratings of teachers were significantly lower in unscheduled observations than scheduled ones. Although classroom observation is widely used the MET study showed that despite being significantly correlated to teacher performance they were one of the least accurate measures of long-run teacher performance. In order to improve this correlation the study recommended the following :
- Choose an observation instrument that sets clear expectations: Define a set of teaching competencies and providing specific examples at different performance levels 2. Require observers to demonstrate accuracy before they rate teacher practice: Teachers need to know observers will be fair and accurate. 3. When high-stakes decisions are being made, multiple observations are necessary: Averaging over multiple lessons reduces spurious evaluations. 4. Track system-level reliability by double scoring some teachers with impartial observers: To ensure reliability and keep teacher support, evaluations should be compared with those from external observers. 5. Regularly verify that teachers with stronger observation scores also have stronger student achievement gains on average: Even a great observation instrument can be implemented poorly.
- The MET study found that pupil surveys correlated more with future pupil outcomes than classroom observations even when the observations were conducted by highly trained and independent observers. The debate is to what extent do surveys simply reflect the pupil’s expected test scores rather than their actual learning, for example high ability students know they are going to get good grades and so score their teacher highly thus providing high correlation but questionable causality. Another potential flaw in the surveys is that teachers who teach to the test rather than engage in real teaching may score highly as they provide the lowest effort route to good grades. This can be accounted for in the questions posed in the survey to tease out good teaching practice, such as questions on the frequency of homework and marking. With their good correlation to student outcomes they ought to be part of the appraisal toolkit but perhaps treated with some caution.
- Finally, let’s return to the contentious issue of gains in student test scores, since it is used in 95% of teacher appraisal systems it is worth addressing its ability to judge teacher quality.
- Richard Murphy in his report for the Sutton Trust asked the question whether value added test scores were an unbiased measure of teacher quality and suggested that in order for it to be unbiased you needed to make the following assumptions : Firstly teachers need to be unaffected by their working environment? – he found that they are affected but this can be accounted for, especially when comparing teachers within the same school. Secondly, the allocation of pupils to teachers needs to be random? – this is only appears to be a problem when focusing on single year measures from one class. Thirdly, test scales should be invariant and linear – meaning that value added is equal regardless of the baseline performance of the pupils being measured Finally, for the measure to be unbiased teachers should be equally effective with all pupils - studies show that a teacher’s impact on student learning can be dependent on the pupil-teacher match So the author found that value added test scores could show elements of bias but when the scores were adjusted for these factors they were not significantly different on a qualitative basis from unadjusted basic scores. So these elements of bias do not disqualify valued added test scores as a measure of teacher quality. He then posed the question as to whether value added is a consistent measure over time.
- If teachers were equally effective every year – and test scores were an accurate reflection of pupil ability – all teachers would stay in the same quintile from one year to the next. However a study by Koedel and Betts found that a majority of teachers move between performance quintiles each year. 13% of teachers in top quintile in the first year ended up in the bottom quintile the following year and conversely 13% of the worst performers were the best performers the following year.
- Further studies how that existing single year VA measures are not stable. A 2010 study found a 25% increase in the chance of an “average” teacher being labelled as exceptionally bad (or good) when using a single year measure compared to a three year measure. This difference in consistency was also demonstrated by the McCaffery study which showed stability increased between 40 and 60% when aggregating data over two years and a further 18 to 23% when a third year was included. So the clear message is that value added test measures can be consistent but only when we look at results over a number of years. He then asked the question as to whether value added measures are an accurate reflection of teacher quality.
- Value Added test scores are good at identifying teachers at the extremes – they can identify the bottom and top 5% of teachers but they struggle to accurately identify teachers around the mean when single year scores are used. Again, in order to increase accuracy significant improvements are made by using three year averages. The clear message from these studies is that value added test scores must be analysed over longer periods of time and with as large a group of pupils as possible to smooth out the noise and volatility from single year results. Despite these shortcomings a study by Chetty in 2011 tracked one million children from the 4th grade and found that in the long run students assigned to higher VA teachers are more likely to attend college, earn higher salaries and live in better neighbourhoods.
- So, in conclusion the research shows that some of the most common appraisal tools used by schools can be validly used as accurate judgments of teaching quality, especially when they are combined. Value added test scores have a strong correlation but in order to get it right they must be used over an average of a number of years to filter out the effects of noise created by small samples sizes. When combined with other forms of evidence such as classroom observations and student surveys, schools have the tools available to make accurate judgments on the quality of teaching in their school which in turn enables them to identify areas of development for teachers. Creating an objective and transparent system with appropriate feedback will result in teachers improving their teaching which will lead to improved student outcomes but there is still work to be done to ensure this is happening in more schools. After the break we will hear from three experts on appraisal in their jurisdictions – we have Billie Miller from the Charter Schools USA, Professor Kai Ming Cheng from the University of Hong Kong and Nigel Middleton from Educate. We will now take a break for coffee, if you could be back in the room for 10.30 we will start with what promises to be a very interesting session.