Principles of assessment (Belinda Cerdá, Cambridge English) - Simposio CiEnglish
•Download as PPTX, PDF•
2 likes•939 views
Presentación del Simposio de Ciencias e Inglés en Evaluaciones Internacionales celebrado los días 27 y 28 de noviembre en el IES Ramiro de Maeztu.
More Related Content
What's hot
What's hot (20)
Similar to Principles of assessment (Belinda Cerdá, Cambridge English) - Simposio CiEnglish
Similar to Principles of assessment (Belinda Cerdá, Cambridge English) - Simposio CiEnglish (20)
More from Instituto Nacional de Evaluación Educativa
More from Instituto Nacional de Evaluación Educativa (20)
Recently uploaded
Recently uploaded (20)
Principles of assessment (Belinda Cerdá, Cambridge English) - Simposio CiEnglish
- 1. The Principles of Assessment Belinda Cerdá, Cambridge English
- 2. Who are you? 54 28 0 10 20 30 40 50 60 Primary School children Secondary School children Do you teach: Primary School children Secondary School children
- 8. Tests the level and is not too difficult and not too easy Realistic, authentic and meaningful Student friendly Should be pretested and well constructed Relevant Motivating Gives students and teachers info about what they can do What makes a good exam? Must test real life skills and language It must measure what it claims to measure
- 9. What makes a good exam? Cover the four skills Not too long Clear Relevance and significance for people They have to be fair and objective Reliable Tests students´ knowledge fairly and accurately It must teach not only grade Precisely tailored and interesting questions that motivate students and allow them to be creative at the same time Is a mix of open and closed questions
- 10. How can we be sure we are getting it right?
- 11. VRIP Validity = fitness for purpose
- 13. Measurement scale 90 80 70 60 50 40 30 Intermediate Preliminary Breakthrough Standards consistently applied Test 3 Test 1 Test 2 Tests at appropriate level Item bank links all levels . . . . . . Learners located on scale An item banking system
- 14. VRIP Validity = fitness for purpose Reliability = repeatability
- 15. Exam day
- 17. VRIP Validity = fitness for purpose Reliability = repeatability Impact= effect on individuals and society Practicality = Cost? Time? Resource?
- 18. VRIP A candidate´s essay is marked by 2 different examiners and gets wildly different results A test of English that contains a mathematical problem Tests that cost too much to deliver The high % of candidates receiving an ‘A’ grade means that universities cannot choose between the candidates
- 19. VRIP R = A candidate’s essay is marked by 2 different examiners and gets wildly different results V = A test of English that contains a mathematical problem P = Tests that cost too much to deliver I = The high % of candidates receiving an ‘A’ grade means that universities cannot choose between the candidates
- 22. Question paper production cycle Commissioning of material Pre-editing Pretesting & Trialling Pretest Review Test Construction Exam Overview Editing Rejection Revision
- 24. What do you want to take from this event? Knowledge and strategies to further teaching English as a foreign language New ideas to improve the way I assess my students, to share with colleagues the way we assess Practical ideas to assess the students more efficiently
- 25. fdsfdsfsf What do you want to take from this event? Find out how teachers are feeling about bilingualism Sharing experiences and resources Strategies to prepare my students for the exams
- 27. Questions
Editor's Notes
- You know something about me but I also know something about you in the audience
- About 70% or 65% of you teach English in primary school
- Approximately 80% of you teach in state schools from Madrid and around Spain
- The majority of English teachers and probably science teachers too already teach in a bilingual programme ou do already
- Many of you are English and Science teachers but you also occupy other roles in the school…. Such as:
- Bilingual coordinators – in yellow and then in second place Heads of departments and other which may mean librarian, assessment technicians for regional governments perhaps…
- And finally how long you have been teaching…… Approx. 30% of you have been teaching between 7-15 years and then another 20% of you have been teaching for 20 plus years which means that we have a very experienced audience and I think this is great because it shows real dedication and commitment to continuous professional development and a desire to do what is right for your students
- Now we move on to talk about the Principles of Assessment and understanding these. iN order to do this, I wanted to see what you thought and posed the question in the survey: What makes a good exam. This was the second most important question we asked. In addition to asking you this question, we randomly asked teachers last week and produced a very short home video about what they said. I would like to share with you: Play video: So, They talked about …. GO THROUGH EACH ONE SLOWLY And when I asked you what you thought in the survey this is what you said and it was of course similar:
- So, essentially, these are the PRINCIPLES OF ASSESSMENT. One of the things that was mentioned more than once is this idea of balance and it is something I will come back to later. So I could stop there but I will do for the remaining 15 minutes is elaborate on some of the core principles and show how an international organisation such as Cambridge English ensures we adhere to all of the points mentioned by you.
- We have a commitment to students, teachers, parents and other stakeholders to design good exams and we do this in a useful framework which we refer to as VRIP Q. This should be a useful shorthand for assessment specialists to work with and work towards…..I will now go on to briefly explain what this is and how we ensure we ahere to this.
- Different academics may talk about these in different ways but at a basic level these all mean ‘fitness for purpose’ Validity = fitness for purpose. I will talk about three types of validity = construct validity, situational and criterion validity VALIDITY IN AN EXAM MEANS: They assess what they need to assess and do so in an authentic way (cognitive related validity and situational validity). They are an accurate reflection of a candidate’s true ability So a valid test should target the language abilities of interest (or the observable traits) and it should do so by providing a variety of task types and that is why we do not rely on Multiple choice items alone – we always have open and closed questions). For example if we look at the exams which Cambridge provides we see fitness for purpose. The suite of tests ensures that tests are neither too difficult nor too easy and reflect progression in learning and so targets the language abilities of interest.
- Item banking is the methodology that now supports all Cambridge English tests and exams. In the early 1990s we started constructing a single measurement scale to cover the whole range of exam levels. This enabled us to add precision and reliability to a system which had grown up over many years based on subjective expert judgment. Relates tests to a common frame of reference. Allows for creation of multiple equatable forms.
- Now let’s move on to look at reliability and this is essentially: The extent to which test scores are consistent and accurate. This means they are free from bias or random errors and can therefore be relied on for making decisions about test takers: How do we achieve this in Cambridge? We achieve this in many way but I will highlight two: One is through our test administration procedures which ensures that the experience for each candidate is the same.
- On the exam day itself, we have a series of standardised procedures to ensure that every candidate has a similar experience. This ensures reliability, since wherever and whenever the candidates take one of our exams, they will take the exam under similar conditions. CLICK We provide detailed instructions as to how to set up the exam room itself, including the distances between candidates, and what can and cannot be on display within the exam room. A record is kept of where each candidate was sitting for each part of the test, and sent with the answer sheets. CLICK We are very clear about what candidates can have with them at their desk during the exam. Candidates can have water (in a plastic bottle), writing materials and any medication they may need to take, but bags, school books and of course mobile phones must be left in a designated area away from the candidate. CLICK Candidates must be able to control the timings of their exam, so a clock must be visible in the exam room. At the beginning of the test, the invigilator writes on the board the start and end times for the test, using the clock on the wall, so every candidate has the same time reference. Invigilators announce to candidates when ten and five minutes remain before the end of the test.
- SLIDE:EXAMINER ASSESSMENT If we reconsider the teacher’s role during the writing process, this section can also be applied to the role that the teacher adopts at the end of the writing process. The key here is that the raters (writing assessors) should work to some agreed standard, so that all students / candidates are marked according to the same yardstick (as we saw at the beginning of the talk in the ideas put forward by Hamp-Lyons). For high-stakes tests, in order to ensure a positive impact, the process needs to be clear and reliable. To this end, Cam Eng has a system of recruitment, training and continual monitoring of examiner performance that allows us to maintain what we believe is a high level of reliability. Examiners have to meet strict professional standards and then successfully complete a course with an exam at the end. Their performance is then checked every year and there is follow up training as necessary. How is this managed by Cambridge English? STANDARDISATION All writing examiners are trained extensively on how to work with the marking criteria at the different CEFR levels before they begin working as writing examiners. They are also required to undergo refresher training every year. COORDINATION DUPLICATION Each candidate text is marked by two examiners independently. In the case that there is a wide discrepancy in the marks between the two examiners, the script will be marked by a third examiner.
- Impact = effect on individuals and society Assessment can have important effects and consequences both within education systems (test takers, universities, teaching and learning) And the wider society (employers, government agencies) What kind of impact do the tests have – on the candidates, on the teaching and learning process? Do they encourage good teaching and learning practice, do they encourage the development of skills in a way which will be useful in the world outside and to the candidates‘ development? Do they have positive washback in the widest sense? PRACTICALITY Practicality is all about how practical a test is in terms of the resources needed to produce and administer it in its intended context. Practicality covers the development, administration and validation of the examinations. For example: Do the number and complexity of items match the time allowed? Are the task replicable and sustainable? Can they be used again? Is the level of fees to be paid by test takers competitive? Does the availability of the exams in terms of dates, frequency and location meet the needs of the test takers and other stakeholders?
- Here are some examples about tests. One of these represents Validity, one reliability and so-on. Take a minute to think to yourself which is which? (only allow 20 seconds or so if that is enough)
- Here are ‘the answers’ R = A candidate’s essay is marked by 2 different examiners and gets wildly different results If a candidate’s essay would get 2 different results in this situation we can see that it would not be regarded as ‘fair’. V = A test of English that contains a mathematical problem We can see that this also would not be regarded as ‘fair’. If we state that we are testing English and maths then ok, but if we do not then this is not ok. P = Tests that cost too much to deliver If a school cannot afford to pay invigilators for the time required then the test will not be used. If we want people to be able to take our tests, and to extend access to more people then practicality must be considered. I = The high % of candidates receiving an ‘A’ grade means that universities cannot choose between the candidates This has an impact on society as universities have to come up with other ways of choosing students. Of course, these answers are not this straightforward. Each example impacts on other aspects of VRIP and each of these 4 qualities interacts with the others.
- We have a commitment to students, teachers, parents and other stakeholders to design good exams and we do this in a useful framework which we refer to as VRIP Q. This should be a useful shorthand for assessment specialists to work with and work towards…..I will now go on to briefly explain what this is and how we ensure we ahere to this.
- SLIDE:EXAMINER ASSESSMENT If we reconsider the teacher’s role during the writing process, this section can also be applied to the role that the teacher adopts at the end of the writing process. The key here is that the raters (writing assessors) should work to some agreed standard, so that all students / candidates are marked according to the same yardstick (as we saw at the beginning of the talk in the ideas put forward by Hamp-Lyons). For high-stakes tests, in order to ensure a positive impact, the process needs to be clear and reliable. To this end, Cam Eng has a system of recruitment, training and continual monitoring of examiner performance that allows us to maintain what we believe is a high level of reliability. Examiners have to meet strict professional standards and then successfully complete a course with an exam at the end. Their performance is then checked every year and there is follow up training as necessary. How is this managed by Cambridge English? STANDARDISATION All writing examiners are trained extensively on how to work with the marking criteria at the different CEFR levels before they begin working as writing examiners. They are also required to undergo refresher training every year. COORDINATION DUPLICATION Each candidate text is marked by two examiners independently. In the case that there is a wide discrepancy in the marks between the two examiners, the script will be marked by a third examiner.
- So to sum up: What makes a good exam was the second most important question. The most important question was: What do you want to take from this even and this is what you said: Knowledge and strategies – we aim to pass on a lot of strategies New ideas to improve the way I assess – we will be sharing ideas for assessing speaking and writing from A2 – B1 in tomorrow’s workshops. Practical ideas – we’ll have a look at practical ideas for assessing speaking and writing tomorrow 3 GOALS TOMORROW OUR GOAL TOMORROW IS TO RAISE AWARENESS OF THE ASSESSMENT CRITERIA and SCALES for assessing speaking and writing between A2-C1 in order to give you practice assessing real students highlight features of good speaking and writing items PROVIDE AN UNFORGETTABLE EXPERIENCE _ we sincerely hope you take a lot from tomorrow and are all, in the INEE and in Cambridge English, looking forward to working with you.
- Give you ideas for exam preparation Share your feelings and thoughts about topics and issues which affect you – this is something which you can take charge of in the opportunities for discussion and in the mingling and networking which will take place throughout the two days Sharing experiences and resources – We will be doing this throughout the day