Recommended
More Related Content
What's hot
What's hot (14)
Viewers also liked
Viewers also liked (20)
Similar to Thom Kiddle: Responding to the challenge of testing and assessing speaking
Similar to Thom Kiddle: Responding to the challenge of testing and assessing speaking (20)
More from eaquals
More from eaquals (20)
Recently uploaded
Recently uploaded (20)
Thom Kiddle: Responding to the challenge of testing and assessing speaking
- 1. Eaquals International Conference, Lisbon, 21 – 23 April 2016 Responding to the challenge of testing and assessing speaking www.eaquals.org
- 2. Eaquals International Conference, Lisbon, 21 – 23 April 2016 Oral Upper Secondary Level Leaving Exam in Austria Gerda Piribauer, Head of Administration and Organization, CEBS gerda.piribauer@cebs.at www.eaquals.orgwww.eaquals.org
- 3. Oral Upper Secondary Level Leaving Exam in Austria • Exam Subject – PLURILINGUALISM • Which “commonly” used criteria can be integrated in an assessment scale, what new criteria need to be introduced? Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 4. What you have to know about the exam… • Two equal parts: • Sustained monologue • Interactive dialogue • Interlocutors are assessors: • One assessment grid for the two examiners • One vote: • A joint final assessment 4 Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 5. Criteria • Fulfilment of the task • All content points are addressed • Amount of details for each language depends on the situation/interlocutors • Language switch and interaction • Can take the initiative and lead the conversation • Can switch between languages spontaneously and flexibly Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 6. Criteria - continued • Can mediate between the two interlocutors • Shows awareness of intercultural differences • Range and accuracy of spoken language • Grouped together to allow equal weighting of criteria • Compensation strategies are of particular interest Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 7. 7Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 8. Eaquals International Conference, Lisbon, 21 – 23 April 2016 Challenges for implementation of in-house marking criteria for spoken competence Tim Goodier, Head of Academic Development, Eurocentres. tgoodier@eurocentres.com ©Eaquals 06/08/20148
- 9. Agenda I. In house proficiency criteria vs. public examination benchmarking II. Using a profiling approach to developing criteria III. Key considerations ©Eaquals 06/08/2014 9
- 10. I. In house proficiency criteria vs. public examination benchmarking ©Eaquals 06/08/2014 10
- 11. I. In house proficiency criteria vs. public examination benchmarking ©Eaquals 06/08/2014 11 Eurocentres spoken assessment Public examinations Certificated division of spoken production and spoken interaction Normally one overall level provided for speaking ‘Bias for best’ criteria at all levels, in accordance with scale benchmarking to CEFR Criteria formulated positively & negatively around an explicit or implied threshold Task based Interview (+ mini task) based In-house training / standardisation from employer Training / standardisation as career move Certificate score 1-10 as outcome benchmarked to CEFR Public exam result as outcome, benchmarked to CEFR
- 12. II. Using a profiling approach to developing criteria Task-based speaking test (‘R.A.D.I.O.’) grades students for: R: Range A: Accuracy D: Delivery I: Interaction O: Organisation The assessor refers to a grid of defined criteria to grade R, A, D & I (or O) from 1 to 10. Then converts to 2 scores for production and interaction
- 13. II. Using a profiling approach to developing criteria Public examinations offer alternative (but similar) dimensions that can become an assessment ‘lingua franca’
- 14. III. Key considerations • Progress reporting for examination students - whether to only use exam scheme of results / mock tests • Legal / contractual considerations for examiners delivering mock results • Whether to use a (more granular) numeric in- house level scale at all • How to present & standardise ‘bias for best’ profiled assessment criteria in an accessible way ©Eaquals 2016 14
- 15. Eaquals International Conference, Lisbon, 21 – 23 April 2016 How to reconcile a ‘language use’ criterion with the CEFR Anthea Wilson Head of Test Production, Trinity College London anthea.wilson@trinitycollege.co.uk www.eaquals.org
- 16. 4 questions: • What guidance does the CEFR give about assessing grammar? • How does Trinity assess linguistic competence? • How were the Trinity ISE Speaking & Listening rating scales developed? • How can this relate to your context? Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 17. What guidance does the CEFR give about assessing grammar? • ‘an illustrative scale for accuracy’ (Council of Europe, 2001, p114) • ‘a language learner has to acquire both form and meanings’ (Council of Europe, 2001, p. 116) Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 18. How does Trinity assess linguistic competence in the ISE exams? Eaquals International Conference, Lisbon, 21 – 23 April 2016 •a scale for every level •what the learner achieves with the language •as part of a whole
- 19. How were the Trinity ISE Speaking & Listening rating scales developed? Eaquals International Conference, Lisbon, 21 – 23 April 2016 • collaboratively • based on evidence • using iterative processes
- 20. How can these principles be applied to your setting? Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 21. Eaquals International Conference, Lisbon, 21 – 23 April 2016 Student-facing Assessment Criteria Nadine Early Academic Director, ATC Language Schools nadine@atcireland.ie www.eaquals.org
- 22. The Learners’ Perspective What do our learners want? What will be useful to our learners? Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 23. What do learners want? 1. Feedback on familiar aspects of speaking performance: Eaquals International Conference, Lisbon, 21 – 23 April 2016 CEFR Table 3. Range Accuracy Fluency Interaction Coherence ATC Feedback Form Vocabulary Grammar Pronunciation Fluency Coherence / Interaction
- 24. What do learners want? 2. Accessible descriptors: Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 25. What will be useful to learners? 1. Develop a solid understanding of what speaking well entails: Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 26. What will be useful to learners? 2. The opportunity to reflect and plan: So far, so good! Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 27. Eaquals International Conference, Lisbon, 21 – 23 April 2016 How we approach criteria in non-test-based measurement of speaking www.eaquals.org Varinder Unlu Director of Studies, General English, International House London Varinder.Unlu@ihlondon.com
- 28. This was the look on the faces of teachers in the staffroom when I asked them how we approach criteria in non-test-based measurement of speaking... 28 Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 29. • I assess my students speaking skills by: • The criteria that I use to assess students’ communication skills are: • The problems and challenges I have with evaluating my students’ speaking skills are: 29 Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 30. • Challenges and problems facing assessing speaking: • Subjective and vague • Artificial • Not done 30 Eaquals International Conference, Lisbon, 21 – 23 April 2016
- 31. When, what and where to assess … • Classroom participation • Pronunciation • Vocabulary • Class presentations/debates • ? 31 Eaquals International Conference, Lisbon, 21 – 23 April 2016
Editor's Notes
- First of all let me give you some introductory information: in this plurilingual exam the candidates use English and a second foreign language which is taught in their school. The expected level for English is B2 while it‘s B1 for the second foreign language. The plurilingual exam consists of 2 equally important parts… Therefore the candidates have to demonstrate two different types of speech – from planned to spontaneous. Paramount of the assessment is the successful communication in a given plurilingual context. So the main criterion of the exam is whether the candidate can master this situation in both languages and not whether he or she has attained level B2 in English and B1 in the second foreign language. As I mention on this slide, these two parts are „equal“ so they are also taken equally in consideration for the final assessment. As there are two examiners – one for each language - there are no additional assessors. This decision has been taken by the Austrian ministry of Education and not by the team of CEBS. But, that is why only one assessment grid which is holistic rather than analytic, has been developed by our team for the two examiners. The two examiners have one vote and therefore must agree on a common final assessment in the form of a grade following the exam. So you can see that there were a number of aspects which had to be taken into consideration during the development of the assessment grid. Let me sum them up: First of all, how can we evaluate the performances in two languages which are spoken on a different level? Secondly, how to formulate descriptors which describe as precisely as possible an oral performance in two languages but can, at the same time, be handled by the examiners. And last but not least, how to include not only productive skills but also receptive skills and personal attitudes which enable the candidates to act in a plurilingual context, as a successful interface between other users of language who are monolingual.
- Let me now switch to the different criteria of our assessment grid. The first one is the fulfilment of the task. One aspect that is commonly assessed is whether all content points have been addressed and to what extent. In addition to this, the relevance of what is said, the clarity and the identification with the context/situation are taken into consideration. What is new or different is that the amount of… because in this plurilingual conversation the most important thing is that the interlocutors receive all the desired information. The second criterion,….includes flexibility. spontaneity and initiative – aspects which are very often taken into consideration when you are assessing speaking performances. What makes the difference in our assessment is that the examiners should also take into account to what extent the candidate is able to switch between two languages or even between three languages – because the task itself and most of the input are in German.
- The next two aspects of this criterion are not yet commonly included in assessment grids for speaking. The first one is the ability to mediate between two interlocutors who speak different languages. But mediation does not only mean to pass on information but also enabling interaction between the two interlocutors e.g. explaining terms which are not accessible when they are only translated. The second aspect is awareness of intercultural differences. Brian North and his group of experts are currently working on new descriptors for these two aspects of language performance. So I would say that CEBS made a “first step” when including them into our assessment grid, but after the publication of the work of the European Council/Brian and his group, we will certainly rethink our grid, perhaps adapt or even improve it. Range… is our third criterion and it includes… of structures and vocabulary to allow… Within this criterion compensation strategies, as for example the attempt to transfer words from one language to the other, are taken into consideration even if they are difficult to assess. In addition to that, there is a combination of two different levels (B2 +B1) and so the examiners have to apply descriptors bearing in mind this difference of levels. Nevertheless, the most important thing is that the candidate has mastered the plurilingual situation. At the end of this short presentation I would like to show you what our grid looks like.
- In this 10 minutes we’ll look at four questions, and probably raise more questions than answers.
- Not very much; certainly nothing prescriptive. But the CEFR isn’t meant to be a checklist, or an assessment tool in itself. It can be starting point or a guide, and here the key words are ‘illustrative’ and ‘meaning’. So how to measure ‘both form and meaning’?
- The ISE Speaking & listening rating scale, and also the Writing scale, are specific to the level – A2, B1, B2 and C1. So the performance of an A2 candidate is measured against a scale for A2 candidates. Each scale has four levels, giving an indication of performance that falls above, at or below the CEFR level. The ‘language control’ criterion focusses on what the candidate can do with their linguistic resources, i.e. what language functions they can achieve, rather than which specific tense they can use in an utterance. Also, linguistic competence is part of communicative competence, and is part of the interaction between the test-taker, his or her cognitive processes, the context, the task, the rating scales and how they are interpreted by the rater.
- Collaboratively – all stakeholders were consulted and part of the process; based on evidence – i.e. performances of language learners; academic research; the CEFR; drawn up, trialled, evaluated, adapted; trialled again by examiners, CEFR experts, academics and testing experts. And now they are live, the process of evaluating and testing them continues.