Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Thom Kiddle & Sue Hackett: Assessing listening skills: what, why and how

76 views

Published on

Thom Kiddle & Sue Hackett: Assessing listening skills: what, why and how

Published in: Education
  • Be the first to comment

  • Be the first to like this

Thom Kiddle & Sue Hackett: Assessing listening skills: what, why and how

  1. 1. Assessing Listening The Eaquals Assessment Group Sue Hackett, Quality and Qualifications Ireland (QQI) Thom Kiddle, Norwich Institute for Language Education Alex Thorp, Trinity College London ©Eaquals Eaquals International Conference | Madrid | 11 -13 April 2019 #eaquals19madrid
  2. 2. Welcome • Who we are (today’s presenters and the Eaquals Assessment Group) • The purpose of the session: • Assessing listening at the macro-level – cognitive process and context • Assessing listening at the meso-level – the CEFR + CV and illustrative descriptors • Assessing listening at the micro-level – tasks and tests ©Eaquals #eaquals19madrid
  3. 3. ©Eaquals #eaquals19madrid
  4. 4. ©Eaquals #eaquals19madrid The receptive process: The perception of speech: sound and word recognition; The identification of the text, complete or partial, as relevant; The semantic and cognitive understanding of the text as a linguistic entity; The interpretation of the message in its specific context. What does the CEFRL say about Listening skills?
  5. 5. ©Eaquals #eaquals19madrid
  6. 6. ©Eaquals #eaquals19madrid What does this mean for assessment? A Framework for conceptualising & investigating listening test validity (adapted from Weir 2005a) Before … Context Validity Cognitive Validity Linguistic demandsTask Setting Cognitive Processes The Test Taker Purpose of task & rubric Response method Weighting Knowledge of criteria Order of items Presentation type Text length Time constraint & no of times heard Security Admin Exam conditions Task input and output Overall text type & purpose Discourse mode Lexical & grammatical resources Functional resources Nature of information Content knowledge Re Speaker Rate of speech and accent No of speakers Sociolinguistic aspects Decoding acoustic / phonetic (& visual) input Lexical search Syntactic recognition Establishing propositional meaning Constructing a meaning representation Constructing a discourse representation
  7. 7. After … Scoring Validity Test difficulty Item bias Internal consistency Error of measurement Grading and awarding Score / Grade Consequential Validity Criterion-related ValidityScore interpretation Washback on individual and group Impact on society and institutions Comparison with all versions of the same test Comparability with external standards and frameworks
  8. 8. Listening as a receptive skill Overall listening comprehension LISTENING TO AUDIO MEDIA AND RECORDINGS LISTENING AS A MEMBER OF A LIVE AUDIENCE LISTENING TO ANNOUNCEMENTS AND INSTRUCTIONS UNDERSTANDING CONVERSATION BETWEEN OTHER SPEAKERS
  9. 9. Zoom in – “key concepts operationalized in the scale…” UNDERSTANDING CONVERSATION BETWEEN OTHER SPEAKERS identifying when people agree and disagree, points made; identifying chronological progression, e.g. a story; catching enough to identify the topic, changes of topic; picking up and connecting words, phrases etc.; identifying attitudes and sociocultural implications (C levels).
  10. 10. Additions in the new Companion Volume
  11. 11. Listening as a part of interaction UNDERSTANDING THE INTERLOCUTOR Includes a consideration of the degree of accommodation by the interlocutor: from sympathetic repetition and taking the trouble to help, to just confirming details if the accent is less familiar. e.g. A1: Can understand everyday expressions aimed at the satisfaction of simple needs of a concrete type, delivered directly to him/her in clear, slow and repeated speech by a sympathetic speaker. A2+ : Can generally understand clear, standard speech on familiar matters directed at him/her, provided he/she can ask for repetition or reformulation from time to time. B1: Can follow clearly articulated speech directed at him/her in everyday conversation, though will sometimes have to ask for repetition of particular words and phrases.
  12. 12. Listening as a component in mediation MEDIATION Mediating a text Mediating concepts Mediating communication
  13. 13. Listening as a component in mediation
  14. 14. Mediation across two languages? “For all the descriptors in the scales in [Mediating a Text], Language A and Language B may be two different languages, two varieties of the same language, two registers of the same variety, or any combination of the above. However, they may also be identical: the CEFR is clear that mediation may be in one language.” CEFR CV 2018, p. 107
  15. 15. Listening at the meso-level “The CEFR descriptors… are not specifically linked to what research has told us about the nature of the listening skill. This is by no means to deny the usefulness of the CEFR criteria…It makes sense to supplement the CEFR specifications with an additional set of criteria that can both guide item writing and ensure that the grading of recordings, tasks and items takes place in a principled way.” Field, J. (2019) Rethinking the Second Language Listening Test, p. 9
  16. 16. Listening assessment in a high-stakes international test
  17. 17. Bachman & Palmer (2010): communicative competence model Communicative Competence Linguistic competence Socio-linguistic competence Discourse competence Strategic competence ISE Construct Communicative competence
  18. 18. Trinity ISE Reading & Writing Speaking & Listening 4 tasks with the same structure across all four levels: 2x Reading tasks, 1x Reading into writing task, 1x Writing task 2 hours 2x Speaking tasks 2x Listening tasks 13 minutes 2x Speaking tasks 2x Listening task 18 minutes 3x Speaking tasks 1x Listening task 20 minutes 3x Speaking tasks 1x Listening task 25 minutes ISE Foundation A2 ISE I B1 ISE II B2 ISE III C1 • Mapped from CEFR (not to). • On-going calibration to CEFR CV
  19. 19. Factual listening text(s); takes notes before reporting orally to the examiner* Examiner selects topic from list and starts a conversation A prompt sets the scene for a test-taker led discussion Candidate prepares ANY topic for discussion with the examiner ISE Speaking and listening Interactive and Independent listening Topic task Collaborative task Conversation task Independent listening task
  20. 20. Rating Scale categories Applied to Topic, Interactive and Conversation tasks
  21. 21. Analytic rating scales at ISE III (C1) Banded not sliding
  22. 22. Analytic rating scales and the CEFR (Example at CEFR C1)
  23. 23. Scales at ISE II (B2)
  24. 24. Factual listening text(s); takes notes before reporting orally to the examiner* Examiner selects topic from list and starts a conversation A prompt sets the scene for a test-taker led discussion Candidate prepares ANY topic for discussion with the examiner ISE Speaking and listening Topic task Collaborative task Conversation task Independent listening task
  25. 25. Independent listening across levels
  26. 26. ISE Independent listening • Objective and subjectiveISE Foundation • Objective and subjectiveISE I • Subjective onlyISE II • Subjective onlyISE III Increasingcognitiveload
  27. 27. At A2 (ISE Foundation) Task 1 – listen twice, take notes 1: Complete graphic (5 letters – matching) Objectively scored Task 2 – listen twice, take notes 1: Report 5 facts 2: Answer 3 questions (orally) Subjectively scored ISE Foundation tasks
  28. 28. ISE Foundation (A2) Task 1 Selected-response (Matching) Visually scaffolded Note-taking
  29. 29. Task 1 – listen twice, take notes 1: Answer 6 questions Objectively scored Task 2 – listen twice, take notes 1: Report 6 facts 2: Answer 4 questions Subjectively scored ISE I (B1) tasks
  30. 30. Task 1a – listen once 1: Report gist of text Subjectively scored Task 1b – listen again, take notes 1: Report details (examiner prompt) Subjectively scored ISE II & III (B2 & C1) tasks
  31. 31. Independent listening at B2 (ISE II)
  32. 32. • At what point does cognitive load, notably engagement of working memory, constitute CIV (Construct-irrelevant variance)? • Cognitive validity – how well does item elicit target cognitive processes (i.e. academic domain) • How well does the item (and construct) reflect real- world skill use (Are learners exposed to selected- response outside of ELT)? • Role of note-taking (enhance construct validity, risk of CIV, i.e. note taking skills) • Where is the test-taker (Centrality of learner in socio- cognitive framework)? ISE Independent listening Considerations
  33. 33. Listening assessment in a high-stakes international test
  34. 34. ISE – Summary of listening Application of Socio-cognitive framework (CRELLA / John Field) Context and Cognitive validity arguments - academic domain Interactive and independent listening* assessed separately *Recorded monologic text – oral response *Note taking permitted *Gist question (summary) to detail - macro to micro Rating scales mapped from CEFR
  35. 35. Example rating scale at B2 (ISE II)

×