Language Acquisition Resource Center, profile picture
Uploaded byLanguage Acquisition Resource Center
8,120 views

How do we assess task-based performance? with Dr. John Norris

The document discusses task-based language assessment (TBLA), highlighting its significance in evaluating language performance within a communicative context. It details the evolution from traditional assessment methods to more integrated and authentic approaches that reflect real-world language use. Challenges and opportunities in implementing TBLA are also addressed, alongside the importance of specifying intended assessment uses and integrating various assessment tasks.

Embed presentation

How do we assess task- based performance? Recent advances in language assessment LARC/CALPER webinar April 3, 2014 John M. Norris – Georgetown University Copyright © John M. Norris, 2014: Please cite as: Norris, J. M. (2014, April). How do we assess task-based performance? Invited LARC/CALPER testing and assessment webinar.
Plan: 0. What is assessment and why do we do it? 1. Traditions, alternatives  Task-based assessment 2. The range of uses for task-based language assessment 3. Issues in assessing task-based language performance 4. Some challenges and opportunities in TBLA Task-based language assessment: the elicitation and evaluation of language use (across all modalities) for expressing and interpreting meaning, within a well-defined communicative context (and audience), for a clear purpose, towards a valued goal or outcome.
What is assessment? What isn’t assessment (in language education)? CEFR B1
What is assessment and why to we do it? Robert Mislevy (2004) Assessment is itself a particular kind of narrative: • an evidentiary argument • about aspects of what students know and can do, • based on a handful of particular things they have said, done, or made. What is educational assessment? Which narratives do we want to tell in language assessment?
2. The European Union and the United States: Possible Comparisons and Lessons You have been invited to give a lecture at the Society for German-American Relations in Weimar next month. For your contribution, the organizers are looking for a presentation on the much-discussed topic “The European Union and the Founding of the United States – A Model?” Your talk is limited to 15 minutes, to be followed by discussion of your remarks. What is assessment and why do it? 1. Fill in the missing letters to create complete words that make sense in the passage. This is an example C-test passage. Starting wi_____ the sec____ word o____ this sent____, the la____ half fr____ each consec____ word h____ been del____. C-tests a____ typically comp____ of mult____ texts wh____ become increa____ difficult. Ea____ text usu____ contains bet____ 20 a____ 25 it____. So, wh____ do y____ think ge____ tested on a C-test? Which is the ‘better’ language assessment?
The ‘correct’ answer: IT DEPENDS ON THE INTENDED USE of the TEST!
Specifying intended uses for assessment (Norris, 2000) INTENDED ASSESSMENT USE WHO? Test Users WHAT? Test Information IMPACT? Test Consequences WHY? Test Purposes The ‘correct’ answer: IT DEPENDS ON THE INTENDED USE of the TEST!
2. The European Union and the United States: Possible Comparisons and Lessons You have been invited to give a lecture at the Society for German-American Relations in Weimar next month. For your contribution, the organizers are looking for a presentation on the much-discussed topic “The European Union and the Founding of the United States – A Model?” Your talk is limited to 15 minutes, to be followed by discussion of your remarks. What is assessment and why do it? 1. Fill in the missing letters to create complete words that make sense in the passage. This is an example C-test passage. Starting wi_____ the sec____ word o____ this sent____, the la____ half fr____ each consec____ word h____ been del____. C-tests a____ typically comp____ of mult____ texts wh____ become increa____ difficult. Ea____ text usu____ contains bet____ 20 a____ 25 it____. So, wh____ do y____ think ge____ tested on a C-test? Intended use: quick estimate of language proficiency, e.g., for use in placing students into language courses…not task- based assessment. Which is the ‘better’ language assessment?
2. The European Union and the United States: Possible Comparisons and Lessons You have been invited to give a lecture at the Society for German-American Relations in Weimar next month. For your contribution, the organizers are looking for a presentation on the much-discussed topic “The European Union and the Founding of the United States – A Model?” Your talk is limited to 15 minutes, to be followed by discussion of your remarks. What is assessment and why do it? 1. Fill in the missing letters to create complete words that make sense in the passage. This is an example C-test passage. Starting wi_____ the sec____ word o____ this sent____, the la____ half fr____ each consec____ word h____ been del____. C-tests a____ typically comp____ of mult____ texts wh____ become increa____ difficult. Ea____ text usu____ contains bet____ 20 a____ 25 it____. So, wh____ do y____ think ge____ tested on a C-test? Intended use: quick estimate of language proficiency, e.g., for use in placing students into language courses…not task- based assessment. Intended use: extended, authentic, and integrated performance with the target language, e.g., for end-of-unit achievement testing and feedback…yes, task-based assessment. Which is the ‘better’ language assessment?
Making decisions about students admissions placement course credit fulfilling requirements grading /achievement advancement/retention certification of abilities awarding funding ETC. INDIVIDUAL DECISION- MAKING EDUCATIONAL USES FOR ASSESSMENT
Making decisions about students admissions placement course credit fulfilling requirements grading /achievement advancement/retention certification of abilities awarding funding ETC. Informing classroom teaching and learning diagnosing learner needs motivating students understanding learning outcomes improving instruction defining learning expectations focusing students’ learning/study ETC. INDIVIDUAL DECISION- MAKING CLASSROOM FEEDBACK EDUCATIONAL USES FOR ASSESSMENT
Making decisions about students admissions placement course credit fulfilling requirements grading /achievement advancement/retention certification of abilities awarding funding ETC. Informing classroom teaching and learning diagnosing learner needs motivating students understanding learning outcomes improving instruction defining learning expectations focusing students’ learning/study ETC. Improving, ensuring, demonstrating quality demonstrating student learning clarifying program mission/needs revising curriculum/ pedagogy improving teacher performance communicating beyond, about the program responding to program review ETC. INDIVIDUAL DECISION- MAKING CLASSROOM FEEDBACK PROGRAM EVALUATION aligning, articulating program practices EDUCATIONAL USES FOR ASSESSMENT
Making decisions about students admissions placement course credit fulfilling requirements grading /achievement advancement/retention certification of abilities awarding funding ETC. Informing classroom teaching and learning diagnosing learner needs motivating students understanding learning outcomes improving instruction defining learning expectations focusing students’ learning/study ETC. Improving, ensuring, demonstrating quality demonstrating student learning clarifying program mission/needs revising curriculum/ pedagogy improving teacher performance communicating beyond, about the program responding to program review ETC. INDIVIDUAL DECISION- MAKING CLASSROOM FEEDBACK PROGRAM EVALUATION aligning, articulating program practices EDUCATIONAL USES FOR ASSESSMENTWhat different kinds of assessment are needed to meet these different uses? How might tasks play a role?
METHODS MCT Matching Fill-in-the-blank Short answer Cloze/C-test Performance assessment Interviews Observations Essays Portfolios True-false Self-assessment What are the possibilities for assessment? Projects Peer-assessment Rubrics
Alternatives in assessment: The move to task
Early interest in ‘tasks’ for assessment “If we want to find out how well a person can perform a task, we can put him to work at that task, and observe how well he does it and the quality and quantity of the product he turns out.” Cureton (1950): “Validity” e.g., in language testing: Foreign Service Institute Oral Interview (1950s) Using tasks to elicit a holistic estimate of functional language ability, from 0 (no ability) to 5 (educated native speaker ability)
Standardized testing and psychometrics But, the primary tradition of assessment in the 20th century: Large-scale standardized testing: minimal competency, accountability emphasis on efficiency, technology (e.g., scoring), discrete-point primary concern with psychometrics, norm-referencing validity = narrowly focused on construct representation, reliability e.g., in language testing: Test of English as a Foreign Language (1964)* MCT: vocabulary, reading, listening, structure, grammar (emphasis on testing language knowledge) *But see later iterations of TOEFL…
Influence of traditional testing When we ask teachers what an assessment looks like, what do they tell us? • True-false • Matching • Multiple choice • Fill-in the blank • Exam • Test • Giving grades • Accountability • ETC.
Influence of traditional testing When we ask teachers what an assessment looks like, what do they tell us? Washback of testing traditions: Narrowing teaching and learning Misrepresenting assessment Disconnect between teaching- testing-learning • True-false • Matching • Multiple choice • Fill-in the blank • Exam • Test • Giving grades • Accountability • ETC.
“It is the simplification of competencies to what can be ‘objectively’ tested which has a detrimental effect: the learning of trivial facts, the reduction of subtle understanding to generalizations and stereotypes, the lack of attention to interaction and engagement because these are not tested.” e.g., Michael Byram (1997): Need for change in intercultural assessment Challenges to ‘traditional’ testing
Reactions to ‘traditional’ testing Portfolios Performance-based assessments Essays CBTs Self-assessments Interviews Rating scales, rubrics Alternative assessments 1980s-90s •Authenticity •Performance •Can-do (vs. know) •Classroom •Washback
http://www.learner.org/libraries/tfl/assessment/analyze.html? pop=yes&pid=2003 The move to task as assessment medium Grant Wiggins (1998): Educative assessment The primary assessment medium should be authentic tasks which have learners demonstrate what they can do with what they are learning; enables meaningful integration of teaching, learning, and assessment assessment becomes educative (informative) rather than just an authoritarian mechanism for control.
The move to task-based language assessment “…the process of evaluating, in relation to a set of explicitly stated criteria, the quality of the communicative performances elicited from learners as part of goal-directed, meaning-focused language use requiring the integration of skills and knowledge” Geoff Brindley (1994): Task-centered assessment aligned with CLT, TBLT movements in pedagogy emphasis on authenticity of L2 use validity = capturing ability for use in valued tasks classroom applications obvious, but beyond?
Operationalizing task-based language assessment “…genuinely task-based language assessment takes the task itself as the fundamental unit of analysis, motivating item selection, test instrument construction, and the rating of task performance” Long & Norris (2000): Task-based language assessment 1. Specify intended uses for the assessment: who, what, why, impact 2. Select and analyze key target tasks/features from needs analysis 3. Design tests and items: emphasize authentic performances 4. Determine real-world criteria for rating task performance qualities 5. Pilot-test and revise instruments and procedures (raters/rating) 6. Evaluate validity in terms of intended uses, and especially impact on teaching and learning.
Norris et al. (1998) Tasks respond to diverse needs/uses in assessment • Focus on what L2 users can actually do with the language • Integrate language form and meaning into what gets assessed • Provide useful feedback to teachers, learners, curriculum • ‘Wash back’ on teaching and learning by making outcomes ‘real’ • Counter negative influence of traditional knowledge assessment • Align testing with new, evolving language pedagogies Task-based language assessment: the elicitation and evaluation of language use (across all modalities) for expressing and interpreting meaning, within a well-defined communicative context (and audience), for a clear purpose, towards a valued goal or outcome. Why task-based language assessment?
Up Next: How do we assess task-based performance in language classes and programs? Issues, examples, suggestions… But First: Any Questions?
Task-based assessment in language classes and programs Issues 1. Which tasks do we assess? 2. How do we elicit task-based performance? 3. What gets assessed in task performance? 4. How do we score task performance?
1. Which tasks do we assess? e.g.: EAP pragmatics assessment (Youn, 2009) 1. Write a recommendation request e-mail to professor 2. Write an e-mail to potential employer to send application packet 3. Write an e-mail to refuse professor’s request 4. Write constructive comments on cover letter written by classmate 5. Give oral peer-feedback on classmate’s request e-mail to professor 6. Interacting with classmates in situations of making suggestions and disagreement 7. Interacting with a professor in situations of making requests and refusal Why these tasks?
1. Which tasks do we assess? Why these tasks?Which tasks? Based on needs analysis for specific English language program context (individual & interactive tasks) High priority tasks reflecting pragmatic challenges for college-level international students Each task serves as end-of-unit achievement assessment Combined, all of the tasks can be used for diagnostic assessment
You have been given the opportunity of a lifetime to apply for an athletic training camp in [e.g., Spain], tuition free! This camp trains young people in all sports, from the extreme (snowboarding, bicycle motor cross, rollerblading) to team sports of all kinds (basketball to volleyball). You name it, they help you train for it! To be accepted into the camp, all applicants have to convince the admissions office that they have good exercise and nutrition habits. First, you will read about health and nutrition from the perspective of the Spanish – speaking world. Then you will discuss your eating and exercise regimen with your partner to compare your nutrition and exercise, perhaps event to get some ideas. You will then write your application letter to the summer camp describing your nutrition and training regimen, convincing them that you are well prepared for the camp and need to be accepted. 1. Which tasks do we assess? e.g., Integrated performance assessment in US schools Overall introduction to a sequence of interrelated pedagogic/assessment tasks. Note the distinct task phases. Consortium for assessing performance standards http://flenj.org/CAPS/?page=parent
1. Which tasks do we assess? e.g., Integrated performance assessment in US schools Consortium for assessing performance standards http://flenj.org/CAPS/?page=parent Individual reading task Pair discussion task Letter writing task Why these tasks?
1. Which tasks do we assess? e.g., Integrated performance assessment in US schools Consortium for assessing performance standards http://flenj.org/CAPS/?page=parent Individual reading task Pair discussion task Letter writing task Self-assessment: understanding task content, language Peer-assessment: feedback on development and expression of ideas Teacher-assessment: effectiveness of letter, language use Why these tasks?
1. Which tasks do we assess? Which tasks? Based on the U.S. Standards for Foreign Language Learning, by specific age/grade level and proficiency 3 integrated tasks—Interpretive, interpersonal, presentational—reflect distinct modes of performance Tasks are closely integrated with what is being taught: this is why you are learning what you are learning Tasks provide the opportunity for self-, peer-, and teacher-assessment and feedback Assessment raises awareness about effectiveness of teaching and learning See Adair-Hauck, et al. (2006) Why these tasks?
2. How do we elicit task-based performance? The European Union and the United States: Possible Comparisons and Lessons Task You have been invited to give a lecture at the Society for German-American Relations in Weimar next month. For your contribution, the organizers are looking for a presentation on the much-discussed topic “The European Union and the Founding of the United States – A Model?” Your talk is limited to 15 minutes, to be followed by discussion of your remarks. Please observe your position as a speaker to this particular audience. While you should expect good background knowledge from your German audience regarding the situation in the EU, you primarily contribute the perspective of an American who is familiar with the foundational history of the USA and subsequent development of the county. In its particularities this is most likely to be less familiar to your audience. Therefore, it is precisely those aspects that you must bring to the awareness of your audience in order to link them to the present situation in Europe. In other words, continuously keep in mind this occasion for your speech, this starting point for your presentation, and this intention of your talk. e.g.: Georgetown German: Prototypical Performance Writing Task Level IV course, “Text in Context” What does this description accomplish?
Example: CNaVT Certificate in Dutch as a Foreign Language Tourist domain Listening comprehension task “Visit to the museum” Item: Identify the paintings based on the descriptions you hear from an audio museum guide. What is the purpose of the paintings?
Task Title: Recycling Theme: Environment Level: Novice-Mid (Focus Age Group: 11 – 14) Communicative Mode: Presentational Time Frame: One 42 minute period or more depending on the class and the type of presentations chosen. Description of Task: Now that you’ve learned about saving the planet and the need to reuse and recycle, you want to share with your family your ideas for reusing some throw away things. You have cleverly thought of a few uses for reusing ____________. Now you will share your new ideas. Create and develop your idea and choose one from the following types of presentations to present it. Continued on next page… Consortium for assessing performance standards http://flenj.org/CAPS/?page=parent e.g.: assessing young/beginning learners, US schools 2. How do we elicit task-based performance?
Title: ¿Qué puedo hacer con_________? (What can I do with_______?) Include the following in your presentation: • four or more ways of re-using something • specific uses • descriptions/elaboration Three options, you choose: Power point presentation Poster Live oral 3-D presentation Materials Needed: computers, markers, poster board, realia Why give the student these options? 2. How do we elicit task-based performance?
2. How do we elicit task-based performance? Task-based performance elicitation? Give a purpose for communicating (motivate) For interactive/presentational tasks, indicate or provide an audience Provide a structure or procedures for doing the task Utilize authentic input and other realia for situating the task in a meaningful context Indicate desired outcomes or products of the task Where feasible, encourage ownership over the task through individual choice in its accomplishment
3. What gets assessed in task performance? Test task: Order pizza for the class. Take orders Place orders Task accomplishment?
3. What gets assessed in task performance? Test task: Order pizza for the class. XTake orders Place orders Task accomplishment?
3. What gets assessed in task performance? e.g.: classroom assessments for middle-school learners in Flanders (Jordens, 2007) Type task: (from learning goals) Pupils can report to the teacher on a subject, previously discussed at school Test task: Pupils report to the teacher on a film about scuba diving, the report being based on photographs extracted from the film.
e.g.: classroom assessments for middle-school learners in Flanders Parameters for assessment What does pupil say? Content relevance, completeness Clarity Organization How does pupil say it? Fluency Articulation Manner of speaking Variation and extensiveness Accuracy: word/sentence Interaction skills 3. What gets assessed in task performance? How is this rubric used?
Give a presentation in Spanish about your life and interests. Indicate how these activities reflect your personality. Include some possibilities for your future in terms of career, study, travel, pursuing personal interests. e.g.: performance assessment for 8th graders in US school 3. What gets assessed in task performance? Source: http://depts.washington.edu/mellwa/Events/20081105 /sandrock_ipa_handout.pdf
EXCEEDS EXPECTATIONS MEETS EXPECTATIONS DOES NOT MEET EXPECTATIONS Do we understand you? (Comprehensibility) My message is easily understood by my audience. I am able to communicate my message by reproducing memorized material. I rely heavily on visuals to convey my message. How do I use the Spanish language? (Language Control & Vocabulary Use) I can write/speak in simple sentences correctly. My presentation is rich in appropriate vocabulary. I demonstrate some accuracy in written/oral presentation when I reproduce memorized words and phrases. My vocabulary reveals basic information. My presentation is correct only at the word level. My vocabulary is limited and / or repetitive. I use English. How do I organize the presentation? (Communication strategies) My presentation is supported with many enriched examples. My presentation has some visuals/realia and examples. My presentation lacks organization. How well do I impact the audience? (Impact) I use visuals/ realia and gestures, and tone of voice to maintain audience’s attention. I use some visuals/realia and gestures to maintain audience’s attention. My tone of voice is acceptable. I use a small amount of effort to maintain audience’s attention. What is the purpose of these criteria?
Criteria for judging performance? Consider real-world criteria for task success or accomplishment, including as appropriate: Performance products: Does something happen, does something get produced? Performance steps: Do critical elements of the task happen? Performance qualities: linguistic and paralinguistic features (e.g., accuracy, fluency, gesture, pragmatics), content (e.g., meaning), other attributes (e.g., visual support, efficiency of task completion) Performance levels: How well does the task performance meet expectations, standards, etc.? 3. What gets assessed in task performance?
e.g., Georgetown Univ. German writing assessments Writing task 1 ______ ______________ ______________ ______________ _____ Writing task 2 ______ ______________ ______________ ______________ _____ Writing task 3 ______ ______________ ______________ ______________ _____ Writing task 4 ______ ______________ ______________ ______________ _____ Writing task 5 ______ ______________ ______________ ______________ _____ End-of-level Prototypical task _________ _________________ _________________ _________________ _________________ _________________ ____ 4. How do we score task performance?
Final Grade Feedback: Rating rubric, In-text Initial Grade Task –based assignment Level scoring rubric Writing task v. 1 ______ ______________ ______________ ______________ _____ Writing task v. 2 ______ ______________ ______________ ______________ _____ 4. How do we score task performance? Formative assessment: Process writing Providing rich feedback to enable learning Teacher Students
End-of-level Prototypical task _________ _________________ _________________ _________________ _________________ _________________ ____ Example Journalistic portrait: The Vietnamese immigrant experience in contemporary Germany 2 raters from the curricular level + 2 raters from another level 4. How do we score task performance? Summative assessment: Performance writing Providing reliable judgments of language + task abilities
Scoring and score reporting for different purposes? Consider how ‘scores’ will be used before deciding how to score: Formative assessment: Use public criteria and scales, and encourage self-, peer-, teacher-scoring for feedback Summative assessment: Use well-defined criteria and scales, and multiple, trained raters in scoring for judgment Holistic scoring (overall judgment) provides and indication of task success or accomplishment or completion Analytic scoring (by categories and qualities) provides a useful basis for remediation Rich and detailed score reporting enables learning from assessment (e.g., in person, score + justification) 4. How do we score task performance?
Up Next: The range of current uses for task-based language assessment But First: Additional Questions?
Tasks as targets: Policies & standards V. Workplace Tasks: Reading --Scan basic charts, tables, maps or schedules for information. (5) --Find routine information on the computer screen/scanner screen/computerized display screen, if available. (6) --Read information in the reception/appointment book to find available openings for a new appointment. (6) --Read a checklist to verify if all the steps in the procedure have been completed. (6) --Follow one page of clear familiar instructions. (7) --Read a reminder or complaint letter/memo, take appropriate action. (7) --Scan complex charts, tables and schedules for several specific pieces of information for comparison/contrast. (7) --Follow instructions on evacuation procedures, fire drills, or on using simple machinery/equipment.(7) --Use plain language manual with familiar topic and content in own field of knowledge to find specific information. (8) ETC… Who should be able to do these tasks? For example…
Tasks as targets: Policies & standards V. Workplace Tasks: Reading --Scan basic charts, tables, maps or schedules for information. (5) --Find routine information on the computer screen/scanner screen/computerized display screen, if available. (6) --Read information in the reception/appointment book to find available openings for a new appointment. (6) --Read a checklist to verify if all the steps in the procedure have been completed. (6) --Follow one page of clear familiar instructions. (7) --Read a reminder or complaint letter/memo, take appropriate action. (7) --Scan complex charts, tables and schedules for several specific pieces of information for comparison/contrast. (7) --Follow instructions on evacuation procedures, fire drills, or on using simple machinery/equipment.(7) --Use plain language manual with familiar topic and content in own field of knowledge to find specific information. (8) ETC… Who should be able to do these tasks? Canadian Language Benchmarks: English Language Demands for Nursing For example…
Tasks as targets: Policies & standards (e.g., Canadian Language Benchmarks) “The CLB is task-based: • In syllabus design, tasks are considered to be basic building blocks.• Tasks in language learning promote the integration of all aspects of communicative competence, and multilevel language processing.” CEFR europe CLB canada ACTFL SFLL U.S. TESOL/NCATE teacher development Why? Tasks provide meaningful targets: Ability to use the language Washback on teaching and learning TOC hong kong
Tasks in large-scale proficiency assessment 4. Integrated Speaking: Read a passage from a psychology textbook and listen to the lecture that follows it. Then answer the question. Reading: Flow In psychology, the feeling of complete and energized focus in an activity is called flow. People who enter a state of flow lose their sense of time and have a feeling of great satisfaction. They become completely involved in an activity for its own sake rather than for what may result from the activity, such as money or prestige. Contrary to expectation, flow usually happens not during relaxing moments of leisure and entertainment, but when we are actively involved in a difficult enterprise, in a task that stretches our mental or physical abilities. THEN For example…
Listen to the Lecture (Male professor): I think this will help you get a picture of what your textbook is describing. I had a friend who taught in the physics department, Professor Jones, he retired last year. . . . Anyway, I remember . . . this was a few years ago . . . I remember passing by a classroom early one morning just as he was leaving, and he looked terrible: his clothes were all rumpled, and he looked like he hadn’t slept all night. And I asked if he was OK. I was surprised when he said that he never felt better, that he was totally happy. He had spent the entire night in the classroom working on a mathematics puzzle. He didn’t stop to eat dinner; he didn’t stop to sleep . . . or even rest. He was that involved in solving the puzzle. And it didn’t even have anything to do with his teaching or research; he had just come across this puzzle accidentally, I think in a mathematics journal, and it just really interested him, so he worked furiously all night and covered the blackboards in the classroom with equations and numbers and never realized that time was passing by. Question: Explain flow and how the example used by the professor illustrates the concept. Tasks in large-scale proficiency assessment Q: Who should be able to do these tasks? For example…
Lecture (Male professor): I think this will help you get a picture of what your textbook is describing. I had a friend who taught in the physics department, Professor Jones, he retired last year. . . . Anyway, I remember . . . this was a few years ago . . . I remember passing by a classroom early one morning just as he was leaving, and he looked terrible: his clothes were all rumpled, and he looked like he hadn’t slept all night. And I asked if he was OK. I was surprised when he said that he never felt better, that he was totally happy. He had spent the entire night in the classroom working on a mathematics puzzle. He didn’t stop to eat dinner; he didn’t stop to sleep . . . or even rest. He was that involved in solving the puzzle. And it didn’t even have anything to do with his teaching or research; he had just come across this puzzle accidentally, I think in a mathematics journal, and it just really interested him, so he worked furiously all night and covered the blackboards in the classroom with equations and numbers and never realized that time was passing by. Question: Explain flow and how the example used by the professor illustrates the concept. Tasks in large-scale proficiency assessment Q: Who should be able to do these tasks? Test of English as a Foreign Language: English for Academic Purposes Example…
Tasks in large-scale proficiency assessment (e.g., TOEFL iBT) “Working directly with higher education institutions, we designed the TOEFL test to simulate actual academic classroom tasks.” TOEFL iBT U.S. CNaVT Holland/Belgium IELTS U.K. NMET China AMES Australia Why? Tasks provide improved interpretations, construct validity Washback on teaching and learning
Tasks in professional language certification Example…
Tasks in professional language certification http://www.anglo-continental.com/en/uk/forms/Aviation/Part%202%20%20ready.mp3 Q: Who should be able to do these tasks? Example…
International Civil Aviation Organization: English for Aviation Purposes Tasks in professional language certification http://www.anglo-continental.com/en/uk/forms/Aviation/Part%202%20%20ready.mp3 For example…
Tasks in professional language certification (e.g., ICAO, 2007) “Because of the high stakes involved, pilots and controllers deserve to be tested in a context similar to that in which they work and test content should therefore be relevant to their roles in the work-place” FBI Translation CLB- Nursing NCATE L2 teaching English Aviation Why? Tasks demonstrate ‘on-the-job’ abilities for high-stakes decisions Washback on teaching and learning OET Medical
Overview of task In groups, learners design a brochure for the Hong Kong Tourism Board describing 4 attractions in Hong Kong which would appeal to young people of their own age. Writing a Tourist Brochure Imagine the Hong Kong Tourism Board has asked your class to design a brochure that would be of interest to young visitors of your own age. In groups of 4, design the brochure describing FOUR sites suitable to young people of your own age coming to Hong Kong. Complete this task by following the steps below: Step 1: Group Task. Discuss in your groups which sites young people would want to visit in Hong Kong. Choose one site each to investigate. For homework find out as much as you can about the site, where the site is, when it is open, what one can see / do there, what the facilities are, how one gets there, etc. Bring this information to the next class. For example… Tasks in classrooms and programs
Step 2: Group Task. Exchange information with your group members. Tell them about the site you have found out about. Then decide how you are going to present the information in your brochure, what order you want to put your sites in, what illustrations you need, what title you want to give the brochure, etc. Step 3: Individual Task. Write a description about your chosen site (120 words). Remember to say why it is interesting. Proofread it carefully, then hand it to your teacher. Step 4: Group Task. In your groups edit your work based on your teacher’s comments. Then put together your brochure. Your brochures will be assessed on the following basis: • Task fulfilment: would your selected sites appeal to young people? • Accuracy of language and information provided: is the brochure written in good English? Is the information provided accurate? • Attractiveness of final written submission: is your brochure really attractive? Can you make it more appealing? For example… Tasks in classrooms and programs Q: Who should be able to do these tasks?
Step 2: Group Task. Exchange information with your group members. Tell them about the site you have found out about. Then decide how you are going to present the information in your brochure, what order you want to put your sites in, what illustrations you need, what title you want to give the brochure, etc. Step 3: Individual Task. Write a description about your chosen site (120 words). Remember to say why it is interesting. Proofread it carefully, then hand it to your teacher. Step 4: Group Task. In your groups edit your work based on your teacher’s comments. Then put together your brochure. Your brochures will be assessed on the following basis: • Task fulfilment: would your selected sites appeal to young people? • Accuracy of language and information provided: is the brochure written in good English? Is the information provided accurate? • Attractiveness of final written submission: is your brochure really attractive? Can you make it more appealing? Task-Based Assessment for English Language Learning: Hong Kong, Secondary Schooling Example… Tasks in classrooms and programs Q: Who should be able to do these tasks?
Tasks in classrooms and programs (e.g., Hong Kong Special Administrative Region) “Task-based assessment for English language learning at secondary level: One of the key principles underpinning the school English language curriculum in Hong Kong is that of task-based language teaching. In this package we introduce the features of task-based assessment (TBA) within a school-based curriculum…” http://cd1.edb.hkedcity.net/cd/eng/TBA_Eng_Sec/index.html Local practice TBLT research Can-do e.g., DIALANG Teacher guides ACTFL IPAs Why? Tasks enable feedback on learning, teaching, curriculum Washback on teaching and learning
Task-based language assessment in practice Administrators Staff Classroom: TBLAssessment For Learning Program: Tasks as summative indicators of teaching effectiveness Profession: Tasks for certification of important abilities Large-scale: Tasks for demonstrating proficiency in societal domains (e.g., education) Standards: Tasks as valued targets for learning
Some challenges and opportunities In Task-Based Language Assessment…
2002 (19.4) • How do we interpret task-based L2 performance? authentic criteria, proficiency, L2 production? • Can task-based performances be rated reliably? analytic/holistic rating scales, rater training? • What is the nature of task difficulty? cognitive complexity, learner individual diffs? • Can we generalize and/or extrapolate based on tasks? domain-sampling, ECD, underlying constructs? • How does task-based assessment impact instruction? teacher, learner, curricular consequences? Basic validity questions for TBLA Norris (2008) “Fundamentally…assessments are only good insofar as their use does good, in terms of supporting educational efforts and outcomes, and it is my intent that validity evaluation helps to ensure that they do.”
How do we interpret task-based L2 performance? Can do a specific task Has a certain proficiency Displays X language qualities High-fidelity task simulation or replication Test of aviation English Domain sampling, ‘type’ tasks TOEFL, CNAVT Implicit, spontaneous elicitation Classroom practice, SLA research
Can task-based performances be rated reliably? YES!!! Individual task accomplishment: single task Highly specified indigenous criteria, steps, etc. Teacher, tester, learner awareness-raising Ability for L2 use: multiple/type tasks Cross-task holistic/analytic rating scales Rater training
What is the nature of task difficulty? Why is this task so hard? Task features e.g., number of elements, number of steps, planning time, structure, reasoning requirements, code frequency/complexity Task conditions e.g., number of participants, open/closed task, convergent/divergent, familiarity, gender, status Individual differences e.g., task familiarity, motivation, testing stakes, aptitude, personality More difficult = better or worse performance???
Can we generalize and/or extrapolate based on tasks? Of course we can  and we do it all the time! Driving testseries of tasks (1-time!) Generalizationcan drive a car Extrapolation can negotiate a roundabout in London??? …be careful!
Can we generalize and/or extrapolate based on tasks? 1. Listening comprehension? 2. Listening comprehension in a particular domain? 3. Listening comprehension on a specific task? 1. Adequate construct definition 2. Adequate domain sampling3. Adequate task representation
How does task-based assessment impact instruction? Local impact (Georgetown German): “So I think it has filtered down to smaller writing assignments. First of all so that students get used to it and it’s a coherent system. Whereas before we would say, ‘Okay, now write one page about your family and your Heimat’. Now we actually make up a sheet and use the same features.” Global impact (TOEFL) “…considerable changes occurred in the amount of attention the teachers paid to the development of speaking and to the integration of different skills.”
In sum, task-based performances and assessments may not (should not) meet all purposes for assessment within language education, but they probably offer a lot of advantages for certain purposes: Task-based performances tell us things that other item types do not (e.g., ability to accomplish particular tasks under particular conditions) Task-based performances offer rich opportunities for eliciting and observing language use with meaningful content in context (there is always more to language use than just language, and tasks capture that) Task-based performances can reveal multiple aspects of language ability and/or development within a single task (e.g., accuracy, complexity, fluency; content knowledge; procedural knowledge; expertise) Task-based performances are in many cases the most indicative targeted expectations of language classes and programs (if not, then what exactly are we teaching?) Task-based assessments provide relevant and comprehensive frames of reference for learners and teachers to generate feedback on form-function-meaning relationships (formative, learning-oriented uses) Task-based assessments communicate to teachers and learners and others what is valued within a given educational setting (motivational, educative uses) Task-based assessments may impact curriculum and educational systems, causing a rethinking of what and how teaching is occurring (washback uses) Task-based assessments generate interpretable data regarding the worthwhile outcomes of language learning programs (summative uses, e.g., for accountability, certification, program evaluation)
Why task-based assessment? Tasks are useful: • align to instructional practices and values • provide rich samples of language use data • inform how well we teach and learn • show what L2 users can do, and motivate them to learn Task-based assessments can: • improve accuracy of interpretations • ensure abilities to use language for specific purposes • guide educational practices • counter negative washback of other assessments • enable a variety of uses, decisions, actions Q: Task-based assessment is a lot of work, and we can make somewhat accurate assessments of learners’ abilities and knowledge with short-cut tests; why should we bother with TBLA?
Thank you!

More Related Content

PPT
Task Based Syllabus
byoanhbinh
 
PPTX
Standards based approach
by1101989
 
PPTX
Kinds of tests and testing
byMaury Martinez
 
PPT
Syllabus Types
byDavid Deubelbeiss
 
PPTX
Task based syllabus
byFariba Chamani
 
PPT
Task-based Syllabus
byyektayeganeh
 
PPTX
A framework for materials writing
byMohammed Mallah
 
PPTX
task-based language teaching
byAlex Chacon
 
Task Based Syllabus
byoanhbinh
 
Standards based approach
by1101989
 
Kinds of tests and testing
byMaury Martinez
 
Syllabus Types
byDavid Deubelbeiss
 
Task based syllabus
byFariba Chamani
 
Task-based Syllabus
byyektayeganeh
 
A framework for materials writing
byMohammed Mallah
 
task-based language teaching
byAlex Chacon
 

What's hot

PPTX
Approaches to Language Testing
byAnn Liza Sanchez
 
PPTX
Language Assessment : Kinds of tests and testing
byMusfera Nara Vadia
 
PPTX
Writing assessment
byAnabel Aqramonte
 
PPTX
Needs Analysis
byCarlos Mayora
 
PPTX
Assessing listening
byShama Siddiqui
 
PDF
ESP Course- chapter 3 - English for academic purposes
byMar Iam
 
PPTX
Language Testing
byTeguh Ekosetio
 
PPTX
Task based syllabus
byUspan Sayuti
 
PPTX
Kinds of Language Tests
byJennefer Edrozo
 
PPTX
Modified Input (Foreigner Talk)
bySawsan Ali
 
PPTX
Task-Based Language Teaching (TBLT)
byRonald Suplido Jr
 
PPT
Assessing reading
byronadelarosa
 
PPT
Grading task
byEika Matari
 
PPT
Chapter 3(designing classroom language tests)
byKheang Sokheng
 
PPT
ENGLISH FOR SPECIFIC PURPOSES
byCarrie Carraway
 
PPT
Notional functional syllabus design
byMazharul Islam
 
PPTX
Grammatical based syllabus. Akram Jabar Najim
byAkramEnglish
 
PPTX
Articulating beliefs
byAnnasta Tastha
 
DOCX
vocabulary testing
byMuhammadiyah University Of Jember
 
PPTX
TASK BASED LEARNING
bymeymirah
 
Approaches to Language Testing
byAnn Liza Sanchez
 
Language Assessment : Kinds of tests and testing
byMusfera Nara Vadia
 
Writing assessment
byAnabel Aqramonte
 
Needs Analysis
byCarlos Mayora
 
Assessing listening
byShama Siddiqui
 
ESP Course- chapter 3 - English for academic purposes
byMar Iam
 
Language Testing
byTeguh Ekosetio
 
Task based syllabus
byUspan Sayuti
 
Kinds of Language Tests
byJennefer Edrozo
 
Modified Input (Foreigner Talk)
bySawsan Ali
 
Task-Based Language Teaching (TBLT)
byRonald Suplido Jr
 
Assessing reading
byronadelarosa
 
Grading task
byEika Matari
 
Chapter 3(designing classroom language tests)
byKheang Sokheng
 
ENGLISH FOR SPECIFIC PURPOSES
byCarrie Carraway
 
Notional functional syllabus design
byMazharul Islam
 
Grammatical based syllabus. Akram Jabar Najim
byAkramEnglish
 
Articulating beliefs
byAnnasta Tastha
 
vocabulary testing
byMuhammadiyah University Of Jember
 
TASK BASED LEARNING
bymeymirah
 

Viewers also liked

PDF
Tim McNamara
byTouchstone Institute
 
PPT
Self-assessment in language learning
bywillyccardoso
 
PPTX
Language Assessment - Standards-Based Assessment by EFL Learners
byEFL Learning
 
PDF
Developing Rubrics for Language Assessment with Dr. JD Brown
byLanguage Acquisition Resource Center
 
PPT
Introduction to Language Assessment by Brown
byEFL Learning
 
PPTX
Assessment & Language Learning
byBishara Adam
 
PPTX
Language Assessment Types
byOla Sayed Ahmed
 
PPT
technology in standardized language assessment
byHuang YaLi
 
PPTX
Language Assessment - Beyond Test-Alternatives Assessment by EFL Learners
byEFL Learning
 
PPT
Beyond tests alternatives in assessment
byYamith José Fandiño Parra
 
PPTX
Oral language assessment
byyonfreddygarzon
 
PDF
Types of language assessment
byFrancis Calubayan
 
PPTX
Language Assessment_Formal and Informal
byMæäSii Mööì
 
PPTX
Language Assessment Principles and Issues
byMaury Martinez
 
PPTX
Principles of language assessment
byAstrid Caballero
 
PPT
Informal Assessment
bysongoten77
 
PPT
Portfolio Based Language Assessment (PBLA): Maximizing Assessment for Learning
byJoanne Pettis
 
PPTX
Types of Assessment
byCin Banares
 
PPTX
Types of Language Assessment
byJhun Ar Ar Ramos
 
PPTX
Principles of Language Assessment
byA Faiz
 
Tim McNamara
byTouchstone Institute
 
Self-assessment in language learning
bywillyccardoso
 
Language Assessment - Standards-Based Assessment by EFL Learners
byEFL Learning
 
Developing Rubrics for Language Assessment with Dr. JD Brown
byLanguage Acquisition Resource Center
 
Introduction to Language Assessment by Brown
byEFL Learning
 
Assessment & Language Learning
byBishara Adam
 
Language Assessment Types
byOla Sayed Ahmed
 
technology in standardized language assessment
byHuang YaLi
 
Language Assessment - Beyond Test-Alternatives Assessment by EFL Learners
byEFL Learning
 
Beyond tests alternatives in assessment
byYamith José Fandiño Parra
 
Oral language assessment
byyonfreddygarzon
 
Types of language assessment
byFrancis Calubayan
 
Language Assessment_Formal and Informal
byMæäSii Mööì
 
Language Assessment Principles and Issues
byMaury Martinez
 
Principles of language assessment
byAstrid Caballero
 
Informal Assessment
bysongoten77
 
Portfolio Based Language Assessment (PBLA): Maximizing Assessment for Learning
byJoanne Pettis
 
Types of Assessment
byCin Banares
 
Types of Language Assessment
byJhun Ar Ar Ramos
 
Principles of Language Assessment
byA Faiz
 

Similar to How do we assess task-based performance? with Dr. John Norris

PPTX
Meeting 1 (Test, Assessing, and Teaching).pptx
byTsaltsaNakita
 
PDF
1. presentacion de Evaluation 1st part.pdf
byInesMato
 
PPTX
Educ 1814 assessment new
byCynthia Hatch
 
PPT
Plenary 1 presentation alexis a. lopez
byCarlos Estrada
 
PDF
Neus Figueras: Principled assessment - What it is and what it involves (plenary)
byeaquals
 
PPTX
Group 1 - Devini.AR , Henny, Wahyuni - Language Testing - Mrs.Tiara Dian Sari...
bytiara dian
 
PPTX
Unit1(testing, assessing, and teaching)
byKheang Sokheng
 
PPT
A AND E.ppt
byGulnazFatima6
 
PDF
Eaquals apr 14 barker
byeaquals
 
PDF
Learning_activity1_Erick_Mora.pdf
byERICKJOELMORACALVOPI
 
PDF
Assessing Task-based Language Teaching
byParisa Mehran
 
PDF
ASSESSMENT CONCEPTS AND ISSUES
byAndre Philip Tacderas
 
PDF
languagetestingpresentationintroducinglangauageandassessment-210119155750.pdf
byAttallah Alanazi
 
PPTX
Purposeful language assessment selecting the right alternative test
byyoselin vasuqez
 
PDF
Assessnent literacy apr 14 (fiona barker)
byeaquals
 
PPT
Language Testing
byedac4co
 
PPTX
Language Assessment to help teachers and students
byelizabethhuembes94
 
PPT
Hoffmann magno testing_apiba_27_10_12
byMarisa Hoffmann
 
PPTX
introducing language testing and assessment
byNajah M. Algolaip
 
PPTX
Introduction to Test and Assessment
bysoerdepoer
 
Meeting 1 (Test, Assessing, and Teaching).pptx
byTsaltsaNakita
 
1. presentacion de Evaluation 1st part.pdf
byInesMato
 
Educ 1814 assessment new
byCynthia Hatch
 
Plenary 1 presentation alexis a. lopez
byCarlos Estrada
 
Neus Figueras: Principled assessment - What it is and what it involves (plenary)
byeaquals
 
Group 1 - Devini.AR , Henny, Wahyuni - Language Testing - Mrs.Tiara Dian Sari...
bytiara dian
 
Unit1(testing, assessing, and teaching)
byKheang Sokheng
 
A AND E.ppt
byGulnazFatima6
 
Eaquals apr 14 barker
byeaquals
 
Learning_activity1_Erick_Mora.pdf
byERICKJOELMORACALVOPI
 
Assessing Task-based Language Teaching
byParisa Mehran
 
ASSESSMENT CONCEPTS AND ISSUES
byAndre Philip Tacderas
 
languagetestingpresentationintroducinglangauageandassessment-210119155750.pdf
byAttallah Alanazi
 
Purposeful language assessment selecting the right alternative test
byyoselin vasuqez
 
Assessnent literacy apr 14 (fiona barker)
byeaquals
 
Language Testing
byedac4co
 
Language Assessment to help teachers and students
byelizabethhuembes94
 
Hoffmann magno testing_apiba_27_10_12
byMarisa Hoffmann
 
introducing language testing and assessment
byNajah M. Algolaip
 
Introduction to Test and Assessment
bysoerdepoer
 

More from Language Acquisition Resource Center

PPTX
Assessing Vocabulary with Dr. John Read
byLanguage Acquisition Resource Center
 
PDF
Designing Writing Assessments and Rubrics with Dr. Deborah Crusan
byLanguage Acquisition Resource Center
 
PDF
Assessing Language Using Computer Technology with Dr. Dr. Volker H. Hegelheimer
byLanguage Acquisition Resource Center
 
PPTX
Pathways to Excellence: Using Backward Design Principles for Instruction and ...
byLanguage Acquisition Resource Center
 
PPTX
Diagnosing Strengths and Weaknesses of Foreign/Second Language Readers with D...
byLanguage Acquisition Resource Center
 
PPT
Assessing Listening with Dr. Larry Vandergrift
byLanguage Acquisition Resource Center
 
PDF
Foreign Language Classroom Assessment in Support of Teaching and Learning wit...
byLanguage Acquisition Resource Center
 
PPTX
Integrating Performance Assessment into World Language Learning and Teaching ...
byLanguage Acquisition Resource Center
 
PPTX
NO PROGRAMMING NECESSARY: Tools for creating and sharing online language reso...
byLanguage Acquisition Resource Center
 
PPTX
Using MOBILE PHONES for Language Teaching and Learning
byLanguage Acquisition Resource Center
 
PPTX
What’s So Epic About This Legend? Blending Technologies to Teach Language and...
byLanguage Acquisition Resource Center
 
PPSX
LARC ILR at Ed presentation
byLanguage Acquisition Resource Center
 
PPTX
Don't Put Your Cell Phone Away: Polleverywhere and Google Voice for the Langu...
byLanguage Acquisition Resource Center
 
PPTX
Introducing Project GO - ROTC Language and Culture Project
byLanguage Acquisition Resource Center
 
PPT
Project GO - The ROTC Language & Culture Project
byLanguage Acquisition Resource Center
 
Assessing Vocabulary with Dr. John Read
byLanguage Acquisition Resource Center
 
Designing Writing Assessments and Rubrics with Dr. Deborah Crusan
byLanguage Acquisition Resource Center
 
Assessing Language Using Computer Technology with Dr. Dr. Volker H. Hegelheimer
byLanguage Acquisition Resource Center
 
Pathways to Excellence: Using Backward Design Principles for Instruction and ...
byLanguage Acquisition Resource Center
 
Diagnosing Strengths and Weaknesses of Foreign/Second Language Readers with D...
byLanguage Acquisition Resource Center
 
Assessing Listening with Dr. Larry Vandergrift
byLanguage Acquisition Resource Center
 
Foreign Language Classroom Assessment in Support of Teaching and Learning wit...
byLanguage Acquisition Resource Center
 
Integrating Performance Assessment into World Language Learning and Teaching ...
byLanguage Acquisition Resource Center
 
NO PROGRAMMING NECESSARY: Tools for creating and sharing online language reso...
byLanguage Acquisition Resource Center
 
Using MOBILE PHONES for Language Teaching and Learning
byLanguage Acquisition Resource Center
 
What’s So Epic About This Legend? Blending Technologies to Teach Language and...
byLanguage Acquisition Resource Center
 
LARC ILR at Ed presentation
byLanguage Acquisition Resource Center
 
Don't Put Your Cell Phone Away: Polleverywhere and Google Voice for the Langu...
byLanguage Acquisition Resource Center
 
Introducing Project GO - ROTC Language and Culture Project
byLanguage Acquisition Resource Center
 
Project GO - The ROTC Language & Culture Project
byLanguage Acquisition Resource Center
 

Recently uploaded

PPTX
TLE 8 W1 Q4 D2.pptx COMMON FLOOR PLAN SYMBOLS PLUMBING SYMBOLS
byraquelanaaltres1
 
PDF
Chapter 05 Drug Acting on CNS Sedatives & Hypnotics | Antipsychotics.pdf
bySandeshSul
 
PDF
Risks and opportunities of artificial intelligence in education: A critical view
byYannis
 
PPTX
Day 2 ppt english.powerpoint presentation ppt
byJeahMaeMedez
 
PDF
West Hatch High School - GCSE French Specification
byWestHatch
 
PDF
Darwinism: Theory of Natural Selection and Origin of Species
byIPGDCWA,NAMPALLY
 
PPTX
West Hatch High School - GCSE Media Studies
byWestHatch
 
PPTX
Mohan - "A man of silence and authority"
bynidhibachauhan46
 
PPTX
Appreciations - Jan 26 01.pptxkkkmmkmkmkmkm
bypreetheshparmar
 
PPTX
Drug Distribution in Pharmacology: Mechanisms, Barriers, Factors & Volume of ...
byamrin3379
 
PPTX
West Hatch High School: Year 9 Art Course
byWestHatch
 
PPTX
Poster Based Ethical Reflection - Dharma and Values Poster.pptx
byRajshreeBeneymadoo
 
PPTX
West Hatch High School - GCSE Art Presentation
byWestHatch
 
PPTX
28 January 2026 Rebecca Frankum Are high-stakes exams and assessments still r...
byEduSkills OECD
 
PDF
West Hatch High School - GCSE Media Specification
byWestHatch
 
PPTX
YSPH VMOC Special Report - Measles - The Americas 1-25-2026
byYale School of Public Health - The Virtual Medical Operations Center (VMOC)
 
PPTX
Exploring Higher education and its pathways
byEVAMAEBONGHANOY5
 
PPTX
PREVENTIVE PEDIATRICS.pptx
byBANDITA PATRA
 
PPTX
Unit 3- Culture.pptx....................
byAkanksha Kotangale
 
PPTX
"Aristotle : Father Of Western Philosophy"
bymitalbarathod28
 
TLE 8 W1 Q4 D2.pptx COMMON FLOOR PLAN SYMBOLS PLUMBING SYMBOLS
byraquelanaaltres1
 
Chapter 05 Drug Acting on CNS Sedatives & Hypnotics | Antipsychotics.pdf
bySandeshSul
 
Risks and opportunities of artificial intelligence in education: A critical view
byYannis
 
Day 2 ppt english.powerpoint presentation ppt
byJeahMaeMedez
 
West Hatch High School - GCSE French Specification
byWestHatch
 
Darwinism: Theory of Natural Selection and Origin of Species
byIPGDCWA,NAMPALLY
 
West Hatch High School - GCSE Media Studies
byWestHatch
 
Mohan - "A man of silence and authority"
bynidhibachauhan46
 
Appreciations - Jan 26 01.pptxkkkmmkmkmkmkm
bypreetheshparmar
 
Drug Distribution in Pharmacology: Mechanisms, Barriers, Factors & Volume of ...
byamrin3379
 
West Hatch High School: Year 9 Art Course
byWestHatch
 
Poster Based Ethical Reflection - Dharma and Values Poster.pptx
byRajshreeBeneymadoo
 
West Hatch High School - GCSE Art Presentation
byWestHatch
 
28 January 2026 Rebecca Frankum Are high-stakes exams and assessments still r...
byEduSkills OECD
 
West Hatch High School - GCSE Media Specification
byWestHatch
 
YSPH VMOC Special Report - Measles - The Americas 1-25-2026
byYale School of Public Health - The Virtual Medical Operations Center (VMOC)
 
Exploring Higher education and its pathways
byEVAMAEBONGHANOY5
 
PREVENTIVE PEDIATRICS.pptx
byBANDITA PATRA
 
Unit 3- Culture.pptx....................
byAkanksha Kotangale
 
"Aristotle : Father Of Western Philosophy"
bymitalbarathod28
 

How do we assess task-based performance? with Dr. John Norris

  • 1.
    How do weassess task- based performance? Recent advances in language assessment LARC/CALPER webinar April 3, 2014 John M. Norris – Georgetown University Copyright © John M. Norris, 2014: Please cite as: Norris, J. M. (2014, April). How do we assess task-based performance? Invited LARC/CALPER testing and assessment webinar.
  • 2.
    Plan: 0. What isassessment and why do we do it? 1. Traditions, alternatives  Task-based assessment 2. The range of uses for task-based language assessment 3. Issues in assessing task-based language performance 4. Some challenges and opportunities in TBLA Task-based language assessment: the elicitation and evaluation of language use (across all modalities) for expressing and interpreting meaning, within a well-defined communicative context (and audience), for a clear purpose, towards a valued goal or outcome.
  • 3.
    What is assessment? Whatisn’t assessment (in language education)? CEFR B1
  • 4.
    What is assessmentand why to we do it? Robert Mislevy (2004) Assessment is itself a particular kind of narrative: • an evidentiary argument • about aspects of what students know and can do, • based on a handful of particular things they have said, done, or made. What is educational assessment? Which narratives do we want to tell in language assessment?
  • 5.
    2. The EuropeanUnion and the United States: Possible Comparisons and Lessons You have been invited to give a lecture at the Society for German-American Relations in Weimar next month. For your contribution, the organizers are looking for a presentation on the much-discussed topic “The European Union and the Founding of the United States – A Model?” Your talk is limited to 15 minutes, to be followed by discussion of your remarks. What is assessment and why do it? 1. Fill in the missing letters to create complete words that make sense in the passage. This is an example C-test passage. Starting wi_____ the sec____ word o____ this sent____, the la____ half fr____ each consec____ word h____ been del____. C-tests a____ typically comp____ of mult____ texts wh____ become increa____ difficult. Ea____ text usu____ contains bet____ 20 a____ 25 it____. So, wh____ do y____ think ge____ tested on a C-test? Which is the ‘better’ language assessment?
  • 6.
    The ‘correct’ answer: ITDEPENDS ON THE INTENDED USE of the TEST!
  • 7.
    Specifying intended usesfor assessment (Norris, 2000) INTENDED ASSESSMENT USE WHO? Test Users WHAT? Test Information IMPACT? Test Consequences WHY? Test Purposes The ‘correct’ answer: IT DEPENDS ON THE INTENDED USE of the TEST!
  • 8.
    2. The EuropeanUnion and the United States: Possible Comparisons and Lessons You have been invited to give a lecture at the Society for German-American Relations in Weimar next month. For your contribution, the organizers are looking for a presentation on the much-discussed topic “The European Union and the Founding of the United States – A Model?” Your talk is limited to 15 minutes, to be followed by discussion of your remarks. What is assessment and why do it? 1. Fill in the missing letters to create complete words that make sense in the passage. This is an example C-test passage. Starting wi_____ the sec____ word o____ this sent____, the la____ half fr____ each consec____ word h____ been del____. C-tests a____ typically comp____ of mult____ texts wh____ become increa____ difficult. Ea____ text usu____ contains bet____ 20 a____ 25 it____. So, wh____ do y____ think ge____ tested on a C-test? Intended use: quick estimate of language proficiency, e.g., for use in placing students into language courses…not task- based assessment. Which is the ‘better’ language assessment?
  • 9.
    2. The EuropeanUnion and the United States: Possible Comparisons and Lessons You have been invited to give a lecture at the Society for German-American Relations in Weimar next month. For your contribution, the organizers are looking for a presentation on the much-discussed topic “The European Union and the Founding of the United States – A Model?” Your talk is limited to 15 minutes, to be followed by discussion of your remarks. What is assessment and why do it? 1. Fill in the missing letters to create complete words that make sense in the passage. This is an example C-test passage. Starting wi_____ the sec____ word o____ this sent____, the la____ half fr____ each consec____ word h____ been del____. C-tests a____ typically comp____ of mult____ texts wh____ become increa____ difficult. Ea____ text usu____ contains bet____ 20 a____ 25 it____. So, wh____ do y____ think ge____ tested on a C-test? Intended use: quick estimate of language proficiency, e.g., for use in placing students into language courses…not task- based assessment. Intended use: extended, authentic, and integrated performance with the target language, e.g., for end-of-unit achievement testing and feedback…yes, task-based assessment. Which is the ‘better’ language assessment?
  • 10.
    Making decisions about students admissions placementcourse credit fulfilling requirements grading /achievement advancement/retention certification of abilities awarding funding ETC. INDIVIDUAL DECISION- MAKING EDUCATIONAL USES FOR ASSESSMENT
  • 11.
    Making decisions about students admissions placementcourse credit fulfilling requirements grading /achievement advancement/retention certification of abilities awarding funding ETC. Informing classroom teaching and learning diagnosing learner needs motivating students understanding learning outcomes improving instruction defining learning expectations focusing students’ learning/study ETC. INDIVIDUAL DECISION- MAKING CLASSROOM FEEDBACK EDUCATIONAL USES FOR ASSESSMENT
  • 12.
    Making decisions about students admissions placementcourse credit fulfilling requirements grading /achievement advancement/retention certification of abilities awarding funding ETC. Informing classroom teaching and learning diagnosing learner needs motivating students understanding learning outcomes improving instruction defining learning expectations focusing students’ learning/study ETC. Improving, ensuring, demonstrating quality demonstrating student learning clarifying program mission/needs revising curriculum/ pedagogy improving teacher performance communicating beyond, about the program responding to program review ETC. INDIVIDUAL DECISION- MAKING CLASSROOM FEEDBACK PROGRAM EVALUATION aligning, articulating program practices EDUCATIONAL USES FOR ASSESSMENT
  • 13.
    Making decisions about students admissions placementcourse credit fulfilling requirements grading /achievement advancement/retention certification of abilities awarding funding ETC. Informing classroom teaching and learning diagnosing learner needs motivating students understanding learning outcomes improving instruction defining learning expectations focusing students’ learning/study ETC. Improving, ensuring, demonstrating quality demonstrating student learning clarifying program mission/needs revising curriculum/ pedagogy improving teacher performance communicating beyond, about the program responding to program review ETC. INDIVIDUAL DECISION- MAKING CLASSROOM FEEDBACK PROGRAM EVALUATION aligning, articulating program practices EDUCATIONAL USES FOR ASSESSMENTWhat different kinds of assessment are needed to meet these different uses? How might tasks play a role?
  • 14.
  • 15.
  • 16.
    Early interest in‘tasks’ for assessment “If we want to find out how well a person can perform a task, we can put him to work at that task, and observe how well he does it and the quality and quantity of the product he turns out.” Cureton (1950): “Validity” e.g., in language testing: Foreign Service Institute Oral Interview (1950s) Using tasks to elicit a holistic estimate of functional language ability, from 0 (no ability) to 5 (educated native speaker ability)
  • 17.
    Standardized testing andpsychometrics But, the primary tradition of assessment in the 20th century: Large-scale standardized testing: minimal competency, accountability emphasis on efficiency, technology (e.g., scoring), discrete-point primary concern with psychometrics, norm-referencing validity = narrowly focused on construct representation, reliability e.g., in language testing: Test of English as a Foreign Language (1964)* MCT: vocabulary, reading, listening, structure, grammar (emphasis on testing language knowledge) *But see later iterations of TOEFL…
  • 18.
    Influence of traditionaltesting When we ask teachers what an assessment looks like, what do they tell us? • True-false • Matching • Multiple choice • Fill-in the blank • Exam • Test • Giving grades • Accountability • ETC.
  • 19.
    Influence of traditionaltesting When we ask teachers what an assessment looks like, what do they tell us? Washback of testing traditions: Narrowing teaching and learning Misrepresenting assessment Disconnect between teaching- testing-learning • True-false • Matching • Multiple choice • Fill-in the blank • Exam • Test • Giving grades • Accountability • ETC.
  • 20.
    “It is thesimplification of competencies to what can be ‘objectively’ tested which has a detrimental effect: the learning of trivial facts, the reduction of subtle understanding to generalizations and stereotypes, the lack of attention to interaction and engagement because these are not tested.” e.g., Michael Byram (1997): Need for change in intercultural assessment Challenges to ‘traditional’ testing
  • 21.
    Reactions to ‘traditional’testing Portfolios Performance-based assessments Essays CBTs Self-assessments Interviews Rating scales, rubrics Alternative assessments 1980s-90s •Authenticity •Performance •Can-do (vs. know) •Classroom •Washback
  • 22.
    http://www.learner.org/libraries/tfl/assessment/analyze.html? pop=yes&pid=2003 The move totask as assessment medium Grant Wiggins (1998): Educative assessment The primary assessment medium should be authentic tasks which have learners demonstrate what they can do with what they are learning; enables meaningful integration of teaching, learning, and assessment assessment becomes educative (informative) rather than just an authoritarian mechanism for control.
  • 23.
    The move totask-based language assessment “…the process of evaluating, in relation to a set of explicitly stated criteria, the quality of the communicative performances elicited from learners as part of goal-directed, meaning-focused language use requiring the integration of skills and knowledge” Geoff Brindley (1994): Task-centered assessment aligned with CLT, TBLT movements in pedagogy emphasis on authenticity of L2 use validity = capturing ability for use in valued tasks classroom applications obvious, but beyond?
  • 24.
    Operationalizing task-based languageassessment “…genuinely task-based language assessment takes the task itself as the fundamental unit of analysis, motivating item selection, test instrument construction, and the rating of task performance” Long & Norris (2000): Task-based language assessment 1. Specify intended uses for the assessment: who, what, why, impact 2. Select and analyze key target tasks/features from needs analysis 3. Design tests and items: emphasize authentic performances 4. Determine real-world criteria for rating task performance qualities 5. Pilot-test and revise instruments and procedures (raters/rating) 6. Evaluate validity in terms of intended uses, and especially impact on teaching and learning.
  • 25.
    Norris et al.(1998) Tasks respond to diverse needs/uses in assessment • Focus on what L2 users can actually do with the language • Integrate language form and meaning into what gets assessed • Provide useful feedback to teachers, learners, curriculum • ‘Wash back’ on teaching and learning by making outcomes ‘real’ • Counter negative influence of traditional knowledge assessment • Align testing with new, evolving language pedagogies Task-based language assessment: the elicitation and evaluation of language use (across all modalities) for expressing and interpreting meaning, within a well-defined communicative context (and audience), for a clear purpose, towards a valued goal or outcome. Why task-based language assessment?
  • 26.
    Up Next: How dowe assess task-based performance in language classes and programs? Issues, examples, suggestions… But First: Any Questions?
  • 27.
    Task-based assessment inlanguage classes and programs Issues 1. Which tasks do we assess? 2. How do we elicit task-based performance? 3. What gets assessed in task performance? 4. How do we score task performance?
  • 28.
    1. Which tasksdo we assess? e.g.: EAP pragmatics assessment (Youn, 2009) 1. Write a recommendation request e-mail to professor 2. Write an e-mail to potential employer to send application packet 3. Write an e-mail to refuse professor’s request 4. Write constructive comments on cover letter written by classmate 5. Give oral peer-feedback on classmate’s request e-mail to professor 6. Interacting with classmates in situations of making suggestions and disagreement 7. Interacting with a professor in situations of making requests and refusal Why these tasks?
  • 29.
    1. Which tasksdo we assess? Why these tasks?Which tasks? Based on needs analysis for specific English language program context (individual & interactive tasks) High priority tasks reflecting pragmatic challenges for college-level international students Each task serves as end-of-unit achievement assessment Combined, all of the tasks can be used for diagnostic assessment
  • 30.
    You have beengiven the opportunity of a lifetime to apply for an athletic training camp in [e.g., Spain], tuition free! This camp trains young people in all sports, from the extreme (snowboarding, bicycle motor cross, rollerblading) to team sports of all kinds (basketball to volleyball). You name it, they help you train for it! To be accepted into the camp, all applicants have to convince the admissions office that they have good exercise and nutrition habits. First, you will read about health and nutrition from the perspective of the Spanish – speaking world. Then you will discuss your eating and exercise regimen with your partner to compare your nutrition and exercise, perhaps event to get some ideas. You will then write your application letter to the summer camp describing your nutrition and training regimen, convincing them that you are well prepared for the camp and need to be accepted. 1. Which tasks do we assess? e.g., Integrated performance assessment in US schools Overall introduction to a sequence of interrelated pedagogic/assessment tasks. Note the distinct task phases. Consortium for assessing performance standards http://flenj.org/CAPS/?page=parent
  • 31.
    1. Which tasksdo we assess? e.g., Integrated performance assessment in US schools Consortium for assessing performance standards http://flenj.org/CAPS/?page=parent Individual reading task Pair discussion task Letter writing task Why these tasks?
  • 32.
    1. Which tasksdo we assess? e.g., Integrated performance assessment in US schools Consortium for assessing performance standards http://flenj.org/CAPS/?page=parent Individual reading task Pair discussion task Letter writing task Self-assessment: understanding task content, language Peer-assessment: feedback on development and expression of ideas Teacher-assessment: effectiveness of letter, language use Why these tasks?
  • 33.
    1. Which tasksdo we assess? Which tasks? Based on the U.S. Standards for Foreign Language Learning, by specific age/grade level and proficiency 3 integrated tasks—Interpretive, interpersonal, presentational—reflect distinct modes of performance Tasks are closely integrated with what is being taught: this is why you are learning what you are learning Tasks provide the opportunity for self-, peer-, and teacher-assessment and feedback Assessment raises awareness about effectiveness of teaching and learning See Adair-Hauck, et al. (2006) Why these tasks?
  • 34.
    2. How dowe elicit task-based performance? The European Union and the United States: Possible Comparisons and Lessons Task You have been invited to give a lecture at the Society for German-American Relations in Weimar next month. For your contribution, the organizers are looking for a presentation on the much-discussed topic “The European Union and the Founding of the United States – A Model?” Your talk is limited to 15 minutes, to be followed by discussion of your remarks. Please observe your position as a speaker to this particular audience. While you should expect good background knowledge from your German audience regarding the situation in the EU, you primarily contribute the perspective of an American who is familiar with the foundational history of the USA and subsequent development of the county. In its particularities this is most likely to be less familiar to your audience. Therefore, it is precisely those aspects that you must bring to the awareness of your audience in order to link them to the present situation in Europe. In other words, continuously keep in mind this occasion for your speech, this starting point for your presentation, and this intention of your talk. e.g.: Georgetown German: Prototypical Performance Writing Task Level IV course, “Text in Context” What does this description accomplish?
  • 35.
    Example: CNaVT Certificate inDutch as a Foreign Language Tourist domain Listening comprehension task “Visit to the museum” Item: Identify the paintings based on the descriptions you hear from an audio museum guide. What is the purpose of the paintings?
  • 36.
    Task Title: Recycling Theme:Environment Level: Novice-Mid (Focus Age Group: 11 – 14) Communicative Mode: Presentational Time Frame: One 42 minute period or more depending on the class and the type of presentations chosen. Description of Task: Now that you’ve learned about saving the planet and the need to reuse and recycle, you want to share with your family your ideas for reusing some throw away things. You have cleverly thought of a few uses for reusing ____________. Now you will share your new ideas. Create and develop your idea and choose one from the following types of presentations to present it. Continued on next page… Consortium for assessing performance standards http://flenj.org/CAPS/?page=parent e.g.: assessing young/beginning learners, US schools 2. How do we elicit task-based performance?
  • 37.
    Title: ¿Qué puedohacer con_________? (What can I do with_______?) Include the following in your presentation: • four or more ways of re-using something • specific uses • descriptions/elaboration Three options, you choose: Power point presentation Poster Live oral 3-D presentation Materials Needed: computers, markers, poster board, realia Why give the student these options? 2. How do we elicit task-based performance?
  • 38.
    2. How dowe elicit task-based performance? Task-based performance elicitation? Give a purpose for communicating (motivate) For interactive/presentational tasks, indicate or provide an audience Provide a structure or procedures for doing the task Utilize authentic input and other realia for situating the task in a meaningful context Indicate desired outcomes or products of the task Where feasible, encourage ownership over the task through individual choice in its accomplishment
  • 39.
    3. What getsassessed in task performance? Test task: Order pizza for the class. Take orders Place orders Task accomplishment?
  • 40.
    3. What getsassessed in task performance? Test task: Order pizza for the class. XTake orders Place orders Task accomplishment?
  • 41.
    3. What getsassessed in task performance? e.g.: classroom assessments for middle-school learners in Flanders (Jordens, 2007) Type task: (from learning goals) Pupils can report to the teacher on a subject, previously discussed at school Test task: Pupils report to the teacher on a film about scuba diving, the report being based on photographs extracted from the film.
  • 42.
    e.g.: classroom assessmentsfor middle-school learners in Flanders Parameters for assessment What does pupil say? Content relevance, completeness Clarity Organization How does pupil say it? Fluency Articulation Manner of speaking Variation and extensiveness Accuracy: word/sentence Interaction skills 3. What gets assessed in task performance? How is this rubric used?
  • 43.
    Give a presentationin Spanish about your life and interests. Indicate how these activities reflect your personality. Include some possibilities for your future in terms of career, study, travel, pursuing personal interests. e.g.: performance assessment for 8th graders in US school 3. What gets assessed in task performance? Source: http://depts.washington.edu/mellwa/Events/20081105 /sandrock_ipa_handout.pdf
  • 44.
    EXCEEDS EXPECTATIONS MEETS EXPECTATIONS DOES NOT MEET EXPECTATIONS Dowe understand you? (Comprehensibility) My message is easily understood by my audience. I am able to communicate my message by reproducing memorized material. I rely heavily on visuals to convey my message. How do I use the Spanish language? (Language Control & Vocabulary Use) I can write/speak in simple sentences correctly. My presentation is rich in appropriate vocabulary. I demonstrate some accuracy in written/oral presentation when I reproduce memorized words and phrases. My vocabulary reveals basic information. My presentation is correct only at the word level. My vocabulary is limited and / or repetitive. I use English. How do I organize the presentation? (Communication strategies) My presentation is supported with many enriched examples. My presentation has some visuals/realia and examples. My presentation lacks organization. How well do I impact the audience? (Impact) I use visuals/ realia and gestures, and tone of voice to maintain audience’s attention. I use some visuals/realia and gestures to maintain audience’s attention. My tone of voice is acceptable. I use a small amount of effort to maintain audience’s attention. What is the purpose of these criteria?
  • 45.
    Criteria for judgingperformance? Consider real-world criteria for task success or accomplishment, including as appropriate: Performance products: Does something happen, does something get produced? Performance steps: Do critical elements of the task happen? Performance qualities: linguistic and paralinguistic features (e.g., accuracy, fluency, gesture, pragmatics), content (e.g., meaning), other attributes (e.g., visual support, efficiency of task completion) Performance levels: How well does the task performance meet expectations, standards, etc.? 3. What gets assessed in task performance?
  • 46.
    e.g., Georgetown Univ.German writing assessments Writing task 1 ______ ______________ ______________ ______________ _____ Writing task 2 ______ ______________ ______________ ______________ _____ Writing task 3 ______ ______________ ______________ ______________ _____ Writing task 4 ______ ______________ ______________ ______________ _____ Writing task 5 ______ ______________ ______________ ______________ _____ End-of-level Prototypical task _________ _________________ _________________ _________________ _________________ _________________ ____ 4. How do we score task performance?
  • 47.
    Final Grade Feedback: Rating rubric, In-text Initial Grade Task–based assignment Level scoring rubric Writing task v. 1 ______ ______________ ______________ ______________ _____ Writing task v. 2 ______ ______________ ______________ ______________ _____ 4. How do we score task performance? Formative assessment: Process writing Providing rich feedback to enable learning Teacher Students
  • 48.
    End-of-level Prototypical task _________ _________________ _________________ _________________ _________________ _________________ ____ Example Journalistic portrait: The Vietnamese immigrant experiencein contemporary Germany 2 raters from the curricular level + 2 raters from another level 4. How do we score task performance? Summative assessment: Performance writing Providing reliable judgments of language + task abilities
  • 49.
    Scoring and scorereporting for different purposes? Consider how ‘scores’ will be used before deciding how to score: Formative assessment: Use public criteria and scales, and encourage self-, peer-, teacher-scoring for feedback Summative assessment: Use well-defined criteria and scales, and multiple, trained raters in scoring for judgment Holistic scoring (overall judgment) provides and indication of task success or accomplishment or completion Analytic scoring (by categories and qualities) provides a useful basis for remediation Rich and detailed score reporting enables learning from assessment (e.g., in person, score + justification) 4. How do we score task performance?
  • 50.
    Up Next: The rangeof current uses for task-based language assessment But First: Additional Questions?
  • 51.
    Tasks as targets:Policies & standards V. Workplace Tasks: Reading --Scan basic charts, tables, maps or schedules for information. (5) --Find routine information on the computer screen/scanner screen/computerized display screen, if available. (6) --Read information in the reception/appointment book to find available openings for a new appointment. (6) --Read a checklist to verify if all the steps in the procedure have been completed. (6) --Follow one page of clear familiar instructions. (7) --Read a reminder or complaint letter/memo, take appropriate action. (7) --Scan complex charts, tables and schedules for several specific pieces of information for comparison/contrast. (7) --Follow instructions on evacuation procedures, fire drills, or on using simple machinery/equipment.(7) --Use plain language manual with familiar topic and content in own field of knowledge to find specific information. (8) ETC… Who should be able to do these tasks? For example…
  • 52.
    Tasks as targets:Policies & standards V. Workplace Tasks: Reading --Scan basic charts, tables, maps or schedules for information. (5) --Find routine information on the computer screen/scanner screen/computerized display screen, if available. (6) --Read information in the reception/appointment book to find available openings for a new appointment. (6) --Read a checklist to verify if all the steps in the procedure have been completed. (6) --Follow one page of clear familiar instructions. (7) --Read a reminder or complaint letter/memo, take appropriate action. (7) --Scan complex charts, tables and schedules for several specific pieces of information for comparison/contrast. (7) --Follow instructions on evacuation procedures, fire drills, or on using simple machinery/equipment.(7) --Use plain language manual with familiar topic and content in own field of knowledge to find specific information. (8) ETC… Who should be able to do these tasks? Canadian Language Benchmarks: English Language Demands for Nursing For example…
  • 53.
    Tasks as targets:Policies & standards (e.g., Canadian Language Benchmarks) “The CLB is task-based: • In syllabus design, tasks are considered to be basic building blocks.• Tasks in language learning promote the integration of all aspects of communicative competence, and multilevel language processing.” CEFR europe CLB canada ACTFL SFLL U.S. TESOL/NCATE teacher development Why? Tasks provide meaningful targets: Ability to use the language Washback on teaching and learning TOC hong kong
  • 54.
    Tasks in large-scaleproficiency assessment 4. Integrated Speaking: Read a passage from a psychology textbook and listen to the lecture that follows it. Then answer the question. Reading: Flow In psychology, the feeling of complete and energized focus in an activity is called flow. People who enter a state of flow lose their sense of time and have a feeling of great satisfaction. They become completely involved in an activity for its own sake rather than for what may result from the activity, such as money or prestige. Contrary to expectation, flow usually happens not during relaxing moments of leisure and entertainment, but when we are actively involved in a difficult enterprise, in a task that stretches our mental or physical abilities. THEN For example…
  • 55.
    Listen to theLecture (Male professor): I think this will help you get a picture of what your textbook is describing. I had a friend who taught in the physics department, Professor Jones, he retired last year. . . . Anyway, I remember . . . this was a few years ago . . . I remember passing by a classroom early one morning just as he was leaving, and he looked terrible: his clothes were all rumpled, and he looked like he hadn’t slept all night. And I asked if he was OK. I was surprised when he said that he never felt better, that he was totally happy. He had spent the entire night in the classroom working on a mathematics puzzle. He didn’t stop to eat dinner; he didn’t stop to sleep . . . or even rest. He was that involved in solving the puzzle. And it didn’t even have anything to do with his teaching or research; he had just come across this puzzle accidentally, I think in a mathematics journal, and it just really interested him, so he worked furiously all night and covered the blackboards in the classroom with equations and numbers and never realized that time was passing by. Question: Explain flow and how the example used by the professor illustrates the concept. Tasks in large-scale proficiency assessment Q: Who should be able to do these tasks? For example…
  • 56.
    Lecture (Male professor):I think this will help you get a picture of what your textbook is describing. I had a friend who taught in the physics department, Professor Jones, he retired last year. . . . Anyway, I remember . . . this was a few years ago . . . I remember passing by a classroom early one morning just as he was leaving, and he looked terrible: his clothes were all rumpled, and he looked like he hadn’t slept all night. And I asked if he was OK. I was surprised when he said that he never felt better, that he was totally happy. He had spent the entire night in the classroom working on a mathematics puzzle. He didn’t stop to eat dinner; he didn’t stop to sleep . . . or even rest. He was that involved in solving the puzzle. And it didn’t even have anything to do with his teaching or research; he had just come across this puzzle accidentally, I think in a mathematics journal, and it just really interested him, so he worked furiously all night and covered the blackboards in the classroom with equations and numbers and never realized that time was passing by. Question: Explain flow and how the example used by the professor illustrates the concept. Tasks in large-scale proficiency assessment Q: Who should be able to do these tasks? Test of English as a Foreign Language: English for Academic Purposes Example…
  • 57.
    Tasks in large-scaleproficiency assessment (e.g., TOEFL iBT) “Working directly with higher education institutions, we designed the TOEFL test to simulate actual academic classroom tasks.” TOEFL iBT U.S. CNaVT Holland/Belgium IELTS U.K. NMET China AMES Australia Why? Tasks provide improved interpretations, construct validity Washback on teaching and learning
  • 58.
    Tasks in professionallanguage certification Example…
  • 59.
    Tasks in professionallanguage certification http://www.anglo-continental.com/en/uk/forms/Aviation/Part%202%20%20ready.mp3 Q: Who should be able to do these tasks? Example…
  • 60.
    International Civil AviationOrganization: English for Aviation Purposes Tasks in professional language certification http://www.anglo-continental.com/en/uk/forms/Aviation/Part%202%20%20ready.mp3 For example…
  • 61.
    Tasks in professionallanguage certification (e.g., ICAO, 2007) “Because of the high stakes involved, pilots and controllers deserve to be tested in a context similar to that in which they work and test content should therefore be relevant to their roles in the work-place” FBI Translation CLB- Nursing NCATE L2 teaching English Aviation Why? Tasks demonstrate ‘on-the-job’ abilities for high-stakes decisions Washback on teaching and learning OET Medical
  • 62.
    Overview of task Ingroups, learners design a brochure for the Hong Kong Tourism Board describing 4 attractions in Hong Kong which would appeal to young people of their own age. Writing a Tourist Brochure Imagine the Hong Kong Tourism Board has asked your class to design a brochure that would be of interest to young visitors of your own age. In groups of 4, design the brochure describing FOUR sites suitable to young people of your own age coming to Hong Kong. Complete this task by following the steps below: Step 1: Group Task. Discuss in your groups which sites young people would want to visit in Hong Kong. Choose one site each to investigate. For homework find out as much as you can about the site, where the site is, when it is open, what one can see / do there, what the facilities are, how one gets there, etc. Bring this information to the next class. For example… Tasks in classrooms and programs
  • 63.
    Step 2: GroupTask. Exchange information with your group members. Tell them about the site you have found out about. Then decide how you are going to present the information in your brochure, what order you want to put your sites in, what illustrations you need, what title you want to give the brochure, etc. Step 3: Individual Task. Write a description about your chosen site (120 words). Remember to say why it is interesting. Proofread it carefully, then hand it to your teacher. Step 4: Group Task. In your groups edit your work based on your teacher’s comments. Then put together your brochure. Your brochures will be assessed on the following basis: • Task fulfilment: would your selected sites appeal to young people? • Accuracy of language and information provided: is the brochure written in good English? Is the information provided accurate? • Attractiveness of final written submission: is your brochure really attractive? Can you make it more appealing? For example… Tasks in classrooms and programs Q: Who should be able to do these tasks?
  • 64.
    Step 2: GroupTask. Exchange information with your group members. Tell them about the site you have found out about. Then decide how you are going to present the information in your brochure, what order you want to put your sites in, what illustrations you need, what title you want to give the brochure, etc. Step 3: Individual Task. Write a description about your chosen site (120 words). Remember to say why it is interesting. Proofread it carefully, then hand it to your teacher. Step 4: Group Task. In your groups edit your work based on your teacher’s comments. Then put together your brochure. Your brochures will be assessed on the following basis: • Task fulfilment: would your selected sites appeal to young people? • Accuracy of language and information provided: is the brochure written in good English? Is the information provided accurate? • Attractiveness of final written submission: is your brochure really attractive? Can you make it more appealing? Task-Based Assessment for English Language Learning: Hong Kong, Secondary Schooling Example… Tasks in classrooms and programs Q: Who should be able to do these tasks?
  • 65.
    Tasks in classroomsand programs (e.g., Hong Kong Special Administrative Region) “Task-based assessment for English language learning at secondary level: One of the key principles underpinning the school English language curriculum in Hong Kong is that of task-based language teaching. In this package we introduce the features of task-based assessment (TBA) within a school-based curriculum…” http://cd1.edb.hkedcity.net/cd/eng/TBA_Eng_Sec/index.html Local practice TBLT research Can-do e.g., DIALANG Teacher guides ACTFL IPAs Why? Tasks enable feedback on learning, teaching, curriculum Washback on teaching and learning
  • 66.
    Task-based language assessmentin practice Administrators Staff Classroom: TBLAssessment For Learning Program: Tasks as summative indicators of teaching effectiveness Profession: Tasks for certification of important abilities Large-scale: Tasks for demonstrating proficiency in societal domains (e.g., education) Standards: Tasks as valued targets for learning
  • 67.
    Some challenges andopportunities In Task-Based Language Assessment…
  • 68.
    2002 (19.4) • Howdo we interpret task-based L2 performance? authentic criteria, proficiency, L2 production? • Can task-based performances be rated reliably? analytic/holistic rating scales, rater training? • What is the nature of task difficulty? cognitive complexity, learner individual diffs? • Can we generalize and/or extrapolate based on tasks? domain-sampling, ECD, underlying constructs? • How does task-based assessment impact instruction? teacher, learner, curricular consequences? Basic validity questions for TBLA Norris (2008) “Fundamentally…assessments are only good insofar as their use does good, in terms of supporting educational efforts and outcomes, and it is my intent that validity evaluation helps to ensure that they do.”
  • 69.
    How do weinterpret task-based L2 performance? Can do a specific task Has a certain proficiency Displays X language qualities High-fidelity task simulation or replication Test of aviation English Domain sampling, ‘type’ tasks TOEFL, CNAVT Implicit, spontaneous elicitation Classroom practice, SLA research
  • 70.
    Can task-based performancesbe rated reliably? YES!!! Individual task accomplishment: single task Highly specified indigenous criteria, steps, etc. Teacher, tester, learner awareness-raising Ability for L2 use: multiple/type tasks Cross-task holistic/analytic rating scales Rater training
  • 71.
    What is thenature of task difficulty? Why is this task so hard? Task features e.g., number of elements, number of steps, planning time, structure, reasoning requirements, code frequency/complexity Task conditions e.g., number of participants, open/closed task, convergent/divergent, familiarity, gender, status Individual differences e.g., task familiarity, motivation, testing stakes, aptitude, personality More difficult = better or worse performance???
  • 72.
    Can we generalizeand/or extrapolate based on tasks? Of course we can  and we do it all the time! Driving testseries of tasks (1-time!) Generalizationcan drive a car Extrapolation can negotiate a roundabout in London??? …be careful!
  • 73.
    Can we generalizeand/or extrapolate based on tasks? 1. Listening comprehension? 2. Listening comprehension in a particular domain? 3. Listening comprehension on a specific task? 1. Adequate construct definition 2. Adequate domain sampling3. Adequate task representation
  • 74.
    How does task-basedassessment impact instruction? Local impact (Georgetown German): “So I think it has filtered down to smaller writing assignments. First of all so that students get used to it and it’s a coherent system. Whereas before we would say, ‘Okay, now write one page about your family and your Heimat’. Now we actually make up a sheet and use the same features.” Global impact (TOEFL) “…considerable changes occurred in the amount of attention the teachers paid to the development of speaking and to the integration of different skills.”
  • 75.
    In sum, task-basedperformances and assessments may not (should not) meet all purposes for assessment within language education, but they probably offer a lot of advantages for certain purposes: Task-based performances tell us things that other item types do not (e.g., ability to accomplish particular tasks under particular conditions) Task-based performances offer rich opportunities for eliciting and observing language use with meaningful content in context (there is always more to language use than just language, and tasks capture that) Task-based performances can reveal multiple aspects of language ability and/or development within a single task (e.g., accuracy, complexity, fluency; content knowledge; procedural knowledge; expertise) Task-based performances are in many cases the most indicative targeted expectations of language classes and programs (if not, then what exactly are we teaching?) Task-based assessments provide relevant and comprehensive frames of reference for learners and teachers to generate feedback on form-function-meaning relationships (formative, learning-oriented uses) Task-based assessments communicate to teachers and learners and others what is valued within a given educational setting (motivational, educative uses) Task-based assessments may impact curriculum and educational systems, causing a rethinking of what and how teaching is occurring (washback uses) Task-based assessments generate interpretable data regarding the worthwhile outcomes of language learning programs (summative uses, e.g., for accountability, certification, program evaluation)
  • 76.
    Why task-based assessment? Tasksare useful: • align to instructional practices and values • provide rich samples of language use data • inform how well we teach and learn • show what L2 users can do, and motivate them to learn Task-based assessments can: • improve accuracy of interpretations • ensure abilities to use language for specific purposes • guide educational practices • counter negative washback of other assessments • enable a variety of uses, decisions, actions Q: Task-based assessment is a lot of work, and we can make somewhat accurate assessments of learners’ abilities and knowledge with short-cut tests; why should we bother with TBLA?
  • 77.