2. Outline
1. Our task
2. Widgets
3. TBLT & TBLA
4. Approach to TBLA
5. Needs Analysis
6. Test design and purpose
7. The tasks and their assessment criteria
8. Limitations
9. References
2
3. Our task
Design a task-based language test for a group of students who have done
the first unit of the task-based course in practical English Widgets
+ Appendix C: Video Scripts
-design decisions
-Instrument(s)
-tasks will the students have to do
-judgment criteria
-generalisability, task difficulty, and task variability
3
4. Widgets
http://www.widgets-inc.com/teacher/tblt.php
“What makes Widgets truly different from other language
teaching textbooks is its focus on creating a believable,
“real-life” English-speaking environment in the classroom.
Students imagine that they are new employees at an
exciting international company, Widgets Incorporated,
where they must work together in small groups to perform
various practical, yet fun, tasks.”
4
5. Widgets (2)
Initial reservations with the course:
Are the tasks truly authentic when they often centre round make-believe, weird products
We did not find these fantastical products either fun or motivating
Language is given before a task to be used in that task (not TBL)
There is little evidence of language being fed in when the need arises (this is TBL)
TBL is only one type of methodology for teaching language.
Shouldn’t it be presented with others to provide a balanced syllabus?
Is it desirable that a course book confines itself to one methodology?
5
6. TBLT & TBLA
“Task-based language assessment takes the task itself as the fundamental
unit of analysis motivating item selection, test instrument construction, and
the rating of task performance. Task-based assessment does not simply
utilize the real-world task as a means for eliciting particular components of
the language system, which are then measured or evaluated; instead, the
construct of interest is performance of the task itself.”
Brown, Hudson Bonk and Norris
“Many traditional methodologies begin by teaching grammatical forms and
then go on to set communicative activities in which they believe learners
will be able to use these forms. The initial aim of TBT is to encourage
learners to engage in meaning with the language resources they already
have. This makes learners acutely aware of what they need to learn.”
Dave and Jane Willis
6
7. Approach to TBLA
7
1. Specify intended uses for the assessment: who, what, why, impact
2. Select and analyze key target tasks/features from needs analysis
3. Design tests and items: authentic elicitation, evaluation conditions
4. Determine real-world criteria for rating task performance qualities
5. Pilot-test and revise instruments and procedures (raters/rating)
6. Evaluate validity in terms of intended uses, and especially impact
on teaching and learning.
(Long & Norris 2000)
8. 1) Intended assessment use
8
Who?
Test takers:
-12-40 Asian students
-between 16-24 years old
“Stake holders” :
-Teachers
-parents
What?
Similar tasks that the students have
been exposed to/ dealt with in the
teaching
Impact?
-Washback effect
-Low stakes
Why?
The purpose of the test is to gather
information about the student learning,
i.e. whether the students, after having
completed the first unit in Widgets, are
able to perform these tasks and how
language is used among them.
Moreover, the assessment might
support further teaching and learning.
9. 2) Key target tasks & features from NA
9
Can do statements from Widgets
10. 10
2) Key target tasks & features from NA (2)
Objectives from Widgets
11. 3) & 4) Test design & tasks
11
Importance of task assessment being based on tasks known or familiar to the
students
Achievement test
Transfer of the tasks from the context of the company (as in the unit)
into a different contextual framework: biannual trade fair
assessment spin-off to unit 1
12. Tasks and their assessment criteria
Task 1 – speaking (role play):
Task title: Catching up over coffee
Theme: Chat
Level: intermediate
Time frame: 3-5 minutes per pair
Responds to: water cooler chat
Communicative Mode: Role Play
Description of task:
As an employee of Widgets Inc. you are attending a biannual trade fair in New
York where you meet people from other companies. You are acquainted with most
of them. On the first day during a conference break you meet a former colleague
of yours who now works at another Widgets subsidiary. You two enjoy the chance
to catch up socially over coffee. In pairs exchange information about events in
your lives over the last 12 months. The prompt cards provided will help you find
topics to talk about (e.g. work, holidays etc.).
12
13. 13
Tasks and their assessment criteria (2)
Task 1 – assessment criteria
a) Task-dependent criteria (Norris et al 2002)
appropriateness of questions
ability to produce follow-up questions that show the speaker A has listened to speaker B
ability to respond to follow-up questions appropriately rather than beginning a new topic
ability to maintain dialogue and begin a new topic when necessary
b) Task-independent criteria
appropriateness of vocabulary and structure for intermediate
discourse markers
turn taking elements
rejoinders
14. 14
Tasks and their assessment criteria (3)
Task 1 – Scoring
5-0 scale
Reason: reflects the self-assessment task at the end of stage 1
15. Tasks and their assessment criteria (4)
Task 2.1 – listening and note taking:
Task title: Competing products
Theme: Advertisement
Level: Intermediate
Time frame: depends on the length of recordings (approx. 6-7 min)
Responds to: “Water cooler chat” & “That’s my department”
Communicative Mode: Aural reception/ information gap filling
Description of task:
As a conference participant you decide to listen to the presentation of three products from a
competitor. You and your company have heard some things of the products before, but are
still missing more detailed information about the products. You listen to the presentation to
find out more details about the product and to tell your superiors more about the products
later on. Therefore, you will listen to the presentations twice. While listening to the
presentation for the first time, take notes on the form provided. When listening a second
time make notes that you will need to tell your boss more about the competitive products.
15
17. 17
Tasks and their assessment criteria (5)
Task 2.1 – assessment criteria
a) Task-dependent criteria (Norris et al. 2002)
-Completeness of information
-Listening for detail/ keyword/ gist
-I can take short notes in English
b) Task-independent criteria
-Alternative spellings
18. 18
Tasks and their assessment criteria (6)
Task 2.1 – Scoring
a) Information gap
-Point for each blank
-Cutoff: 3 correct
b) Note taking
- Will be taken into consideration in the context of the presentation
Task 2.1 = scaffolding for the following presentational task
19. Tasks and their assessment criteria (7)
Task 2.2 - presentation:
Task title: Product presentation
Theme: Marketing
Level: intermediate
Time frame: Preparation 20 min; task 3-5 min
Responds to: “Sell, sell, sell”
Communicative Mode: Presentational
Description of task:
You have returned to your company site from the conference. Your superiors are
eager to know what you learned and which of the products you consider to be the
most likely competitor. Go back to your notes from the listening task and prepare
a short presentation for your group on the product you consider to be the most
competitive one for Widgets.
19
20. 20
Tasks and their assessment criteria (8)
Task 2.2 – assessment criteria
a) Task-dependent criteria (Norris et al. 2002)
-speaks smoothly without any pauses
-structure of the presentation: introduction, description, good and bad points, why it's a
competitor, conclusion
-quality of delivery - longer turns, lack of hesitation, clarity of pronunciation
- does the presentation content reflect a success of the listening
-b) Task-independent criteria
-signpostings
-should we include body language/gestures?
21. 21
Tasks and their assessment criteria (9)
Task 3 – Scoring
-Scale numerical 5-0
-Descriptors such as from excellent to not sufficient
22. Limitations
22
General limitations:
Not Constructed the materials
Interlocutor frame for assessment
Outline of task, rather than creation of tasks
Descriptors on rating scales need to be considered
Concerns expressed in the literature no easy answers
Limitations based on Widgets:
No clear language objectives outlined in Can do statements
Level of learners is not well-defined
Can Do statements are very general
suggestion to have an overall written report on the entire assessment series
23. References
Elder, C., Iwashita, N. & McNamara, T. (2002). Estimating the difficulty of oral proficiency tasks: what does
the test-taker have to offer? Language Testing, 19 (4), 347 – 348.
Long, M. H., & Norris, J. M. (2000). Task-based teaching and assessment. In M. Byram (Ed.), Encyclopedia of
language teaching (pp. 597-603). London: Routledge.
Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee abilities and
task difficulty in task-based second language performance assessment. Language
Testing, 19(4), 395-418.
Wigglesworth, G. (2008). Task and performance based assessment. In S. Shohamy & N. H. Hornberger
(Eds.), Encyclopedia of language and education (2nd Ed.), Volume 7: Language testing and assessment
(pp.111-122). Springer Science+Business Media.
Willis, D & Willis, J. (2007) Doing task-based Teaching, Oxford
http://flenj.org/CAPS/?page=parent
23
Editor's Notes
Stages 3 and 4 of the approach to TBLA shows how, as an achievement test, our instrument relates directly to tasks in the course book. Given that it is important that task assessment is based on tasks known or familiar to the students, the assessment should arise out of the content and tasks dealt with in unit 1.
In terms of context, there is a slight shift from the context of the company to that of a biannual trade fair. So – similar performances will be elicited in a different context.
Here is task 2. Task 2 consists of a task in 2 steps and this is the first step. The task combines a listening and note taking task and a presentation task that build upon one another. Students need to know that the listening is a pre-step presentation task. In the task the students will listen to the presentation twice. We realize that this is not very authentic. However, given the level of pre-intermediate or intermediate, the division into Task 2.1 and 2.2 functions as a scaffolding as students can listen for detail during the first time and for gist as they listen to the presentation again.
To familiarize yourself with the task, I give you a moment now to read through this slide and focus on the task.
…..
This is the form that students have to complete while they are listening and it very much mirrors the information gap activity in first part of the “water cooler chat” outlined in unit 1 of widgets. Students are required to listen for detail helping them to basically structure the information they hear and serve as a scaffolding device. What can be seen from this part of the task is that students are required to integrate skills: listening, writing and speaking as well as to some extent reading from their own notes. This integration of skills is a key feature of TBLA.
