1. Welcome!
The Grid - Reloaded
Competence-based test design and
assessment

2. The theory: what we need for a...
Validity
Authenticity
Good Reliability
test
Feasibility

3. Validity
= The degree to which the test actually
measures what it intends to measure.

4. Reliability
= The consistency of results that a
particular test produces.
Reliability of
Test reliability test conditions
Reliability
Scorer Student-related
reliability reliability

5. Feasibility
= is the test practical...
◦ to set up?
◦ to administer?
◦ to assess/mark?
◦ to finance?

6. Authenticity
àHow convincingly does the test simulate
“real-world” tasks?
— Can be increased through:
◦ Contextualisation
– “Authentic” topics, situations
– Linked sentences, not isolated ones
à thematic organisation (story, episodes...)
◦ Layout and instructions of test tasks

7. Keep in mind:
The importance of specifying:
- Form
- review, story, letter...
- Audience
- familiarity, distance
- Topic
- Purpose
- entertain, persuade, describe...

8. Any remaining questions?
Good, because now it’s YOUR turn...

9. What you need to do now:
1. Use the marking grid you are given to
assess and mark the three student
productions.
2. Use the Think Aloud Protocol to record
any reflections, justifications... that
underpin your decision(s).
3. Compare and discuss your findings with
the other members of your group.

10. Conclusion:
Why work with marking grids?
— Criterion-referenced assessment
◦ Systematic, nuanced
◦ Qualitative and quantitative considerations
◦ Accountability
— Avoiding dangers of ‘gut-feeling’ assessment
— Increasing scorer reliability
— Standardising assessment procedure
— Chance for focused, detailed feedback
— A positive look at the “bigger picture”

11. BAND BAND BAND
BAND 1 BAND 3 BAND 5
0 2 4
Only 1/3 of the content elements dealt with Only 2/3 of the content elements dealt with: All content elements covered appropriately:
and / or hardly any content elements dealt message only partly communicated to the message clearly communicated to reader.
with successfully. reader; and /or not all content elements dealt
Message only partly communicated to the with successfully. Support data might be The desired effect on reader is fully achieved.
reader and / or considerable effort required on missing or there might be irrelevant detail. Overall, no effort required of the reader.
behalf of the reader. Some effort required of the reader. The format is appropriate
Format only partly respected. Awareness of format, but occasional
inappropriacies.
CONTENT
• Task achievement
• Effect on reader
• Format (if required)
Response may contain some inconsistencies. Overall coherent response. Coherent response.
Only short simple sentences. At times limited Mostly good control of simple sentences. Good control of simple sentences and in
No rateable language. Totally incomprehensible. Totally irrelevant
control of simple sentences. Complex sentence forms attempted, but they general also of complex sentences.
The sentences are simply listed rather than tend to be less accurate than simple Coherence is achieved by use of simple
connected and presented as a text. sentences. linking devices.
ORGANISATION:
Use of simple linking devices such as ‘and’, Response is well organised.
COHERENCE AND COHESION
‘or’, ‘so’, ‘but’ and ‘because’.
• Logic
Serious effort by reader required.
Information presented with some
Fluency
Little effort by reader required.
•
Features of bands 1 and 3.
Features of bands 3 and 5.
organisation.
• Control of linking devices,
referencing...
• Paragraphing
• Punctuation
Limited range which is minimally adequate for In general, appropriate and adequate range Confident and ambitious use of a wide range
the task and which may lead to repetition. for the task. of vocabulary within the task set.
Limited control of spelling and / or word Errors, which do not reduce communication Few minor errors which are non-impeding,
formation. and are mainly due to inattention or risk perhaps due to ambitious attempts at more
taking. complex language.
LEXIS
• Appropriacy
• Range
• Accuracy
Limited range of structures. Sufficient range of structures for the task. Confident and ambitious use of a wide range
Frequent grammatical errors and faulty Errors, which do not reduce communication of structures within the task set.
punctuation, which cause difficulty in terms of and are mainly due to inattention or risk Few minor errors which are non-impeding,
communication. taking. perhaps due to ambitious attempts at more
complex language.
GRAMMATICAL STRUCTURES
• Appropriacy
• Range
• Accuracy

12. BAND 0 BAND 1 BAND 2 BAND 3 BAND 4 BAND 5
Meaning only conveyed in very familiar Communication handled in everyday Communication on familiar topics is handled
everyday situations. situations. well. Communication on unfamiliar topics is
Response is only partly relevant. In general, response is relevant. limited.
Serious effort on behalf of the listener Generally meaning is conveyed successfully, Contributions are relevant despite some
CONTENT
required. occasional effort repetition.
• Task response
Meaning is conveyed successfully
• Relevance
• Effect on listener
Speech is mostly intelligible, despite limited Generally intelligible though mispronunciation Generally intelligible. Intonation is generally
control of phonological features. Considerable may occasionally cause momentary strain for appropriate. L1 is perceptible but does not
effort on behalf of the listener required. the listener. interfere with intelligibility.
Noticeable pauses and slow speech, frequent Produces simple speech fluently. Usually Produces extended stretches of language but
repetition and self-correction. maintains flow of speech but uses repetition, may lose coherence at times due to
Very short basic utterances, which are self-correction and / or slow speech to keep occasional repetition, self-correction or
juxtaposed rather than connected or linked going. hesitation.
PRONUNCIATION AND DISCOURSE through repetitious use of simple connectives. May over-use or under-use certain Uses connectives and discourse markers but
MANAGEMENT Requires prompting and support. connectives and discourse markers. not always appropriately.
• Pronunciation Requires little prompting and support. Requires no prompting and support.
No rateable language. Totally incomprehensible. Totally irrelevant.
• Fluency (speech rate and Initiates and responds appropriately.
continuity)
• Effort to link ideas and language
so as to form a coherent,
connected speech
• Prompting and support
Serious effort by listener required.
Little effort by listener required.
• Interaction
Features of bands 1 and 3.
Features of bands 3 and 5.
Limited range of vocabulary which is minimally In general, adequate range of vocabulary for Has a wide enough range of vocabulary to
adequate for the task and which may lead to familiar topics. Attempt to vary expressions, discuss familiar topics at length and make
repetition. but with inaccuracy. meaning clear in spite of some
Paraphrase rarely attempted. Paraphrase attempted, but with mixed inappropriacies.
success. Unfamiliar topics may provoke inaccuracies
LEXIS and errors.
• Appropriacy
• Range
Generally paraphrases successfully.
• Accuracy
• Paraphrase
Only a limited range of structures is used. In general, adequate range for familiar topics. Uses simple and complex structures, but with
limited flexibility.
Limited control of basic forms and sentences. Good degree of control of simple grammatical May make mistakes with complex structures,
Subordinate structures are rare and tend to forms and sentences. Both simple and though these rarely cause comprehension
lack accuracy. complex forms are used. problems.
GRAMMATICAL STRUCTURES Frequent grammatical errors. Minor and occasional errors.
• Appropriacy
• Range
• Accuracy

14. Thank you for attending!