Test methods in Language Testing

Test Methods
(Five facets affecting
performance on language
tests)

Chapter 5 - Test Methods
FOCUS
Test method facets (features, aspects) are the
primary focus of this chapter.
Factors affecting test performance:
Example:
-‘Live’ versus “recorded” presentation of aural
material,
-Personality of examiner,
-Filling in the blanks in isolated sentences as opposed
to completing blanks in a text
-The amount and type of text to be processed

*What is known: the effects of test methods on test performance
Example:
*The amount and type of preparation or prior experience with a
given test
*TOEFL (Test of English as a Foreign Language) : may not have any
prior preparation (proficiency test)
*CPE (Certificate of Proficiency in English): taking a course whose
syllabus is based upon the CPE (achievement test)
*In order to more fully understand variation in language test
performance, it is necessary to develop a framework for
describing the specific features (facets) of test method.

*There have been a number of descriptions of the
characteristics of language testing methods over the years
(Carroll, 1968; Ciark, 1972; Cohen, 1980, 1979; Hawkey,
1982; Mackey, 1968; Morrow, 1977; Savard, 1968; Weir,
1983).
*In the current book: the framework of test method facets
is an extension and recasting of Caroll’s, Clark’s and
Weir’s taxonomies, incorporating more recent views of the
nature of language.

Five Facets Affecting Performance on Language Tests
(Bachman, 1990)
1. Testing environment
2. Test rubric
3. The nature of the input the test taker receives
4. The nature of the expected response to that input
5. The relationship between input and response

1. Testing Environment
Test takers might be expected to perform differently under
differing environmental conditions.
Testing environment includes 4 facets (aspects):
1.a. Familiarity of the place and equipment used in
administering the test (familiar place: less threatening –
equipment: pencil versus computer)
1.b. The personnel involved in the test: a superior, a peer, or
a subordinate)
1.c. The time of testing: early in the day, just after a heavy
noon meal
1.d. Physical conditions: noise, temperature, humidity, seating
arrangement, lighting)

2. Test Rubric
Test rubric specifies how test takers are expected to
proceed in taking the test.
Test rubric includes 3 main facets:
2.a. Test organization (i.e. salience of parts - sequence of
parts - relative importance of parts)
2.b. Time allocation
2.c. Instructions (i.e. language – channel - specification of
procedures & tasks - explicitness of criteria for correctness)

2. Test Rubric
2.a. Test organization: Tests consists of different parts such as
items, questions, or sub-tests. So,
- The salience of (these) parts : whether parts are explicitly marked
and identified or not…
Ex. Test A – A test of how well you can recognize a well organized
English paragraph
“Listening Comprehension” “Reading Comprehension” etc.
- The sequence of (these) parts: easy to difficult – power tests;
random or same difficulty – speeded tests; standard tests – no
opportunity, fixed order of subtests
- Their relative importance: The parts of a test are not always
weighted equally
Ex. A part in which nearly everyone receives the same score, and
which is highly correlated with the other parts of the test will
contribute relatively little to the total test score.
effecttestperformance

2. Test Rubric
2.b. Time allocation: The amount of time allocated for the
test or its parts is likely to affect test performance.
- Speeded tests: Speed is a variable – Not all the test takers can
complete in the allocated time.
-Power tests: Sufficient time is allocated for almost all.
2.c. Instructions: Performance depends on how well the test
conditions are understood. Facets of instruction includes:
-Language: Instructions in test-takers’ native or tested language
-Channel: Aural or visual channel, or both
-Specification of procedures & tasks: procedure - whether
circling or checking the alternative; task – type (selection,
construction) and form of the response (native/target language)
-Explicitness of criteria for correctness: multiple choice-
the criteria for correctness is clear; writing tests - vague
Testtakersshouldbe
awareofthis

3. Input: the information contained in a given test
task, to which the test taker is expected to respond.
Input includes 2 main facets:
3.a. Input format
3.b. Nature of language input

3. Input: the information contained in a given test task, to
which the test taker is expected to respond. Two facets:
3.a. Input format
- Channel of presentation: aural, visual (e.g. listening
comprehension test)
- Mode of presentation: Input will be in receptive mode
- Form of presentation: language (e.g. reading passage),
nonlanguage (e.g. pictures), both (e.g. a passage with tables)
- Vehicle of presentation: live vs. canned human input (e.g. tape
recording)
- Language of presentation: native, target, both
- Identification of the problem: specific (e.g. correcting errors
underlined) – general (e.g. identifiying and correcting an
unspecified number of errors)
- Degree of speededness: Input may be perceived as speeded if test
taker perceives speed as a factor in performance. The perception
of speededness vary from one test-taker to another.

3. Input
3.b. Nature of language input: When the form of the input
is language, that language can be characterized by its
-Length: Not a critical facet, but longer language samples
include more potential effects in terms of…
-Propositional content: Vocabulary (frequency, specialization),
degree of contextualization (embedded/ reduced), distribution
of new information (compact/diffuse), type of information
(concrete/abstract, positive/negative, factual/counterfactual),
topic, genre
-Organizational characteristics: grammar, cohesion,
rhetorical organization
-Pragmatic characteristic: illocutionary force, sociolinguistic
characteristics (dialect or variety)
Thesecharacteristicsdecidethecomprehensibilityofinput.

4. Expected Response
Expected response includes 3 main facets:
4.a. Format of expected response
4.b. Nature of language
4.c. Restrictions on response

4.a. Format of expected response
-Channel: aural, visual (Listening comprehension test: marking
the correct choices on multiple-choice questions / responding
speaking
-Mode: productive
-Type of response: selected (multiple choice tests),
constructed (production of a language sample – interview tests
of communicative proficiency)
-Form of expected response: language, non-language
(making a mark, drawing a picture), both
-Language of response: native, target, both

4.b. Nature of language
-Length: Not a critical facet, but longer language samples include
more potential effects in terms of…
-Propositional content: Vocabulary (frequency, specialization:
technical registers, slangs), degree of contextualization
(embedded/ reduced), distribution of new information
(compact/diffuse), type of information (concrete/abstract,
positive/negative, factual/counter-factual), topic, genre
-Organizational characteristics: grammar, cohesion, rhetorical
organization
-Pragmatic characteristic: illocutionary force, sociolinguistic
characteristics (dialect or variety, register)

4.c. Restrictions on response
-Channel: variety of conditions (face-to-face, in quiet rooms, phone
conversations)
-Format: restricted: selection/identification response types;
unrestricted: composition test
-Organizational characteristics: grammatical forms (multiple
choice vocabulary item: test-taker deals with the meanings of specific
words), organization of discourse (following a specific rhetorical pattern:
comparing & contrasting, arguing for & against)
-Propositional and illocutionary characteristics: Language tests
restrict illocutionary force of responses: Picture description tests: to
describe what is given in a picture
-Time or length of response: In testing, situations, administrative
considerations place additional restrictions on time and length.

5. The Relationship between Input and Response
The relationship between input and response
includes 3 main facets:
5.a. Reciprocal input and response
5.b. Nonreciprocal input and response
5.c. Adaptive input and response

5.a. Reciprocal input and response
can be defined as the use of language by one individual to
produce an effect in another individual through the reduction
of uncertainty with knowledge of results.
*one individual - another individual: a sender and a receiver
need to be involved.
*to produce an effect: indicates that the language use has a
communicative goal, or illocutionary intention.
*reduction of uncertainty: characterizes the means by which
the communicative goal is effected, that is, by means of a
change in information at the disposal of the receiver.
*There is an interaction and a feedback: Ex. a well
conducted oral interview

5.b. Nonreciprocal
*Nonreciprocal language use: there is no interaction
between language users.
*There is no feedback: Ex. reading a book.
*Ex. A cloze or dictation test

5.c. Adaptive input and response
*The relationship between input and response is adaptive if
the input is influenced by the response, but without the
feedback that characterizes a “reciprocal relationship”.
*An individual taking an adaptively administered multiple-
choice test, for example, typically begins with an item
that is of medium difficulty.
*If she answers this item correctly, the next item presented
will be slightly more difficult.
*If she misses that item, the next item presented will be
slightly easier…

Applications of this framework to language testing
The applications of this framework to language testing are
potentially in 4 areas:
1. The description of language tests
2. Language test design
3. The validation of language tests
4. The formulation of hypotheses for language testing
research

1. The description of language tests
*What is my purpose?: Test users are frequently faced with
the need to compare and evaluate different tests with
respect to their possible utility for a given testing need.
*A common framework could be very useful for describing
and comparing the types of tasks included in two similar
tests or in a test and a language program.
*Ex. Using subjective ratings of experienced language
testers, Bachman, Davidson, and Lynch (1988) found
considerable differences between the structure sections
on one form of the TOEFL and one form of the First
Certificate in English (FCE) in several facets of rubric:
*Relative importance of parts - specification of procedures
and tasks - explicitness of criteria for correctness.

2. Language test design
*An essential step in the development of language tests is
operationalizing the constructs that we want to measure.
*The framework of test facets (we have talked about) may
provide language testers
-a detailed basis for the description of different tests for
purposes of selection, and for the design of specific
language tests
-an appropriate means of codifying and describing the tests
they are developing, using, or researching.

3. The validation of language tests
*A detailed description of the aspects of the test method is
as essential to the validation process as is that of the
abilities to be measured.
*The validation of language tests must therefore be based
on a detailed description of both the abilities to be
measured and the facets of the test methods.
*That’s the reason why the facets of test methods are
applicable to the validation process.

4. The formulation of hypotheses for language testing
research
*As stated in Chapter 1, one of the goals of language
testing research is to investigate the factors that affect
performance on language tests.
*Considerable research has already demonstrated that test
methods affect performance on language tests, and much
of this has examined the effects of specific facets, either
by themselves, or as they interact with other facets.
*As these facets have a role in performance, test-
developer should make use of them effectively while
preparing tests.

Test methods in Language Testing

More Related Content

What's hot

Viewers also liked

Similar to Test methods in Language Testing

More from Seray Tanyer

Recently uploaded

Test methods in Language Testing

Editor's Notes