A brief summary of the Test Methods and Test Facets affecting testing performance (Source: Fundamental Considerations in Language Testing - Lyle F. Bachman)
2. Chapter 5 - Test Methods
FOCUS
Test method facets (features, aspects) are the
primary focus of this chapter.
Factors affecting test performance:
Example:
-‘Live’ versus “recorded” presentation of aural
material,
-Personality of examiner,
-Filling in the blanks in isolated sentences as opposed
to completing blanks in a text
-The amount and type of text to be processed
3. Chapter 5 - Test Methods
*What is known: the effects of test methods on test performance
Example:
*The amount and type of preparation or prior experience with a
given test
*TOEFL (Test of English as a Foreign Language) : may not have any
prior preparation (proficiency test)
*CPE (Certificate of Proficiency in English): taking a course whose
syllabus is based upon the CPE (achievement test)
*In order to more fully understand variation in language test
performance, it is necessary to develop a framework for
describing the specific features (facets) of test method.
4. Chapter 5 - Test Methods
*There have been a number of descriptions of the
characteristics of language testing methods over the years
(Carroll, 1968; Ciark, 1972; Cohen, 1980, 1979; Hawkey,
1982; Mackey, 1968; Morrow, 1977; Savard, 1968; Weir,
1983).
*In the current book: the framework of test method facets
is an extension and recasting of Caroll’s, Clark’s and
Weir’s taxonomies, incorporating more recent views of the
nature of language.
5. Chapter 5 - Test Methods
Five Facets Affecting Performance on Language Tests
(Bachman, 1990)
1. Testing environment
2. Test rubric
3. The nature of the input the test taker receives
4. The nature of the expected response to that input
5. The relationship between input and response
6. Five Facets Affecting Performance on Language Tests
1. Testing Environment
Test takers might be expected to perform differently under
differing environmental conditions.
Testing environment includes 4 facets (aspects):
1.a. Familiarity of the place and equipment used in
administering the test (familiar place: less threatening –
equipment: pencil versus computer)
1.b. The personnel involved in the test: a superior, a peer, or
a subordinate)
1.c. The time of testing: early in the day, just after a heavy
noon meal
1.d. Physical conditions: noise, temperature, humidity, seating
arrangement, lighting)
7. 2. Test Rubric
Test rubric specifies how test takers are expected to
proceed in taking the test.
Test rubric includes 3 main facets:
2.a. Test organization (i.e. salience of parts - sequence of
parts - relative importance of parts)
2.b. Time allocation
2.c. Instructions (i.e. language – channel - specification of
procedures & tasks - explicitness of criteria for correctness)
Five Facets Affecting Performance on Language Tests
8. 2. Test Rubric
2.a. Test organization: Tests consists of different parts such as
items, questions, or sub-tests. So,
- The salience of (these) parts : whether parts are explicitly marked
and identified or not…
Ex. Test A – A test of how well you can recognize a well organized
English paragraph
“Listening Comprehension” “Reading Comprehension” etc.
- The sequence of (these) parts: easy to difficult – power tests;
random or same difficulty – speeded tests; standard tests – no
opportunity, fixed order of subtests
- Their relative importance: The parts of a test are not always
weighted equally
Ex. A part in which nearly everyone receives the same score, and
which is highly correlated with the other parts of the test will
contribute relatively little to the total test score.
effecttestperformance
Five Facets Affecting Performance on Language Tests
9. 2. Test Rubric
2.b. Time allocation: The amount of time allocated for the
test or its parts is likely to affect test performance.
- Speeded tests: Speed is a variable – Not all the test takers can
complete in the allocated time.
-Power tests: Sufficient time is allocated for almost all.
2.c. Instructions: Performance depends on how well the test
conditions are understood. Facets of instruction includes:
-Language: Instructions in test-takers’ native or tested language
-Channel: Aural or visual channel, or both
-Specification of procedures & tasks: procedure - whether
circling or checking the alternative; task – type (selection,
construction) and form of the response (native/target language)
-Explicitness of criteria for correctness: multiple choice-
the criteria for correctness is clear; writing tests - vague
Testtakersshouldbe
awareofthis
Five Facets Affecting Performance on Language Tests
10. 3. Input: the information contained in a given test
task, to which the test taker is expected to respond.
Input includes 2 main facets:
3.a. Input format
3.b. Nature of language input
Five Facets Affecting Performance on Language Tests
11. 3. Input: the information contained in a given test task, to
which the test taker is expected to respond. Two facets:
3.a. Input format
- Channel of presentation: aural, visual (e.g. listening
comprehension test)
- Mode of presentation: Input will be in receptive mode
- Form of presentation: language (e.g. reading passage),
nonlanguage (e.g. pictures), both (e.g. a passage with tables)
- Vehicle of presentation: live vs. canned human input (e.g. tape
recording)
- Language of presentation: native, target, both
- Identification of the problem: specific (e.g. correcting errors
underlined) – general (e.g. identifiying and correcting an
unspecified number of errors)
- Degree of speededness: Input may be perceived as speeded if test
taker perceives speed as a factor in performance. The perception
of speededness vary from one test-taker to another.
Five Facets Affecting Performance on Language Tests
12. 3. Input
3.b. Nature of language input: When the form of the input
is language, that language can be characterized by its
-Length: Not a critical facet, but longer language samples
include more potential effects in terms of…
-Propositional content: Vocabulary (frequency, specialization),
degree of contextualization (embedded/ reduced), distribution
of new information (compact/diffuse), type of information
(concrete/abstract, positive/negative, factual/counterfactual),
topic, genre
-Organizational characteristics: grammar, cohesion,
rhetorical organization
-Pragmatic characteristic: illocutionary force, sociolinguistic
characteristics (dialect or variety)
Thesecharacteristicsdecidethecomprehensibilityofinput.
Five Facets Affecting Performance on Language Tests
13. 4. Expected Response
Expected response includes 3 main facets:
4.a. Format of expected response
4.b. Nature of language
4.c. Restrictions on response
Five Facets Affecting Performance on Language Tests
14. 4. Expected Response
4.a. Format of expected response
-Channel: aural, visual (Listening comprehension test: marking
the correct choices on multiple-choice questions / responding
speaking
-Mode: productive
-Type of response: selected (multiple choice tests),
constructed (production of a language sample – interview tests
of communicative proficiency)
-Form of expected response: language, non-language
(making a mark, drawing a picture), both
-Language of response: native, target, both
Five Facets Affecting Performance on Language Tests
15. 4. Expected Response
4.b. Nature of language
Five Facets Affecting Performance on Language Tests
-Length: Not a critical facet, but longer language samples include
more potential effects in terms of…
-Propositional content: Vocabulary (frequency, specialization:
technical registers, slangs), degree of contextualization
(embedded/ reduced), distribution of new information
(compact/diffuse), type of information (concrete/abstract,
positive/negative, factual/counter-factual), topic, genre
-Organizational characteristics: grammar, cohesion, rhetorical
organization
-Pragmatic characteristic: illocutionary force, sociolinguistic
characteristics (dialect or variety, register)
16. 4. Expected Response
4.c. Restrictions on response
-Channel: variety of conditions (face-to-face, in quiet rooms, phone
conversations)
-Format: restricted: selection/identification response types;
unrestricted: composition test
-Organizational characteristics: grammatical forms (multiple
choice vocabulary item: test-taker deals with the meanings of specific
words), organization of discourse (following a specific rhetorical pattern:
comparing & contrasting, arguing for & against)
-Propositional and illocutionary characteristics: Language tests
restrict illocutionary force of responses: Picture description tests: to
describe what is given in a picture
-Time or length of response: In testing, situations, administrative
considerations place additional restrictions on time and length.
Five Facets Affecting Performance on Language Tests
17. 5. The Relationship between Input and Response
The relationship between input and response
includes 3 main facets:
5.a. Reciprocal input and response
5.b. Nonreciprocal input and response
5.c. Adaptive input and response
Five Facets Affecting Performance on Language Tests
18. 5. The Relationship between Input and Response
5.a. Reciprocal input and response
can be defined as the use of language by one individual to
produce an effect in another individual through the reduction
of uncertainty with knowledge of results.
*one individual - another individual: a sender and a receiver
need to be involved.
*to produce an effect: indicates that the language use has a
communicative goal, or illocutionary intention.
*reduction of uncertainty: characterizes the means by which
the communicative goal is effected, that is, by means of a
change in information at the disposal of the receiver.
*There is an interaction and a feedback: Ex. a well
conducted oral interview
Five Facets Affecting Performance on Language Tests
19. 5. The Relationship between Input and Response
5.b. Nonreciprocal
*Nonreciprocal language use: there is no interaction
between language users.
*There is no feedback: Ex. reading a book.
*Ex. A cloze or dictation test
Five Facets Affecting Performance on Language Tests
20. 5. The Relationship between Input and Response
5.c. Adaptive input and response
*The relationship between input and response is adaptive if
the input is influenced by the response, but without the
feedback that characterizes a “reciprocal relationship”.
*An individual taking an adaptively administered multiple-
choice test, for example, typically begins with an item
that is of medium difficulty.
*If she answers this item correctly, the next item presented
will be slightly more difficult.
*If she misses that item, the next item presented will be
slightly easier…
Five Facets Affecting Performance on Language Tests
21. Chapter 5 - Test Methods
Applications of this framework to language testing
The applications of this framework to language testing are
potentially in 4 areas:
1. The description of language tests
2. Language test design
3. The validation of language tests
4. The formulation of hypotheses for language testing
research
22. Chapter 5 - Test Methods
Applications of this framework to language testing
1. The description of language tests
*What is my purpose?: Test users are frequently faced with
the need to compare and evaluate different tests with
respect to their possible utility for a given testing need.
*A common framework could be very useful for describing
and comparing the types of tasks included in two similar
tests or in a test and a language program.
*Ex. Using subjective ratings of experienced language
testers, Bachman, Davidson, and Lynch (1988) found
considerable differences between the structure sections
on one form of the TOEFL and one form of the First
Certificate in English (FCE) in several facets of rubric:
*Relative importance of parts - specification of procedures
and tasks - explicitness of criteria for correctness.
23. Chapter 5 - Test Methods
Applications of this framework to language testing
2. Language test design
*An essential step in the development of language tests is
operationalizing the constructs that we want to measure.
*The framework of test facets (we have talked about) may
provide language testers
-a detailed basis for the description of different tests for
purposes of selection, and for the design of specific
language tests
-an appropriate means of codifying and describing the tests
they are developing, using, or researching.
24. Chapter 5 - Test Methods
Applications of this framework to language testing
3. The validation of language tests
*A detailed description of the aspects of the test method is
as essential to the validation process as is that of the
abilities to be measured.
*The validation of language tests must therefore be based
on a detailed description of both the abilities to be
measured and the facets of the test methods.
*That’s the reason why the facets of test methods are
applicable to the validation process.
25. Chapter 5 - Test Methods
Applications of this framework to language testing
4. The formulation of hypotheses for language testing
research
*As stated in Chapter 1, one of the goals of language
testing research is to investigate the factors that affect
performance on language tests.
*Considerable research has already demonstrated that test
methods affect performance on language tests, and much
of this has examined the effects of specific facets, either
by themselves, or as they interact with other facets.
*As these facets have a role in performance, test-
developer should make use of them effectively while
preparing tests.
Editor's Notes
Some test takers, for example, may perform better in the context of an oral interview than they would sitting in a language laboratory speaking into a microphone in response to statements or questions.
1. A familiar place to be less threatening than an unfamiliar one. Test takers tested with familiar equipment such as paper and pencils or tape recordings might perform better than those tested with unfamiliar equipment, such as computers 2. research indicates that the number of individuals involved in interactive tests, such as an oral interview, can affect test takers’ perceptions of the test, and thus, presumably, their performance
Standard deviation
Research in EFL reading comprehension tests suggests that test ’takers perform significantly better when questions about a reading passage in the target language are presented in their native language (Shohamy 1984a)
Degree of contextualization: whether the test taker bring relevant information to the task or not. To the extend the reader has this knowledge and is able to recall it, the discourse will be context-embedded. / Distribution of new information: compact- a group oral interview, each of the participants speaking one or two sentences; diffuse – listening to a lecture and then summarizing its content. / Counterfactual sentences: conditionals: possible to be actual in the real world but again they are actual. / TOPIC: subject matter of discourse - passage effect: if you are familiar, you will get better results. / GENRE Unfamiliarity with the genre is challenging. / Organizational characteristics: The input material in tests that require a constructed response is generally accurately organized; however input materials that require a selected response may include errors / Illocutionary force: the language function performed by an exam question is not to elicit information but to find whether the test-taker knows sth.
Degree of contextualization: whether the test taker bring relevant information to the task or not. To the extend the reader has this knowledge and is able to recall it, the discourse will be context-embedded. / Distribution of new information: compact- a group oral interview, each of the participants speaking one or two sentences; diffuse – listening to a lecture and then summarizing its content. / Counterfactual sentences: conditionals: possible to be actual in the real world but again they are actual. / TOPIC: subject matter of discourse - passage effect: if you are familiar, you will get better results. / GENRE Unfamiliarity with the genre is challenging. /
Virtually all language use, even in situations which might be characterized as ‘authentic’ or ‘natural’, is restricted in various ways by the context or situation. Upon meeting a colleague on the way to class, for example, we are usually not free to utter whatever may come into our head, but are restricted both in the illocutionary act we perform (greeting), and in the register we use (formal, informal).
