Method effects on reading comprehension test performance: text organization and response format

1. Method effects on reading
comprehension test
performance:
text organization and
response format
Instructor: Dr. Mahdavi
M. Vedad Taghavi
Miyoko Kobayashi University of Warwick

2. Introduction
According to Bachman, 5 factors affect test performance and
endanger validity:
1. Testing environment
2. Test rubrics
3. The nature of input
4. The nature of expected response
5. The interaction between input and response
2

3. If tests are to provide an accurate measure of learners’
language abilities, examiners must minimize the influence of
intervening factors on test results.
3
￮ text organization
￮ response format

4. The purpose of the article is to investigate
the effects of these two factors on second
language learners’ performance in
reading comprehension tests.
4

5. coherence
The degree of unity (how well a text holds together)
How can we characterize the coherence?
1. Examining how the sentences are related?
2. Quantifying and mapping the links between key words.
3. Rating coherence holistically.
5

6. The top-level rhetrorical structure( text types)
Meyer believes that idea units are organized in a hierarchical
manner on the basis of their rhetorical relationship.
1. Collection
2. Causation
3. Response (problem-solution)
4. Description
5. Comparison
These top-level relationships are thought to represent
patterns in the way we think.
6

7. Response format
1. Cloze test
2. Open-ended
3. Summary writing
Researchers believe that:
 different kinds of tests measure different aspects of
language ability.
 open-ended questions can measure the readers overall
comprehension whereas cloze tests will touch only upon
local understanding.
7

8. Pilot study
It was conducted before the main study
Its purpose:
￮ to examine the viability of the research questions
￮ to identify potential pitfalls in the proposed research
methodology
8

9. Pilot study
relevant variables included:
￮ topic areas of reading passages
￮ text length
￮ text readability
￮ the number of questions
￮ the nature of questions
￮ students’ language proficiency
￮ appropriacy of test level for the students
9

10. Pilot study
￮ the findings suggested that text structure and
response format had an important impact on
reading comprehension
￮ In addition to this relatively large-scale pilot
study, the preparation involved a series of
mini-pilots and reviews by expert judges to
ensure the quality of the test materials.
10

11. The purpose of study
 To investigate weather two factors text
organization and response format exercise a
systematic influence on test result.
 If the systematic interaction is observed this
suggests that text organization and test format
will have a significant effect on reading
comprehension test performance.
 Effect of learners language proficiency level is
another factor that will affect test performance.
11

12. Participants
￮ 754 Japanese university students
￮ the majority being 18–19 years of age
￮ in the first or second years of their courses
￮ six years of English language learning at
secondary schools
￮ randomly divided into twelve groups, with each
student receiving one of a selection of reading
comprehension tests
12

13. Materials - English proficiency test
Aim: to establish the comparability of the twelve
groups
￮ It contained 50 multiple choice grammar and
vocabulary items (it was designed to fit the level
of the students)
￮ there were no significant differences between 12
groups in English language proficiency and
students were divided into 3 groups according to
the rank order of their scores: high –middle-low
13

14. Materials - Reading comprehension tests
￮ texts were specially prepared to maximize
control over the variables identified in the pilot
study
￮ 6topics, 4 texts for each topic were selected.
￮ The mean length of the texts was 369.3 words
and the mean score was 64.4 on the Flesch
Reading Ease Formula, which is one of the most
widely recognized readability indices.
14

15. 15
Flesch Reading Ease was created in 1948 as a readability test. The
score on the test will tell you roughly what level of education someone
will need to be able to easily read a piece of text.
The Reading Ease formula generates a score between 1 and 100.
Although, it is possible to generate scores below and above this
banding. A conversion table is then used to interpret this score.
What is a Flesch Reading Ease score?

16. 16
The formulas are based on two factors:
• sentence length as judged by the average number of words in a
sentence
• word length as judged by the average number of syllables in a
word.
What is a Flesch Reading Ease score?

17. Expert judgement
￮ Use of expert judgement is a fairly recent
development in the second language testing
field
￮ expert judges were asked to assist at different
stages, ranging from text selection and item
analysis to establishing marker reliability.
￮ Most of the judges had MAs in applied
linguistics and were currently engaged in EFL
teaching, materials development or testing
consultancy.
17

18. Procedure
￮ The test was administered by classroom
teachers who were given detailed written
directions.
￮ Written instructions were also prepared
for students. Both sets of instructions were
written in Japanese and piloted to
minimize the risk of misunderstanding or
confusion.
18

19. Statistical analysis
The cloze test were marked by the
semantically and syntactically acceptable
word scoring method. The result were
analyzed using SPSS/PC.
To assess the reliability of marking, 15% of
papers of open-ended questions and
summary writing were independently marked
by other expert judges.
19

20. Results
1. Overall results
On the whole, reliability values were higher in cloze
tests regardless of text types (a = .86 ~ .90) in
comparison to open-ended questions and
summary writing (a = .69 ~ .79). However, this
seemed to be because there were more items in
the cloze test.
In association texts, the mean scores were highest
in cloze test and lowest in problem solution test.
20

21. In problem solution the mean scores were
highest in open-ended and lowest in cloze
test.
(comprehension performance was better
were the text structure is loosely
organized and where it is tightly
organized. The mean scores were biter in
open-ended and summary writing)
21

22. 22
Figure 2 – Comparison of text types: mean scores

23. 2. Hypothesis testing 1
The effects of text organization on the result of test
performaformats were significant in all text types
except for causation texts.
The most important and interesting aspect of the
results is that the two-way interaction between the
two effects proved to be statistically significant.
Text type and nce were observed. the effect of
response response format not only have significant
effects on reading comprehension performance but
they also interact with each other.
2.
23

24. 24
Table 2 – Results of one-way ANOVA: effects of text organization

25. 3. Effects of learners’ English proficiency level
In cloze tests (see Figure 3), higher proficiency learners
performed consistently better than those with lower
language proficiency, with more or less regular
distances between them, even though the three
proficiency groups varied in their test performance
across different types of texts (especially the variation
in the High and Low groups was significant) (see
Table5). This suggests that the distinction between
different proficiency groups was clear regardless of
the variation across text types within each group.
25

26. 26
Figure 3 – Cloze test results (as a percentage) by proficiency levels

27. With open-ended questions (see Figure 4), the text-type effects
varied according to the groups:
￮ the low group performed best with Description texts but
there was little variation among the other three text types.
￮ By comparison, the two higher groups performed most
poorly in Association texts but equally well in the other three
text types.
This suggests that in open-ended questions higher proficiency
learners will be disadvantaged if loosely-orgenized texts are
used as an input for reading comprehension because they
cannot demonstrate their ability fully.
27

28. 28
Figure 4 – Open-ended questions results (as a percentage) by proficiency levels

29. In summary writing (see Figure 5), the diversity in the
pattern
across the three groups is striking.
While the lowest group showed very little variation
across text types, higher groups performed
significantly better with more tightly-organized
texts.
29

30. 30
Figure 5 – Summary writing results (as a percentage) by proficiency levels

31. 4. Correlation with the proficiency test
For cloze tests, correlations with the proficiency test did
not vary so much across text types, though Association
texts tended to produce slightly lower correlations with
the proficiency test.
By contrast, in open-ended questions and summary
writing, results varied considerably depending on what
type of text structure was involved. When texts with looser
structures were used, the reading comprehension
measured by these response formats did not correspond
to general language proficiency as much as when more
tightly organized texts were used.
31

32. 32
Table 4 – Comparison of text types: correlation coefficients with the proficiency test

33. 33
Table 6 – the results of analysis of variance: response format effects by proficiency levels

34. Conclusions
￮ This research has employed Bachman’s
influential model of language ability as an
organizing framework.
￮ The findings have provided data supporting two
aspects of his model:
￮ the nature of input
￮ the nature of expected response
34

35. ￮ there is a systematic relationship between the
students’ test performance and the two
variables examined:
￮ text type
￮ response format
￮ It is therefore vitally important for language
testers, or anyone involved in assessment, to
pay great attention to the test methods they
use.
35

36. 36
The End
31 August 2018