This is a poster I use at the 3rd Asian Association of Language Assessment.

1. Item π* r* 1 r* 2 r* 3 r* 4 r* 5 r* 6 r* 7 IF ID B-index
1 0.44 0.16 0.21 0.48 0.43
2 0.86 0.37 0.53 0.49 0.45
3 0.47 0.52 0.33 0.28 0.31
4 0.88 0.38 0.53 0.56 0.52
5 0.76 0.79 0.68 0.21 0.18
6 0.63 0.58 0.48 0.32 0.32
7 0.46 0.25 0.27 0.43 0.40
8 0.98 0.62 0.78 0.50 0.31
9 0.78 0.47 0.52 0.38 0.38
10 0.83 0.42 0.54 0.53 0.50
11 0.95 0.70 0.79 0.42 0.24
12 0.62 0.37 0.41 0.42 0.42
13 0.91 0.25 0.56 0.56 0.49
14 0.92 0.52 0.65 0.41 0.34
15 0.74 0.46 0.49 0.40 0.40
16 0.82 0.87 0.53 0.55 0.47 0.39
17 0.99 0.87 0.94 0.27 0.08
18 0.75 0.65 0.59 0.32 0.27
19 0.17 0.64 0.17 0.08 0.06
20 0.94 0.56 0.71 0.48 0.37
21 0.70 0.33 0.40 0.45 0.41
22 0.99 0.77 0.87 0.44 0.18
23 0.76 0.59 0.60 0.35 0.23
24 0.94 0.60 0.70 0.45 0.30
25 0.88 0.34 0.52 0.56 0.53
26 0.80 0.58 0.61 0.38 0.34
27 0.78 0.66 0.64 0.36 0.25
28 0.97 0.56 0.74 0.52 0.34
29 0.60 0.76 0.43 0.19 0.09
30 0.67 0.56 0.49 0.39 0.36
31 0.74 0.37 0.47 0.41 0.35
32 0.48 0.89 0.95 0.37 0.09 0.07
33 0.83 0.53 0.59 0.41 0.41
34 0.85 0.44 0.56 0.49 0.44
35 0.16 0.91 0.15 -0.06 0.01
36 0.31 0.64 0.24 0.20 0.13
37 0.54 0.38 0.32 0.31 0.32
38 0.65 0.60 0.51 0.32 0.24
39 0.72 0.46 0.78 0.42 0.44 0.47
40 0.63 0.43 0.41 0.36 0.29
41 0.99 0.64 0.78 0.52 0.31
42 0.65 0.46 0.45 0.41 0.35
43 0.82 0.55 0.72 0.53 0.49 0.44
44 0.26 0.89 0.19 0.12 0.06
45 0.95 0.57 0.67 0.49 0.35
46 0.53 0.46 0.38 0.33 0.37
47 0.87 0.61 0.72 0.58 0.47 0.36
48 0.74 0.67 0.79 0.54 0.39 0.35
49 0.58 0.39 0.36 0.37 0.37
50 0.65 0.69 0.84 0.89 0.45 0.33 0.30
51 0.74 0.40 0.42 0.46 0.45
52 0.29 0.82 0.26 0.10 0.10
53 0.53 0.61 0.41 0.32 0.33
54 0.48 0.62 0.36 0.26 0.19
55 0.61 0.65 0.49 0.29 0.27
Using a cogni*ve diagnos*c model to examine remedial Japanese university
students’ gramma*cal knowledge
Takaaki KUMAZAWA (Kanto Gakuin University)
ktakaaki@kanto-gakuin.ac.jp
In many Japanese universiBes, teaching grammar is crucial because not all the students have yet mastered the grammaBcal knowledge they were supposed to during their
secondary educaBon. Thus, the purpose of this study is to examine remedial Japanese university students’ grammaBcal knowledge using a cogniBve diagnosBc model. The
parBcipants were 548 Japanese university freshmen in the department of law at a private university. They were divided into two groups: remedial, and non-remedial, and
the equivalent grammar test was administered twice as a pretest and posQest: pre remedial, pre non-remedial, post remedial, and post non-remedial (n = 142, 175, 121,
and 110, respecBvely). The results of the item analysis showed that most of 55 criterion-referenced mulBple-choice items were working on the test-takers, and the test
was reliable (α = .89) and dependable (Φ(λ) = .90) for making diagnosBc and achievement decisions. The fusion model showed that although two out of the 55 entries in
the Q-matrix were quesBonable, it was theoreBcally and staBsBcally valid. Only the remedial students were able to improve their mastery levels parBally due to the
effecBveness of remedial intervenBon.
Abstract
sGrammar criterion-referenced multiple-choice test
1. Fill-in MC items (k = 16):
He (a. is, b. study, c. studies, d. is study) math every day
2. English-Japanese translation equivalent MC items (k = 8):
We were not members of this hockey team (a. We are members of this
hockey team, b. We were members of this hockey team, c. Were we
members of this hockey team?, d. We were not members of this hockey
team [the four choices were in Japanese]
3. Japanese-English translation equivalent MC items (k = 8):
He must be old [this sentence was in Japanese] (a. He may be old, b. He
must be old, c. He can be old, d. He will be old)
4. Error identification MC items (k = 8):
There is (a. an idea) to make (b. a building) near (c. the Tokyo station).
(d. The plan) is great.
5. Sentence-reordering MC items (k = 8):
If I (a. money, b. had, c. I, d. would) buy a house.
6. Cloze MC items (k = 7):
The one in Waikiki was expensive and not very good, (a. but, b. or, c.
because, d. when) I had the best one in North Shore.
Table 1
Q-matrix and item analysis
Summary of the results
• Only two items (items 32 and 35) were not working because the IF, ID, B-
index values were too low and because π and r parameters were too low
and high.
• The test was reliable (α = .89) and dependable (Φ(λ) = .90) for making
diagnostic and achievement decisions
• Based on the means, the remedial group improved their score by 6.24
and the non-remedial group dropped their score by 5.89
• Based on the PPMs, the remedial group mastered passive voice the most.
The non-remedial group dropped their overall mastery but their mastery
on comparatives did not decrease drastically.
Group n M SD Min Max α Φ（λ）
Pre remedial 142 20.15 5.41 2 43 0.62 0.93
Pre non-remedial 175 33.53 8.86 11 50 0.87 0.85
Post remedial 121 26.39 5.28 12 40 0.61 0.90
Post non-
remedial
110 29.33 11.91 9 47 0.93 0.86
Total 548 27.64 9.66 2 50 0.89 0.90
Table 2
Descriptive Statistics for the Grammar Test (k = 55)
.0000
.1000
.2000
.3000
.4000
.5000
.6000
.7000
.8000
.9000
1.0000
1 2 3 4 5 6 7
Average PPMs for four groups
Pre-remedial
Pre-non remedial
Post-remedial
Post-non remedial
Figure 1. Remedial and non-remedial students’ posterior probability of mastery (PPM)
on average.
Par$cipants
• 548 Japanese university freshmen in the law department at a
private university
• 3 class levels: Every year, based on a placement test, about 50 low-
scoring students and 60 high-scoring students were placed in
remedial and advanced classes. The rest were placed in
intermediate classes
• Their TOEIC® scores approximately ranged from 200 to 450
(beginner to lower intermediate)
Procedure
• Pre-posQests, control-experimental group design: pre remedial, pre
non-remedial, post remedial, and post non-remedial [n = 142, 175,
121, and 110, respecBvely]
• In a remedial class, 30 90-minute lessons on teaching mainly
grammarIn intermediate classes, 30 90-minute lessons on teaching
reading mainly through intensive reading
Methods
Results