Shiroza, S. (2023). Accommodating Englishes in high-stakes proficiency tests: A study of the listening component in the Common Test for University Admissions in Japan. Paper presented at the 21st Asia TEFL International Conference. Aug. 17–20. Daejeong, Korea.

1. Accommodating Englishes
in high-stakes proficiency tests:
A study of the listening component in
the Common Test for University Admissions in Japan
Saran SHIROZA
International Christian University
saran.shiroza@icu.ac.jp
Aug. 20, 2023 Asia TEFL

2. Outline
• Background
• Previous studies
• Research Questions
• Method
• Results
• Discussion
• Suggestions

3. Background
• Global expansion of English use
à linguistic diversification
– Challenging conventional native-speaker (NS) norms in
teaching and learning of English (Alsagoff, et al., 2012; Matsuda, 2012)
• “Testing time for testers” (Jenkins, 2006)
– Questioning validity, appropriateness, fairness
of measuring Non-NS English proficiency against
idealized NS standard (Brown, 2014, Davies et al., 2003, Lowenberg, 2002)

4. Recent developments
• The ‘weak’ approach (Hu, 2012)
– Test accommodations without altering the test
construct based on the NS standard
• IELTS: Incorporating reading/listening texts that
reflect social/regional variations (restricted to the
Inner Circle (Kachru, 1985)) and material writers from UK,
AUS, NZ + including proficient non-NS examiners in
oral/written tests (Taylor, 2002)
• TOEFL: Exploring the effects of including non-NS
accents in listening section (Major et al., 2012)

5. Recent developments
• The ‘strong’ approach(Hu, 2012)
– A “more radical reorientation towards the test
construct” (Hu, 2012, p. 132)
– Not dependent on Inner Circle varieties
– Emphasizing strategic competence,
performance tasks, functional effectiveness (Elder
&Davies, 2006)

6. Research gap
• Previous studies center on Outer Circle contexts
• Studies in the Expanding Circle focus on the use of
international standardized English proficiency tests
(e.g., TOEFL, IELTS, TOEIC, etc.) (Lowenberg, 2002)
• Little research on locally developed and
administered standard English proficiency tests
– Davies et al. (2003), Zhang (2022) on China
– Few studies in Japanese context

7. The present study: The Common Test
• A standardized exam that developed by the Natioanl
Center for University Entrance Examinations.
• Replaced the National Center Test [1990–2020] in 2021
• Held annually in mid-January
• Compulsory for applicants to public universities and
optional for private universities c. 500,000 examinees
• Covers 30 subjects in six areas (i.e. Japanese Language,
Geography and History, Civics, Mathematics, Science,
and Foreign Languages (English, French, German,
Korean, Chinese)

8. Changes in the listening component
The National Center Test [–2020]
• 50 points
(Reading 200 points)
• All passages read twice
• All based on American
English only
The Common Test [2021–]
• 100 points
(Reading 100 points)
• Some passages read twice,
others only once
• “Use modern standard
English by a variety of
speakers” (DNC, 2019)

9. Curriculum policy: The Course of Study
• “consideration should be given to the fact that
various forms of English are widely used
internationally as a means of communication.”
(MEXT, 2009)
• [Students should be taught] “to realize that
present-day English is widely used around the
world as a means of communication, and that
there is diversity in vocabulary, spelling,
pronunciation, grammar, etc.” (MEXT, 2017)

10. Research Questions
1. How does the listening component of the
Common Test accommodate diversity in
English in terms of content and audio?
2. How do stakeholders evaluate the
accommodation of English varieties in the
Common Test?

11. Method: Data collection
1. Test items: Six sets of listening tests (2021-A/B, 2022-
A/B, and 2023-A/B), including the instructions and
questions, scripts, and audio recordings.
2. Commentaries from the External Evaluation
Committee, an ELT-related organization, and exam
preparation board.
3. Information website for the Common Test, PR
materials by the test administrator, websites targeting
examinees preparing for the Common Test, etc.

12. Summary of listening component
Allocated
marks
Content
Type
code
# of items
# of
recordings
# of times
read
Question 1A
25 points
Matching sentence ST 4 4 2
Question 1B Matching figure PIC 3 3 2
Question 2 16 points Short conversation + Q SC+PIC 4 4 2
Question 3 18 points Short conversation SC2 6 6 1
Question 4A
12 points
Short passage + Figure PASS1 8 2 1
Question 4B Short passage PASS2 1 1 1
Question 5 15 points Lecture + Table/Figures LECT 7 1 1
Question 6A
14 points
Longer Conversation LC1 2 1 1
Question 6B
Longer Conversation +
Table/Figure
LC2 2 1 1
100 points 37 23 × 6 = 138

13. Method: Data analysis
• Test item analysis
– (Script) Setting: GLOBAL (Inner Circle, Outer
Circle, Expanding Circle), LOCAL (i.e., Japan),
OTHER
– (Voice) Accent: American English, British
English, Japanese English, Other (if any)
• Thematic analysis
– Focus on how the inclusion of varieties of
English is evaluated

14. Results: Topic distribution
TOPIC CATEGORIES
• OTHER: 59% (n=82)
• GLOBAL: 31% (n=42)
– English names
(place/person, etc.)
– Reference to specific
regions/countries
• GLOBAL+LOCAL: 7% (10)
• LOCAL: 2% (n=3)
– Japanese culture
specific terms
OTHER
59%
GLOBAL
31%
GLOBAL+
LOCAL
7%
LOCAL
2%
GLOBAL+OTHER
1%
TOPIC CATEGORIES OF
LISTENING TESTS 2021–2023
Example:
OTHER;
2023B-Q7:
Choose a
matching
figure (Script:
“Turn left at
the tree and go
straight. The
apartment
building will be
on the right.” )

15. Results: Breakdown of “GLOBAL”
Examples:
• UNSPECIFIED
2021A-Q12: A conversation between
Tina and Mr. Corby about their
schedule
• IC(UK)+Germany
2023A-Q15 (Script excerpt)
W: So, are you originally from here? I
mean, London?
M: Yes, but my family moved to
Germany after I was born.
W: Then, you must be fluent in
German.
32
4
3
1
1
1
0 5 10 15 20 25 30 35
UNSPECIFIED
IC(US)
IC(UK)
IC(US)+UNSPECIFIE
D
IC(CANADA)
IC(UK)+GERMANY
# of Occurrence
REGIONS
BREAKDOWN OF
“GLOBAL” TOPIC CATEGORY

16. Results: Accent distribution
1. Statements/passages read in
American English: 43% (n=60)
2. Conversations between two
American English speakers:
39% (n=54)
3. Statements/conversations by
Japanese and British English
speakers with two American
English speakers: 9% (n=12)
4. Conversations between two
British English speakers: 9%
(n=12)
AmE
43%
AmE &
AmE
39%
JpE, AmE,
AmE, BrE
9%
BrE &
BrE
9%
VARIETIES OF ENGLISH USED
PER PASSAGE/CONVERSATION

17. Discussion: English as a lingua franca
English used in everyday situations
• Directions
• Public announcements
• Discussing schedule
• Comparing tour options
• Listening to lectures,
etc.

18. Discussion: Limited/limiting diversity
• GLOBAL & LOCAL topics: predominantly
spoken in American English
à People with English names=AmE speakers?
• Dialogues: Between AmE or between BrE;
No inter-varietal conversations
àEnglish is for NS-NS interaction? AmE as norm?
• No other varieties used
à Limited/limiting representation of diversity

19. Discussion: “Washback” on pedagogy
• External Evaluation committee (High-school
English teachers):
– “The inclusion of British English and non-native
English speakers in this test is a very good trend. It
will be necessary to devise teaching materials for
the test, such as using the Internet to expose the
students to a variety of modern English sounds in
actual communication.” (2021, p. 371)
• Washback (or backwash): “the impact that a
test has on the teaching and learning done in
preparation for it”(Green, 2013, p. 40)

20. Discussion: “Washback” on pedagogy
• Test writers’ responses: Positive washback
on pedagogy intended
• Actual washback: No supporting evidence
– The inclusion of British English mentioned by
major test-prep institutions/publishers (Z-kai,
Obunsha, Yotsuya gakuin, etc.)
– Negative comments (by examinees?) on online
platforms (Yahoo!BBS, Twitter)

21. Conclusions
1. The listening component of the Common Test
accommodates diversity in English only to a limited
extent in terms of content and audio.
– Efforts to de-centralize the Inner Circle cultural topics
– Still predominantly reliant on American English
– Interaction with NS assumed
2. Stakeholders very highly evaluate the accommodation
of English varieties in the Common Test.
– Positive washback effects expected
– Questionable understanding of the significance of
assessing proficiency in English as a global language
– Attitudinal changes to Japanese English unconfirmed

22. Suggestions
Future studies need to investigate
• examinee attitudes to inclusion of non-US
(UK) and NNS varieties
• pedagogical shifts, if any, in high-school ELT
classrooms
• effectiveness/validity of expanding varietal
representations

23. References
Alsagoff, L., McKay, S. L., Hu, G., & Renandya, W. A. (Eds.), (2012). Principles and practices
for teaching English as an international language. NY: Routledge.
Brown, J. D. 2014. ‘The future of world Englishes in language testing.’ Language Assessment
Quarterly, 11(1), 5–26.
Davies, A., Hamp–Lyons, L. & Kemp, C. (2003). ‘Whose norms? International proficiency
tests in English.’ World Englishes, 22(4), 571–584.
Elder, C., & Davies, A. (2006), Assessing English as a lingua franca. Annual Review of
Applied Linguistics, 26, 282–301.
Green, A. (2013). Washback in language assessment. International Journal of English
Studies, 13(2), 39-51. doi:https://doi.org/10.6018/ijes.13.2.185891
Hu, G. (2012). Assessing English as an international language. In Alsagoff et al., pp. 123–143.
Jenkins, J. (2006). ‘The spread of EIL: A testing time for testers.’ ELT journal, 60(1), 42–50.
Kachru, B.B. (1985). Standards, codification, and sociolinguistic realism, In R. Quirk & H.G.
Widdowson, English in the World (pp, 11–30).
Lowenberg, P. H. (2002). ‘Assessing English proficiency in the expanding circle.’ World
Englishes, 21(3), 431–435.
Major, R.C., Fitzmaurice, S.F., Bunta, F., & Balasubramanian, C. (2012). The effects of
nonnative accents on listening comprehension: Implications for ESL assessment. TESOL
Quarterly, 36(2), 173–190.
Matsuda, A. (Ed.) (2012). Principles and practices of teaching English as an international
language. Bristol, UK: Multilingual Matters.
Zhang, Q. (2021). Impacts of World Englishes on local standardized language proficiency
testing in the Expanding Circle. English Today, 38(4), 254–270.
Slides ↓