1A Study of the impacts of IELTS, especially on candidates and teachersDr Roger Hawkey, Consultant in language testing to University of Cambridge ESOL ExaminationsIntroductionUniversity of Cambridge ESOL Examinations has been conducting a study of the impacts of theInternational English Language Testing System (IELTS) test. The study explores the effects of thetest on candidates, on preparation courses and on receiving institutions.This paper describes the structure and methodology of the IELTS impact study and presents selectedfindings relevant to the context of the British Council Going Global International Education conference,Edinburgh, with a particular focus on matters of interest to international candidates and teachers ofEnglish. These include: test fairness and difficulty; test anxiety and motivation; test washback onIELTS preparation course content, methods, materials and satisfaction.The use of the term ‘impact’ in the title of the study acknowledges Hamp-Lyons’ (2000: 586) definitionof the term as concerning the “wider influences of tests”, that is matters beyond learning and teaching.Test impact includes ‘washback’, which Hamp-Lyons defines as “influences on teaching, teachers, andlearning (including curriculum and materials)”.IELTS Characteristics and UseThe IELTS test is owned, developed and delivered through the partnership of the British Council, IDPEducation Australia and University of Cambridge ESOL Examinations (IELTS Annual Review, 2003).IELTS is a high-stakes test, used globally by around 500,000 candidates a year to assess their Englishlanguage ability for study or work in places where English is the language of communication. In 2002,76% of the candidates taking the IELTS test were seeking an English language qualification for theiracademic studies, 24% for immigration, training or employment purposes.IELTS assesses the four language macro-skills, listening, reading, writing, speaking, but has nopass/fail scores. The institutions or organisations receiving IELTS candidates themselves decide,according to their perceptions of the language demands of each course or occupation concerned,which of the nine IELTS band scores should be their cut-off point.The IELTS Impact StudyResearch aim and focusThe IELTS impact study is part of a continuous programme of research to ensure that the test is asvalid, effective and ethical as possible. IDP Education Australia and the British Council fund jointlyagreed research projects on the IELTS test, and Cambridge ESOL undertakes research projectsincluding the impact study which is the subject of this paper.The study seeks information in response to research questions on: characteristics and attitudes of IELTS candidates the views of English language teachers preparing these candidates for the test methods and materials used on preparation programmes receiving institution administrator experiences with IELTS.Research approach, structure, validationAs in all research projects, methodological choices had to be made. Figure 1 (Lazaraton, 2001 afterLarsen Freeman and Long, 1991) usefully represents quantitative vs qualitative research approachesas continua of by no means mutually exclusive method characteristics.
2 Figure 1: Continua of research approaches Quantitative Research Qualitative Research controlled ----------------- -----------------*-- naturalistic experimental ---------------- ------------------*-- observational objective ---------------- *------------------- subjective inferential ---------------- --*----------------- descriptive outcome-oriented ---------*------- -------------------- process-oriented reliable ----------------- ------------------- valid particularistic ----------------- --*---------------- holistic ‘hard’, ‘replicable’ data ---------------- -*----------------- ‘real’, ‘rich’, ‘deep’ dataAs will become clear below, the IELTS impact study used both qualitative and quantitative researchapproaches, and from various points along the continua in Figure 1. The asterisk (*) signs on the chartsuggest approximately how the approaches of the IELTS impact study might be categorised (noasterisk appears on the reliable – valid continuum as the two do not appear to the writer to beconceptually or practically opposed, reliability being, rather, part of validity).The IELTS impact study was structured in three phases.1. In Phase One of the study, Cambridge ESOL Examinations commissioned Professor Charles Alderson and his research team at Lancaster University (see references below) to develop data collection instruments and pilot them on small groups of participants locally.2. In Phase Two, these instruments were trialled on larger international participant groups similar to the target populations for the study. The trial data were analysed for instrument validation, revision and rationalisation by Cambridge ESOL Research and Validation Group staff and outside consultants (see, for example Gardiner, 1999, Kunnan, 1999, Milanovic and Saville, 1996: Purpura, 1996 and Hawkey, 2004) .3. Phase Three of the IELTS impact study saw the administration and analysis of the revised data collection instruments.In a paper commissioned as part of the IELTS impact study, How might impact study instruments bevalidated? (Alderson and Banerjee, 1996), the writers express some surprise at the apparent lack of “awell-developed literature on how the reliability and validity“ of “questionnaires, interviews, surveys andclassroom observation schedules” should be established. The IELTS impact study sought to ensurevalid data collection instrumentation, and useful research lessons were learnt during this processabout the differences between the validation of data collection instruments and the validation of tests.In data collection instruments, the veracity rather than accuracy or appropriacy of what participantssay or write is the key aim. However, since responses to questionnaire items may not always, for avariety of reasons, reflect what participants actually think or feel, checks are made, where feasible,through related items or through triangulation, that is the use of more than one method or datasource to investigate a particular phenomenon. A further distinction between test and datainstruments is that since tests may include several items on the same construct or trait, whereasquestionnaires may ask a question only once, some test item analysis operations may not be notappropriate for the validation of questionnaires.During Phase 2 of the study the IEL:TS impact study data collection instruments were subjected,where appropriate, to a range of validating measures, including: descriptive analyses (mean,standard deviation, skew, kurtosis, frequency); factor analysis; test-retest reliability analysis;convergent – divergent and multi-trait, multi-method validation; brainstorming, expert opinion andreview.Research instrumentation and participationPhase Three of the IELTS impact study saw the administration of the following revised data collectioninstruments: a modular IELTS test-taker questionnaire seeking information on background; language learning and testing experience, strategies, attitudes; preparation courses a teacher questionnaire, covering background, views on IELTS, experience of IELTS-preparation programmes an instrument for the evaluation of books used to prepare students for IELTS an IELTS-preparation lesson observation analysis instrument a receiving institute IELTS administrator questionnaire.
3A total of 572 IELTS candidates from all world regions prominent in the actual IELTS test-takerpopulation (see Figure 2) completed the IELTS impact study student questionnaires in 2002. Therewas a reasonable match between the two populations, though with imbalances in South Asia andWestern Europe candidate representation. Figure 2: Comparison of IELTS impact study and IELTS candidate regions World Region Impact study % IELTS 2002 % (CSA) Central and Southern Africa 2 2 (EAP) East Asia Pacific 73 61 Central and Eastern Europe 3 3 Central and South America 1 1 Middle East and North Africa 1 4 North America and Mexico 1 1 South Asia 8 21 Western Europe 8 4 Total 97 97Figure 3 presents a summary profile of the impact study candidate participants. Figure 3: Profile of IELTS impact study candidate participants Gender: % Female 55% male 45% Age: 15-20 35% 21-25 37% 26-30 16% 31-35 6% 36-40 4% 41-45 1% 46-50 0.3% First languages 35 Countries of origin 28 Fields Business, commerce, finance, marketing 30% Health, social sciences, law 26% Science, Engineering 14% Education, language 13% Computer science, IT 10% Art, design 7% Years of English: kindergarten to college/university 10% primary school to college/university 16% secondary school to college/university 38% kindergarten to secondary 3% kindergarten only 1% primary only 6% primary + secondary only 4% secondary only 16% college/ university only 4% outside classes only 2% with extra language classes? 56% Educational levels* pre-university 6% undergraduate 46% post-graduate 48% Pre-IELTS 64% Post-IELTS 36% IELTS Module taken* academic 89% general training 11% * Only post IELTS candidates askedThe gender balance reverses the actual IELTS current male : female percentages, but ages, socio-linguistic histories, fields of study and educational levels reflected the overall IELTS population quite
4well. The IELTS test band scores of impact study candidates who had taken the test were close to theglobal averages. The focus of the study was mainly on IELTS Academic rather than General Trainingcandidates.Eight-three teachers, mainly teaching English on the preparation courses attended by the studentparticipants, completed the impact study teacher questionnaire. Forty-three teachers completed thetextbook evaluation instrument. To enhance and triangulate questionnaire data from students andteacher participants, 120 students, 21 teachers and 15 receiving institution administrators participatedin face-to-face interviews and focus groups as part of the study. In addition, 12 IELTS preparationlessons were observed, video-ed and analysed using the impact study classroom observation andanalysis instrument.Selected IELTS impact study findingsThe findings cited here are selected for their potential interest to IELTS preparation teachers and fortheir relevance to British Council Going Global conference themes.Perceptions of test fairness and difficultiesIIS candidate participants who had already taken IELTS were asked whether they thought IELTS a fairway to test their proficiency in English. Figure 4 summarises the responses (of the 190 test-takersconcerned) to this question and to the follow-up question “If no, why not?” Figure 4: IELTS takers’ perceptions of the fairness of the test Do you think IELTS is a fair way to test your proficiency in English? (N=190) YES 72% NO 28% If No, why not? 1 opposition to all tests 2 pressure, especially of time 3 topics 4 rating of writing and speaking 5 no grammar testThe 72% : 28% division on perceived test fairness may be considered a rather positive response,especially if, as is useful practice when developing questionnaire items, one attempts to predict theresponse of test-takers in general to such a question. Seeking a related view from the impact studyteachers, it was found that 70% of their students’ actual IELTS overall band score results wereperceived as in accordance with their teachers’ expectations.When the IELTS test-takers’ specific objections to the test are pursued, the most frequent is,interestingly, opposition to all tests, followed by time pressure. The next most frequent concerns aremore language oriented, worry about test topics, like time pressure, being a recurring theme in IELTSimpact study data (see below). Figure 5 gives interesting related findings on factors candidatesconsidered actually affected their IELTS performance, suggesting again that time pressure and topicunfamiliarity are seen as significant problms. Figure 5: Factors affecting IELTS candidate performance (%) Time pressure 40% Unfamiliarity of topics 21% Difficulty of questions 15% Fear of tests 13% Difficulty of language 9%Figure 6 suggests that IELTS candidates and preparation teachers have closely similar perceptions ofthe relative difficulties of the IELTS skills modules.
5 Figure 6: Student and teacher perceptions of IELTS module difficulty Most difficult IELTS Module? (%) Students Teachers Reading 49% 45% Writing 24% 26% Listening 18% 20% Speaking 9 9Interesting here is that the reading test is seen as significantly more difficult than the other skillmodules. Whether this should be the case or not raises key validity questions about, for example,whether the reading activities of students once they have entered English-medium higher education,are actually more demanding than activities in the three other macro-skills. To seek answers to suchquestions, relevant points from the impact study face-to-face data are examined, collected from visitsto a selection of the 40 study centres involved in the study.A frequently expressed view in the impact study interviews and focus groups was that it is, bydefinition, difficult to test reading and writing skills directly in ways that validly relate them to actualacademic reading and writing activities. These tend to involve long, multi-sourced texts, handledreceptively and productively, and heavy in reference and statistics, thus difficult to replicate innormal, time-constrained test contexts. Feeling that, on balance, time pressure on the IELTS readingwas too great, an impact study language centre teacher focus group suggested that the test shouldbe allocated 1 hour 20 minutes rather than the current one hour. The same focus group proposed1½ hours for the IELTS writing test (an increase of half an hour), also on the grounds that the timepressure was both excessive and unlike the target writing events in real university life.Cultural diversity issuesWe may also review impact study data on IELTS test fairness for implications relating to theinternational education theme of the British Council Going Global conference. In response, forexample, to a post-IELTS candidate questionnaire item on whether the test is appropriate for allnationalities / cultures, 73% of the 97 students responding answered positively. Most of the negativeresponses referred to learning approaches implied by the test, which may have been unfamiliar to thecandidates, though they may well reflect the realities of academic life in the target countriesconcerned. From the language learning attitudes and strategies section of the student questionnaires,it emerged that 81% of the participants (n=528) claimed to try to learn about the cultures of Englishspeakers, 75% to read for pleasure in English. And from among 156 related face-to-face commentsfrom impact study students and teachers, only the following few were related to cultural diversityissues: occasional inappropriate test content such as questions on family to refugee candidates target culture bias of IELTS topics and materials non-European language speaker disadvantages re Latin-based vocabulary some IELTS-related micro-skills, e.g. the rhetoric of arguing one’s opinions, new to candidates.Both teachers and students, however, recognised that most such matters needed to be prepared foras intrinsic to the international education challenge facing candidates.Test anxiety and motivationLearner and teacher views on the question of test anxiety compare interestingly, as the responsesummaries in Figure 7 indicate. Figure 7: Candidate and teacher views of IELT test anxiety and motivation Candidates Do you worry about taking the IELTS test? Very much 41% Quite a lot 31% A little 19% Very little 9% Teachers Does the IELTS test cause stress for your students ? Yes 53% No 33% Don’t know 14% Teachers Does the IELTS motivate your students? Yes 84% No 10%
6 Don’t know 6%The table suggests that 72% of the candidates were very much or quite a lot worried about the test,which, like the similar figure cited above for test fairness, may be seen as a somewhat predictablerather than an exceptional proportion. Rather fewer of the preparation course teachers see IELTS ascausing their students stress but 84% see the test as a source of motivation for the students. Thestress : motivation balance is clearly an interesting in high-stakes test washback issue.The IELTS preparation coursesThe coursesIt soon became clear that courses referred to by the selected impact study centres as “IELTS courses”came in various forms, as summarised in the table in Figure 8. Figure 8: Preparation course types (n=233) IELTS-specific 108 EAP / Study skills 86 General English with IELTS prep elements 39Teacher questionnaire response data indicated that the three most common class sizes on IELTSpreparation courses were, in order of frequency, 11 to 15, 16 to 20 and 6 to 10.IELTS test washbackNinety per cent of the participant teachers agreed that IELTS influenced the content of their lessons,63% that it also influenced their methodology. Fifty-five of 80 open-ended responses to the questionon course content refer specifically to teaching in direct support of IELTS test components. Thiscompares with 23 responses referring to teaching IELTS formats, techniques, skills, and 7 referencesto administering practice tests. Seven teachers referred to student demands for such a close testfocus.The message is thus that there appears to be strong IELTS washback on the preparation courses interms of both content and methodology. But to avoid possible over-statement or over-simplificationof the complex washback relationship between the courses and the test, it should be noted thatresponses to the content and methodology questions also featured references to teachers whoemphasise improving students’ general proficiency (5), encourage study skills and wide reading (4),or set out to challenge, and confidence build ( 3 ). To theses examples of less test-orientedapproaches should be added the 10% of teachers who claimed that the IELTS test did not influencetheir course content, and the 37% who said it did not influence their methodologyActivities and micro-skillsFigure 9 summarises and compares IIS candidate and teacher selections, from a list on theirquestionnaires of 28 classroom activities, of those that are prominent in their IELTS preparationcourses. Figure 9: Candidate and teacher perceptions of prominent IELTS-preparation activities Activities Students % Teachers % Reading questions and predicting text and answer types 89 86 Listening to live, recorded talks and note-taking 83 63 Analysing text structure and organisation 74 90 Interpreting and describing statistics / graphs / diagrams 74 90 Learning quick and efficient ways of reading texts 73 93 Reading quickly to get the main idea of texts 77 96 Learning how to organise essays 82 99 Practising making a point and providing supporting examples 78 88 Group discussion / debates 83 76 Practising using words to organise a speech 74 83
7The influences of the IELTS test may still be seen in these activities, for example in their relationshipswith the four test modules (reading, listening, writing, speaking). But the potential is clear for theactivities listed to facilitate classroom lessons with an interesting set of communicative activities,appropriate to students’ needs. This impression is reinforced by data from the IELTS impact studyinstrument for the analysis of textbook materials (Saville and Hawkey, 2004). Teachers completing theinstrument on IELTS-related textbooks, found that the books covered, in particular, micro-skills suchas those listed, in rank order of selections, in Figure 10. Figure 10: Perceived micro-skills coverage in IELTS preparation books Micro-skills No. of selections (N=43) identifying main points 40 identifying overall meaning 38 predicting information 36 retrieving and stating factual information 34 planning and organising information, and 34 distinguishing fact from opinion. 31 drawing conclusions 30 making inferences 29 evaluating evidence 27 identifying attitudes 23 understanding, conveying meaning through stress 15 and intonation recognising roles 12Activities in the IELTS-related course books which the teacher analysers consider encouragecommunicative opportunity are, in rank order, essays, group work, pair work, reports, reviews,reading, listening, viewing for personal interest, and role plays.It is part of the theory-based validity claim of the IELTS that it reflects the language macro- and micro-skills needed in candidates’ target activities. We have seen, in the student and teacher datapresented here, evidence that the IELTS preparation courses and their purpose-designed textbookmaterials are both influenced by the test and characterised by an emphasis on communicativeactivities and micro-skills.Classroom obervationsA check may be made on whether the IELTS preparation classes are, in action, as they are perceivedby students and teachers, using evidence from the IELTS-preparation lessons recorded and analysedusing the IELTS impact study classroom observation instrument. This describes lessons in terms of:participations, timings, modes, activities and communicative opportunity time. The analyses of IELTSpreparation lessons suggest that, while communicative activities across the four skills in line with thetest tasks were prominent, the opportunities for learners to communicate on their own behalf variedquite significantly.General impressions from the analyses of the IELTS preparation lessons may be summarised asfollows: learners who are motivated but sometimes to the extent of wanting a more narrow IELTS focus teachers making confident choices of task-based, often inter-related skills activities, involving IELTS-relevant micro-skills. materials from within and beyond the textbook, often with multi-media treatments an ethos of focused activity within a coherent, apparently learner-centred, institutional approach to preparation for IELTS multi-cultural learning and communicating activity often one of the most attractive features of mixed-nationality classes.Preparation course satisfactionThere is a high 83% positive response from the 282 candidates already studying in an English-medium situation to an open-ended item on whether their IELTS preparation courses provided themwith the language knowledge and skills they needed. But this positive view of preparation coursesdoes not, it seems, mean that candidates are fully satisfied with their own performance on thecourses. When all the students were asked whether they felt they were successful on the courses,only 49.5% said yes.
8The main criteria for success mentioned by those responding positively to this item and adding theircomments (n = 141) were: perceived improvement in English proficiency level (26 responses) or skills(14 responses) and increased familiarity with the test (33 positive responses). This result suggests aninteresting balance in students’ perceptions of success between improvement in their target languagelevel, and gaining more knowledge of and practice with the IELTS test. But a closer analysis of thereasons given for perceived lack of success on preparation courses showed the candidates focusingmainly on problems of their own, rather than with the course itself (for example, in order, insufficienttime or hard work, inadequate initial TL level, personal characteristics).As far as the IELTS impact study teachers are concerned, their IELTS preparation courses areconsidered successful compared with other courses they teach, with only 9% of the 83 teachersfeeling otherwise. The reasons given by the teachers for the perceived success of their IELTSpreparation courses were: clear course goals and focus (21) high student motivation (16) clearcut student achievement potential (12) and course validity (11) in terms of student target language needs, topics and skills.ConclusionThis summary of the aims, approaches and selected findings of the IELTS impact study hasattempted to illustrate some of the processes, problems and revelations of impact research. Thegeneral impression of the IELTS test created by the findings of the study is that the test: is recognised as a competitive, high-stakes, four skills, communicative task-based test is seen as mainly fair, though hard, especially in terms of time pressures assesses mainly target-domain content and micro-skills may need to reconsider relationships between target and test reading and writing tasks should maintain continuous test validation activities, including through impact studies.From the analyses of the qualitative and quantitative data collected, hypotheses will be developed onmany aspects of IELTS impact. Findings and recommendations that are felt to need further inquirywill be compared with related IELTS research or receive it in a possible Phase 4 of the impact study.Contacts for further information on the IELTS impact study can be obtained from: www. CambridgeESOL.org, Turner.email@example.com or firstname.lastname@example.org co.uk. The impact study will bedescribed in detail with a full analysis of data and findings in Hawkey, 2005, forthcoming.ReferencesAlderson, J. and J. Banerjee. 1996: How might Impact study instruments be validated? Paper commissioned by the University of Cambridge Local Examinations Syndicate as part of the IELTS Impact Study.Banerjee, J V. 1996: The Design of the Classroom Observation Instruments, internal report, Cambridge: UCLES.Bonkowski, F. 1996: Instrument for the Assessment of Teaching Materials, unpublished MA assignment, Lancaster University.British Council, IDP IELTS Australia, University of Cambridge ESOL Examinations. 2003: Annual Review.Hamp-Lyons, L. 2000: Social, professional and individual responsibility in language testing. System 28, 579-591.Hawkey, R. 2004: ‘An IELTS Impact Study: Implementation and some early findings’. Research Notes, 15. Cambridge: Cambridge ESOL.Hawkey, R. 2005: The theory and practice of impact studies, with special reference to the IELTS and the Progetto Lingue 2000 studies. Cambridge: Cambridge University Press.Herington, R. 1996: Test-taking strategies and second language proficiency: Is there a relationship? Unpublished MA Dissertation, Lancaster University.Horak, T. 1996: IELTS Impact Study Project, unpublished MA assignment, Lancaster University.
9Lazaraton, A.. 2001: ‘Qualitative research methods in language test development and Validation’, in European Language Testing in a Global Context: proceedings of the ALTE Barcelona Conference July 2001. Cambridge: Cambridge University Press.Milanovic, M and Saville, N (1996): Considering the Impact of Cambridge EFL Examinations, internal report, Cambridge: UCLES.Gardiner, K. 1999: Analysis of IELTS impact test-taker background questionnaire. UCLES IELTS EFL Validation Report .Kunnan, A. 2000: IELTS impact study project. Report prepared for UCLES.Purpura, J. 1996: ‘What is the relationship between students’ learning strategies and their performance on language tests?’. Unpublished paper given at UCLES Seminar, Cambridge, June 1996.Larsen-Freeman, D and M Long. 1991: An Introduction to Second Language Acquisition Research. London: Longman.Saville, N and Hawkey, R. 2004: A study of the impact of the International English Language Testing System, with special reference to its washback on classroom materials, in Cheng, L, Watanabe, Y and Curtis, A (eds): Concept and method in washback studies: the influence of language testing on teaching and learning, Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc.Winetroube, S (1997): The design of the teachers attitude questionnaire. Internal report. Cambridge: UCLES.Yue, W (1997): An investigation of textbook materials designed to prepare students for the IELTS test: a study of washback, unpublished MA dissertation, Lancaster University.