Downloaded 290 times
Uploaded byWenlie Jean
Testing writing (for Language Teachers)
The document discusses the key considerations for properly testing writing ability. It identifies four main problems in testing: 1) using representative tasks, 2) eliciting valid writing samples, 3) ensuring scores are valid and reliable, and 4) providing feedback. For each, it outlines various factors that test designers should take into account such as specifying all content domains, including a representative task sample, restricting candidates, using appropriate scoring scales, and calibrating scorers. The goal is to develop writing tests that accurately measure students' abilities.
More Related Content
Similar to Testing writing (for Language Teachers)
Testing writing (for Language Teachers)
- 1. Testing Writing Testing for languageteachers by Arthur Hughes Reporter: Tumana, Wenlie Jean
- 2. The best wayto test people’s writing ability is to get them to write
- 3. 1. We haveto set writing tasks that are properly representative of the population of tasks that we should expect the students to be able to perform. 2. The tasks should elicit valid samples of writing (i.e. which truly represent the students’ ability). 3. It is essential that the samples of writing can and will be scored validly and reliably. 3 parts in testing problem
- 4. 1. Representative tasks (i)Specify all possible content –We have to be clear at the outset just what these tasks are that they should be able to perform. These should be identified in the test specifications.
- 5. - From thestandpoint of content validity, the ideal test would be one which required candidates to perform all the relevant potential writing tasks. The total score obtained on that test (the sum of the scores on each of the different tasks) would be our best estimate of a candidate’s ability. (ii) Include a representative sample of the specified content
- 6. So if wearen’t able to include every tasks (that usually actually happens) and happen to choose a task or tasks that the students are good or bad at, then the outcome will be very different. The more tasks (reasonable number of tasks) that we set, the more representative of a candidate’s ability will be the totality of the samples we obtain.
- 7. Testing problem 1. Representativetasks (i) Specify all possible content (ii) Include a representative sample of the specified content
- 8. 2. Elicit avalid sample of writing ability (i) Set as many separate tasks as is feasible –This requirement is closely related to the need to include a representative sample of the specified content.
- 9. (ii) Test onlywriting ability, and nothing else – In language testing we are not normally interested in knowing whether students are creative, imaginative, or even intelligent, have wide general knowledge, or have good reasons for the opinions they have. Therefore, for the sake of validity, we should not set tasks which measure these abilities.
- 10. 1. Write theconversation you have with a friend about the holiday you plan to have together. 2. You spend a year abroad. While you are there, you are asked to talk to a group of young people about life in your country. Write down what you would say to them. 3. ‘Envy is the sin which most harms the sinner.’ Discuss. 4. Discuss the advantages and disadvantages of being born into a wealthy family.
- 11. Another abilitythat at times interferes with the accurate measurement of writing ability is that of reading. While it is perfectly acceptable to expect the candidate to be able to read simple instructions, we have to ensure that these can be fully understood by everyone whose ability is of sufficiently high standard.
- 12. Testing problem 2. Elicita valid sample of writing ability (i) Set as many separate tasks as is feasible (ii) Test only writing ability, nothing else
- 14. (iii) Restrict candidates There are so many significantly different ways of developing a response to the stimulus. Writing tasks should be well defined: candidates should know just what is required of them, and they should not be allowed to go too far. - A useful devise is to provide information in the form of notes or pictures.
- 15. Testing problem (iii) Restrictcandidates Full sentences avoided
- 16. Tasks shouldnot only fit well with the specifications, but they should also be made as authentic as possible. When thinking of authenticity, it is important to take into account the nature of the candidates and their relationship for whom the task requires to write.
- 17. Testing problem 3. Ensurevalid and reliable scoring (i) Set tasks which can be reliably scored Set as many tasks as possible Restrict candidates Ensure long enough samples Give no choice of tasks Suggestions
- 18. (i) Set taskswhich can be reliably scored Suggestions: 1. Set as many tasks as possible 2. Restrict candidates. The greater the restriction imposed on the candidates, the more directly comparable will be the performance of different candidates. 3. Give no choice of tasks. Making them perform all tasks also makes comparisons between candidates easier. 4. Ensure long enough samples. The samples of writing that are elicited have to be long enough for judgements to be made reliably.
- 19. Testing problem (ii) Createappropiate scales for scoring Two approaches • A single scoring to a piece of writing Holistic scoring • A method that requires a separate score for each aspect of a task Analytic scoring
- 20. Holistic scoring – hasthe advantage of being very rapid. This means that it is possible for each piece of work to be scored more than once because this is also necessary. Not every scoring system will give equally valid and reliable results in every situation. The system has to be appropriate to the level of the candidates and the purpose of the test. Testers have to be prepared to modify existing scales to suit their own purposes. The purpose is purely to measure proficiency, regardless of how it has been achieved.
- 23. Analytic scoring – Methodof scoring which require a separate score for each of a number of aspects of a task. Advantages of analytical scoring: 1. It disposes of the problem of uneven development of subskills in individuals. 2. Scores are compelled to consider aspects of performance which they might otherwise ignore. 3. The very fact that the scorer has to give a number of scores will tend to make the scoring more reliable.
- 24. Disadvantages: 1. The maindisadvantage of the analytic method is the time that it takes. Even with practice, scoring will take longer than with the holistic method. 2. Concentration on the different aspects may divert attention from the overall effect of the piece of writing.
- 25. What will we choosebetween holistic and analytic scoring?
- 26. The choicebetween holistic and analytic scoring depends in part on the purpose of the testing. If diagnostic information is required directly from the ratings given, then analytic scoring is essential. The choice also depends on the circumstances of scoring. If it is being carried out by a small, well-knit group at a single site, then holistic scoring, which is likely to be more economical of time, may be the most appropriate. But if scoring is being conducted by a heterogeneous, possibly less well trained group, or in a number of different places analytic scoring is probably called for.
- 28. Testing problem (iii) Calibratethe scale to be used Collect samples and members of the testing team cover the full range of the scales (iv) Select and train scorers Trainee scores should be native speakers, be sensitive to the languages, have had experience of teaching writing.
- 29. (iv) Follow acceptable scoringprocedures Once the test is completed, a search should be made to identify ‘benchmark’ scripts that typify key levels of ability on each writing task. Copies of these should then be presented to the scorers for an initial scoring. Only when there is agreement on these benchmark scripts should scoring begin.
- 30. Testing problem (v) Followacceptable scoring procedures If the differences are small the two scores can be averaged but if they are big senior members will decide the score. Two o more scorers to give a score independently A senior member of the them identify if there are discrepancies Multiple scoring should ensure scorer reliability
- 31. It isimportant that scoring should take place in a quiet, well-lit environment. Scorers should not be allowed to become too tired. While holistic scoring can be very rapid, it is nevertheless extremely demanding if concentration is maintained.
- 32. Testing problem 4. Feedback Insome situations feedback is very useful to the students The content that appears in the feedback can be decided during calibration
Editor's Notes
- #3 The best way to test people’s writing ability is to get them to write.
- #4 READ THIS FIRST: Since we have to test writing ability directly, we have to consider 3 problems in testing.
- #5 FIRST: In order to judge whether the tasks we set are representative of the tasks that we expect students to be able to perform
- #6 1.We also cannot assume that the students’ scores are equal because people will simply be better at some tasks than others.
- #7 . This is why we select a representative set of tasks. In other words, the more reasonable number of tasks that we set, the more valid it will be. That’s the principle. And if the test includes a wide ranging representative sample of specifications, the test is more likely to have a beneficial backwash effect.
- #8 This is taken from the handbook of the Cambridge in communicative skills in English. This is the description and the specifications of the communicative writing skills.
- #9 1. Remember that peoples performance even on the same task is unlikely to be perfectly consistent. Therefore we have to offer students as many ‘fresh starts’ as possible. By doing this, we will achieve greater reliability and so greater validity. BUT THERE HAS TO BE BALANCE BETWEEN WHAT IS DESIRABLE AND WHAT IS PRACTICAL.
- #10 FIRST: This advice assumes that we do not want to test anything other than the ability to write.
- #11 FIRST: Look at the following tasks and tell the class if what it measures is writing only. WHAT OTHER ABILITIES THEN ARE ASKED FOR IN THIS TASK? - There is creativity, imagination and script-writing ability.
- #12 An example that would contribute to reading ability is giving instructions that are too long. 2. so how do we prevent reading ability interfering in the writing tasks? ….. One way is to use illustrations.
- #13 1. This test is from the Assessment and Qualification Alliance which was intended for science students. 2. As you can see the instructions are short but comprehensible to the students because of the use of illustrations.
- #14 This may take form of a quite realistic transfer of information from graphic form to continuous prose.
- #16 Here is an example of a devise. A note. 1. Notes are given not to provide students with too much of what they need in order to carry out the task. Full sentences are generally to be avoided.
- #19 FIRST: There are a number of suggestions made to obtain a representative performance that will facilitate reliable scoring.
- #20 FIRST: There are two approaches to scoring
- #21 SHOW SAMPLE FROM THE PDF
- #24 SHOW SAMPLE FROM THE PDF
- #27 AFTER: now whichever is used, if high accuracy is sought, multiple scoring is desirable.
- #28 FRST: constructing a valid rating scale is not easy. Here is a practical guide to scale construction. Also, this can be used for the construction of oral rating scales. AFFTER: Any scale which is used, whether holistic or analytic, should reflect the particular purpose of the test and the form that the reported scores on it will take. Because valid scales are not easy to construct, it is eminently reasonable to begin by reviewing existing scales and choosing those that are closest to one’s needs. LAST: since rating scales are in effect telling candidates that ‘these are the criteria by which we will judge you’, their potential for backwash is considerable, since candidates are aware of them.
- #29 Calibrate – means collecting samples of performance collected under test conditions. Train scorers – they should be sensitive to language, have had experience of teaching writing and marking written work.
- #30 FIRST: now let us assume that scorers have already been trained. Once the test is completed, a search should be made to
- #31 Each task of every student should be scored independently by two or more scorers (as many scorers as possible should be involved in the assessment of each student’s work), the scores being recorded on separate sheets.