The document discusses the key considerations for properly testing writing ability. It identifies four main problems in testing: 1) using representative tasks, 2) eliciting valid writing samples, 3) ensuring scores are valid and reliable, and 4) providing feedback. For each, it outlines various factors that test designers should take into account such as specifying all content domains, including a representative task sample, restricting candidates, using appropriate scoring scales, and calibrating scorers. The goal is to develop writing tests that accurately measure students' abilities.

1. Testing
Writing
Testing for language teachers by
Arthur Hughes
Reporter: Tumana, Wenlie Jean

2. The best way to test
people’s writing
ability is to get them
to write

3. 1. We have to set writing tasks that are properly
representative of the population of tasks that we
should expect the students to be able to perform.
2. The tasks should elicit valid samples of writing (i.e.
which truly represent the students’ ability).
3. It is essential that the samples of writing can and will
be scored validly and reliably.
3 parts in testing
problem

4. 1. Representative tasks
(i) Specify all possible content
–We have to be clear at the outset
just what these tasks are that they
should be able to perform. These
should be identified in the test
specifications.

5. - From the standpoint of content validity,
the ideal test would be one which required
candidates to perform all the relevant
potential writing tasks.
The total score obtained on that test (the
sum of the scores on each of the different
tasks) would be our best estimate of a
candidate’s ability.
(ii) Include a representative sample of
the specified content

6. So if we aren’t able to include every tasks
(that usually actually happens) and happen
to choose a task or tasks that the students
are good or bad at, then the outcome will be
very different.
The more tasks (reasonable number of tasks)
that we set, the more representative of a
candidate’s ability will be the totality of the
samples we obtain.

7. Testing problem
1. Representative tasks
(i) Specify all possible
content
(ii) Include a
representative sample
of the specified content

8. 2. Elicit a valid sample of writing
ability
(i) Set as many separate tasks as is feasible
–This requirement is closely related to
the need to include a representative
sample of the specified content.

9. (ii) Test only writing ability, and
nothing else
– In language testing we are not normally
interested in knowing whether students
are creative, imaginative, or even
intelligent, have wide general knowledge,
or have good reasons for the opinions
they have. Therefore, for the sake of
validity, we should not set tasks which
measure these abilities.

10. 1. Write the conversation you have with a friend about the
holiday you plan to have together.
2. You spend a year abroad. While you are there, you are asked
to talk to a group of young people about life in your country.
Write down what you would say to them.
3. ‘Envy is the sin which most harms the sinner.’ Discuss.
4. Discuss the advantages and disadvantages of being born into
a wealthy family.

11.  Another ability that at times interferes
with the accurate measurement of writing
ability is that of reading. While it is perfectly
acceptable to expect the candidate to be
able to read simple instructions, we have to
ensure that these can be fully understood
by everyone whose ability is of sufficiently
high standard.

12. Testing problem
2. Elicit a valid sample of writing ability
(i) Set as many separate tasks as is feasible
(ii) Test only writing ability, nothing else

14. (iii) Restrict candidates
 There are so many significantly different
ways of developing a response to the
stimulus. Writing tasks should be well
defined: candidates should know just what is
required of them, and they should not be
allowed to go too far.
- A useful devise is to provide information in
the form of notes or pictures.

15. Testing problem
(iii) Restrict candidates
Full sentences
avoided

16.  Tasks should not only fit well with the
specifications, but they should also be
made as authentic as possible. When
thinking of authenticity, it is important to
take into account the nature of the
candidates and their relationship for
whom the task requires to write.

17. Testing problem
3. Ensure valid and reliable scoring
(i) Set tasks which can be reliably scored
Set as many tasks as
possible
Restrict candidates
Ensure long enough
samples
Give no choice of
tasks
Suggestions

18. (i) Set tasks which can be reliably
scored
Suggestions:
1. Set as many tasks as possible
2. Restrict candidates. The greater the restriction imposed on the
candidates, the more directly comparable will be the performance
of different candidates.
3. Give no choice of tasks. Making them perform all tasks also
makes comparisons between candidates easier.
4. Ensure long enough samples. The samples of writing that are
elicited have to be long enough for judgements to be made reliably.

19. Testing problem
(ii) Create appropiate scales for scoring
Two approaches
• A single scoring to a
piece of writing
Holistic
scoring
• A method that
requires a separate
score for each
aspect of a task
Analytic
scoring

20. Holistic scoring
– has the advantage of being very rapid. This means that it is
possible for each piece of work to be scored more than once
because this is also necessary.
 Not every scoring system will give equally valid and reliable
results in every situation. The system has to be appropriate to
the level of the candidates and the purpose of the test.
Testers have to be prepared to modify existing scales to suit
their own purposes.
The purpose is purely to measure proficiency, regardless of how
it has been achieved.

23. Analytic scoring
– Method of scoring which require a separate score for each of a
number of aspects of a task.
Advantages of analytical scoring:
1. It disposes of the problem of uneven development of
subskills in individuals.
2. Scores are compelled to consider aspects of performance
which they might otherwise ignore.
3. The very fact that the scorer has to give a number of scores
will tend to make the scoring more reliable.

24. Disadvantages:
1. The main disadvantage of the analytic
method is the time that it takes. Even
with practice, scoring will take longer
than with the holistic method.
2. Concentration on the different aspects
may divert attention from the overall
effect of the piece of writing.

25. What will we
choose between
holistic and
analytic
scoring?

26.  The choice between holistic and analytic scoring
depends in part on the purpose of the testing. If
diagnostic information is required directly from the
ratings given, then analytic scoring is essential.
The choice also depends on the circumstances of
scoring. If it is being carried out by a small, well-knit
group at a single site, then holistic scoring, which is
likely to be more economical of time, may be the most
appropriate.
But if scoring is being conducted by a heterogeneous,
possibly less well trained group, or in a number of
different places analytic scoring is probably called for.

28. Testing problem
(iii) Calibrate the scale to be used
Collect samples and members of the testing
team cover the full range of the scales
(iv) Select and train scorers
Trainee scores should be native speakers, be
sensitive to the languages, have had experience
of teaching writing.

29. (iv) Follow acceptable
scoring procedures
 Once the test is completed, a search
should be made to identify ‘benchmark’
scripts that typify key levels of ability on
each writing task.
Copies of these should then be presented
to the scorers for an initial scoring.
Only when there is agreement on these
benchmark scripts should scoring begin.

30. Testing problem
(v) Follow acceptable scoring procedures
If the differences are
small the two scores
can be averaged but
if they are big senior
members will decide
the score.
Two o more
scorers to give a
score
independently
A senior
member of the
them identify if
there are
discrepancies
Multiple
scoring should
ensure scorer
reliability

31.  It is important that scoring should take
place in a quiet, well-lit environment.
Scorers should not be allowed to become
too tired.
While holistic scoring can be very rapid, it
is nevertheless extremely demanding if
concentration is maintained.

32. Testing problem
4. Feedback
In some
situations
feedback is
very useful
to the
students
The content
that appears
in the
feedback can
be decided
during
calibration