1. The document discusses rating scales used to evaluate student performance on extended tasks. It covers different types of rating scales including holistic, analytic, and trait scales. 2. Key aspects of developing rating scales are addressed, such as basing descriptors on the test construct, deciding between holistic and analytic scales, and using empirical or intuitive approaches. 3. Effective training of raters is also discussed to ensure consistent understanding and application of the rating scale. Raters should be given examples to rate independently and receive feedback.

1. RATING SCALES FOR EXTENDED
PRODUCTION TASKS
ELT508 English Language Testing
Asst. Prof. Dr. Emrah CİNKARA
BY AHMAD YOUSSEF

2. INTRODUCTION
 since most of the tasks are generally rated using human
judgment. Much of the purpose of this chapter is to point out
ways to reduce the degree of that subjectivity .
 This chapter discusses how to do this by explaining various
types of rating scales, and how to develop or adapt rating
scales for use in a particular testing situation.

3. Rating Scales and Scoring Rubrics
 These terms all refer to a set of generic descriptions of
student performance which can be used to assign scores to an
individual student s performance in a systematic fashion.
 A rating scale has several levels or score bands, each of which
has its own description of what performance looks like at that
level. The descriptors, or descriptions of performance within a
particular score band, should be based on the construct
definition that was written as part of the context and purpose
specifications .

4. Analytic vs. Holistic Rating Scales
holistic
scoring
• (sometimes referred to as
“global,” or “unitary”)
analytic
scoring
• (sometimes called
“componential”)

5. 1.holistic scoring
 In holistic scoring, a student’s performance on a task is
judged in terms of its overall quality, or the overall level of
ability displayed, and a single rating is given to their
performance of the task, usually in the form of a grade or
number.
 All of the descriptors for a given level are grouped together.

6. 2.analytic scoring
 the descriptors are grouped into several subscales, which are
essentially smaller scoring rubrics that are each concerned
with a specific performance feature or portion of the overall
construct, Each subscale is considered separately, and
receives its own rating.
 The difference between analytic and holistic scoring is
whether parts of the construct definition are each considered
separately, or taken together as a whole.

7. Figure 7. I Example of a Holistic Rating Scale
Figure 1.2 Example of an Analytic Rating Scale
(used by permission of the ESL Service Courses, University of California, Los Angeles)

9. 3.“Trait” Scales
 another way of classifying rating scales which they are tailored to a
specific test task.
Primary
trait scoring
• refers to using a holistic rating scale that includes
features of the specific test task.
multiple-trait
scoring,
• which refers to using an analytic rating scale that
includes features of the specific test task

10. 4. “Objective” Scoring Approaches For
Speaking and Writing
 This way aim to speed up the rating process, and
making it less subjective and more mechanical.
 One example includes the use of several yes-no
questions, such as “Does the student use
appropriate grammar in the essay?” The number of
“yes” answers is counted, and then this number is
used to provide a score.

11. Developing or Adapting
Rating Scales
 When it comes to writing a new scoring rubric, or adapting
an existing one, the first place to start is with the construct
definition contained in the test context and purpose
specifications.
 If an existing rating scale can be found that covers all of its
components, and no extraneous (irrelevant ) ones, the next
step is to consider whether it will work with the tasks that
will be included on the test.

12.  For example, a rating scale designed for use with essays
is probably not appropriate for use in scoring
paragraphs, letters, or summaries of reading passages.
 Most likely, no preexisting rating scale will be a perfect
fit to the specifications for a different test, unless it was
designed for use in a similar program, for similar
purposes, with similar students. If one can be found, of
course, that is wonderful.

13.  More commonly, it is necessary to make
modifications to an existing rating scale, or to create
a new one entirely.
 Adapting an existing rubric can involve just minor
wording changes, or extensive modifications
resembling the creation of a completely new rating
scale.

14. Developing Rating Scales: First Steps
 Developing a scoring rubric should begin with the construct definition
contained in the test specifications. The components of the construct
definition will dictate what points need to be addressed in the descriptors.
Various parts of the construct definition may be combined together, or
treated separately .
 A related decision that must also be made early on is whether to use an
analytic or holistic rating scale.

15. Fulcher (2003) classifies approaches to creating rubrics as either
intuitive or empirical.
Intuitive
 is when teachers or experts
base the scale on their
expert judgment,
knowledge of the teaching
and testing context, and so
on.
empirical
 generally involve looking
at a sample of actual learner
performances, dividing them into
groups based on the level of
performance, and then describing
the characteristics of the written
or spoken responses at each level.

16. When the performances are being divided into different levels or
bands, there are two ways to go about separating them :
 If the divisions are based on features within the
performances themselves, this would be
a norm-referenced approach .
 In contrast, if the writing or speaking performances were
divided on the basis of preexisting levels in the program
the approach would be
criterion-referenced approach .

17. Writing the Descriptors
 Descriptors should be kept clear and as brief as possible.
 Furthermore, descriptors should be parallel across levels.
 in a rating scale the role of the interlocutor SHOULD BE CONSIDERED.
 Something to avoid, is using the “educated native speaker” as a
standard for accuracy, or as an ideal interlocutor in the rating scale.

18. Characteristics of standards descriptors :
 describe evidence in the student's response
 describe the quality of the student's response in terms of the
criteria suited to the task
 give meaning to the mid-range or typical standards
 use words which are descriptive and comparative NOT just
comparative
 contain positive statements about student achievement
 use language that is not derogatory
 use unambiguous language which students' understand

19. Figure 7.3 ACTFL Proficiency
Guidelines: Speaking (American
Council on the Teaching of Foreign
Languages 1999

20. Rater Training
 Once the rating scale has been written, and raters have
been selected, there still remains a problem. Even with the
clearest rating scale, different raters may not understand
or apply it the same way.
 The untrained raters are less consistent, and that training
is important with inexperienced raters, and can help them
to become more consistent in their ratings.

21.  Raters observation skills should be improved .
 Raters evaluation skills should be improved .
 Reduce the judgmental biases of the raters.
 Raters should then be shown examples of compositions
or recorded speaking tasks that have been judged to be
clear , and there should be discussion or explanation
after that .

22.  After examples of each level have been reviewed, the
raters should be shown several examples to rate on
their own, with this followed by more discussion.
 the training should also advise the raters not to
compare test takers to each other; instead, they should
compare an examinees performance on each task to the
descriptors in the rating scale.

23. rater norming : ?
 Its when experienced, fully-trained raters still review the rating
scale before using it again, and perhaps do some practice
scoring, especially if it has been a long time since the last time
they rated. This procedure is sometimes referred o as rater
norming .
 although this term also refers to a group of raters meeting to
review a sample of responses, and then determine how strictly
or leniently they should score on that particular occasion .

24. Rating Procedures
 Its important to have multiple ratings of each
performance, whether it be written or spoken.
 Having two raters instead of one will improve reliability,
and is an important step in helping to minimize the
subjectivity of ratings.
 In the case of speaking assessments there should be a
recording.

25.  in the case of disagreements, it possible to bring in a third rater to
settle the matter .
 the best way to handle the disagreement is not by negotiation or
compromise, but by bringing in a third rater.
 The two closest scores are then used, and the more disparate one
removed .
 Negotiation and compromise are only appropriate when there is
simply no trained third rater available .

26. THANKS