1.Distinctions and similarities among tests, measurement, and evaluation. 2. Different measurement scales and their properties 3. ....

2. Chapter 2 Measurement
◊ Introduction
◊ Definition of terms: measurement, test, evaluation
♦ Measurement
◊ Quantification
◊ Characteristics
◊ Rules and procedures
♦ Test
♦ Evaluation
◊ Essential measurement qualities
◊ Properties of measurement scales
◊ Characteristics that limit measurement
◊ Steps in measurement
◊ Summary
Instructor: Professor Khoshsima
Presenter: Omidi, A
Sat 19/04/2014

3. Introduction
◊ In developing language tests, we must take into account considerations and follow procedures
that are characteristics of tests and measurement in the social sciences in general. Likewise our
interpretation and use of the results of language tests are subject to the same general limitations
that characterize measurement in the social sciences.
◊ The purpose of this chapter is to introduce the fundamental concepts of measurement, an
understanding of which is essential to the development and use of language tests.
◊ Fundamental Concepts of Measurement
► The terms ‘measurement’ , ‘test’ , and ‘evaluation’ and how these are distinct from each other
► Different types of measurement scales and their properties
► The essential qualities of measures – reliability and validity
► The characteristics of measures limiting our interpretations of test results

4. Definition of terms: measurement, test, evaluation
◊ The terms ‘measurement’ , ‘test’ , and ‘evaluation’ are often used synonymously; in deed they
may, in practice, refer to the same activity. When we ask for an evaluation of an individual’s language
proficiency, for example, we are frequently given a test score. This attention to the superficial
similarities among these terms, however, tends to obscure the distinctive characteristics of each. So
an understanding of the distinctions among the terms is vital to the proper development and use of
language tests.
♦ Measurement
► Measurement in the social sciences is ‘the process of quantifying the characteristics of persons
.according to explicit procedures and rules’
► This definition includes three distinguishing features :
quantification, characteristics, and explicit procedures and rules
(1) Quantification
Quantification involves assigning numbers, and this distinguishes measures from qualitative
descriptions such as verbal accounts or nonverbal, visual representations. Non-numerical categories
or rankings such as letter grades (‘A,B.C,..’), or labels (for example, ‘excellent , good, average,…’) may
have the characteristics of measurement. When we actually use categories or rankings, we
frequently assign numbers to them in order to analyze and interpret them.

5. (2) Characteristics
We can assign numbers to both physical and mental characteristics of persons. Physical attributes
such as height and weight can be observed directly. In testing, however, we are almost always interested
in quantifying mental attributes and abilities, sometimes called traits or constructs, which can only be
observed indirectly. These mental attributes include characteristics such as aptitude, intelligence,
motivation, field dependence/independence, attitude, native language, fluency in speaking, and
achievement in reading comprehension. Whatever attributes or abilities we measure, it is important to
understand that it is these attributes or abilities and not the persons themselves that we are measuring.
(3) Rules and procedures
The third distinguishing characteristic of measurement is that quantification must be done according to
explicit rules and procedures. That is, the blind or haphazard assignment of numbers to characteristics
of individuals cannot be regarded as measurement.
► In order to be considered a measure, an observation of an attribute must be replicable:
for other observers,
in other contexts,
and with other individuals.

6. ♦ Test
Carroll’s (1968)definition of a test:
► A psychological or educational test is ‘a procedure designed to elicit certain behavior from which one
can make inferences about certain characteristics of an individual.’
► As one type of measurement, a test necessarily quantifies characteristics of individuals according
to explicit procedures.
► What distinguishes a test from other types of measurement is that :
it is designed to obtain a specific sample of behavior.

7. ♦ Evaluation
Evaluation can be defined as the systematic gathering of information for the purpose of making
decisions. (Weiss, 1972).
► Evaluation is involved only when the results of tests are used as a basis for making a decision.
► One aspect of evaluation is the collection of reliable and relevant information.
► Evaluation does not necessarily entail testing.
► Tests in and of themselves are not evaluative.
► Tests are often used for pedagogical purposes either as a means of motivating students to study or
as a means of reviewing material taught.
► Tests may also be used for purely descriptive purposes.
Information-providing function of measurement versus decision-making function of evaluation

9. Essential measurement qualities
Reliability is a quality of test scores themselves.
Validity is a quality of test interpretation and use.

10. unlike physical attributes, such as height, weight, voice pitch, and temperature, we cannot directly
observe intrinsic attributes or abilities, and we therefore must establish our measurement scales by
definition, rather than by direct comparison.
 The scales we define can be distinguished in terms of four properties:
1. A measure has the property of distinctiveness if different numbers are assigned to persons with
different values on the attribute.
2. It is ordered in magnitude if larger numbers indicate larger amounts of the attribute.
3. The measure has equal intervals if equal differences between ability levels are indicated by equal
differences in numbers.
4. The measure has an absolute zero point if a value of zero indicates the absence of the attribute.
 Measurement specialists have defined four types of measurement scales –
nominal, ordinal, interval, and ratio – according to how many of these four properties they possess.
Properties of measurement scales

11. Nominal/ categorical scale
A nominal scale comprises numbers that are used to ‘name’ the classes or categories of a given
attribute. That is, we can use numbers as a shorthand code for identifying different categories.
Nominal scales possess the property of distinctiveness.
► A special case of a nominal scale is a dichotomous scale, in which the attribute has only two
categories, such as ‘sex’ (male and female), or ‘status of answer’ (true and false) on some types of
tests.
The attribute ‘native language’
native language background number
Chinese = 1
Bengali = 2
French = 3
Arabic = 4
… …

12. Ordinal scale
An ordinal scale comprises the numbering of different levels of an attribute that are ordered with
respect to each other.
The most common example of an ordinal scale is a ranking, in which individuals are ranked ‘first’,
‘second’, ‘third’, and so on, according to some attribute or ability.
The use of subjective ratings in language tests is another example of ordinal scales. The points or
levels on an ordinal scale can be characterized as ‘greater than’ or ‘less than’ each other.
Ordinal scales possess the property of ordering in addition to the property of distinctiveness.
The attribute ‘speaking ability’
student rank
Ali 1st
Reza 2nd
Mehdi 3rd
… …

13. Interval scale
An interval scale is a numbering of different levels in which the distances, or intervals, between
the levels are equal. That is, in addition to the ordering that characterizes ordinal scales, interval scales
consist of equal distances or intervals between ordered levels. Thus they possess the properties of
distinctiveness, ordering, and equal intervals.

14. Ratio scale
The distinguishing feature of a ratio scale is an absolute zero point.
The reason we call a scale with an absolute zero point a ratio scale is that we can make comparisons in
terms of ratios with such scales.

15. Characteristics that limit measurement
As test developers and test users, we must know that our tests:
► are not perfect indicators of the abilities we want to measure and
► that test results must always be interpreted with caution.
The most valuable basis for keeping this clearly in mind can be found in:
► an understanding of the characteristics of measures of mental abilities and
► the limitations these characteristics place on our interpretation of test scores.
1. limitations in specification, and
2. limitations in observation and quantification
 These limitations are of two kinds:

16. Limitations in specification
In any language testing situation, as with any non-test situation in which language use is involved, the
performance of an individual will be affected by a large number of factors. The most important factor
affecting test performance, with respect to language testing, is the individual’s language ability.
► In order to measure a given language ability, we must be able to specify what it is.
♦ This specification generally is at two levels:
(1) At the theoretical level, we need to specify the ability in relation to, or in contrast to, other
language abilities and other factors that may affect test performance.
(2) At the operational level, we need to specify the instances of language performance that we
are willing to interpret as indicators of the ability we wish to measure. This level of specification
defines the relationship between the ability and the test score.

17. Limitations in observation and quantification
In addition to the limitations related to the underspecification of factors that affect test performance,
there are characteristics of the processes of observation and quantification that limit our
interpretations of test results. These derive from the fact that all measures of mental ability are
necessarily indirect, incomplete, imprecise, subjective, and relative.
► Indirectness
► Incompleteness
► Imprecision
► Relativeness
► Subjectivity

18. Steps in measurement
Interpreting a language test score as an indication of a given level of language ability involves being
able to infer, on the basis of an observation of that individual’s language performance, the degree to
which the ability is present in the individual. The limitations discussed above restrict our ability to make
such inferences.
A major concern of language test development is to minimize the effects of these limitations.
To accomplish this, the development of language tests needs to be based on a logical sequence of
procedures linking the putative ability to the observed performance.
◊ This sequence includes three steps:
(1) Identifying and defining the construct theoretically;
(2) Defining the construct operationally,
(3) Establishing procedures for quantifying observations

19. ◊ The first step – defining constructs theoretically
The first step in the measurement of a given language ability is to distinguish the construct we wish to
measure from other similar constructs by defining it clearly, precisely, and unambiguously.
This can be accomplished by determining what specific characteristics are relevant to the given
construct.
♦ Two distinct approaches to defining language proficiency:
(1) The ‘real life’ approach:
In this approach, language proficiency itself is not defined, but rather, a domain of actual, or
‘real life’ language use is identified that is considered to be characteristic of the performance
of competent language users.
Example: ILR oral proficiency interview, ACTFL oral proficiency interview
(2) The ‘interactional/ability’ approach:
In this approach, language proficiency is defined in terms of its component abilities.
Example: the functional framework of Halliday, or the communicative framework of Munby

20. ◊ The second step – defining constructs operationally
This step enables us to relate the constructs we have defined theoretically to our observations of
behavior. It involves determining how to isolate the construct and make it observable.
We must decide what specific procedures, or operations, we will follow to elicit the kind of perform-
ance that will indicate the degree to which the given construct is present in the individual.
 The theoretical definition itself will suggest relevant operations.
 The context in which the language testing takes place also influences the operations we would
follow.
 For an operational definition to provide a suitable basis for measurement, it must elicit language
performance in a standard way, under uniform conditions.

21. ◊ The third step – quantifying our observations
This step involves establishing procedures for quantifying or scaling our observations of performance.
◊ The primary concern in establishing scales for measuring mental abilities:
defining the units of measurement
◊ The units of measurement of language tests are typically defined in two ways:
(1) One way is to define points or levels of language performance or language ability on a scale.
(2) Another common way of defining units of measurement is to count the number of tasks
successfully completed.

22. Relevance of steps to the interpretation of test results
◊ The first step – defining constructs theoretically
provides the basis for evaluating the validity of the uses of test scores
◊ The second step – defining constructs operationally
the observed relationship among different measures of the same theoretical construct provide the
basis for investigating concurrent relatedness
◊ The third step – quantifying our observations
directly related to reliability

23. Summary
► Fundamental measurement terms and concepts – measurement, test, and evaluation
► Four properties of measurement scales – distinctiveness, ordering, equal intervals and
an absolute zero point
► Four types of scales or levels of measurement – nominal, ordinal, interval, and ratio
► Essential qualities of measurement scales – reliability and validity
► characteristics of measures of ability limiting our interpretation and use of test scores
► Fundamental steps in the development of tests in order to minimize the effects of the limitations
and to maximize the reliability of test scores