This is a presentation about the development and evaluation of the scales/instruments used in psychology and psychiatry

1. DEVELOPMENT AND EVALUATION
OF SCALES/INSTRUMENTS IN
PSYCHIATRY
DR. PAWAN SHARMA
pawan60@gmail.com

2. • INTRODUCTION
• STEPS FOR DEVELOPMENT OF SCALES
• WHOQOL-100: AN EXAMPLE

3. • INTRODUCTION
• STEPS FOR DEVELOPMENT OF SCALES
• WHOQOL-100: AN EXAMPLE

4. INTRODUCTION
• The term 'rating scale' was originally used to define a series of
items which quantified or placed in rank order, the manifestations
of a single variable e.g. aggressiveness
Hamilton, 1976
• Psychological testing is something that requires one to perform a
behavior to measure some personal attribute, trait or characteristic
or to predict an outcome

5. INTRODUCTION
Intelligence test
• Binet-Simon scale
• Ability in global
areas like verbal
comprehension,
performance,
reasoning etc.
Aptitude
• Capability in
specific task or skill
• Supplement global
intelligence tests
• Vocational test
• General aptitude
battery
Achievement
• Degree of learning,
success
• Civil services
• Resemble
intelligence tests
more closely
• SLD batteries
Personality test:
• Trait, quality or behavior
• Self report inventories, check lists
• Performance or situational test
• Projective technique

6. INTRODUCTION
• In terms of user
– Self rating
– Observer rating
• Form of items
– Graded items- degree of severity
– Checklist- present or absent
– Forced choice items - choose 2 alternatives as applicable
• Content
– Behavior
– Social adjustment
– Functional capacity etc

7. INTRODUCTION
Most important scale as per function in clinical setting
1. Intensity scales – severity and response to treatment : BPRS,
PANSS
2. Prognostic scales- prediction of response to treatment: Strauss
and Carpenter Prognostic Scale
3. Scales for selection of treatment by means of differential
indicators: Conner's rating scale for ADHD
4. Scales for diagnosis and classification : IQ scales
Hamilton, 1976

8. • INTRODUCTION
• STEPS FOR DEVELOPMENT OF SCALES
• WHOQOL-100: AN EXAMPLE

9. SCALE DEVELOPMENT
Defining the test
Selecting a scaling method
Constructing the items
Testing the items
Revising the test Publishing

10. DEFINING THE TEST
• Over the years thousands of scale or tests developed
• Clear idea of what the test or scale is supposed to measure
(purpose)
• How is the new test different from others and what contribution
the test or scale provides to the existing field

11. SELECTING A SCALING METHOD
Representative scaling method
1. Expert Ranking : Glasgow Coma Scale for scaling the depth of
coma
– Panel of neurologists to list patient behaviors associated with
different level of consciousness
2. Method of equal appearing intervals: (Thurstone scaling
approach)
– For an scale of attitude collect as many true false statement as
possible
– Known judges or experts determine the favorability usually one
out of 11 categories (1 to 11): extremely favorable to extremely
unfavorable
– The mean favorability rating and Standard deviation for each
item determined
– Items with large standard deviation dropped

12. SELECTING A SCALING METHOD
Method of absolute scaling:
• Used in aptitude testing and group achievement testing
• Measure of absolute item difficulty based upon results of different
age groups of test takers
• Administration of common set of questions to two or more age
groups
• The relative difficulty of these items between any two age groups
serves as a basis for making a series of interlocking comparisons
• One age group as an anchor
• Item difficulty measured in common units as SD units of the ability
for anchor group

13. SELECTING A SCALING METHOD
Likert scaling
• Proposed by Likert 1932
• Presents the examinee with 5 responses ordered on an
agree/disagree or approve/ disapprove continuum
• Score of 5 to extreme response and 1 to opposite extreme
• Total scale score is obtained by adding scores from individual
items so known as summative scale

14. SELECTING A SCALING METHOD
Guttman scale:
• Respondents who endorse one statement also agree with the
milder statement pertinent to the same underlying continuum
• Produced by selecting items that fall into an ordered sequence of
examinee endorsement
• Perfect Guttman scale is seldom achieved – errors of measurement
• Example : Beck Depression Inventory
• I occasionally feel sad or blue
• I often feel sad or blue
• I feel sad or blue most of the time
• I always feel sad and I cant stand it
A client who endorses the last
certainly agrees with milder form

15. SELECTING A SCALING METHOD
Method of Empirical Keying:
• Test items selected based on how well they contrast a criterion
group from a normative sample
Pool of person experiencing Major Depression gathered
to answer the pool of true-false questions
Endorsement frequency of depression group is
compared to normative group
Items showing large difference in endorsement
frequency selected for the depression scale – keyed in
the direction favored by depression subjects
Raw score for depression-number of items
answered in keyed direction

16. SELECTING A SCALING METHOD
Rational Scale construction (Internal consistency):
• Popular method for self-report personality inventories
• Rationale: All scale items correlate positively with each other and
with the total score
• Starting with review of literature - to scale a character like
leadership
• True false statements
• Administered to large sample of individual similar to target sample
• Items with weak or negative correlation to the total score are
discarded
• Item-total correlation is recalculated the to verify the homogeneity
of remaining items

17. CONSTRUCTING THE ITEMS
Initial questions:
• Homogeneity vs heterogeneity:
– Depends on how the test developer has defined the new
instrument
– For example culture reduced form of general intelligence will
have varied items whereas theory based test of spatial
thinking will have homogenous items
• Range of difficulty :
– Meaningful differentiation of examinees of both extremes
– Graded series of very easy items passed by all to a difficult
items passed by virtually no one

18. CONSTRUCTING THE ITEMS
• Ceiling effect
– Significant number of examinee obtain near perfect score
– No distinction between high scoring individual even though there
might be substantial difference in underlying trait
• Floor effect:
– Significant number of examinee obtain scores at bottom or near
bottom
– Example: WAIS-R had serious floor effect so the discrimination
between moderate, severe and profound level of intelligence is
difficult

19. CONSTRUCTING THE ITEMS
Table of specification
• Enumerates the tasks on which examinees are to be assessed
• Content by process matrix: lists the exact number of items n
relevant contact areas and details the precise composite of
items
Hypothetical 10 item Science Achievement Test(Content by process
Content area Factual knowledge Information
competence
Inferential reasoning
Astronomy 8 3 3
Botany 6 7 2
Chemistry 10 5 4
Geology 10 5 2
Physics 8 5 6
Zoology 8 5 3
Total 50 30 20

20. CONSTRUCTING THE ITEMS
ITEM FORMAT
1. Multiple choice methodology:
• Quick and objective
• Measure conceptual as well as factual knowledge
• Fairness can be proved
• But difficulty in writing good distractor options
• Presence of response may cue a half knowledgeable respondent to
the correct answer

21. CONSTRUCTING THE ITEMS
2. Matching Questions :
• Good in classroom testing but has serious psychometric
shortcomings
• Responses are not independent
• Missing one match compels the examinee to miss another
• Options must be closely related

22. CONSTRUCTING THE ITEMS
3. Short answer Objective item :
– Individually administered test
– Simplest and most straight forward types of questions
– the best reliability and validity
4. True/False Questions:
– Useful in personality tests
– Easy to understand and simple to answer
• Socially desirable responses minimized by “Forced choice
methodology
e.g. choose between 2 equally desirable or undesirable options
Which would u do? a. Mop the floor b. Volunteer for half day

23. TESTING THE ITEMS
Reliability:
• Consistency of the score
– Reexamining with same test on different occasions
– Different sets of equivalent items
– Under other variable examining conditions
• Concerned with degree of consistency or agreement between two
independently derived set of scores
• Can be expressed in terms of correlation coefficient (0 to +1)
– Degree of correspondence or relationship between two sets of
scores
– Sometimes negative as in time score correlated with amount
scores (time to solve maximum no of problem of mathematics)

24. TESTING THE ITEMS
1. Test-Retest reliability
• Reliability coefficient is simply the correlations between the same
person on the two administration of the same test
• In a test the interval between the administration of test should be
always be specified
• Difficulties: practice leads to improvement, recall
• Best for sensory discrimination or motor test

25. TESTING THE ITEMS
2. Alternate form reliability:
• Using alternate form of test
• Correlation between the two test scores
• Temporal stability and consistency of response to items
3. Split-Half Reliability
• Two scores for each individual by diving the test into two halves
• Also called coefficient of internal consistency

26. TESTING THE ITEMS
4. Kuder-Richardson Reliability and Coefficient Alfa
• Examination of each item in the test
• Single administration and single test
5. Scorer Reliability:
• Good deal of judgment in the scorer like in projective testing
Error of measurement: margin of error to be expected in an individual
score as a result of unreliability of the test
A reliability coefficient of .85 means that 85% of variance in test
scores depends on true variance in trait and 15% on the error
variance

27. TESTING THE ITEMS
Item reliability index
• Point biserial correlation coefficient
• Higher the correlation coefficient between an item and total
score the more useful is the item from the stand point of
internal consistency
• In case of dichotomous items if the item approaches a 50 -50
split of right and wrong scores greater is the standard deviation
• Product of these two indices, correlation and standard
deviation; is item reliability index
• Item with high reliability will have high internal consistency and
high SD
• So items with low reliability index can be discarded

28. TESTING THE ITEMS
VALIDITY
1. Content validity:
• systematic examination of the test content to determine whether it
covers a representative sample of the behavior domain to be
measured
• Important in achievement test; less in personality or aptitude
• Example the different domain of IQ in IQ questionnaire
• Another validity that is confused with content validity is face
validity
• It is what the test appears superficially to measure, e.g. a scale for
children might have low face validity for adult

29. TESTING THE ITEMS
2. Criterion validity:
• Effectiveness of a test in predicting an individual’s performance in
specified activities
– Concurrent: diagnosis or existing status
– Predictive: prediction for future e.g. selection or hiring of new
personnel
• Compares the test with other measures or outcomes (the criteria)
already held to be valid

30. TESTING THE ITEMS
3. Construct validity:
• The extent to which the test may be said to measure a theoretical
construct or trait like intelligence, depression, psychopathology
• Test must correlate with other variables or tests with which it
shares an overlap of constructs- convergent validity
• Test must not correlate with the variables from which it should
differ-discriminant validity
Measured by validity coefficient which is the correlation between test score and
criterion measured
Error of estimate: margin of error to be expected in predicted criterion score as a
result of imperfect validity of test

31. TESTING THE ITEMS
Item validity index
• To identify predictively useful test items
• Compute point biserial correlation between the item score and
score on the criterion variable
• More is the value more useful is the item in view of item validity
• Item validity index is the product of SD and the point biserial
correlation

32. TESTING THE ITEMS
Item difficulty index
• Proportion of examinee in a large tryout sample who get that item
correct
• Varies from 0.0 to 1.0
• If the index is 0 , every individual has answered it; so item
becomes psychometrically unproductive – same with index 1
• Index should hover between .3 to .7
• In a true false or multiple choice test, difficulty index of 0.5 can
result due to guessing so difficulty index must be around 0.75

33. TESTING THE ITEMS
Item Characteristic curves
• Graphical display of relationship between the probability of correct
response and the examinee’s position on the underlying trait
measured by the test
• Used for identifying items that perform differently for subgroups
of examinees
• Example: sex biased question involving football facts – For man
desired slope but for woman flat curve so the items that differ can
be eliminated

34. TESTING THE ITEMS
Ability level
Probability
of correct
response
b
a
c
Item Characteristic curves

35. TESTING THE ITEMS
• An ideal item is the one that most of the high scorers pass and most of the
low scorers fail
• For an ideal item it should have an ogive curve
• But visual inspection is not completely objective
Item discrimination Index:
U=no of examinees in upper range who
answered correctly
L=no in lower range who answered the item
correctly
N= total no of examinee in upper range or lower
range
• -1 to +1
d=(U-L)/N
D=0 : cant discriminate between low and high scoring
Closer to 1 is good
Negative score items need to be replaced

36. TESTING THE ITEMS
Response biases
• Wide range of cognitive biases that influence the responses of
participants away from an accurate or truthful response
• Socially desirable response, mainly prevalent in self report
inventories
• Steps to prevent:
– Relatively subtle or socially neutral items
– Use of forced choice items
– Use of special scales within the inventory to see socially
desirable responding
– Rapport during administration

37. STANDARIZATION
• Compared with some norm obtained by applying same test in the
sample supposed to represent the population
• Major problem – application of the norm of the large majority to
the minority population
• Context specific not stable over time
• Z score :
– Raw score expressed in units that indicate the position of an
individual relative to distribution of scores
– Score 0 =score at mean
– Score 1 =1 SD above mean
– Score -1=1 SD below mean
Z= (variable – group mean)/2
Fischer & Milfont, 2010

38. STANDARIZATION
1. Within subject:
Transformation of scores of each individual using the mean for that
individual across all variables
Relative endorsement of item = (Raw score – average across all
variables of an individual )
2. Within culture standardization:
Mean across all items and individuals in a group
3. Double standardization:
Combination of both
Used for the culture free dimensions
Ipsatization
Fischer & Milfont, 2010

39. REVISING THE TEST
• After the try out sample many items identification of unproductive
items so that they can be revised, eliminated, or replaced
• Next step: collect new data (2nd tryout sample)
• Examinees similar to test sample
• The main purpose of collecting additional data is to repeat the item
analysis procedure anew
• If major changes needed it might be desirable to collect data from
3rd or even 4th tryout sample

40. REVISING THE TEST
1. Cross validation:
– Fresh and independent confirmation of test validity
– The practice of using the original regression equation in a new
sample to determine whether the test predicts the criterion as
well as it did in original sample
2. Validity shrinkage :
– In the cross validation research the test predicts the relevant
criterion less accurately with new sample than original sample
– Inevitable
– Major problem when derivation and cross validation samples are
small, large number of items
3. Feed back from examiner

41. PUBLISHING THE TEST
Production of testing material
Technical manual and user’s manual
• Describe rationale, cautions against anticipated misuses
• Cite representative studies
• Identify special qualification needed to administer and interpret
test
• Provide revisions, ammendations, supplements
• Appropriate interpretive aids
• Essential data on validity and reliability

42. CULTURAL ISSUES
• Reliability and validity of measures within the dominant culture
• Measurement of various psychological constructs is also likely to be
influenced by cultural characteristics
• When utilizing standardized measures, assumption is the client is
similar to the standardized population - violated when assessing a
client from another culture
• Test translated in another language tends to decrease the reliability
and validity of the test
• One important way of tackling the cultural issues is cross cultural
development of the scale
Suzanne M. Skevington, 2002

43. • INTRODUCTION
• STEPS FOR DEVELOPMENT OF SCALES
• WHOQOL-100: AN EXAMPLE

44. AN EXAMPLE:WHOQOL-100
• Develop a reliable, valid, and responsive assessment of quality of
life that is applicable across cultures
• The aim was to simultaneously develop the assessment in several
different cultures and languages rather than translating
• CONCEPT CLARIFICATION:
– First phase of work : international collaborative review to
establish a definition of quality of life and an approach to
international quality of life assessment
– Protocol development
Group, T. W. (1998)

45. AN EXAMPLE:WHOQOL-100
Qualitative pilot : second phase of work
WHOQOL Facets:
• Focus groups discussion conducted in each of the 15 centers
• Facets were listed and the definition created with consensus from
health professionals, general population, population with disease or
impairment
• Maximum of twelve questions written in each center for each facet
in the local language and translated in English
• Global question pool = 1800 questions
• Evaluated to see what extent they met the criteria for WHOQOL
questions=1000 question
Group, T. W. (1998)

46. AN EXAMPLE:WHOQOL-100
Generation of the response scales
• Five point Likert scales
• Two anchor points selected (like very satisfied to very dissatisfied or
not at all to extremely never to always)
First anchor
point
second
anchor point
Best descriptors for 25%, 50%,
75% in each language
Group, T. W. (1998)

47. AN EXAMPLE:WHOQOL-100
Piloting and psychometric evaluation
• 15 field centers in different countries : 236 questions, 6 domains, 29
facets
• Separate 41 questions to indicate how important the each facet
was to the quality of life
• The centers could add up to 2 additional national/ regional
questions per facet
• Scale reliability using SPSS and MAP (multi trait analysis program)
• Items dropped
– failed to discriminate between sick and well population
– Showing <10% of the responses on aggregate
– Criterion for the global data, but failed to meet it for more than
50% of the centers
Group, T. W. (1998)

48. AN EXAMPLE:WHOQOL-100
• Reliability analysis second wave
• Validity analysis
– Groups discriminant validity in the form of a comparison
between healthy and unhealthy individuals
– Items not significantly distinguishing healthy from unhealthy
individuals were highlighted for possible elimination
• Facet and domain inter-correlation
• Decision was taken to select four items per facet,
• These decisions therefore led to the selection of 25 x 4 =100 items
• Revised field trial known as whoqol-100
Group, T. W. (1998)

49. AN EXAMPLE:WHOQOL-100
• Factor analysis yielded 4 factors with Eigen values greater than 1
• Confirmatory Factor analysis:
– The 4 factor model fitted to the conceptual 6 factor model with
the value of little less than 0.9
– Six domain model CFI=0.88
– Six conceptual domains expected to load onto one single factor
(a hypothetical quality of life construct) with comparative fit
index of 0.975
Group, T. W. (1998)

50. AN EXAMPLE:WHOQOL-100
Group, T. W. (1998)

51. CONCLUSION
• Different scales are used in psychiatry broadly the personality and
intelligence
• Scales can be divided into many types as per their use, function or
form of items
• The development of scale is a daunting task that needs the
expertise in the respective field and statistics
• Cross cultural development of scale is very useful in overcoming the
cultural confounders
• To develop a new test over existing test there are hindrances like
copy right laws, economic issues as developing is a costly venture

52. Thank you