Reliability

Reliability Testing for Item Analysis Dr. Debdulal Dutta Roy , Ph.D. Psychology Research Unit Indian Statistical Institute 203, B.T. Road Kolkata - 700108 E-mail: ddroy@isical.ac.in Fax : 91 - 33 - 25776680 Tel (o) : 91 - 33 - 2575 3454 Presentation at the Department of Clinical Psychology, Ram Chandra University, Chennai 21.2.09

Reliability Analysis

Reliability refers to the consistency of scores obtained by the same persons when reexamined with the same test on different occasions, or with different sets of equivalent items, or under other variable examining conditions (Anastasi, 1990). It indicates the extent to which individual differences in test scores are attributable to “true” differences in the characteristics under consideration and the extent to which they are attributable to chance errors. Reliability of a test is given by the proportion of true variance resulting from the presence of specific situation under consideration and error variance resulting from the presence of some factors irrelevant to the present situation.

Reliability Analysis

Test-Retest Reliability

Johnny, Johnny, Yes, Papa, Eating sugar? No, Papa Telling lies? No, Papa Open your mouth O Ha! Ha! Ha!

Test retest reliability (8 months interval) Test retest reliability of Reading Motivation Questionnaire (n=72) using both t-test and correlation coefficients 0.93** -0.27 1.01 4.5 1.05 4.46 Total -1.41 0.38 0.83 0.44 0.74 19A 2.43 0.49 0.63 0.41 0.79 16B 0.53 0.35 0.86 0.32 0.89 14B -0.83 0.32 0.89 0.36 0.85 4A 0 0.23 0.94 0.23 0.94 2A -1.41 0.44 0.35 0.44 0.25 1B Knowledge 0.706 -0.23 1.01 4.86 1.14 4.82 Total -2.99 0.26 0.93 0.42 0.78 21A 0.19 0.46 0.69 0.46 0.71 17A -1.84 0.23 0.94 0.36 0.85 9B 1 0.28 0.92 0.2 0.96 7A 0.81 0.45 0.72 0.42 0.78 6B 1.41 0.48 0.65 0.44 0.75 1A Application Correlation t-ratio(df=71) SD Mean SD Mean Second session First session

Test-Retest Multi-item response

All items do not behave in same fashion always.

Identify inconsistent items in the set across periods..

Last supper: Leonardo Da Vinci

Test-Retest Multi-item response Consistency (8 months interval)

Test-Retest Multi-Trait Consistency (8 months interval) After 8 moths Tool: Reading motivation questionnaire (Dutta Roy, 2002); N=72 students of same school

Alternate form

Correlating scores of two parallel forms of a single test

Number of items in both forms should be same.

Both have uniform content, range of agreement and disagreement

Means and standard deviations of both forms should be equal

Mode of administration and scoring of both should be uniform

Split-half

Upper and lower part of the questionnaire sometimes differ in item content.

All items do not reflect same content always.

Split-half reliability

Correlating equal halves of test

Correlating odd and even no. of items

2 X reliability of half test

rtt= -------------------------------------------

1 + reliability of half test

Reliability of half test = Product moment corr of two halves

Advantages : On-the-spot relibility

Disadvantages : Failure to assess temporal stability

Split-half Canonical correlation

Split-half Canonical correlation provides knowledge about the percent of variance in the one set explained by the other set of variables along a given dimension .

Study: Split-half Canonical correlation

12-item Likert type 5 point scale assessing attitude towards workers education was administered to 1600 rural workers of WB.

Split-half rtt=0.85; Cronbach’s alpha = 0.87

Canonical correlation coefficient between the sets (first 6 and last 6 items) = 0.78, Chisq(36)=1558.3, p<0.0000.

Internal Consistency

It measures whether several items that propose to measure the same general construct produce similar scores. For example, if a respondent expressed agreement with the statements "I like to ride bicycles" and "I've enjoyed riding bicycles in the past", and disagreement with the statement "I hate bicycles", this would be indicative of good internal consistency of the test.

Coefficient alpha

Cronbach’s coefficient alpha is an useful index to assess internal consistency of the scale. It is equivalent of Hoyt’s ANOVA procedure. The formula is:

N Sum of item variance

Alpha = -------------- X 1 - --------------------------------------------

N -1 Variance of total composite

EXAMPLE

Item-Item correspondence: Internal consistency among 42 items of Reading Motivation questionnaire Correspondence Map shows cluster of intrinsic reading motivation items and extrinsic motivation items are scattered widely.

Correspondence map of traits

Rational Equivalence

The formula is given below:

rtt = (n/(n-1)) X ((  2t-  pq) /  2t)

in which,

rtt= reliability coefficient of the whole test

n= number of items

 t= the SD of the total scores

p= proportion of the group giving ‘yes’ responses

q= (1-p)= the proportion of the group giving ‘no’ responses

Reliability Coefficients of Attitude towards School Infrastructure Questionnaire (N=175) Ref: Dutta Roy,D. (2008). Attitude towards school infrastructure in rural areas. Unpublished project report submitted to Indian Statistical Institute, P45. 0.5 10 Willingness to Participate 0.63 5 Equal Opportunity 0.68 7 Easiness 0.5 5 Reliability 0.5 12 Exploring 0.58 12 Adequacy 0.42 5 Comfort 0.68 7 Safety 0.58 5 Cleanliness Kuder Richardson’s Reliability coefficients No. of Items Attitudes

Factors influencing test scores