Reliability

1. By
Vishnu Achutha Menon
Research Scholar
Department of Media and Communication
CUTN
Understanding, Calculating, and
Evaluating Reliability
and Objectivity

2. Methods for Evaluating Reliability andMethods for Evaluating Reliability and
Their CalculationsTheir Calculations

3. TestTest–Retest–Retest
• Most straightforward way to determine
reliability
• Must have:
– No major changes in the construct being
measured
– Sufficient recovery time between
measurements

4. Evaluating TestEvaluating Test–Retest Reliability–Retest Reliability
• Reliability coefficient:
– A ratio that shows the relationship between two
measurements, indicating the consistency (or
reliability) between them.
• Intraclass correlation:
– A statistical technique used to compute the
reliability coefficient to assess the relationship
between measures of the same class as in a test–
retest study.

5. Calculating the Intraclass CorrelationCalculating the Intraclass Correlation
R=[(SSa/n–1) – (SSw/(n*(k–1)))]/ (SSa/n–1)
SSa=(ΣT2
/k) – ((ΣX) 2
/nk)
SSw= ΣX2
– (ΣT2
/k)
Where R is the intraclass reliability
Σ represents the sum
N = the number of test subjects
k = the number of trials for each person (usually two)
ΣT2
= the sum of all the squared total scores for each
person
ΣX2
= the sum of all the scores of everyone tested

6. An Alternative forAn Alternative for
Calculating the Intraclass CorrelationCalculating the Intraclass Correlation
• Try this website:
http://department.obg.cuhk.edu.hk
and go the Statistics Tool Box link.

7. Results of a CalculationResults of a Calculation
of Intraclass Reliabilityof Intraclass Reliability

8. Evaluating Reliability withEvaluating Reliability with
a Single Test Administrationa Single Test Administration
• Split-half reliability:
– Compare one half of
a test with the other
half
– Spearman-Brown
Prophecy Formula
• Internal consistency
reliability:
– Average all possible
split-half estimates
– Cronbach’s alpha

9. Evaluating the ReliabilityEvaluating the Reliability
of Criterion-Referenced Measurementsof Criterion-Referenced Measurements
• Calculate percentage of agreement between the test
and the retest.
Percentage of Agreement= [(Cboth + NCboth) / (Cboth + NCboth +
C/NC + NC/C)] * 100
Where Cboth = people scored as competent in both Trials 1 and 2
NCboth = people scored as not competent in both trials
NC/C = people scored as not competent in Trial 1 but
competent in Trial 2
C/NC = people scored as competent in Trial 1 but not
competent in Trial 2

10. Example Diagram for Evaluating the ReliabilityExample Diagram for Evaluating the Reliability
of a Criterion-Referenced Measurementof a Criterion-Referenced Measurement

11. Standard Error of MeasurementStandard Error of Measurement
• Defined:
– An estimation of the error inherent in any
individual’s test score.
SEM = SD * √1–rrc
where SD = the standard deviation for the test
rrc= the reliability coefficient for the test

12. Increasing ReliabilityIncreasing Reliability
• Repeat a measurement several times—
– To improve both validity and reliability
– To discover and minimize errors
– To average out the errors

13. Methods for Evaluating ObjectivityMethods for Evaluating Objectivity

14. Calculating ObjectivityCalculating Objectivity
• Objectivity can be considered a special case of
reliability.
– Inter-rater reliability
• Most techniques used to evaluate reliability
can be used to evaluate objectivity.

15. Calculating ObjectivityCalculating Objectivity
of Different Types of Measuresof Different Types of Measures
• For continuous measures:
– Intraclass correlation
• For discrete measures:
– Calculate the percent agreement between
test administrators

16. Validity, Reliability, and ObjectivityValidity, Reliability, and Objectivity
• It is possible to have high reliability or
objectivity without high validity.
• Good reliability or objectivity will always be
present with a valid measurement.
• Good reliability and objectivity do not
establish good validity; they simply suggest
that a measurement may be valid.

17. Your ViewpointYour Viewpoint
• Can you think of any times in your life when
you have had to evaluate the reliability or
objectivity of something or someone?
• What did you do with the results of this
evaluation? Did it cause you to make any
changes in your daily routines or change your
mind about a decision?