This document outlines the fundamentals of measurement and instrument development, focusing on true-score theory, measurement error, reliability, and validity. It provides five strategies for developing effective survey instruments, emphasizing clarity, purpose-driven question design, and understanding the target population to avoid irrelevant data collection. The importance of detailed survey design is highlighted to ensure accurate and meaningful results.

1. Fundamentals of
Measurement & Instrument
Development
Patrick B. Barlow, PhD
Assistant Professor, Internal Medicine
Program Evaluation & Research
Consultant, The Office of Consultation &
Research in Medical Education (OCRME)

2. First, an example of why we need
quality measurement….

3. In This Presentation
• Reviewing the Basics of True-Score Theory
– Measurement Vocabulary
– Measurement Error
– Reliability
– Validity
• Five Simple Strategies to Developing an Instrument
– Fit your question stems and response options to your
purpose
– Clarity is key, “If it’s nice to know, it’s gotta go!”
– Leave NO detail unexplained
– Know your population
– Garbage in, garbage out.
• SURVEY 911! Activity

4. REVIEWING THE BASICS OF
TRUE-SCORE THEORY
Measurement Error
Reliability
Validity
Survey Data

5. Measurement Vocabulary
• Measurement:
– Broadly, involves assigning numeric values to objects or events
in an effort to make meaning and understanding of a
particular variable
• Scale:
– A number of individual measurement items are combined to
create a single, composite instrument
• Latent Variable (aka “construct” or “latent trait”)
– Responses to individual items on these scales are combined to
create a single score meant to measure theoretical or latent
variables or traits.
– A latent trait is one that cannot be easily observed directly,
and is therefore estimated by an individual’s observed score on
the scale

6. True-Score Theory
• Also known as “Classical Test Theory
(CTT)” or “Classical Measurement
Model”
• Views an individual’s actual location
on a latent variable as:
𝑂𝑏𝑠𝑒𝑟𝑣𝑒𝑑 𝑆𝑐𝑜𝑟𝑒 = 𝑇𝑟𝑢𝑒 𝑆𝑐𝑜𝑟𝑒 + 𝑀𝑒𝑎𝑠𝑢𝑟𝑒𝑚𝑒𝑛𝑡 𝐸𝑟𝑟𝑜𝑟

7. Measurement Error
• CTT assumes error values are…
– Random across all scale items
– Independent of one another
– Independent of the true score itself
• Random vs. Systematic Error
– Random error is uncontrolled “noise” that does not
dramatically impact the accuracy of the measurement
– Systematic error such as extraneous/confounding
variables and biasing artificially “trend” the
measurement in one direction or another

8. Reliability
• Essentially, the consistency of scores
produced by a given instrument.
– The degree to which differences in
respondents’ observed scores are consistent
with those on their true/trait scores
– All things being equal, an individual should
score consistently on the same measure
across multiple administrations
– Necessary but not sufficient for validity!

9. Reliability vs Validity

10. Types of Reliability
• Test-retest
– Correlation between pretest and posttest
• Inter-rater
– Kappa Statistic
– Inter-rater agreement
• Internal Consistency
– Cronbach’s Alpha, KR20
• Parallel Forms
– Correlation between the two forms

11. Validity – The Basics
• There is an entire field of study based on
validity and its various components; however,
we will not be our focus today.
• Essentially, validity refers to extent to which
the interpretation of an instrument’s scores
accurately reflect the construct of interest.
• Considered to be an ongoing process of
gathering increasingly varied sources of
“Validity Evidence”, not “validate and be
done”.

12. Types of Validity
• First, everything is “construct validity”
• Content Validity
• Criterion Validity
– Convergent
– Concurrent
– Predictive
– Discriminant

13. Fit your question stems and response options to your purpose
Clarity is key, “If it’s nice to know, it’s gotta go!”
Leave NO detail unexplained
Know your population
Garbage in, garbage out.
FIVE SIMPLE STRATEGIES TO
DEVELOPING AN INSTRUMENT

14. Strategy One
Fit your question stems and response
options to your purpose
Question Stem
• Should clearly ask a single
question, or make a single
statement to which the
participant will respond.
• Consider grouping multiple
statements/questions into a
single matrix to save time
and space.
• Use Bold or italics to highlight
key words or phrases
Response Options
• Should flow naturally from
the stem
• Include all possible options
when you can.
• More precise levels of
measurement are going to
yield better results (see
examples)
• Group items with the same
response sets to avoid
confusion whenever possible.

15. Examples…
The same stem can be written as both a
question and a statement:
“How confident are you that the teacher
training improved your skills?”
OR
“I am confident that the teacher training
improved my skills”
The choice depends on what you want to
know from the participants.

16. Examples…
“How confident are you that
the teacher training
improved your skills in…?”
Here the participants could rate
their confidence in various skills
that the training addressed
from:
1 = “Not at all confident”
2 = “Somewhat confident”
3 = “Moderately confident”
4 = “Very confident”
5 = “Extremely confident”
“I am confident that the
teacher training improved my
skills in…”
This approach could have
participants rate their
agreement with the statement.
1 = “Strongly disagree”
2 = “Disagree”
3 = “Neither agree nor disagree”
4 = “Agree”
5 = “Strongly agree”

17. Examples…
Similarly, the same piece of
data can be measured many
different ways!
Take “Teacher experience” for
example…
Stem: What is Your Smoking History?
“New Teacher” or
“Returning Teacher”
“First year teacher,” “2 to 5
years,” “5 to 10 years,”
“More than 10 years”
“1 to 3 years” “4 to 6
years,” “7 to 9 years,” “10
to 12 years”
How many years have you
been a teacher? (write a
number) ______ Years.
MorePrecise

18. Strategy Two
“If it’s nice to know, it’s gotta go!”
• Oftentimes we want to ask additional questions
because we may use the information later. This leads
to…
– Increased time to take the survey
– Irrelevant questions being placed in the survey that may
distract the participants
– More work on the data collector to write and analyze the
extra items
– Increased “participant fatigue”

19. Strategy Three
Leave NO Detail Unexplained
• Always assume that the participants do not
understand how to take your survey, even if it
seems very self-explanatory. In other words,
“Play to the lowest common denominator.”
• A lack of clarity can lead to participants giving
incorrect information, which in turn will weaken
the results of your survey.

20. Examples…
• Always add clear instructions at any point where there
may be some lack of clarity. Examples of when to use
instructions include:
– Whenever you change type of question or response set
• “Directions: Please place a in the box that best describes your
opinion of the teacher training workshop.”
– Whenever the participant is given an opportunity to write in
their own response
• “What is your age? (Please write a number) _____”
– When conducting an online survey, include instructions that
remind the participant to click “Submit” prior to closing their
browser.
• “Please fill out the background information on the form below. Once
you are done, click “Submit” at the bottom of the page to submit your
responses before closing your web browser.”

21. Strategy Four
Know Your Population
• Properly researching the target population will be
essential to a successful survey because it:
– Lets you target your writing style, vocabulary, and
question type to fit the education or skillset of the
population
– Helps you choose the best mode of administration
(internet, “snail mail,” in-person, etc.) to get the
maximum number of respondents
– Gives you a frame of reference for any generalizations or
conclusions the survey is meant to make.
– Helps to avoid including unnecessary or irrelevant questions
that can be gathered from other sources.
• For example: if the population is school children in this district,
then there is no need to ask their GPA since it could be pulled
from district records.

22. Strategy Five
“Garbage in, garbage out.”
• Many surveys are created in the last minute without
attention to the details shared in this presentation.
• Failure to attend to these details leads to a “Garbage”
survey
• A garbage survey will produce garbage data that cannot be
used to make any meaningful conclusions regarding your
populations.
• The goal with any type of survey research is to generate
the highest response rate possible within the target
population, and issues such as those described can all
negatively impact the number of people answering your
survey!

23. Parting Reminders…
EVERY element of a survey design from the stems/responses
(Strategy One) to the instructions (Strategy Three), and even
the aesthetics, order, and presentation of the items have a
large body of research supporting best practices.
So,
Be clear, be concise, be professional in your attention to
detail, and you will be able to maximize your return on any
survey project you encounter.

24. Activity: SURVEY911!