This document provides guidelines for writing noncognitive measurement items. It discusses the importance of having a strong conceptual framework to define the construct being measured. Good frameworks are presented in scholarly journals and provide clear descriptions and factors of the variable. The document also discusses characteristics of good items such as being consistently understood by respondents. Response formats that can be used are also presented such as Likert scales, checklists, and ranking scales. Presurvey evaluation through focus groups and testing is recommended to ensure respondents understand the questions.

1. Guidelines in Writing
Items for Noncognitive
Measures
Carlo Magno, PhD
De La Salle University, Manila

2. How do I start writing items?
What variable do you want to measure?
You need a conceptual framework
Good conceptual frameworks are presented in
scholarly journals
A scale becomes faulty if the framework is faulty or
not well-grounded on a theory or model
The framework should provide the description,
definition, and factors of the variable you want to
study.
Get a framework from the original author of the
construct

3. Examples of Framework
Passion by Vallerance et al. (2003)
Obsessive passion
Harmonious passion
Epistemological beliefs by Schommer (1993)
Fixed ability
Simple knowledge
Quick learning
Certain learning

4. Examples of Framework
Achievement emotions by Pekrun (2006)
Positive activating emotions: enjoyment, hope,
pride
Positive deactivating emotions: relief
Negative activating emotions: anger, anxiety,
shame
Negative deactivating emotions: hopelessness,
boredom

5. Look for the definition in the framework
There are good definitions
There are also bad definitions

6. Using definitions
• Good definitions
– Accurate
– Easily understood
– Provides behavioral
manifestation of the
construct
– No contradictions
• Bad definitions
– Overlaps with other
constructs
– Highly technical
– Difficult to frame
examples
– Parsimonious
(exemption to the rule)

7. What is a good item?
Items need to be consistently understood
Items need to be consistently administered or
communicated to respondents
Unless measuring knowledge is the goal of the
question, all respondents should have access to the
information needed to answer the item accurately
Respondents must be willing to provide the
responses called for in the item

8. Gathering factual data
Ensures that respondents have the same
understanding of what is to be reported.
Faulty: I ate butter in the past week.
Better: I ate butter in the past 7 days not counting
any margarine.
Faulty: My income is 1,000 Php.
Better: My total family income including from all
sources, before deductions for taxes, for me and
my family members living with me for 2008 is
1,000 Php.

9. Knowing and remembering
 Form of an answer:
 Faulty: In the past 30 days, were you able to climb a flight
of stairs with no difficulty, with some difficulty, or were you
not able to climb the stairs at all?
 Better: How many miles are you from the nearest hospital?
 Reducing social desirability:
 Ensure confidentiality
 Emphasize in the introduction and in other ways the
importance of the accuracy of answers
 Use self-administration rather than interviewer
administrator, or have respondents enter their answers
directly into the computer.

10. Questions to measure subjective states
Defining what is to be rated:
Example: In general, do you think government
officials care about your interest a lot, some, only a
little, or not at all?
Example: Do you consider crime to be a big problem,
some problem, or no problem at all?
Response task:
Overall, how would you rate your health?-excellent,
very good, good, fair, poor?

11. Questions to measure subjective states
Agree-disagree format:
Example: My health is excellent
Example: I like my mother
Rank order:
Which candidate do you prefer?
Here are some factors some people consider when
deciding where to live. Which is most important to
you?
Proximity to work
Quality of school
Parks
Safety
Access to shopping

12. Questions to measure subjective states
Narrative answer
What do you consider to be the most important
problem facing your local city government today?
Measuring responses to ideas:
Higher taxes generally hurt the rich and benefit the
poor. Do you agree or disagree?
In general, would you like to have more money
spent on the parks and playgrounds in your
neighborhood area or not?

13. General rules for designing good survey
instruments
 The strength of the survey research is asking people about
their firsthand experiences. Ask questions to which most
people have informed answer.
 Beware of asking about information that is only acquired
second-hand.
 Beware of hypothetical questions.
 Beware of asking about solutions to complex problems.
 Questions should be asked one at a time.
 Avoid asking two questions at once.
 Avoid questions that impose unwarranted assumptions.
 Beware of questions that include hidden contingencies.

14. General rules for designing good survey
instruments
 A survey question should be worded so that all respondents
are answering the same questions.
 To the extent possible, choose the words in questions so
that all respondents understand their meaning and all
respondents have the same sense of what the meaning is.
 To the extent that words or terms must be used that have
meanings that are likely not to be shared, provide
definitions to all respondents.
 The time period referred to by a question should be
unambiguous.
 If what is to be covered is too complex to be included in a
single question, ask multiple questions.

15. General rules for designing good survey
instruments
 If a survey is to be interviewer administered, wording of the
questions must constitute a complete and adequate script
such that when the interviewer reads the question as
worded, the respondent will be fully prepared to answer the
question.
 If definitions are to be given, give them before the question
itself is asked.
 A question should end with the question itself. If there are
response alternatives, arrange the questions so that they
constitute the final part.
 All respondents should understand the kind of answer that
constitutes an adequate answer to a question.

16. General rules for designing good survey
instruments
Avoid questions that begin with adverbs: how when,
where, where, why, to what extent. Such questions do
not specify the terms of an adequate answer.
Specify the number of responses to be given to
question for which more than one answer is possible.
Survey instruments should be designed so that the task
of reading questions, follow instructions, and recording
answers are as easy as possible for interviewers and
respondents.

17. Presurvey evaluation questions
Before the actual survey is done, testing should be
done to find out if respondents can understand it, if
they can perform the tasks required, and if interviewers
will read it as recorded.
Some steps:
Focus group discussion
Do the questions appropriately cover what respondents
are suppose to describe?
Are the response tasks that the questions will pose tasks
that respondents are able and willing to perform?
Do the words or descriptions proposed in the questions
convey consistent meaning, so that respondents have a
common understanding of question they are to answer?

18. Presurvey evaluation questions
Intensive individual interview
Find out how respondents understand the questions
Respondents are brought to special settings like the
laboratory
Reading questions to respondents
Field pretesting

19. Guide to questionnaire construction
Match your objectives with your questions
Keep the language pitched to the level of the
respondents.
Try to pick words that have the same meaning for
everyone.
Avoid long questions
Do not assume that that your respondents possess
factual information, or first hand opinions.
Establish the frame of reference you have in mind.
In forming a question, either suggest all possible
alternatives to the respondent or don’t suggest any.

20. Presurvey evaluation questions
Protect your respondents ego.
If you’re after unpleasant orientations, give your
respondent a chance to express his positive feelings
first so that he’s not put in an unfavorable light.
Ex. What do you like about X?
What don’t you like about X?
Decide whether you need a direct question, an indirect
question, or an indirect followed by a direct question.
Decide whether the question should be open or
closed.
Decide whether general or specific questions are
needed.

21. Presurvey evaluation questions
Avoid biased or leading questions
Phrase questions so that they are not unnecessarily
objectionable.
Decide whether a personal or impersonal question will
obtain the better response.
Questions should be limited to a single idea or a single
reference.

22. 22
Examples of Response Formats
Multiple response item:
Single-response item

23. 23
Examples of Response Formats
Lickert Scale

24. 24
Examples of Response Formats
Verbal frequency scale

25. 25
Examples of Response Formats
Ordinal scale

26. 26
Examples of Response Formats
Forced Ranking Scale
Paired Comparison Scale

27. 27
Examples of Response Formats
Comparative Scale

28. 28
Examples of Response Formats
Linear Numeric Scale

29. 29
Examples of Response Formats
Semantic differential scale

30. 30
Examples of Response Formats
Adjective checklist
Semantic distance scale

31. 31
Examples of Response Formats
Fixed sum scale

32. 32
Examples of Response Formats
Multiple rating list

33. 33
Examples of Response Formats
Multiple rating matrix

34. 34
Examples of Response Formats
Diagram scale

35. 35
Examples of Response Formats
Graphic scale