The document provides information on designing questionnaires and surveys. It discusses determining research questions, different types of surveys, principles of good survey design, question formatting and layout, advantages and disadvantages of questionnaires, improving response rates, question types, designing effective questions, and data analysis. Key points covered include framing the research question, descriptive vs. analytic vs. exploratory vs. confirmatory surveys, keeping surveys concise, credible, and minimizing burden on respondents.

1. Research Methodology Workshop 22 May 2013
Hampshire Teaching School Alliances
Wan Ching Yee
Questionnaire design & analysis

2. The research question frames the rest
• … the concepts you are researching
• … the relevant research literature
• … design considerations
• … samples
• … methodologies
• … the conclusions that you can draw!

3. Types of survey
• Descriptive – produce relevant data
• Analytic – examine relationships
• Exploratory – patterns explored without prior theory
• Confirmatory – model or causal relationships are
tested

4. What are the principles of good survey
design?
• Realistic and feasible
• KISS
• Select only those variables that are most likely to
affect outcomes
• Minimise cost and effort to respondents
• Credibility is a factor

5. 5
Presentation & Layout:
• Clear presentation e.g use of fonts
• Open questions used as little as possible
• Clear instructions about how to respond e.g. give
examples
• Vertical or horizontal closed answers?
• Keep question and answers together

6. 6
Why use questionnaires?
• Cheap to administer
• Quick to administer
• Absence of interviewer effects
• No interviewer variability
• Convenience for respondents

7. 7
Disadvantages of questionnaires
• Cannot prompt
• Cannot probe
• Cannot ask too many questions that are not salient
to respondents
• Do not know who answers
• Not appropriate for some kinds of respondents
• Greater risk of missing data
• Lower response rates

8. 8
Improving response rates
• Good (personalising?) covering letter
• Accompanied by SAE if postal questionnaire
• Follow up non-respondents
• Shorter questionnaires
• Clear instructions and attractive layout
• Avoid „bulky‟ questionnaire
• Begin with questions which are likely to be of
interest to the respondent
• Incentives

9. 9
Types of questions
• Personal factual questions
• Factual questions about others
• Questions about attitudes
• Questions about beliefs and values
• Questions about knowledge
• Branching question for more detailed questions

10. 10
General rules for designing questions
• Always bear in mind your research questions
(these are at a different level to your questions on
your questionnaire!)
• What do you want to know?
• How would you answer it?
• Aim for clarity, simplicity & neutrality

11. 11
Improve the quality of your questions
Try to avoid:
• Long complex questions
• Double negatives
• Jargon
• Abbreviations
• Culture specific terms
• Words with double meaning
• Leading questions
• Emotionally loaded words

12. 12
Improve the quality of your questions
May want to consider:
Including a response category:
• Don't know
• Not applicable
• Do not want to answer

13. 13
Data gathering
Question types: Closed and open ended text or
open-ended numeric
• Closed: Yes/No, Male/Female
+ Easy to convert to numerical format if using
statistical packages
- Limited choices
• Open ended: What is your major source of
anxiety at the moment?
+ Respondents have more freedom to respond in
their own way
- Responses need to be summarised (more time)

14. Young respondents
14

15. 15
Formatting
• Group together questions on major subject areas
• Begin with emotionally neutral questions which
ideally, are easy and pleasant to answer
• More sensitive or difficult questions in the middle
• Personal characteristics eg income, at the end

16. 16
Piloting and pre-testing questions
• Open questions used to generate fixed-choice
answers
• Identify questions that don‟t work e.g skipped,
unclear
• Identify clarity of instructions
• Use questions employed by other researchers?
• http://www.natcen.ac.uk
• http://surveynet.ac.uk/sqb/surveys/introduction.asp

17. 17
Steps in questionnaire design
• Make a list of variables
• Assemble file of questions or instruments for
measuring each variable e.g available in books, on
web
• Compose a draft
• Shorten the set of instruments
• Pretest
• Validate
• Consider your sample – how will you access them?
And how will you recruit them?

18. 18
Different types of data
Nominal or categorical
Examples Sex, preference type of coffee, colour
Characteristics No sense of order
Can be stored as a word, text or numerical
code
Summarise Frequency or percentage
Cannot use Mean or average value
Display Pie chart
Column chart
Bar chart

19. 19
Different types of data
Ordinal
Examples Satisfaction, rank, fanciness
Characteristics Have a meaningful order but the intervals
between the values in the scale may not
be equal
Summarise Frequency or percentage
Cannot use Mean or average value?
Display NOT as a pie chart
Column chart
Bar chart

20. 20
Different types of data
Interval/Ratio
Examples Number of customers, weight, age, size
Characteristics Includes things that can be measured
rather than classified or ordered
Data can be discrete whole numbers (5
customers) or continuous with fractional
numbers (1.2 miles)
Summarise Mean
Median
Standard deviation
Display Histogram
Bar chart
Line chart (data that occur over time)

21. 21
Data gathering and processing
• Data gathering –
Quantification/measurement
• Data modification –grouping/categorising,
ordering, taxonomies
Displaying – looking for trends
Comparing – with each other, with the
population
Relationships between variables
Use of statistics

22. 22
Grouping
Define and label variables (sex, age)
Assign numbers to each of the possible
responses

23. 23
Grouping
Sex 1=Males
2=Females
Age 1= 21-25 years
2= 26-30 years
3= 31-35 years
4= above 36 years
5= No answer given
Optimising scale (items 1-4) 1 (strongly agree)
2 (agree)
3 (disagree)
4 (strongly disagree)
Closed questions

24. Grouping
Open ended questions
What is your major source of anxiety at the moment?
Scan through questionnaires and try to find common
themes.
Anxiety 1=Grade
2=Pass the course
3=Lack of time

25. 25
Displaying data
• Basic diagrams can be created with word processing
programmes such as Windows
• Most useful diagrams are typically tables, pie charts,
bar charts, trend lines
• Diagrams are best if simple
• Diagrams do not tell you about the importance of the
size of the quantities involved

26. 9

27. 27
Student numbers 2007: ft/pt,
European+Home/International
Inft Inpt EHft EHpt
Counselling 14 0 6 0
EDLP 17 0 7 9
Psych 14 0 17 23
Special Ed 1 0 7 9

28. 28
MEd and MSc Pathways 2007
0
10
20
30
40
50
60
Coun EDLP ICP Math Psyc ResM SpEd ETS SURE TESOL
O/seas ft O/seas pt EH ft EH pt

29. 29

30. 30
Basic Techniques
• Identification of variables
Dependent/Independent
Discrete/Continuous
• Sampling
• Piloting
Reliability
Validity

31. 31
Analysis
• Hypotheses may require comparisons, e.g. difference
between groups, difference between treatments,
relation between scores
• There is always some variation
• Possibility of measurement error, sample error, etc
• Hence we use statistics for an assessment of how
likely it is that the difference/relation found is not
worth thinking about

32. 32
Probability
• Statistical analyses assess probability that found
result is unlikely to be result of random variation in
population
• Convention is that result needs to be such that it
would not occur by chance more than 1 time in 20
(5%) or 1 time in 100 (1%) - or even less likely –
before we work on interpreting it

33. 33
Enhancing chances of getting Significance
• Carefully define your prediction (effectively bet on
outcome, don’t just look for difference say which
direction the difference should be)
• Use good reliable measures
• Select participants properly
• Use a large enough sample to be able to detect the
result you predict

34. Enhancing chances of making a real
significant discovery
Never ever „cherry-pick‟ the data that suit you
Never ever „cherry-pick‟ the results that suit you
Replicate, replicate, replicate
Test, test, test

35. 35
The core point
Scientific research needs to be seen for
what it truly is; a way of preventing me
from deceiving myself in regard to my
creatively formed subjective hunches
which have developed out of the
relationship between me and my material.
Carl Rogers 1961

36. Paired task
Considering the examples of “what not to do” from the
start of the session:
• Using the research question on your proforma,
start to construct a questionnaire to help you
answer this question. Think about the different
types of questions you could use, and how best to
phrase them.
• Pilot your questionnaire with another pair. What
improvements could you make?

37. Useful references
• Dillman, D.A. (1978) Mail and telephone surveys: The total
design method. New York: John Wiley and Sons.
• Oppenheim, A.N. (1992) Questionnaire design, interviewing
and attitude measurement. London: Pinter
• Sheatsley, P.B. (1983) Questionnaire construction and item
writing. In Ross, P.H., Wright, J.D. and Anderson, A.B.(Eds.),
Handbook of Survey Research. New York: Academic Press.
• Sudman, S. and Bradburn, N.M. (1989) Asking questions: A
practical guide to questionnaire design. San Francisco:
Jossey Bass.