This is a modified version of Master Class that Dr Siobhan O'Dwyer delivered at the Griffith University School of Nursing's Annual Research School for postgraduate students.

1. The Art and Science of
Survey Research

2. What is a survey?
Surveys are a bit like porn – hard to define, but
you know one when you see one!

3. But if you want to be all academic
about it…
A survey is “a systematic method for collecting
information from or about people to describe,
compare or explain – in a quantitative way -
their knowledge, attitudes, experiences, and
behaviours”
-adapted from Fink, 2003 & Groves et al., 2004
This presentation will focus on self-administered, cross-sectional
surveys, analysed in SPSS

4. You should use a survey when you
want to…
Ask about knowledge, attitudes, behaviours or
experiences (“stuff”) on a larger scale
Ask about stuff in a standardized way
Quantify stuff
Focus on self-report
Allow participants to be anonymous

5. Once You’ve Decided on a Survey,
You’ll Need to Consider…
The Science
(what you just have to do)
&
The Art
(what you’ll do if you’re passionate
about good research)

6. The Science

7. Step One
Identify the objectives of your study

8. Here are the objectives from one of my surveys…
Identify the rate of suicidal ideation
in family carers of people with dementia
Identify the factors associated with
suicide risk and resilience in this
group
Explore differences in suicidal ideation
(& risk/resilience factors) between
carers with different demographic
profiles

9. Step Two
Define the key concepts in your objectives

10. Here are the key concepts from my objectives…
SUICIDAL IDEATION: having considered killing
yourself or having attempted suicide
FAMILY CARERS: the primary carer for a family
member with dementia who is at home, in care, or
has recently passed away

11. RISK: factors identified in the literature
e.g. depression, hopelessness,
RESILIENCE: factors identified in the literature
e.g. physical health, optimism, social support
DEMOGRAPHIC PROFILES: age, gender, relationship
to PWD, urban/rural, ethnicity, sexuality

12. Step Three: Select Questions
Demographic questions
Best adopted from large bodies (ABS, AIHW, CDC etc) or
previous studies in your field
This allows you to compare the demographics of your
sample with population level data or previous research
Some people put demographic questions at the end of a survey.
Always put them at the beginning! That way, if people don’t finish
you can at least compare the demographic data of completers vs
non-completers.

13. A Note on Scales
In most surveys, the main questions will
take the form of scales.
A scale is a group of questions (usually
referred to as items) that address the
same topic and are added together to give a
single score that reflects the
construct/concept of interest.
Scales are useful when multiple factors
contribute to an overall concept & when you
are unlikely to get a reliable response from
a single question (e.g. Are you depressed?)

14. A Note on Scales Cont.
In most surveys, you’ll have one scale for
each concept/outcome of interest.
In my research I had one scale each for:
depression, anxiety, suicidal ideation,
physical health, social support etc
Your choice of scales should be informed
by your literature review.
Scale creation is a science in itself, so
unless there is good reason, steer clear of
creating your own.

15. Step Three Cont.
For your key concepts, choose scales that:
Are informed by your literature review
Are reliable
Have been validated
Are appropriate for your population
Have clear scoring instructions
Are publicly available (or within your budget)

16. A Note on Reliability
Test-retest
Consistency over time, or the extent to
which a question or scale will produce the
same result for the same person if
repeated under the same conditions.
Anything over 0.8 is good.
Internal consistency
How well the items of a scale go together
or can be considered to reflect the same
construct/concept. Anything over 0.6 is
acceptable and over 0.8 is good. But over
0.95 may indicate too much overlap or
redundant items.

17. A Note on Validity
Content validity
The extent to which a scale represents all the facets of
a given concept.
Face validity
The extent to which a scale looks like it’s measuring
what you tell participants it’s measuring.
Convergent validity
The extent to which scores on this measure align with
scores on measures of related concepts
Divergent (or discriminant) validity
The extent to which scores on this measure differ from
scores on measures of unrelated or opposing concepts

18. Summarising Steps One to Three
Objective:
Key term:
Measure/Scale 1:
Source/Reference:
Reliability:
Validity:
Scoring:
Previous Use:
Availability:
Measure/Scale 2:
Source/Reference:
Reliability:
Validity:
Scoring:
Previous Use:
Availability:
Create a document
that looks like this…
If you can’t fill in all the
blanks, you either
don’t have a clear idea
of what you want to
measure or the scales
you have selected are
not appropriate.

19. Step Four: Pilot testing
Gather together a small group of people & ask them to
complete the survey as if they were a participant. They
should pay attention to the following…
Are the instructions clear?
Do the questions make sense?
Are the response options sufficient/appropriate?
Are skips easy to follow?
How long did it take to complete?
Do they understand how to complete/save/return the
survey?
You should revise your survey accordingly, but do not change
scales that have already been validated.

20. Step Five: Sampling
A good sample is a miniature version of the
population you are researching.
Inclusion criteria: Who do you want?
Exclusion criteria: Who don’t you want?

21. Here are the exclusion criteria from my survey…
Excluded:
• Paid carers
• Non-primary family carers
• Caring for someone with a condition
other than dementia
• Person with dementia deceased more
than 2 years ago
• Unable to read and write English

22. Sampling Cont.
How many do you need?
This should be informed by the existing literature and an
understanding of the impact of power on your analyses
How many will you have to approach?
This is always more than the number you need
How long is this likely to take?
Take your answer to this question, double it, and add three
months

23. My recent experience…
Wanted: 1500
After 12 months of extensive
recruitment….
Got: 566

24. Step Six: Ethics
How will participants provide informed consent?
Are there any risks to completing your survey?
How will you manage these?
How will you ensure participant anonymity?

25. Step Seven: Administer Your Survey
This is the easy part!

26. Step Eight: Data Entry and Analysis
Set up your data set

27. A thing of beauty is a joy forever
-Keats
An ugly dataset is a pain in the arse
forever
-O’Dwyer

28. Data Entry & Analysis Cont.
Set up your data set
Each item in your survey should have its own
variable.
Each variable needs: Name, Type, Label, Values,
Missing, Measure
Your dataset should be detailed enough that
someone else could run your analyses without any
additional info about the variables.

29. This is the dataset from my survey…
If you create scale scores, you should always recode/transform into new/different
variables. If your data set is too big, you can create a new dataset with just the scale
scores, but NEVER completely delete the item level data!

30. Data Entry & Analysis Cont.
Enter your data
Depending on your sample size, you should also
double check or double enter 20% to check accuracy
of data entry
Prepare your data set
Check for missing data
Check for extreme values
Check for implausible response patterns
Create scale scores (SAVE ALL SYNTAX)
Check internal consistency of scale scores

31. Good data preparation is the academic equivalent of
‘Measure Twice, Cut Once’
Don’t skimp. Be prepared for data prep to take longer than data analysis. And never
delete or modify data without (a) a good reason & (b) keeping a copy.

32. Data Entry & Analysis Cont.
Analysis
Should be guided by your objectives and
research questions
Common techniques in survey research are:
descriptives, frequencies, chi-square, t-tests, and
regressions
More advanced techniques include: factor
analysis and structural equation modeling

33. Step Nine: Reporting
Methods Section
– Brief description of each scale (including number of items, range
of scores and direction)
– Reference original scale article
– Internal consistency as reported previously
– Internal consistency in your sample
– Method of imputation (if applicable)
Analysis Section
– Cronbach’s alpha
– Descriptive analyses
– Univariate analyses
– Multivariate analyses

34. Reporting Cont.
Results Section
– Response rate
– Descriptive findings
– Univariate findings
– Multivariate findings

35. How I reported my survey…
O’Dwyer, Moyle, Zimmer-Gembeck & De Leo.(2013).
Suicidal ideation in family careres of people
with dementia: A pilot study.
International Journal of Geriatric Psychiatry,
11, 1182-1188.

36. A Note on Online Surveys
Advantages
• No postage
• No data
entry
• Increased
anonymity
Disadvantages
• Sample bias
• Missing data
• Duplicates
• Ineligible/imp
lausible
Recommendation: Make your survey
available in both formats (paper/online)
and give people the choice, but consider
any differences in your analysis.

37. If you work through these steps, there’s a
reasonable chance you’ll end up with
a good survey.
But who wants a good survey, when you can
have a great survey?
And that’s where The Art comes in…

38. The Art

39. Recruitment
What would appeal to you?

40. An appealing headline,
but a not so appealing study….

41. Recruitment Cont.
What information is NOT necessary?
The goal is to get people to call or click on the link,
everything else can come later
Make it personal
In my work I include a photo & brief bio of each of the
team members on the information sheet
Be available and return calls promptly
If you don’t respect a participant’s time, why should
they respect yours?

42. Format
Group like scales/questions together
Modify instructions to suit your objective/sample
Leave lots of white space
Make sure font size & colour are appropriate
In scales with more than 4 items, highlight
every second one

43. Format Cont.
If questions break over a page or screen, provide
response options again
If you ask an open question, leave enough space for the
response
Online: Let participants know how much they have left
Online: Check layout on laptops and different browsers

44. Aesthetics / Design
DO NOT photocopy scales from journal articles
and staple them together!!!!
Make your survey pleasing to the eye
If it doesn’t look like you care, why should your participants?
Good design does not have to be expensive
Try using Tables in Word, but making the lines invisible

45. Other Tips
Consider providing reminders at regular
intervals after the survey is administered
Offer telephone support for people who may
otherwise be unable to complete the survey
Practice calculated acts of kindness

46. Calculated Acts of Kindness
A calculated act of kindness is an act that benefits both
parties. A bit like giving a lollipop to a child you are about to
vaccinate.

47. A few of the calculated acts of kindness from my survey…
‘Take a break’ page in middle of survey, with a
teabag & sugar sachet stapled in
Hand-written ‘Thank-You’ cards
Hand-written cover notes
Hand-addressed envelopes
List of resources

48. A Note on Calculated Acts
of Kindness
They are a way to acknowledge that you are the only
person for whom this survey is a priority
They are about recognising that being a decent human
being and being a researcher are not mutually exclusive
They improve response rates and reduce missing data
They support research more generally, as participants
who have a positive experience with you are much more
likely to volunteer again. Consider it good research
karma.
If you don’t have time to practice them, you don’t have
time to do the survey!!!

49. A Checklist for the Perfect Survey
 Identify objectives
 Define key terms
 Select questions
 Prepare survey instrument (Don’t forget The Art)
 Pilot test and refine
 Prepare and submit relevant ethics documents (including
information sheet and consent form, if required)
 Administer survey (Don’t forget Calculated Acts of
Kindness!)
 Preparing data (Don’t skimp here!)
 Analyse data
 Report results