HM404

1. © 2017 American Health Information Management Association
© 2017 American Health Information Management Association
Health Informatics Research Methods:
Principles and Practice, Second Edition
Chapter 2: Survey Research

2. © 2017 American Health Information Management Association
Learning Objectives
• Describe survey research and how it is used in health informatics.
• Display and discuss examples of structured (closed-ended) and
unstructured (open-ended) questions used in health informatics research.
• Demonstrate the appropriate organization of survey questions in relation to
content, flow, design, scales, audience, and appropriate medium.
• Apply appropriate statistics to measure the validity and reliability of the
survey questions.
• Plan and carry out the pilot testing of the questionnaire, whether it is used
as a self-survey or interview instrument.
• Calculate the appropriate sample size for the survey instrument.
• Select appropriate follow-up procedures to retrieve a good response rate.
• Depict what statistics can be generated from collecting data via a survey
instrument.

3. © 2017 American Health Information Management Association
Overview of Survey Research
• Choose a topic of study
• Formulate criteria to develop questions about
that topic
• Can explore a disease, community,
organization, culture, health information
system or software, etc.
• Random sample of subjects chosen to
answer the questions in a standardized
format

4. © 2017 American Health Information Management Association
Overview of Survey Research
• Develop the questionnaire
• Develop the cover letter that explains the
study and directions for completing the
questionnaire
• Pretest the questionnaire for validity and
reliability
• Disseminate the survey via e-mail, mobile
app, mail, Internet, fax, or use the survey
during interviews

5. © 2017 American Health Information Management Association
Overview of Survey Research
• Researcher must choose the best medium
to disseminate survey
• Overall goal of survey research is to
collect the most appropriate and accurate
data that will answer the questions
pertaining to the research topic

6. © 2017 American Health Information Management Association
Real-World Case
• Data in figure 2.1 on the next slide is partially derived
from the CDC’s and NCHS’s National Survey of Family
Growth (NSFG).
• NSFG uses survey research to collect data on family
formation, fertility, and reproductive health.
• Data collected from this survey was used to determine
the pregnancy rates by age across the United States.
• Based on interview questions related to pregnancy
history, including fetal loss.
• The data is collected continuously over time across all
age groups and races (Curtin et al. 2015).

7. © 2017 American Health Information Management Association
CDC’s and NCHS’s National
Survey of Family Growth
(NSFG)

8. © 2017 American Health Information Management Association
Survey Creation
• Use/Adapt Existing Surveys
– Health Information National Trends Survey
(HINTS)
• Created a population-based survey that tracked
trends in the use of communication technologies
as a source of cancer information

9. © 2017 American Health Information Management Association
Survey Creation (cont.)
• Use/adapt existing surveys cont’d
– National Center for Health Statistics
• National Health Interview Survey (NHIS)
• National Ambulatory Medical Care Survey
(NAMCS)
• Health Assessment Questionnaire
• Adapt several different questionnaires into
one

10. © 2017 American Health Information Management Association
New Survey Development
• Need to consider the following items
– Content
– Audience
– Medium
– Sample or survey entire population
– Statistics to be generated

11. © 2017 American Health Information Management Association
Advisory Committee
• Focus group of experts in survey design
and the topic under study
• Assist in phrasing the questions
• May also seek assistance from
organizations/departments which provide
assistance and guidance in survey design,
development, and analysis of results
• Inclusion criteria

12. © 2017 American Health Information Management Association
Types of Questions
• Open-ended (unstructured or qualitative)
• Close-ended (structured or quantitative)
• Scales
– Nominal
– Ordinal
– Interval
– Ratio

13. © 2017 American Health Information Management Association
Pilot Test Survey
• Pilot test survey on a small group of
respondents
• The sample should reflect the true sample of
respondents
• Provide accurate simulation of administration
of survey questionnaire
• Review all comments and discuss with
advisory board
• Incorporate into final survey

14. © 2017 American Health Information Management Association
Test Survey for Validity
• Face Validity
• Criterion-related validity
• Construct validity
• Content validity
Examines how survey looks
Accuracy of intended
survey
Agreement between
theoretical concept and
survey
Survey captures the
information intended to
measure

15. © 2017 American Health Information Management Association
Test Survey for Reliability
• Reliability or consistency of
survey
• Reliability Coefficient:
– Cronbach’s Alpha
• Measures whether survey has
internal consistency—Do all
variables measure the same
concept?
• Should be measured on an
interval or ratio scale
• Normally distributed
• Reliability Coefficients close to
1.00 have very high internal
consistency or reliability
– Test-Retest for Reliability
• Measures whether survey is
consistent over time or when
given multiple times
• Correlation coefficient between
the relationship between two total
scores given two different times
• Those coefficients close to 1.00
show strong reliability.

16. © 2017 American Health Information Management Association
Factor Analysis in Refining
Survey
• Sometimes researchers need to refine the
number of questions used in a survey.
• Factor analysis is a statistical technique in
which a large number of variables are
summarized and reduced down to a
smaller number based on similar
relationships among those variables.

17. © 2017 American Health Information Management Association
Audience
• Need to know your audience so that
questions on the survey can be built so that
they are able to answer them
• Use clear, unambiguous terms
• Do not use terms that are unclear, such as
“not sure”
• Pilot test the survey to capture terms that
should not be included in the original
development of the survey

18. © 2017 American Health Information Management Association
Framing of Questions
• Order of questions is important
– Demographic data is usually first, followed by
more broad or general questions, followed by
more specific questions with ranked or ordinal
type responses or open-ended type questions
• Provide a checklist of possible responses

19. © 2017 American Health Information Management Association
Incentives
• Consider whether incentives should be
provided to individuals who complete the
survey
• Some incentives may influence responses so
be careful when choosing the type of
incentive
• Researcher should decide if it is necessary
and what type of incentive should be used if
any

20. © 2017 American Health Information Management Association
Confidential Reponses
• Discuss how the individuals’ responses will be kept
confidential in the cover letter or instructions
• Institutional Review Board (IRB) will also need to see
how information collected will be kept confidential
• Examples
– Individual non-identifying number and password can be
part of the URL in a Web-based survey
– No identifying information linked with survey
– Separate database used to collect demographic
information but not linked to responses
– All information reported in presentations and published
material should be reported in aggregate form

21. © 2017 American Health Information Management Association
Limitations
• Every study design has limitations
• Survey research design includes the
following limitations
– Inaccurate responses due to
• Not understanding the question or instructions
• Not having appropriate time to complete the survey
• Not able to recall past experiences to answer the
question
• Too long and respondents may tire
• May exaggerate responses to questions—i.e. salary

22. © 2017 American Health Information Management Association
Reduce Limitations By
• Ask questions that
– Are unambiguous
– Do not require extensive recollection
– Jog the respondent’s memory with pictures,
graph, table, etc.
– Provide ranges to choose from especially if
the question is more sensitive in nature

23. © 2017 American Health Information Management Association
Type/Medium
• Web-based/mobile app
• E-mail
• Mail
• Fax
• Group

24. © 2017 American Health Information Management Association
Web-based Surveys
• Advantages
– Reduced cost when compared to paper
– Little or no data entry
– Ease of data analysis
– Use of pop-up instructions and drop-down
boxes
– Ability to present questions in random order
• (Gunn 2002)

25. © 2017 American Health Information Management Association
Web-based Surveys
• Disadvantages
– Missing respondents who do not have a computer or
access to Internet
– Increased up front time in the development of the
questionnaire
– Hire a person with skills in web-based survey design
and development
– Difficult making changes to survey once on the web
– Respondents more reluctant to provide responses
over Internet due to lack of confidence in privacy and
security

26. © 2017 American Health Information Management Association
Other Types of Surveys
• Mail—Still used effectively in health informatics
especially when surveying physicians, nurses and
other health care providers
• E-mail—Very similar to web-based surveys but
time to develop may not be as extensive, however,
automatic response to database is limited
• Fax—Similar to mail surveys and can be used so
that one captures all possible respondents
• Group—Paper survey made available to
participants at a conference, retirement
community, physician practice, etc.

27. © 2017 American Health Information Management Association
Examples of Survey Research
in Health Informatics
• EHR/ASTM study by Watzlaf and colleagues
(2004) used all types of surveys (web-based,
mail, fax, group) with the major focus being
web-based. Intent was to increase response
rate using all types of medium
• Criswell and colleagues (2002) used mail
surveys to determine if physicians in family
practice residency programs used PDAs

28. © 2017 American Health Information Management Association
Examples of Survey Research
in Health Informatics
• Couper and colleagues (2007) used both
mail surveys and phone interviews to
collect data on weight management and
also were interested in which method
produced a better response rate
• Results: Phone interview (59% response
rate) and mail (55% response rate)

29. © 2017 American Health Information Management Association
Distribution of Survey
• Researcher should determine how the survey
will be distributed
– E-mail or phone each of the respondents selected
to participate and explain the purpose of the
study and their role
– Ask if they are willing to participate
– Determine the best way to send the survey and
obtain contact information for them
– Assure respondent that their answers will be kept
confidential and that only aggregate data will be
used when reporting results

30. © 2017 American Health Information Management Association
Distribution of Survey (cont.)
• Discuss the cover letter content.
• Discuss incentives if they are used.
• Once the respondent agrees to participate, contact
them via e-mail, mail, fax, etc. and provide a copy
of the cover letter and survey.
• Make sure that the deadline date to return the
survey is bolded in the cover letter/email.
• After two weeks with no response, send follow-up
letter/e-mail and reiterate the importance of study
and the need for their participation.

31. © 2017 American Health Information Management Association
Sample and Sample Size
• Most survey research is performed on a
sample
• Census survey includes the entire population
• If a sample is chosen, make sure that it
includes an accurate representation of the
population under study. In this way, the
characteristics of the sample participants are
similar to the population characteristics.

32. © 2017 American Health Information Management Association
Sampling Methods
• Stratified Random Sampling—Separate
the population by certain characteristics
such as physician specialty, nursing units,
DRGs, and then choose the sample
• Systematic Random Sampling—Draw the
sample from a list of items such as
diagnoses, ICD-10 codes, or discharges
and select every nth case

33. © 2017 American Health Information Management Association
Sampling Methods (cont.)
• Cluster Sampling—Separate first into a city
block, randomly choose residences and then
sample everyone within that residence
• Convenience Sampling—Not random, can
generate quick results but the results should
not be generalized to the population.
Example: When one surveys everyone in a
specific HIT department to determine their
knowledge of HL7

34. © 2017 American Health Information Management Association
Sample Size Calculation
• Refer to table 2.4 for an example of the
sample size calculation for the EHR study

35. © 2017 American Health Information Management Association
Response Rate
• Very important in survey research
• Even a low response rate may prove
beneficial if it is in an area or topic that has
not been researched in depth before
• Selection bias which is the ability of some
participants to choose to answer the surveyor
to be part of the survey research study for
other reasons. This can bias survey results.

36. © 2017 American Health Information Management Association
Response Rates (cont.)
• Methods used to increase response rates include:
– Follow up letters, e-mails, phone calls, fax
– Do not be annoying
– Include the title of the research study, when the
survey questionnaire was sent, the importance of the
study, and how important the respondent’s reply is to
the research study
– Reiterate how data will be kept confidential
– Explain any incentives
– Attach survey again. Do not make the respondent
look for previous survey sent.

37. © 2017 American Health Information Management Association
Statistical Analysis of Survey
Study Data
• Usually quite simple
– Frequencies and percentages
– Correlation coefficients
– Tests of significance
– Confidence intervals
– Open-ended questions should be analyzed
using content analysis

38. © 2017 American Health Information Management Association
Examples of Statistical Analysis
of Survey Study Data
• See tables 2.5 and 2.6 for a simple data
display for the EHR study.

39. © 2017 American Health Information Management Association
Summary
• Development of survey instrument
• New survey or adapt existing one
• Address the following areas:
– Content
– Audience
– How administered (type/medium)
– Sample or population
– Type of statistics

40. © 2017 American Health Information Management Association
Summary (cont.)
• Choose advisory committee of experts to
review the survey
• Determine the different types of questions
and response types or scales
• Pilot test the survey
• Feedback reviewed with advisory
committee
• Make changes based on the pilot survey

41. © 2017 American Health Information Management Association
Summary (cont.)
• Validity and reliability testing
– Cronbach’s alpha
– Test-retest
– Factor analysis
• Other issues to consider
– Incentives
– How to maintain confidentiality
– Minimize bias or error
– Increase response rate through follow-up
– Appropriate statistical analysis