Survey Methods & Applications in Healthcare

  • 162 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
162
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
12
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Survey Methods & Applications in Healthcare Nawanan Theera-Ampornpunt, M.D., Ph.D. Faculty of Medicine Ramathibodi Hospital Mahidol University www.SlideShare.net/Nawanan
  • 2. 2 2003 M.D. (First-Class Honors) (Ramathibodi) 2009 M.S. in Health Informatics (U of MN) 2011 Ph.D. in Health Informatics (U of MN) 2012 Certified HL7 CDA Specialist • Deputy Executive Director for Informatics (CIO/CMIO) Chakri Naruebodindra Medical Institute • Lecturer, Department of Community Medicine Faculty of Medicine Ramathibodi Hospital Mahidol University nawanan.the@mahidol.ac.th http://groups.google.com/group/ThaiHealthIT www.SlideShare.net/Nawanan Introduction
  • 3. 3 • Overview of Surveys • Survey Methodology • A Sample Survey Outline
  • 4. 4 OVERVIEW OF SURVEYS
  • 5. 5 • An activity in which many people are asked a question or a series of questions in order to gather information about what most people do or think about something Survey Merriam Webster Dictionary
  • 6. 6 • A written set of questions that are given to people in order to collect facts or opinions about something Questionnaire Merriam Webster Dictionary
  • 7. 7 • To know something – Personal information – Knowledge, Opinions & Attitudes – Behaviors & Practice – etc. • About someone – Individuals – Organizations • In order to understand, create knowledge, or make decisions Why Do a Survey?
  • 8. 8 Examples of Surveys Say2KFC.com Service Satisfaction Survey (Online)
  • 9. 9 Examples of Surveys http://www.kmutt.ac.th/building/pdf/application_form_2552_1.pdf Service Satisfaction Survey (Paper-based)
  • 10. 10 Examples of Surveys Image Source: http://pixabay.com/en/agent-business-call-center-18741/ Service Satisfaction Survey (Telephone)
  • 11. 11 SURVEY METHODOLOGY
  • 12. 12 • Survey Design – Study Design – Modes of Data Collection • Instrument Design • Sampling • Survey Conduct • Data Analysis • Reports Survey Methodology
  • 13. 13 WHAT WOULD YOU DO TO INCREASE SURVEY RESPONSES?
  • 14. 14 The Tailored Design Method 1978 1999 2007 2008 Dillman et al.
  • 15. 15 The Tailored Design Method 2014 Dillman et al.
  • 16. 16 • “The development of survey procedures that create respondent trust and perceptions of increased rewards and reduced costs for being a respondent, which take into account features of the survey situation and have as their goal the overall reduction of survey error” The Tailored Design Dillman et al. (2007)
  • 17. 17 • Rewards • Costs • Trust Social Exchange Theory Dillman et al. (2007) Image Source: http://horsebusinessschool.com/using-stick-and-carrot-to-motivate-employees/
  • 18. 18 • “The likelihood of responding to the request to complete a self- administered questionnaire, and doing so accurately, is greater when the respondent trusts that the expected fix rewards of responding will outweigh the anticipated costs” Social Exchange Dillman et al. (2007)
  • 19. 19 • Providing Rewards – Show positive regard – Say thank you – Ask for advice – Support group values – Give tangible rewards – Make the questionnaire interesting – Give social validation – Inform respondents that opportunities to respond are scarce Survey & Social Exchange Theory Dillman et al. (2007)
  • 20. 20 • Reducing Social Costs – Avoid subordinating language – Avoid embarrassment – Avoid inconvenience – Make questionnaires appear short & easy – Minimize requests to obtain personal information – Keep requests similar to other requests to which a person has already responded Survey & Social Exchange Theory Dillman et al. (2007)
  • 21. 21 • Establishing Trust – Provide a token of appreciation in advance – Sponsorship by legitimate authority – Make the task appear important – Invoke other exchange relationships Survey & Social Exchange Theory Dillman et al. (2007)
  • 22. 22 • Study Design – Cross-sectional – Longitudinal Survey Design
  • 23. 23 • Mode of Data Collection – Self-administered survey • Paper-based • Online • Telephone (Automated/IVR) – Interviewer-administered survey (structured interview) • In-person • Telephone – Mixed-mode survey Survey Design
  • 24. 24 • Choose simple over specialized words • Choose as few words as possible to pose the question • Use complete sentences to ask questions • Avoid vague quantifiers when more precise estimates can be obtained • Avoid specificity that exceeds respondent’s potential for having an accurate, ready-made answer • Use equal numbers of positive & negative categories Instrument Design Dillman et al. (2007)
  • 25. 25 Problem • Number of years lived in Idaho Years • Your city or town City or Town • Your county County Use complete sentences Dillman et al. (2007)
  • 26. 26 Revision • How many years have you lived in Idaho? Years • In what city or town do you live? City or Town • In what Idaho county do you live? Idaho County Use complete sentences Dillman et al. (2007)
  • 27. 27 Problem • How often did you attend religious services during the past year?  Never  Rarely  Occasionally  Regularly Avoid vague quantifiers Dillman et al. (2007)
  • 28. 28 Revision • How often did you attend religious services during the past year?  Not at all  A few times  About once a month  Two to three times a month  About once a week  More than once a week Avoid vague quantifiers Dillman et al. (2007)
  • 29. 29 Problem • About how many books have you read for leisure during the past year? Number of books Avoid too much specificity Dillman et al. (2007)
  • 30. 30 Problem • About how many books have you read for leisure during the past year?  less that 10  11-25  26-50  51-75  76 or more Avoid too much specificity Dillman et al. (2007)
  • 31. 31 • Distinguish undecided from neutral by placement at the end of the scale • Avoid bias from unequal comparisons • State both sides of attitude scales in the question stems • Eliminate check-all-that-apply question formats to reduce primacy effects • Develop response categories that are mutually exclusive Instrument Design Dillman et al. (2007)
  • 32. 32 Problem • Which one of the following do you feel is most responsible for recent outbreaks of violence in American’s schools?  Irresponsible parents  School policies  Television programs Avoid bias from unequal comparisons Dillman et al. (2007)
  • 33. 33 Revision • Which one of the following do you feel is most responsible for recent outbreaks of violence in American’s schools?  The way children are raised by parents  School policies  Television programs Avoid bias from unequal comparisons Dillman et al. (2007)
  • 34. 34 Problem • From which one of these sources did you first learn about the tornado in Derby?  Radio  Television  Someone at work  While at home  While traveling to work Mutually Exclusive Dillman et al. (2007)
  • 35. 35 • Use cognitive design techniques to improve recall • Provide appropriate time referents • Be sure each question is technically accurate • Choose question wordings that allow essential comparisons to be made with previously collected data • Avoid asking respondents to say yes in order to mean no • Avoid double-barreled questions Instrument Design Dillman et al. (2007)
  • 36. 36 Problem • Should the city build a new swimming pool that includes lanes for swimming laps that is not enclosed for winter use?  Yes  No Double Barreled Dillman et al. (2007)
  • 37. 37 • Soften the impact of potentially objectionable questions • Avoid asking respondents to make unnecessary calculations Instrument Design Dillman et al. (2007)
  • 38. 38 • Choose simple over specialized words • Choose as few words as possible to pose the question • Use complete sentences to ask questions • Avoid vague quantifiers when more precise estimates can be obtained Instrument Design Dillman et al. (2007)
  • 39. 39 • Survey Design – Study Design – Modes of Data Collection • Instrument Design • Sampling • Survey Conduct • Data Analysis • Reports Survey Methodology
  • 40. 40 • Target Population • Sample • Inclusion & Exclusion Criteria • Sampling Frame • Sampling Method/Technique Sampling Design
  • 41. 41 • Census • Random (probability) sampling – Simple random sampling – Systematic random sampling – Stratified random sampling – Cluster random sampling – Multi-stage random sampling • Non-probability sampling – Purposive sampling – Convenience sampling – Snowball sampling – etc. Sampling Design
  • 42. 42 Example sequence • Pre-notice Letter • 1st Questionnaire Mailout • Reminder + 2nd Questionnaire Mailout Survey Conduct Simplified from Dillman et al. (2007)
  • 43. 43 • Sampling error • Coverage error • Measurement error • Nonresponse error – Questionnaire effects – Data collection mode effects – Interviewer effects – Respondent effects • Processing error Errors in Surveys Dillman et al. (2007) & Office of Management and Budget (2001)
  • 44. 44 • Sampling error • Coverage error • Measurement error • Nonresponse error • Processing error – Data entry error – Pre-edit coding errors – Editing errors – Imputation errors Errors in Surveys Dillman et al. (2007) & Office of Management and Budget (2001)
  • 45. 45 A SAMPLE SURVEY
  • 46. 46 NAWANAN’S DOCTORAL STUDY
  • 47. 47 Unknown State of IT Adoption in Thai Hospitals Need to improve theoretical knowledge This Study Dual Opportunities
  • 48. 48 • To describe current state of IT adoption in Thai hospitals nationwide • To test proposed conceptual framework & explore relationships between organizational characteristics, IT management, and IT adoption Study Objectives
  • 49. 49 Hypothesized Model
  • 50. 50 • Study Design: Nationwide cross-sectional mail survey • Sample: All hospitals in Thailand except pilot (N = 1,302) • Pilot: 5 hospitals (10 respondents each) • Sampling Frame: List of hospitals from Ministry of Public Health’s Web site • Subjects: Hospital’s staff responsible for managing information systems (CIO/IT manager or equivalent; hospital director if N/A) • Data Collection Period: 16 weeks Design & Population
  • 51. 51 • Modified from original instrument • Face & content validity established (Theera-Ampornpunt, 2009) • Further modified based on pilot findings • Translated to Thai Section 1 Hospital Profile Section 2 IT Adoption & Use Profile Section 3 Respondent’s Information English version Survey Instrument
  • 52. 52 • Managerial: To what extent do you agree or disagree with each statement? e.g., • Those who will use the information systems are fully involved in hospital IT development • Functional: How much is each activity supported by computerized information systems in your hospital? • Technological: To what extent is each technology made available in your hospital? • Integration: To what extent is information shared or transmitted among information systems within/outside your hospital? Sample Questions
  • 53. 53   • 150-baht (~US$5) incentive if completed • Endorsed by President of the Thai Medical Informatics Association • Funded by a leading medical school with known informatics focus • Anonymous unless contact information provided for incentive & results mailing Survey Methodology (Nationwide)
  • 54. 54 • 64% response rate • Some items problematic – Differing within-hospital responses on total & IT budgets, No. of IT staff, quality accreditation status – Poor interrater reliability for some dimensions • Quality accreditation status dropped • Item wording revised & survey shortened • Integration sophistication items restructured Pilot Study Findings
  • 55. 55 Hospital Characteristic Site 1 Site 2 Site 3 Site 4 Site 5 Response rate 40% 50% 70% 70% 90% Hospital beds Authoritative source 30 ± 0 30 120.2 ± 0.4 (120-121) 120 360 ± 0 335 303.1 ± 9.4 (282-307) 305 1,058.1 ± 187.1 (863-1,500) 938 Public Private 100% 0% 0% 100% 100% 0% 100% 0% 100% 0% Accreditation status Not accredited & without plan Not accredited, with plan but no significant progress Not accredited, with plan and significant progress Accredited 25% 75% 0% 0% 0% 40% 40% 20% 0% 0% 0% 100% 0% 14% 86% 0% 0% 0% 0% 100% Number of IT staff None 1-5 6-20 21-50 51 or more 0% 75% 25% 0% 0% 0% 80% 20% 0% 0% 0% 100% 0% 0% 0% 0% 43% 57% 0% 0% 0% 0% 22% 11% 67% Pilot Study Findings
  • 56. 56 Hospital Characteristic Site 1 Site 2 Site 3 Site 4 Site 5 2009 total budget (million baht) 22.0 [n=1] 300.0 [n=1] 578.0 [n=1] 368.4 ± 93.7 [n=3] 7,000.0 ± 1,414.2 [n=2] 2009 IT budget (million baht) 0.4 [n=1] 10.0 [n=1] 2.1 ± 1.6 [n=3] 5.5 ± 0.7 [n=2] 93.0 ± 40.0 [n=3] Number of computers in hospital 23.8 ± 4.8 (20-30) 106.7 ± 90.2 (20-200) 170.0 ± 108.9 (10-300) 207.1 ± 82.2 (100-290) 2,350.0 ± 1,332.3 (100-4,000) Calculated percentage of 2009 IT budget according to provided amount 1.8% [n=1] 3.3% [n=1] 0.5% [n=1] 1.4% [n=2] 1.3% [n=1] Subjective estimated percentage of 2009 IT budget (if amount not provided above) Below 1% 1-4% 5-8% Above 8% 0% 75% 25% 0% 0% 40% 40% 20% 20% 60% 20% 0% 0% 86% 14% 0% 0% 17% 50% 33% Pilot Study Findings
  • 57. 57 Construct Overall Site 1 Site 2 Site 3 Site 4 Site 5 Managerial Sophistication 3.6 ± 0.4 3.2 ± 0.2 4.2 ± 0.4 3.9 ± 0.5 3.8 ± 0.4 3.2 ± 0.9 Technological Sophistication 3.5 ± 0.3 3.1 ± 0.2 3.4 ± 0.6 3.7 ± 1.0 3.4 ± 0.5 3.8 ± 0.5 Functional Sophistication 4.0 ± 0.3 3.5 ± 0.5 4.4 ± 0.4 4.1 ± 0.6 4.2 ± 0.5 4.0 ± 0.3 Integration Sophistication (Within Hospital) 3.8 ± 0.3 3.8 ± 0.3 4.2 ± 1.2 3.8 ± 0.6 3.7 ± 0.5 3.4 ± 0.7 Integration Sophistication (Outside Hospital) 2.3 ± 0.9 1.1 ± 0.04 2.5 ± 1.4 3.6 ± 0.8 2.0 ± 0.9 2.1 ± 0.5 Overall IT Sophistication 3.4 ± 0.4 2.9 ± 0.2 3.7 ± 0.7 3.9 ± 0.6 3.4 ± 0.3 3.3 ± 0.4 Pilot Study Findings
  • 58. 58 • IT Sophistication Items Construct Intraclass Correlation Cronbach’s Alpha Managerial Sophistication 0.26* 0.91 Technological Sophistication 0.04 0.81 Functional Sophistication 0.20 0.93 Integration Sophistication (Within Hospital) 0.00 0.89 Integration Sophistication (Outside Hospital) 0.50* 0.97 Overall IT Sophistication 0.30* 0.96 *p < 0.05 on F-test. Pilot Study Findings
  • 59. 59 • 4 of 1,302 hospitals ineligible • Response rate 69.9% Characteristic Overall Responding Hospitals Non- Responding Hospitals N of eligible hospitals 1,298 908 390 Bed size** 106.9 117.5 82.9 Public status** Private Public 24.0% 76.0% 17.4% 82.6% 39.2% 60.8% Geography* Central East North Northeast South West 33.4% 7.5% 11.1% 27.1% 15.3% 5.6% 31.1% 7.8% 13.5% 26.9% 14.9% 5.8% 39.0% 6.7% 5.4% 27.7% 16.2% 5.1% *p < 0.01, **p < 0.001. Nationwide Findings
  • 60. 60 Characteristic Number of Responses Statistic† Public status Private Public 908 158 750 17.4% 82.6% Teaching status Non-teaching Teaching 901 716 185 79.5% 20.5% Total employees 890 368.2 ± 573.5 (10-5269) IT employees 901 4.3 ± 5.3 (0-60) Total budget (million baht) 443 146.67 ± 313.60 (0.25-3,067) IT budget (million baht) 598 2.77 ± 8.79 (0-100) Ratio of IT budget to total budget‡ < 1% 1-4% 5-8% > 8% 416 135 218 40 23 2.7% ± 4.6% (0-43.3%) 32.5% 52.4% 9.6% 5.5% Extent of overall IT utilization Very low Low Moderate High Very high 905 5 35 169 454 242 0.6% 3.9% 18.7% 50.2% 26.7% Total PCs in use 883 126.1 ± 218.6 (0-3,000) Nationwide Findings
  • 61. 61 Estimate (Partial or Complete Adoption) Nationwide Basic EHR, outpatient 86.6% Basic EHR, inpatient 50.4% Basic EHR, both settings 49.8% Comprehensive EHR, outpatient 10.6% Comprehensive EHR, inpatient 5.7% Comprehensive EHR, both settings 5.3% order entry of medications, outpatient 96.5% order entry of medications, inpatient 91.4% order entry of medications, both settings 90.2% order entry of all orders, outpatient 88.6% order entry of all orders, inpatient 81.7% order entry of all orders, both settings 79.4% Adoption Estimates
  • 62. 62 Final Model
  • 63. 63 • High IT adoption rates • Drastic changes in adoption landscape • Local context might play a role – Supply Side – Demand Side • International Comparison – Relatively higher adoption Discussion
  • 64. 64 • Overview of Surveys • Survey Methodology • A Sample Survey Recap
  • 65. 65 • Survey Design – Study Design – Modes of Data Collection • Instrument Design • Sampling • Survey Conduct • Data Analysis • Reports Recap