Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

WGHA Discovery Series: Ali Mokdad


Published on

Washington Global Health Alliance Discovery Series

Ali Mokdad
June 2, 2008
'Surveillance in a Changing Environment'

Published in: Education, Health & Medicine
  • Be the first to comment

  • Be the first to like this

WGHA Discovery Series: Ali Mokdad

  1. 1. Surveillance in a Changing Environment Ali H. Mokdad, Ph.D. Centers for Disease Control and Prevention Atlanta, GA, USA
  2. 2. Outline <ul><li>Background </li></ul><ul><li>Challenges </li></ul><ul><li>Behavioral Risk Factor Surveillance System (BRFSS) </li></ul><ul><li>Improvements </li></ul><ul><li>Conclusion </li></ul><ul><li>Future </li></ul>
  3. 3. Public Health Surveillance <ul><li>Systematic, ongoing </li></ul><ul><li>Collection </li></ul><ul><li>Analysis </li></ul><ul><li>Interpretation </li></ul><ul><li>Dissemination </li></ul><ul><li>Link to public health practice </li></ul>
  4. 4. Purposes of Public Health Surveillance <ul><li>Assess public health status </li></ul><ul><li>Define public health priorities </li></ul><ul><li>Evaluate programs </li></ul><ul><li>Stimulate research </li></ul>
  5. 5. What problems are we trying to address?
  6. 6. Coverage problems <ul><li>Face-to-face coverage: </li></ul><ul><ul><li>Available household lists not complete </li></ul></ul><ul><ul><li>Need to manually count and list </li></ul></ul><ul><li>Telephone coverage: </li></ul><ul><ul><li>Households with no telephones (2-3%) </li></ul></ul><ul><ul><li>Cell phone only households (13-15%) </li></ul></ul><ul><ul><li>No directory of cell phone numbers </li></ul></ul><ul><ul><li>Number portability and erosion of geographic specificity </li></ul></ul><ul><li>Mail coverage: </li></ul><ul><ul><li>USPS list only readily available source for general populations </li></ul></ul><ul><ul><ul><li>Poor coverage in rural areas </li></ul></ul></ul><ul><li>Email coverage: </li></ul><ul><ul><li>No systematic directory of addresses </li></ul></ul>
  7. 7. Declining response rates <ul><li>Response rates decreasing significantly in the last 10 – 15 years. </li></ul><ul><li>Decline has occurred for most types of surveys— particularly telephone and in-person interviews </li></ul><ul><li>Evidence of trends for mail surveys not as clear due to lack of long-term trend studies. </li></ul><ul><li>Web surveys are too new to provide good trend data. </li></ul><ul><li>Increase in nonresponse is a global problem </li></ul><ul><li>No single or clear explanation for these trends. </li></ul>
  8. 8. Behavioral Risk Factor Surveillance System (BRFSS) <ul><li>Monthly state-based RDD survey of health issues </li></ul><ul><li>50 states, District of Columbia, Puerto Rico, Guam, and Virgin Islands </li></ul><ul><li>430,000+ adult interviews conducted in 2007 </li></ul><ul><li>From 2002 to 2007: </li></ul><ul><ul><li>completed 1,950,000 interviews </li></ul></ul><ul><ul><li>Dialed 18,500,000 telephone numbers </li></ul></ul>
  9. 9. BRFSS Strengths <ul><li>Flexible </li></ul><ul><li>Timely </li></ul><ul><li>Standardized </li></ul><ul><li>Useful </li></ul>
  10. 10. Establish and Track Health Objectives
  11. 11. Prevalence of Obesity* Among U.S. Adults Prevalence of Diabetes* Among U.S. Adults (*Includes gestational diabetes) 1990 1990 1996 1996 2004 2004 No Data < 4% 4%-6% 6-8% 8-10% No Data < 10% 10%–14% 15%–19% 20%-24%  25% >10% (*BMI  30, or about 30 lbs overweight for 5’4” person)
  12. 12. Prevalence of Women Who Never Had a Mammogram, Ages 40 and Older BRFSS 1990–2004
  13. 13. Support Policies and Legislation: Mandatory Insurance Coverage for Screening Mammography 1981 1990 2004 No mandatory insurance coverage for screening mammography. Mandatory insurance coverage for screening mammography. Source: National Cancer Institute — State Cancer Legislative Database Program, Bethesda, MD, 2004.
  14. 14. Support Policies and Legislation: Prevalence of Safety Belt Use, 2002 Areas with primary safety belt laws Prevalence > 80% of always using a safety belt among persons aged > 18 years. Prevalence < 80% of always using a safety belt among persons aged > 18 years. Source: CDC. Impact of primary laws on adult use of safety belts – United States, 2002. MMWR 2004;53:257-260.
  15. 15. Develop Local Programs and Policies: SMART BRFSS in Fargo <ul><li>Fargo, ND – 24.9% binge drinking </li></ul><ul><li>vs. 16.4% nationwide </li></ul><ul><li>Formed community coalition: </li></ul><ul><li>AMP (Alcohol Misuse Prevention) </li></ul><ul><li>Mission: Reduce alcohol use among those under 21 in the Fargo-Moorhead area. </li></ul><ul><ul><li>Anti-binge drinking campaign </li></ul></ul><ul><ul><li>Policy change sanctioning facilities </li></ul></ul><ul><ul><li>Intervention with ER doctors </li></ul></ul>
  16. 16. Healthy People in Healthy Places
  17. 17. People Prepared for Emerging Health Threats
  18. 18. Vaccine Shortage – Timeline <ul><li>Oct 5: Vaccine shortage announced </li></ul><ul><li>Oct 5: Initial discussions within CDC </li></ul><ul><li>Oct 19: Call with BRFSS state coordinators </li></ul><ul><li>Oct 19-26: New questions developed and cognitively tested </li></ul><ul><li>Oct 27: CATI specifications to states </li></ul><ul><li>Nov 1: Data collection began </li></ul>
  19. 19. The Effort Involved … <ul><li>4 months of data collection (Nov ’04-Feb ’05) </li></ul><ul><li>35 data collection centers </li></ul><ul><li>50 states + DC </li></ul><ul><li>92 grant/contract modifications </li></ul><ul><li>400 interviewers trained </li></ul><ul><li>2,000 total staff mobilized </li></ul><ul><li>35,106 child interviews (via proxy) </li></ul><ul><li>105,743 adult interviews </li></ul>
  20. 20. From Implementation to First Report <ul><li>November 1 – Questions implemented by states </li></ul><ul><li>November 8 – Data first submitted by states </li></ul><ul><li>November 10 – First data report available </li></ul><ul><li>November 15 – NIP/BRFSS analysis team develop first executive summary and table </li></ul>
  21. 21. December MMWR <ul><li>Dec 1-11: States collected December data </li></ul><ul><li>Dec 13: Submitted files to CDC </li></ul><ul><li>Dec 16: Dr. Gerberding holds press conference & MMWR released on the CDC website </li></ul>
  22. 27. WEAT Startup Screen
  23. 28. WEAT Crosstabs Application Select Age by General Health
  24. 29. WEAT Crosstabs Application Age by General Health Control Race
  25. 30. WEAT Logistic Application Select Independent Variable
  26. 31. WEAT Logistic Application Output for ‘Overweight’ Dependent, ‘Exercise’ Independent Reference = ‘yes’
  27. 32. Link BRFSS to Other Data Sources <ul><li>EPA </li></ul><ul><li>County data </li></ul><ul><li>Follow up (asthma in 35 states) </li></ul>
  28. 33. Response rate trends: Behavioral Risk Factor Surveillance System (BRFSS)
  29. 34. Percentage of U.S. Households Without Landline Telephones <ul><li>Based on National Health Interview Survey data </li></ul>
  30. 36. Percent Distribution of Household Telephone Status for Adults, July-December 2007 Wireless Only: 14.5% Landline with Some Wireless: 49.2% Landline Only: 19.1% Unknown: 1.3% Phoneless: 1.9% Wireless Mostly: 14.0%
  31. 37. Coverage <ul><li>Sampling frame must include all units of the population of interest </li></ul><ul><ul><li>Frame coverage errors </li></ul></ul><ul><li>Coverage by key modes: </li></ul><ul><ul><li>Face-to-face: expensive to develop (lists help) </li></ul></ul><ul><ul><li>Telephone: cell phones and number portability </li></ul></ul><ul><ul><li>Mail: USPS list not complete, but improving </li></ul></ul><ul><ul><li>Web: no comprehensive list </li></ul></ul>
  32. 38. Sampling <ul><li>Each element has a known and non-zero probability of selection from sampling frame </li></ul><ul><ul><li>Protection against selection bias </li></ul></ul><ul><ul><li>Quantify sampling error </li></ul></ul><ul><li>Sampling by key modes: </li></ul><ul><ul><li>Face-to-face and telephone: well developed techniques </li></ul></ul><ul><ul><li>Mail: within household selection techniques quasi-random for general pop surveys </li></ul></ul><ul><ul><li>Email: tend to be nonprobability; opt-in samples </li></ul></ul>
  33. 39. Nonresponse <ul><li>Inability to obtain data from selected respondent: </li></ul><ul><ul><li>Unit nonresponse </li></ul></ul><ul><ul><li>Item nonresponse </li></ul></ul><ul><li>Nonresponse by key modes: </li></ul><ul><ul><li>Highest in face-to-face </li></ul></ul><ul><ul><li>Lowest in internet </li></ul></ul><ul><li>Item nonresponse varies by mode and question </li></ul>
  34. 40. Measurement <ul><li>Measurement error occurs when a respondent’s answer to a question is inaccurate (departs from the “true” value) </li></ul><ul><li>Modes vary in terms of: </li></ul><ul><ul><li>Interviewer versus self-administered </li></ul></ul><ul><ul><li>Stimuli / manner in which survey question is conveyed to respondent (and response is recorded) </li></ul></ul>
  35. 41. Interviewer administered questions <ul><li>Help to: </li></ul><ul><ul><li>Motivate respondents </li></ul></ul><ul><ul><li>Guide through complex questionnaires </li></ul></ul><ul><ul><li>Clarify questions and instructions </li></ul></ul><ul><ul><li>Probe for detailed answers </li></ul></ul><ul><li>Potential problems: </li></ul><ul><ul><li>Less privacy, less anonymous </li></ul></ul><ul><ul><li>Social desirability </li></ul></ul><ul><ul><li>More positive </li></ul></ul><ul><ul><li>Acquiescence </li></ul></ul>
  36. 42. Self-administered questions <ul><li>Help to: </li></ul><ul><ul><li>Ensure privacy </li></ul></ul><ul><ul><li>Self-paced </li></ul></ul><ul><ul><li>Conduct survey at convenience of respondent </li></ul></ul><ul><li>Potential problems: </li></ul><ul><ul><li>Ensuring correct respondent completes survey </li></ul></ul><ul><ul><li>Little/no option for requesting assistance </li></ul></ul><ul><ul><li>Stray/out-of-range responses </li></ul></ul><ul><ul><li>No means of assess cognitive engagement of respondent </li></ul></ul>
  37. 43. Goal: Optimize survey design to decrease total survey error for a given cost Costs Measurement Nonresponse Coverage Sampling
  38. 44. Costs for single modes <ul><li>Face-to-face (most expensive): </li></ul><ul><ul><li>5-10 times higher than telephone </li></ul></ul><ul><li>Telephone: </li></ul><ul><ul><li>2-3 times more expensive than mail </li></ul></ul><ul><li>Mail: </li></ul><ul><ul><li>higher than Web due to fixed costs plus per interview processing </li></ul></ul><ul><li>Web (least expensive): </li></ul><ul><ul><li>Primarily fixed set-up costs, little per interview costs </li></ul></ul>
  39. 45. Assessing Data Validity
  40. 46. What do we mean by “validity”? <ul><li>The closeness of our survey estimates to the “true value” </li></ul><ul><ul><li>Ideally there is no difference </li></ul></ul><ul><ul><li>Potential survey bias is minimized </li></ul></ul><ul><li>“ Bias” in survey estimates results from product of: </li></ul><ul><ul><li>Level of nonresponse </li></ul></ul><ul><ul><li>Difference between respondents and nonrespondents on measures of interest </li></ul></ul>
  41. 47. Ensuring validity of BRFSS Estimates <ul><li>Monitoring data collection process </li></ul><ul><li>Refining post-survey adjustments </li></ul><ul><li>Benchmarking to other studies </li></ul><ul><li>Testing alternative ways of collecting data </li></ul><ul><ul><li>Cell phone interviewing </li></ul></ul><ul><ul><li>Address-based sampling (ABS) </li></ul></ul>
  42. 48. Monitoring the Data Collection Process
  43. 49. Monitoring 54 monthly surveys <ul><li>BRFSS data collection process is semi-centralized </li></ul><ul><li>States: </li></ul><ul><ul><li>In charge of own data collection </li></ul></ul><ul><ul><li>Conduct front-line monitoring </li></ul></ul><ul><li>Centers for Disease Control (CDC): </li></ul><ul><ul><li>Provides sample </li></ul></ul><ul><ul><li>Weighting </li></ul></ul><ul><ul><li>Quality reports </li></ul></ul>
  44. 50. Web-based systems are key <ul><li>Data transfer via upload/download site </li></ul><ul><li>Automated quality control programs </li></ul><ul><ul><li>State level and CDC level </li></ul></ul><ul><li>Monthly detailed reports to states: </li></ul><ul><ul><li>Key quality indicators </li></ul></ul><ul><ul><li>Deviations from norm and/or past trends within state </li></ul></ul><ul><li>Year-end quality report </li></ul><ul><ul><li>Comparison across states </li></ul></ul><ul><li>Newest tool: Simplified web-based / color coded system </li></ul>
  45. 53. What did we learn? <ul><li>Estimates are only as valid as the process in which the data were collected </li></ul><ul><li>Tools for monitoring the quality of data collection and collecting valid data are only good: </li></ul><ul><ul><li>… if they are actually used </li></ul></ul><ul><ul><li>… and if they are understood </li></ul></ul><ul><ul><li>… and initiate follow-up action </li></ul></ul>
  46. 54. Refining post-survey adjustments
  47. 55. Goals and limits of weighting <ul><li>Weighting and other post-survey adjustments are used to correct for imbalances in the data due to issues of: </li></ul><ul><ul><li>Coverage </li></ul></ul><ul><ul><li>Sampling </li></ul></ul><ul><ul><li>Nonresponse </li></ul></ul><ul><li>Weighting methodology affects the estimates produced </li></ul><ul><li>Can only weight data you have </li></ul><ul><ul><li>Assumes no difference between respondents and nonrespondents on variables of interest </li></ul></ul><ul><li>Can only weight to external standards that exist </li></ul><ul><ul><li>Typically limits weighting to a handful of demographic variables, not “substantive” variables </li></ul></ul>
  48. 56. Current BRFSS Weighting System <ul><li>Use poststratification (cell-based approach) </li></ul><ul><li>Controls for: </li></ul><ul><ul><li>Age by sex </li></ul></ul><ul><ul><li>Race/Ethnicity (in some states) </li></ul></ul><ul><ul><li>Region (in some states) </li></ul></ul><ul><li>Problems: </li></ul><ul><ul><li>Small sample cells produce highly variable weights and require collapsing </li></ul></ul><ul><ul><li>No factor to account for socioeconomic status </li></ul></ul>
  49. 57. Conducted detailed analysis to identify key demographic correlates of BRFSS Measures <ul><li>Age </li></ul><ul><li>Race/ethnicity </li></ul><ul><li>Gender </li></ul><ul><li>Education </li></ul><ul><li>Marital status </li></ul>
  50. 58. New weighting system <ul><li>Uses “Multi-Dimensional Raking” (Sample Balancing) </li></ul><ul><li>Controls for: </li></ul><ul><ul><li>Age by sex </li></ul></ul><ul><ul><li>Race/ethnicity (2.5% rule) </li></ul></ul><ul><ul><li>Region (as necessary) </li></ul></ul><ul><ul><li>Education level </li></ul></ul><ul><ul><li>Marital status </li></ul></ul><ul><ul><li>Telephone service interruption </li></ul></ul>
  51. 59. Does It Represent an Improvement?
  52. 60. Changes in estimate of health status Percentage with Fair or Poor Health Status 19.0 20.0 21.0 22.0 23.0 24.0 25.0 BRFSS Final Weight Add Race Add Education Add Marital Status Add Age by Educ Add Age by Race % At Risk
  53. 61. Implications for users of BRFSS data <ul><li>Break in time series </li></ul><ul><li>Plan to release both classic (old) and new weights </li></ul><ul><li>Full changeover in 2010 </li></ul><ul><li>Health condition and risk factor estimates will likely be higher </li></ul>
  54. 62. What did we learn? <ul><li>Modifications to post-survey adjustments can improve the quality of the estimates produced </li></ul><ul><li>Sometimes need to be innovative in the use of external data in developing population estimates </li></ul>
  55. 63. Benchmarking to external standards
  56. 64. Importance and challenges of benchmarking <ul><li>True standards rarely exist in health surveys – relative standards </li></ul><ul><ul><li>Better coverage, response </li></ul></ul><ul><li>No two studies are identical </li></ul><ul><ul><li>Populations </li></ul></ul><ul><ul><li>Modes / procedures </li></ul></ul><ul><ul><li>Wording / question order </li></ul></ul><ul><ul><li>Post-survey adjustments / population standards </li></ul></ul>
  57. 65. Benchmark surveys for BRFSS <ul><li>National Health Interview Survey (NHIS): </li></ul><ul><ul><li>In-person interviews with adults 17+ </li></ul></ul><ul><ul><li>2004: 94,460 adults in 36,579 households </li></ul></ul><ul><ul><li>Household-level response rate = 86.9% </li></ul></ul><ul><li>National Health and Nutrition Examination Survey (NHANES): </li></ul><ul><ul><li>In-person survey with physical measures at mobile lab </li></ul></ul><ul><ul><li>2003-04: 10,122 adults </li></ul></ul><ul><ul><li>Household-level response rate = 91.0% </li></ul></ul>
  58. 66. Comparison across 15 key health variables <ul><li>Cigarette smoking </li></ul><ul><li>Diabetes </li></ul><ul><li>Height </li></ul><ul><li>Weight </li></ul><ul><li>Body mass index </li></ul><ul><li>Health status </li></ul><ul><li>Asthma </li></ul><ul><li>HIV testing </li></ul><ul><li>Alcohol consumption </li></ul><ul><li>Medical coverage </li></ul><ul><li>Influenza vaccination </li></ul><ul><li>Pneumonia shot </li></ul>
  59. 67. Summary of findings <ul><li>BRFSS vs NHIS estimates: </li></ul><ul><ul><li>Significantly different on 10 of 15 variables </li></ul></ul><ul><ul><li>Relative difference: </li></ul></ul><ul><ul><ul><li>Asthma = +35% </li></ul></ul></ul><ul><ul><ul><li>HIV testing = +26% </li></ul></ul></ul><ul><li>BRFSS vs. NHANES estimates: </li></ul><ul><ul><li>Significantly different on 5 of 6 variables </li></ul></ul><ul><ul><li>Relative difference: </li></ul></ul><ul><ul><ul><li>Current smoking = -12.2% </li></ul></ul></ul><ul><ul><ul><li>Body mass index = -2.1% </li></ul></ul></ul>
  60. 68. Ever smoke cigarettes 42.4% 44.0% 48.0%
  61. 69. Ever told had diabetes 8.1% 8.0% 6.1%
  62. 70. Body Mass Index 27.0 27.0 27.6
  63. 71. Percentage of 18–34 year olds 31.2% 31.5% 41.5%
  64. 72. Percentage of Males 48.4% 48.4% 50.9%
  65. 73. Percentage of whites 71.7% 70.4% 74.9%
  66. 74. What did we learn? <ul><li>There are no “gold standards” in health statistics </li></ul><ul><li>All comparisons are relative </li></ul><ul><ul><li>Surveys can vary in terms of backend processing just as much as on front-end design and operational issues </li></ul></ul><ul><li>Determining if BRFSS compares favorably with other surveys is a matter of perspective </li></ul>
  67. 75. Finding new ways of collecting data: Cell phones & Address-based sampling
  68. 76. The Plague of Cell Phones!!!
  69. 77. Cell phones and telephone surveys <ul><li>Reliance on cell phones increasing </li></ul><ul><li>Conducting surveys via cell phones can be operationally challenging: </li></ul><ul><ul><li>Cell phone frame very inefficient </li></ul></ul><ul><ul><li>Cannot use autodialers </li></ul></ul><ul><ul><li>Charges for incoming calls/minutes used </li></ul></ul><ul><ul><li>Safety concerns </li></ul></ul><ul><ul><li>Potential mode effects / measurement errors </li></ul></ul>
  70. 78. 2007 BRFSS cell phone pilot <ul><li>Conducted in 3 U.S. states </li></ul><ul><li>Target: 600 cell & landline / 600 cell-only </li></ul><ul><li>Abbreviated BRFSS core interview: </li></ul><ul><ul><li>66 questions </li></ul></ul><ul><ul><li>15-17 minutes (on average) </li></ul></ul>
  71. 79. Response rates 14.8% 29.7% 14.1%
  72. 80. Landline and Cell phone populations and frames CELL PHONE LANDLINE A B C
  73. 81. Percent male 46.6 51.1 38.2 37.9 Landline survey Cell phone survey State equalized design weight applied
  74. 82. Percent 18–34 years 24.0 51.4 19.6 14.5 Landline survey Cell phone survey State equalized design weight applied
  75. 83. Percent Hispanic 15.2 21.4 12.2 16.8 Landline survey Cell phone survey State equalized design weight applied
  76. 84. Percent high school or less education 39.8 48.5 33.6 60.3 Landline survey Cell phone survey State equalized design weight applied
  77. 85. Comparison of key survey estimates
  78. 86. Percent any kind of health care coverage 86.0 70.1* 89.0 78.7 Landline survey Cell phone survey State equalized design weight applied
  79. 87. Percent currently smoke cigarettes 19.7 31.1* 17.3 24.8 Landline survey Cell phone survey State equalized design weight applied
  80. 88. Percent ever tested for HIV 43.6 54.2* 36.6 37.5 Landline survey Cell phone survey State equalized design weight applied
  81. 89. Percent binge drink past 30 days 13.0 23.5 21.1 11.0 Landline survey Cell phone survey State equalized design weight applied
  82. 90. What did we learn? <ul><li>The part of the population we are missing due to cell phones is different from those we interview --- and we cannot ignore them </li></ul><ul><li>Missing critical information needed to integrate landline and cell phone samples at the sub-national level </li></ul><ul><ul><li>No reliable external standards denoting telephone usage at subnational level </li></ul></ul>
  83. 91. Address-based sampling
  84. 92. Key questions to address <ul><li>How do ABS mixed-mode surveys and RDD telephone surveys compare in terms of: </li></ul><ul><ul><li>response rates </li></ul></ul><ul><ul><li>respondent demographics </li></ul></ul><ul><ul><li>estimates on key health issues </li></ul></ul><ul><ul><li>costs of implementation </li></ul></ul><ul><li>Can ABS mail surveys reach households without telephones and cell phone-only households? </li></ul>
  85. 93. Response rates by mode: 2003 BRFSS mode pilot (address-matched sample)
  86. 94. Why Not Complete Mail Survey? <ul><li>Letter not received: </li></ul><ul><ul><li>Men </li></ul></ul><ul><ul><li>Blacks/Hispanics </li></ul></ul><ul><ul><li>18-34 </li></ul></ul><ul><li>Other reasons: </li></ul><ul><ul><li>Lost questionnaire </li></ul></ul><ul><ul><li>Too much junk mail </li></ul></ul><ul><ul><li>Just don’t like surveys! </li></ul></ul>No letter No time Other 58% 19% 23%
  87. 95. Why Not Complete Web Survey? <ul><li>No web access: </li></ul><ul><ul><li>65 and older </li></ul></ul><ul><ul><li>Lower SES </li></ul></ul><ul><ul><li>No children </li></ul></ul><ul><li>Letter not received: </li></ul><ul><ul><li>Men </li></ul></ul><ul><ul><li>18-34 </li></ul></ul><ul><ul><li>Upper SES </li></ul></ul><ul><ul><li>Blacks/Hispanics </li></ul></ul><ul><ul><li>HHs with children </li></ul></ul>No letter No time Other No web 33% 38% 12% 17%
  88. 96. Item Nonresponse: Telephone vs Mail (Percent DK / RF / Blank) Note: Percentages are unweighted. Significance: * p<.05, ** p<.01, *** p < .001 1 Questions not asked of respondents age 65 years or older Health condition / risk factor Telephone (%) Mail (%) Asthma 0.2 2.4 *** Diabetes 0.1 0.9 *** High blood pressure 0.2 1.7 *** Obese (BMI > 30) 8.2 3.0*** Current smoker 0.4 1.9 *** Binge drinking 2.3 2.1 Tested for HIV 1 5.6 2.8*** HIV risk behaviors 1 3.7 3.4
  89. 97. Potential mode affects on response: Unadjusted estimates ‡ Questions not asked of respondents age 65 years or older Health condition / risk factor Unadjusted prevalence estimates CATI % (95% CI) Mail Survey % (95% CI) Web survey % (95%CI) Asthma 11.7 (10.3-13.1) 12.0 (9.8-14.2) 11.9 (10.0-13.8) Diabetes 9.5 (8.2-10.8) 11.9 (9.7-14.1) 10.2 (8.4-12.0) High blood pressure 31.1 (29.1-33.1) 38.1 (34.8-41.4) 33.2 (30.5-35.9) Obese (BMI > 30) 21.6 (19.8-23.4) 26.5 (23.5-29.5) 25.6 (23.0-28.2) Current smoker 22.8 (21.0-24.6) 16.9 (14.4-19.4) 17.3 (15.1-19.5) Binge drinking 14.4 (12.9-15.9) 12.3 (10.1-14.5) 21.6 (9.0-24.2) STD prevention 1 8.2 (6.8 - 9.6) 4.3 (2.6 - 6.0) 3.3 (2.2 - 4.4) Tested for HIV 1 38.8 (36.3-41.3) 30.8 (27.0-34.6) 32.1 (29.1-35.1)
  90. 98. Potential mode affects on response: Adjusted estimates * Models are adjusted for respondents’ state of residence, sex, race, age, education, and number of adults in the household. 1 Questions not asked of respondents age 65 years or older Health condition / risk factor Adjusted odds ratios * CATI Mail survey AOR (95%CI) Web survey AOR (95%CI) Asthma 1.0 1.07 (0.84-1.34) 1.06 (0.83-1.38) Diabetes 1.0 1.16 (0.89-1.51) 1.30 (1.01-1.67) High blood pressure 1.0 1.22 (1.01-1.46) 1.30 (1.09-1.54) Obese (BMI > 30) 1.0 1.37 (1.12-1.66) 1.31 (1.10-1.57) Current smoker 1.0 0.83 (0.67-1.03) 0.77 (0.63-0.93) Binge drinking 1.0 1.17 (0.90-1.52) 1.87 (1.50-2.34) STD prevention 1 1.0 0.69 (0.43-1.12) 0.51 (0.33-0.78) Tested for HIV 1 1.0 0.81 (0.65-1.01) 0.85 (0.71-1.03)
  91. 99. BRFSS 2006 Address-based sample (ABS) mixed-mode pilot survey <ul><li>Six states: CA, FL, MA, MN, TX, & SC </li></ul><ul><li>Address-based sampling frame: USPS Delivery Sequence File (sample provided by Marketing Systems Group) </li></ul><ul><li>Mixed mode data collection: </li></ul><ul><ul><li>Initial mail survey </li></ul></ul><ul><ul><li>Postcard reminder </li></ul></ul><ul><ul><li>Second mail survey (to nonrespondent) </li></ul></ul><ul><ul><li>Telephone survey follow-up (of nonrespondents) </li></ul></ul><ul><li>75 questions from BRFSS core questionnaire </li></ul><ul><li>Field period: June 20-Oct 4, 2006 </li></ul><ul><li>Compared to monthly RDD surveys from same time frame </li></ul>
  92. 100. Sample design <ul><li>Probability sample from DSF household frames in each state </li></ul><ul><li>Excluded business addresses identified by USPS or Marketing Systems Group </li></ul><ul><li>Included seasonal units, vacant units, PO Boxes, throwback units, and drop point units </li></ul><ul><li>Stratified sample by county and address type </li></ul><ul><li>Drew 1,870 addresses per state using systematic random sampling (goal: 800 completes per state) </li></ul>
  93. 101. Within household randomization <ul><li>Mail survey (3 approaches tested): </li></ul><ul><ul><li>Most recent birthday (25%) </li></ul></ul><ul><ul><ul><li>One questionnaire </li></ul></ul></ul><ul><ul><li>Next birthday (25%) </li></ul></ul><ul><ul><ul><li>One questionnaire </li></ul></ul></ul><ul><ul><li>All adults in household (50%) </li></ul></ul><ul><ul><ul><li>Three questionnaires </li></ul></ul></ul><ul><ul><ul><li>Toll free number for additional questionnaires </li></ul></ul></ul><ul><li>Telephone follow-up survey: </li></ul><ul><ul><li>Used BRFSS protocol – number of males / number of females </li></ul></ul>
  94. 102. Response rates
  95. 103. RDD telephone and ABS multimode survey AAPOR #4 response rates State RDD telephone survey ABS multimode Survey Mail survey (only) Telephone follow-up (only) CA 25.4 36.1*** 33.1 3.0 FL 32.8 37.4*** 33.4 4.0 MA 26.3 42.4*** 36.5 5.9 MN 48.5 54.1*** 48.5 5.6 SC 49.1 41.8*** 37.3 4.5 TX 28.7 35.3*** 30.8 4.5 Mean 35.1 41.2 36.6 4.6
  96. 104. Comparison of respondent demographics
  97. 105. Percent male 48.6 39.3 40.9 41.3 38.5 RDD and ABS data weighted by state equalized design weight only.
  98. 106. Percent 18-34 years old 31.1 19.0 17.6 16.8 22.7 RDD and ABS data weighted by state equalized design weight only.
  99. 107. Percent 65 year old and older 16.9 22.7 23.9 24.8 18.2 RDD and ABS data weighted by state equalized design weight only.
  100. 108. Percent some college or more 55.7 55.7 69.2 55.0 67.2 RDD and ABS data weighted by state equalized design weight only. 60.8
  101. 109. Percent Hispanic 16.2 14.9 8.9 7.9 14.9 RDD and ABS data weighted by state equalized design weight only.
  102. 110. Percent one adult households 18.6 16.8 17.0 15.7 RDD and ABS data weighted by state equalized design weight only. 21.6
  103. 111. Percent live in non-metropolitan area 15.0 14.9 15.6 13.8 16.1 RDD and ABS data weighted by state equalized design weight only.
  104. 112. Type of household telephone access 1 SJ.Blumberg and JV Luke (2007). “Wireless substitution: Early release of estimates based on data from the National Health Interview Survey, July – December 2006.” National Center for Health Statistics E-States. Household Telephone access National Health Interview Survey 1 (%) BRFSS ABS mixed-mode survey (%) Land line 85.0 88.4 -- Landline only --- 14.5 -- Landline and cellular phone --- 73.9 Cellular phone only 12.8 10.5 No telephone 2.2 1.1
  105. 113. Comparison of Survey Estimates
  106. 114. Comparison of Survey Estimates Significance: * p<.05, ** p<.01, *** p < .001 Note: Data weighted for sample design and post-stratified to sex, age, and race totals for each state. Final weights were ratio adjusted to equalize the number of cases across states. Health condition / risk factor RDD telephone survey ABS multimode survey Total Mail survey (only) Telephone survey (only) Health care coverage 81.9 81.3 82.4 75.5** Asthma 12.4 13.4 13.6 12.6 Diabetes 9.3 10.8 10.3 13.8 Cardiovascular disease 8.3 8.7 8.9 8.0 Obese (BMI > 30) 22.9 26.7*** 26.7 26.6 Current smoker 20.1 19.9 18.1 29.5*** Binge drinking 15.1 18.1*** 18.9 13.7* Tested for HIV 36.7 36.1 35.3 40.2 [n] [21,743] [4,871] [4,327] [544]
  107. 115. 18-34 year olds: Binge drinking & HIV test by household phone access Note: Data weighted for sample design and post-stratified to sex, age, and race totals for each state. Final weights were ratio adjusted to equalize the number of cases across states.
  108. 116. Key findings from ABS mixed-mode design <ul><li>In low response rate states the address-based mixed-mode survey approach can yield response rates superior to RDD rates </li></ul><ul><li>Approach reaches households without land-line telephones </li></ul><ul><ul><li>Alternative to sampling from cell phone directories </li></ul></ul><ul><li>Mixed-mode data collection may result in cost savings </li></ul><ul><ul><li>Dependent on complexity of infrastructure put into place to integrate the approaches </li></ul></ul>
  109. 117. Questions raised by ABS mixed-mode design <ul><li>Does the effort significantly reduce nonresponse and coverage bias? </li></ul><ul><ul><li>Little improvement in demographic characteristics (with the exception of non-landline households) </li></ul></ul><ul><ul><li>Weighted prevalence estimates were similar for 6 of 8 health and risk factors </li></ul></ul><ul><li>Mail survey limits number of questions and complexity of survey </li></ul><ul><li>Less control over within household selection </li></ul><ul><ul><li>Little difference between all adult and birthday selection methods </li></ul></ul>
  110. 118. What did we learn? <ul><li>We can reach cell phone population </li></ul><ul><li>We can reach households with no telephones </li></ul>
  111. 119. Operational considerations
  112. 120. Cost for multimodes <ul><li>Typically design mix of modes to: </li></ul><ul><ul><li>Optimize coverage, response, and costs </li></ul></ul><ul><ul><li>Less expensive to most expensive </li></ul></ul><ul><li>However: </li></ul><ul><ul><li>Set-up costs with each mode </li></ul></ul><ul><ul><li>Per unit costs may be high even for “low cost” mode if few use the mode </li></ul></ul>
  113. 121. Multimode: Operational Considerations <ul><li>Population of interest </li></ul><ul><li>Sequential versus concurrent use of modes </li></ul><ul><li>Comparability </li></ul><ul><ul><li>Within study </li></ul></ul><ul><ul><li>Across studies </li></ul></ul><ul><li>Questionnaire design and reducing measurement error </li></ul>
  114. 122. Reaching population of interest <ul><li>Need to understand certain elements of population you are trying to reach: </li></ul><ul><ul><li>Physical accessibility </li></ul></ul><ul><ul><li>Telephone access </li></ul></ul><ul><ul><ul><li>Landline </li></ul></ul></ul><ul><ul><ul><li>Cell phone </li></ul></ul></ul><ul><ul><li>Literacy level </li></ul></ul><ul><ul><li>Web-enabled </li></ul></ul><ul><li>How do respondents prefer to be interviewed? </li></ul><ul><li>Need to match mode combination to best fit population </li></ul>
  115. 123. Comparability across modes <ul><li>Changing methods over time in longitudinal surveys </li></ul><ul><ul><li>Confounding time and mode effects </li></ul></ul><ul><li>Different modes for different subgroups </li></ul><ul><ul><li>Are groups really different or is it mode effect? </li></ul></ul><ul><li>Different modes for different samples </li></ul><ul><ul><li>Comparing across surveys conducted using different modes </li></ul></ul>
  116. 124. Reducing measurement error <ul><li>Different modes have tradition of different formats </li></ul><ul><li>Question format has effect on response distribution </li></ul><ul><li>Consequences: Designers routinely enhance unwanted mode effects in mixed-mode surveys </li></ul><ul><li>What to do? </li></ul>
  121. 129. Concluding thoughts <ul><li>Producing valid survey estimates is a multi-phase / multifaceted process </li></ul><ul><li>Assessing validity is often quite difficult, involving a mix of scientific rigor and subjective judgment </li></ul><ul><li>Ensuring validity is a necessity for the long-term survival of any health surveillance system </li></ul>
  122. 130. 2008 BRFSS <ul><li>Cell phone in 21 states (250+) </li></ul><ul><li>Mutli mode in 19 states (RDD as base)? </li></ul><ul><li>Physical measurements in 3 states? </li></ul>
  123. 131. Future of BRFSS <ul><li>Mutli mode system </li></ul><ul><li>Web as a refusal conversion </li></ul><ul><li>Physical measurements in every state </li></ul><ul><li>Non response research </li></ul>
  124. 132. Contact: Ali Mokdad [email_address] For more information on BRFSS: