Using the Data that We
Collect
Data Sources and Evaluation Tips
Mike Smith, MSPH
MCH Epidemiologist
Director, Division of ...
Overview
• Why do we need data?
• What types of MCH data are available and
where can we get it?
• How can we evaluate our ...
Why do we need data?
Why do we need data?
• Knowing what the real issues are
– e.g. smoking during pregnancy by race
• Knowing the potential fo...
Why do we need data?
• Knowing whether or not our programs or
interventions work at all (or where to
improve them)
• Knowi...
What types of MCH data are
available?
SC Specific Data Sources
• Vital Records/registries – attempt to
capture data around all events that occur
to state reside...
SC Specific Data Sources
• Surveys – attempt to collect more detailed data
on fewer individuals that represent the larger
...
Vital Records - Background
• The National Center for Health Statistics (NCHS) creates
standard forms that they recommend U...
Example – Comparability
• NCHS revised the standard birth certificate (BC) in 2003.
SC implemented revision in 2004.
• Som...
Vital Records Data Sources
• Birth Certificate
– Includes information such as: maternal height
and weight, race/ethnicity,...
Vital Records Data Sources
• Death Certificate
– Can be combined with birth certificate data to
examine infant mortality
•...
Vital Records Data Sources
• Induced Termination of Pregnancy
– Basic demographic data, gestational age,
informed written ...
Registry Data
• SC Birth Defects Program
– Data available beginning in 2008
– Birth defects recommended by the National
Bi...
Where can we get data?
• SC vital records data available through the
DHEC SCAN system:
http://scangis.dhec.sc.gov/scan/ind...
VR Data for MCH Research
• Vital records data is very commonly used for MCH
research.
• Some strengths:
– Standardized (mo...
Survey Data
• Pregnancy Risk Assessment Monitoring System
(PRAMS)
– Survey data about maternal behaviors, attitudes, and
e...
Survey Data
• Behavioral Risk Factor Surveillance System
(BRFSS)
– Survey data about general health behaviors and
status f...
Survey Data
• Youth Risk Behavior Survey (YRBS)
– Survey data about the health and risk behaviors of
middle and high schoo...
Survey Data
• National Survey of Children’s Health (NSCH)
– Survey data about the health of children (<18) by
state; colle...
Survey Data
• National Survey of Children with Special Health
Care Needs (NSCSHCN)
– Survey data about the health and func...
Survey Data for MCH Research
• Survey data is also very commonly used for MCH
research.
• Some strengths:
– Often national...
Population-Based Data
• All of these vital records, registry,
andsurvey data sources are great for
determining what the re...
How can we evaluate our
programs and intervention?
Does our program or intervention work at all?
Does our program or inter...
Types of Program Evaluation
• Process evaluation
– Are we sticking to the planned model faithfully?
– Measures such as: nu...
Basic Outcome Evaluation
• You have an intervention that you want to
impact an outcome, so you implement the
intervention ...
Basic Outcome Evaluation
• You want to be confident that:
– your intervention is actually what causes
changes in the outco...
Basic Outcome Evaluation
• So, it is important to understand what may
constitute threats to internal and external
validity...
Threats to Internal Validity
• Ambiguous Temporal Precedence: Could the
change in outcome have occurred before the
interve...
Threats to Internal Validity
• Testing: Could the change in outcome be due to
participants learning the test or guessing w...
Other Threats to Consider
• History: Could another event or program have
caused the difference in outcome?
• Maturation: C...
Other Threats to Consider
• Regression Artifacts: If participants are selected
because they scored lower or higher than av...
Threats to External Validity
• Interaction of Causal Relationship with Units: Is the
study population representative of th...
Threats to External Validity
• Interaction of Causal Relationship with Outcomes:
Can an evaluation result for one outcome ...
Threats to External Validity
• Interaction of Causal Relationship with Settings:
Does an intervention work similarly in ur...
Randomization is Magic
• Random sampling of individuals to participate in
the evaluation from the population of interest i...
So what?
Questions and Discussion
Upcoming SlideShare
Loading in...5
×

Using the Data that We Collect. Data Sources and Evaluation Tips

759

Published on

Mike Smith, MSPH
MCH Epidemiologist
Director, Division of Research and Planning
South Carolina Department of Health and Environmental Control

Published in: Health & Medicine, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
759
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Using the Data that We Collect. Data Sources and Evaluation Tips

  1. 1. Using the Data that We Collect Data Sources and Evaluation Tips Mike Smith, MSPH MCH Epidemiologist Director, Division of Research and Planning SC DHEC (803) 898-3740 smithm4@dhec.sc.gov
  2. 2. Overview • Why do we need data? • What types of MCH data are available and where can we get it? • How can we evaluate our programs? • So what?
  3. 3. Why do we need data?
  4. 4. Why do we need data? • Knowing what the real issues are – e.g. smoking during pregnancy by race • Knowing the potential for impact and targeting interventions
  5. 5. Why do we need data? • Knowing whether or not our programs or interventions work at all (or where to improve them) • Knowing whether or not our programs or interventions work in our setting or with out population
  6. 6. What types of MCH data are available?
  7. 7. SC Specific Data Sources • Vital Records/registries – attempt to capture data around all events that occur to state residents – birth certificate – death certificate – reportable fetal deaths – induced terminations of pregnancy – birth defects registry
  8. 8. SC Specific Data Sources • Surveys – attempt to collect more detailed data on fewer individuals that represent the larger population of interest – Pregnancy Risk Assessment Monitoring System (PRAMS) – Behavioral Risk Factor Surveillance System (BRFSS) – Youth Risk Behavior Survey (YRBS) – National Survey of Children’s Health (NSCH) – National Survey of Children with Special Health Care Needs (NSCSHCN)
  9. 9. Vital Records - Background • The National Center for Health Statistics (NCHS) creates standard forms that they recommend US states and territories use to document vital events (but can not require states to use these forms). • Periodically, NCHS releases revised versions of the forms and recommends that states implement them. • Some states implement them earlier than others, leading to potential issues around comparability.
  10. 10. Example – Comparability • NCHS revised the standard birth certificate (BC) in 2003. SC implemented revision in 2004. • Some information from the SC BC from 2004-Present is not available for prior years (ex. pre-pregnancy weight, gestational diabetes). • Some information from the SC BC is available before and after 2004, but is not directly comparable (ex. mother/father’s education, smoking status). • Because some states have not moved to the 2003 NCHS standard birth certificate, across-state comparisons can be difficult for some information.
  11. 11. Vital Records Data Sources • Birth Certificate – Includes information such as: maternal height and weight, race/ethnicity, age, education, county of residence, smoking before and during pregnancy, delivery payment method, number of previous live births, risk factors, infections, delivery method, birthweight, gestational age, abnormal newborn conditions, congenital anomalies
  12. 12. Vital Records Data Sources • Death Certificate – Can be combined with birth certificate data to examine infant mortality • Report of Fetal Death – Required to be reported only if beyond 20 weeks gestation and 350 grams or heavier – Some of the same information as on the birth certificate; causes and conditions contributing to fetal death
  13. 13. Vital Records Data Sources • Induced Termination of Pregnancy – Basic demographic data, gestational age, informed written consent – Many induced terminations of pregnancy to SC residents occur out of state • SC vital records data available through the DHEC SCAN system: http://scangis.dhec.sc.gov/scan/index.aspx
  14. 14. Registry Data • SC Birth Defects Program – Data available beginning in 2008 – Birth defects recommended by the National Birth Defects Prevention Network – Demographic and diagnostic data collected through medical record abstraction
  15. 15. Where can we get data? • SC vital records data available through the DHEC SCAN system: http://scangis.dhec.sc.gov/scan/index.aspx • Contact: Daniela Nitcheva, nitchedk@dhec.sc.gov • SC Birth Defects Program data available through the DHEC Environmental Public Health Tracking Program: http://www.scdhec.gov/administration/epht/BirthDefects. htm • Contact: Mike Smith, smithm4@dhec.sc.gov
  16. 16. VR Data for MCH Research • Vital records data is very commonly used for MCH research. • Some strengths: – Standardized (mostly) national data – Data collected for all events – Available for county or smaller geographic levels • Some weaknesses: – Much of the information is self-reported – Other data quality concerns – Staggered implementation of revisions
  17. 17. Survey Data • Pregnancy Risk Assessment Monitoring System (PRAMS) – Survey data about maternal behaviors, attitudes, and experiences before, during, and shortly after pregnancy – Topic such as physical activity, breastfeeding, postpartum depression symptoms, oral health, stressful life events – Available through DHEC SCAN system: http://scangis.dhec.sc.gov/scan/index.aspx • Contact: Mike Smith, smithm4@dhec.sc.gov
  18. 18. Survey Data • Behavioral Risk Factor Surveillance System (BRFSS) – Survey data about general health behaviors and status for the adult population; often subset to women of reproductive age – Topics such as healthcare access, tobacco use, disease screenings, immunizations, women’s health – Available through SC BRFSS website: http://www.scdhec.gov/hs/epidata/brfss2010.htm • Contact: Shae Sutton, suttonsr@dhec.sc.gov
  19. 19. Survey Data • Youth Risk Behavior Survey (YRBS) – Survey data about the health and risk behaviors of middle and high schoolers; conducted by the SC Department of Education – Topics such as tobacco and alcohol use, diet, physical activity, sexual behaviors – Available through SC YRBS website: http://ed.sc.gov/agency/se/Instructional-Practices- andEvaluations/SouthCarolinaYouthRiskBehaviorSur veyYRBS.cfm • Contact: Kimberly Stewart, kwstewart@ed.sc.gov
  20. 20. Survey Data • National Survey of Children’s Health (NSCH) – Survey data about the health of children (<18) by state; collected by US DHHS HRSA – Topics such as physical and mental health status, access to quality health care, family, neighborhood, social context – Available through HRSA Child Health Data website: http://www.childhealthdata.org/
  21. 21. Survey Data • National Survey of Children with Special Health Care Needs (NSCSHCN) – Survey data about the health and functional status of children with special health care needs; collected by US DHHS HRSA – Topics such as access to quality health care, care coordination of services, access to a medical home, transition services for youth, and impact of the chronic conditions(s) on the child’s family – Available through HRSA Child Health Data website: http://www.childhealthdata.org/
  22. 22. Survey Data for MCH Research • Survey data is also very commonly used for MCH research. • Some strengths: – Often national data – Data collection on a variety of topics with flexibility for changing topics and questions regularly • Some weaknesses: – Nearly all information is self-reported – Often only available at the state level – Complex sampling and weighting schemes (applicable if you are analyzing your own datasets)
  23. 23. Population-Based Data • All of these vital records, registry, andsurvey data sources are great for determining what the real issues and planning and targeting interventions. • These data sources are usually not very good for evaluating the effectiveness of programs that are implemented on smaller levels.
  24. 24. How can we evaluate our programs and intervention? Does our program or intervention work at all? Does our program or intervention work in our setting?
  25. 25. Types of Program Evaluation • Process evaluation – Are we sticking to the planned model faithfully? – Measures such as: number of meetings held, number of attendees, number of clients served, etc. • Outcome evaluation – Have we impacted the outcome of interest? – Measures such as: knowledge gained, low birth weight deliveries, contraceptive use – Often requires rigorous evaluation study designs
  26. 26. Basic Outcome Evaluation • You have an intervention that you want to impact an outcome, so you implement the intervention (X) and observe (O) the outcome X O • Interventions and observations are the key elements to outcome evaluation
  27. 27. Basic Outcome Evaluation • You want to be confident that: – your intervention is actually what causes changes in the outcome (internal validity) – your intervention might result in similar changes in the outcome in other populations or settings (external validity)
  28. 28. Basic Outcome Evaluation • So, it is important to understand what may constitute threats to internal and external validity and how to design evaluations that avoid those threats.
  29. 29. Threats to Internal Validity • Ambiguous Temporal Precedence: Could the change in outcome have occurred before the intervention? – Possible Evaluation Design: add a pre-test O1 X O2 • Selection: Could those receiving the intervention be more or less likely to have the outcome than a random person (e.g. volunteering)? – Possible Evaluation Design: split volunteers into two groups and apply intervention at different times O1 X O2 O1 O2 X
  30. 30. Threats to Internal Validity • Testing: Could the change in outcome be due to participants learning the test or guessing what the investigators are looking for in observation? – Possible Evaluation Design: add a series of tests and a control group or have a long time between tests O1 O2 X O3 O1 O2 O3
  31. 31. Other Threats to Consider • History: Could another event or program have caused the difference in outcome? • Maturation: Could natural changes (e.g. growth, fatigue, experience) have resulted in the outcome? – Possible Evaluation Design: select intervention and comparison groups from the same geographic location and observing outcomes at about the same time so that groups are equally impacted by history
  32. 32. Other Threats to Consider • Regression Artifacts: If participants are selected because they scored lower or higher than average, many will naturally regress back toward the average (e.g. periods of stress). • Attrition: Are participants that drop out more or less likely to have a different outcome? • Threats to internal validity can come in multiples.
  33. 33. Threats to External Validity • Interaction of Causal Relationship with Units: Is the study population representative of the population of interest? Are participants of especially high/low risk? – Implication: You can only generalize to populations that are comparable on key characteristics to those that complete your evaluation, so selection and attrition are very important to monitor.
  34. 34. Threats to External Validity • Interaction of Causal Relationship with Outcomes: Can an evaluation result for one outcome be generalized for a similar outcome? – Example: can an evaluation of a program to increase high school graduate rates be assumed to also improve SAT scores? – Implication: Be cautious in overstating or over-interpreting the results of an evaluation project.
  35. 35. Threats to External Validity • Interaction of Causal Relationship with Settings: Does an intervention work similarly in urban and rural areas? In public and private schools? – Implication: Be sure that an evaluation project is set in an area that allows for broad participation across settings of interest or conduct evaluations in multiple sites.
  36. 36. Randomization is Magic • Random sampling of individuals to participate in the evaluation from the population of interest is the most effective way to achieve high external validity. • Random assignment of participants to intervention and control groups is the most effective way to achieve high internal validity. • Of course, randomization is difficult in the real world.
  37. 37. So what?
  38. 38. Questions and Discussion
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×