Mike Smith, MSPH MCH Epidemiologist Director, Division of Research and Planning South Carolina Department of Health and Environmental Control

1. Using the Data that We
Collect
Data Sources and Evaluation Tips
Mike Smith, MSPH
MCH Epidemiologist
Director, Division of Research and Planning
SC DHEC
(803) 898-3740
smithm4@dhec.sc.gov

2. Overview
• Why do we need data?
• What types of MCH data are available and
where can we get it?
• How can we evaluate our programs?
• So what?

3. Why do we need data?

4. Why do we need data?
• Knowing what the real issues are
– e.g. smoking during pregnancy by race
• Knowing the potential for impact and
targeting interventions

5. Why do we need data?
• Knowing whether or not our programs or
interventions work at all (or where to
improve them)
• Knowing whether or not our programs or
interventions work in our setting or with
out population

6. What types of MCH data are
available?

7. SC Specific Data Sources
• Vital Records/registries – attempt to
capture data around all events that occur
to state residents
– birth certificate
– death certificate
– reportable fetal deaths
– induced terminations of pregnancy
– birth defects registry

8. SC Specific Data Sources
• Surveys – attempt to collect more detailed data
on fewer individuals that represent the larger
population of interest
– Pregnancy Risk Assessment Monitoring System
(PRAMS)
– Behavioral Risk Factor Surveillance System (BRFSS)
– Youth Risk Behavior Survey (YRBS)
– National Survey of Children’s Health (NSCH)
– National Survey of Children with Special Health Care
Needs (NSCSHCN)

9. Vital Records - Background
• The National Center for Health Statistics (NCHS) creates
standard forms that they recommend US states and
territories use to document vital events (but can not
require states to use these forms).
• Periodically, NCHS releases revised versions of the
forms and recommends that states implement them.
• Some states implement them earlier than others, leading
to potential issues around comparability.

10. Example – Comparability
• NCHS revised the standard birth certificate (BC) in 2003.
SC implemented revision in 2004.
• Some information from the SC BC from 2004-Present is
not available for prior years (ex. pre-pregnancy weight,
gestational diabetes).
• Some information from the SC BC is available before
and after 2004, but is not directly comparable (ex.
mother/father’s education, smoking status).
• Because some states have not moved to the 2003
NCHS standard birth certificate, across-state
comparisons can be difficult for some information.

11. Vital Records Data Sources
• Birth Certificate
– Includes information such as: maternal height
and weight, race/ethnicity, age, education,
county of residence, smoking before and
during pregnancy, delivery payment method,
number of previous live births, risk factors,
infections, delivery method, birthweight,
gestational age, abnormal newborn
conditions, congenital anomalies

12. Vital Records Data Sources
• Death Certificate
– Can be combined with birth certificate data to
examine infant mortality
• Report of Fetal Death
– Required to be reported only if beyond 20
weeks gestation and 350 grams or heavier
– Some of the same information as on the birth
certificate; causes and conditions contributing
to fetal death

13. Vital Records Data Sources
• Induced Termination of Pregnancy
– Basic demographic data, gestational age,
informed written consent
– Many induced terminations of pregnancy to
SC residents occur out of state
• SC vital records data available through the
DHEC SCAN system:
http://scangis.dhec.sc.gov/scan/index.aspx

14. Registry Data
• SC Birth Defects Program
– Data available beginning in 2008
– Birth defects recommended by the National
Birth Defects Prevention Network
– Demographic and diagnostic data collected
through medical record abstraction

15. Where can we get data?
• SC vital records data available through the
DHEC SCAN system:
http://scangis.dhec.sc.gov/scan/index.aspx
• Contact: Daniela Nitcheva, nitchedk@dhec.sc.gov
• SC Birth Defects Program data available through
the DHEC Environmental Public Health Tracking
Program:
http://www.scdhec.gov/administration/epht/BirthDefects.
htm
• Contact: Mike Smith, smithm4@dhec.sc.gov

16. VR Data for MCH Research
• Vital records data is very commonly used for MCH
research.
• Some strengths:
– Standardized (mostly) national data
– Data collected for all events
– Available for county or smaller geographic levels
• Some weaknesses:
– Much of the information is self-reported
– Other data quality concerns
– Staggered implementation of revisions

17. Survey Data
• Pregnancy Risk Assessment Monitoring System
(PRAMS)
– Survey data about maternal behaviors, attitudes, and
experiences before, during, and shortly after
pregnancy
– Topic such as physical activity, breastfeeding,
postpartum depression symptoms, oral health,
stressful life events
– Available through DHEC SCAN system:
http://scangis.dhec.sc.gov/scan/index.aspx
• Contact: Mike Smith, smithm4@dhec.sc.gov

18. Survey Data
• Behavioral Risk Factor Surveillance System
(BRFSS)
– Survey data about general health behaviors and
status for the adult population; often subset to women
of reproductive age
– Topics such as healthcare access, tobacco use,
disease screenings, immunizations, women’s health
– Available through SC BRFSS website:
http://www.scdhec.gov/hs/epidata/brfss2010.htm
• Contact: Shae Sutton, suttonsr@dhec.sc.gov

19. Survey Data
• Youth Risk Behavior Survey (YRBS)
– Survey data about the health and risk behaviors of
middle and high schoolers; conducted by the SC
Department of Education
– Topics such as tobacco and alcohol use, diet,
physical activity, sexual behaviors
– Available through SC YRBS website:
http://ed.sc.gov/agency/se/Instructional-Practices-
andEvaluations/SouthCarolinaYouthRiskBehaviorSur
veyYRBS.cfm
• Contact: Kimberly Stewart, kwstewart@ed.sc.gov

20. Survey Data
• National Survey of Children’s Health (NSCH)
– Survey data about the health of children (<18) by
state; collected by US DHHS HRSA
– Topics such as physical and mental health status,
access to quality health care, family, neighborhood,
social context
– Available through HRSA Child Health Data website:
http://www.childhealthdata.org/

21. Survey Data
• National Survey of Children with Special Health
Care Needs (NSCSHCN)
– Survey data about the health and functional status of
children with special health care needs; collected by
US DHHS HRSA
– Topics such as access to quality health care, care
coordination of services, access to a medical home,
transition services for youth, and impact of the chronic
conditions(s) on the child’s family
– Available through HRSA Child Health Data website:
http://www.childhealthdata.org/

22. Survey Data for MCH Research
• Survey data is also very commonly used for MCH
research.
• Some strengths:
– Often national data
– Data collection on a variety of topics with flexibility for changing
topics and questions regularly
• Some weaknesses:
– Nearly all information is self-reported
– Often only available at the state level
– Complex sampling and weighting schemes (applicable if you are
analyzing your own datasets)

23. Population-Based Data
• All of these vital records, registry,
andsurvey data sources are great for
determining what the real issues and
planning and targeting interventions.
• These data sources are usually not very
good for evaluating the effectiveness of
programs that are implemented on smaller
levels.

24. How can we evaluate our
programs and intervention?
Does our program or intervention work at all?
Does our program or intervention work in our setting?

25. Types of Program Evaluation
• Process evaluation
– Are we sticking to the planned model faithfully?
– Measures such as: number of meetings held, number
of attendees, number of clients served, etc.
• Outcome evaluation
– Have we impacted the outcome of interest?
– Measures such as: knowledge gained, low birth
weight deliveries, contraceptive use
– Often requires rigorous evaluation study designs

26. Basic Outcome Evaluation
• You have an intervention that you want to
impact an outcome, so you implement the
intervention (X) and observe (O) the
outcome
X O
• Interventions and observations are the key
elements to outcome evaluation

27. Basic Outcome Evaluation
• You want to be confident that:
– your intervention is actually what causes
changes in the outcome (internal validity)
– your intervention might result in similar
changes in the outcome in other populations
or settings (external validity)

28. Basic Outcome Evaluation
• So, it is important to understand what may
constitute threats to internal and external
validity and how to design evaluations that
avoid those threats.

29. Threats to Internal Validity
• Ambiguous Temporal Precedence: Could the
change in outcome have occurred before the
intervention?
– Possible Evaluation Design: add a pre-test
O1 X O2
• Selection: Could those receiving the intervention be
more or less likely to have the outcome than a
random person (e.g. volunteering)?
– Possible Evaluation Design: split volunteers into two
groups and apply intervention at different times
O1 X O2
O1 O2 X

30. Threats to Internal Validity
• Testing: Could the change in outcome be due to
participants learning the test or guessing what the
investigators are looking for in observation?
– Possible Evaluation Design: add a series of tests and a
control group or have a long time between tests
O1 O2 X O3
O1 O2 O3

31. Other Threats to Consider
• History: Could another event or program have
caused the difference in outcome?
• Maturation: Could natural changes (e.g. growth,
fatigue, experience) have resulted in the outcome?
– Possible Evaluation Design: select intervention and
comparison groups from the same geographic location and
observing outcomes at about the same time so that groups
are equally impacted by history

32. Other Threats to Consider
• Regression Artifacts: If participants are selected
because they scored lower or higher than average,
many will naturally regress back toward the average
(e.g. periods of stress).
• Attrition: Are participants that drop out more or less
likely to have a different outcome?
• Threats to internal validity can come in multiples.

33. Threats to External Validity
• Interaction of Causal Relationship with Units: Is the
study population representative of the population of
interest? Are participants of especially high/low
risk?
– Implication: You can only generalize to populations that
are comparable on key characteristics to those that
complete your evaluation, so selection and attrition are
very important to monitor.

34. Threats to External Validity
• Interaction of Causal Relationship with Outcomes:
Can an evaluation result for one outcome be
generalized for a similar outcome?
– Example: can an evaluation of a program to increase high
school graduate rates be assumed to also improve SAT
scores?
– Implication: Be cautious in overstating or over-interpreting
the results of an evaluation project.

35. Threats to External Validity
• Interaction of Causal Relationship with Settings:
Does an intervention work similarly in urban and
rural areas? In public and private schools?
– Implication: Be sure that an evaluation project is set in an
area that allows for broad participation across settings of
interest or conduct evaluations in multiple sites.

36. Randomization is Magic
• Random sampling of individuals to participate in
the evaluation from the population of interest is
the most effective way to achieve high external
validity.
• Random assignment of participants to
intervention and control groups is the most
effective way to achieve high internal validity.
• Of course, randomization is difficult in the real
world.

37. So what?

38. Questions and Discussion