A Graphical Interactive Tool for Evidence-Based Public Health Program Evaluation by Program Personnel with Minimal Computer Skills

1. A Graphical Interactive Tool for Evidenced
Based Public Health Program Evaluation by
Programmatic Personnel with Minimal
Computer Skills
Leisa J. Stanley, MS
Healthy Start Coalition of Hillsborough County, Inc.
David Darr, MD, MPH
Lawton & Rhea Chiles Center, College of Public Health,
University of South Florida
12th Annual MCH Epidemiology Conference
Atlanta, Georgia
December 6-8, 2006

2. Project Background
 Healthy Start Coalitions – HSC of Hillsborough County
– Established by legislative act, Chapter 383, FS., 1991
– Responsibilities for MCH System
 Chiles Center for Healthy Mothers & Babies
– Part of Department of Community & Family Health, College of Public
Health, USF
– The Chiles Center brings together experts in public health, health care,
program design and evaluation, and maternal and child health from
Florida universities and state, national and international programs.
 Healthy Start Program
– Countywide Program – 16,000 births in 2005
– In 2005-2006, served 6,000 pregnant women and 7,000 infants
– Provides home visitation and care coordination services
– 12 state and 11 local Performance Measures
– 6 state Outcome Measures

3. Problem Statement
 Question: Can data be prepared, so it is easily and reliably used
by non-programmers for programmatic evaluation with software
found on virtually all computers?
 Issues:
– Data Access: vital statistics, service data
– Data Timeliness
– Data Volume
– Staff Training
 System-Wide Issues
– Increasing number of pregnant women without insurance
– Rising Infant Mortality Rate
– Racial Disparity

4. Purpose
 Creation of database with historical data:
– 1992 – forward
– Vital Statistics Data
– Healthy Start Services Data
 Addition of provisional vital statistics data
– Timeliness
 Two Pathways
– Research Database
– Program Evaluation Database

5. Creation of a Database
and Interactive Tool

6. Data Files
• Florida Department of Health Office of Vital Statistics
• Natality Record
• Fetal Death Records
• Infant Death Records
• Provisional Vital Statistics Records
• Florida Department of Health Office of Planning &
Evaluation
• Healthy Start Prenatal Screening Data
• Healthy Start Infant Screening Data
• Healthy Start Services Data
• Hillsborough County Health Department
• Local Healthy Start Prenatal Risk Screening Data
• Healthy Start services data with supplemental local service codes

8. Data Manipulations
 Converted all month/day/year variables into true SAS
dates
 Caculated needed variables
– Kotelchuck Index
– Interpregnancy Intervals
 Created as many categorical variables as possible, with as
many variations as possible to reduce the potential need
for future reprogramming
– Race/Ethnicity
– Age of Mother
– Birth Weight
– Gestation

9. Data Manipulations
 Created a variable to distinguish between a delivery
(maternal data) and a birth (infant data)
 Defined and created variables for service areas or
targeted populations
– Central Hillsborough Healthy Start
– Intensive Teen Unit
– Defined geographic regions
 Created value-added variables (Yes/No)
– Low Birth Weight
– Preterm Birth
– Infant Death
– Early Entry into Prenatal Care

10. Data Manipulations
 Created Pivot Charts in Excel
– Data dictionary is available
– Programming already done
– Limitations: 256 variables; 65,563 records
 Created macros in SAS to calculate from data
and import back into Excel
– Calculated Relative Risks
 Agree on reference groups
– Calculated Attributable Risk
– Calculated Population Attributable Risk

11. ADEQUATE
ADEQUATE PLUS
INADEQUATE
INTERMEDIATE
2001
2002
2003
0
50
100
150
200
250
300
350
400
450
500
2001
2002
2003
CHHS_CENSUS_YN YES
Count of IDNUMBER
KOTELCHUCK_INDEX
YEAR

12. CHHS_CENSUS_YN YES
Count of IDNUMBER YEAR
KOTELCHUCK_INDEX 2001 2002 2003 Grand Total
ADEQUATE 168 150 181 499
ADEQUATE PLUS 467 465 450 1382
INADEQUATE 106 81 79 266
INTERMEDIATE 9 9 6 24
Grand Total 750 705 716 2171
Sample of a Pivot Table

14. Uses by Programmatic Personnel
 Used by Federal Healthy Start Project
 Allows complex questions to be answered
easily
– No programming experience is required
– Minimal training is required
 Questions
– Racial Disparity Gap
– Repeat Pregnancies for Teens
– Entry into Prenatal Care
 Prepare reports for funders and key
stakeholders

15. Data Available versus Data Needed
 Available
– 1998-2003 de-
identified linked
vital statistics data
– 1992-2004 de-
identified unlinked
vital statistics data
– 2001-2003 service
data (incomplete)
 Needed
– Identified vital
statistics data to link
– 1992-present
complete services
data
– Local service data
codes
– Provisional vital
statistics data

16. Data Procurement
 Apply for Institutional Review Board
Approval
– University of South Florida
– Florida Department of Health
– Requires execution of proprietary data usage
agreement(s) with each entity owning the data desired for
linkages
– Requesting identified data and development of a research
database
 When we did not need to apply?
– For programmatic evaluation

17. Provisional Vital Statistics Data
 Specific program evaluation questions:
– Issues of program delivery
– Issues of access to care
– Emerging areas in county
 Do not use for:
– Outcome evaluation
– Be cautious with low numbers and impact on
rates: Wide month to month variations are
possible and small numbers preclude use in
outcomes analyses

18. Lessons Learned
 Initiate process as early as possible
– Anticipate delays, especially in use of
provisional vital statistics data
– If identifiers are to be retained, yearly data
usage agreements may be necessary
 Understand limitations of provisional
vital statistics data

19. Lessons Learned
 Data Security
– Assure original data owner that only de-identified data will be
used by programmatic personnel for their evaluations
– Assure original data owner that there will be no follow-back to
any client
– Assure original data owner that all results will be reported in
aggregate only
– Be very mindful of HIPAA rules, especially when reporting at
zip code or census tract levels for rare conditions or events
 Do not allow original data owner to have veto power
over any reporting of data
 Know and understand the data you are given

20. Contact Information
 Leisa J. Stanley, MS
lstanley@hstart.org
 David Darr, MD, MPH
ddarr@health.usf.edu