maternal mortality sri lanka validating maternal mortality estimates_murray_110210_ihme_1210
Upcoming SlideShare
Loading in...5
×
 

maternal mortality sri lanka validating maternal mortality estimates_murray_110210_ihme_1210

on

  • 821 views

 

Statistics

Views

Total Views
821
Views on SlideShare
728
Embed Views
93

Actions

Likes
0
Downloads
12
Comments
0

6 Embeds 93

http://www.healthmetricsandevaluation.org 82
http://overport.ihme.washington.edu 4
http://www.healthdata.org 4
http://dev.ihme.washington.edu 1
http://translate.googleusercontent.com 1
https://ihmeuw.ihme.washington.edu 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    maternal mortality sri lanka validating maternal mortality estimates_murray_110210_ihme_1210 maternal mortality sri lanka validating maternal mortality estimates_murray_110210_ihme_1210 Presentation Transcript

    • Validating maternal mortality estimates November 2, 2010 Christopher J.L. Murray Institute Director
    • Outline
      • Predictive validity
      • Uncertainty
      • Comparisons with new WHO model
    • Validation of our model approach
      • Given the range of options of the modeling strategy it is essential to objectively evaluate model performance
      • We want an empirical answer to questions like:
        • Are these the right covariates to include in the first stage?
        • Are these the right transformations of the covariates?
        • Does the spatial-temporal stage improve model performance? If so, by how much?
    • Validation
      • To validate our model, we need something to compare the model’s output to
      • Ideally, we would have the “truth” to compare the model to, but we just have observed data points, not the true underlying risk of maternal death
      • Instead, we can “hold back” some of the observed data and then see how well our model, fit to the remaining data, does in predicting the held back data points
    • What do we care about?
      • We care that our model can:
        • Predict for gaps in time
          • For country-years that are missing in the middle of the time series
        • Predict out of time (i.e. forecast and backcast)
          • For countries where we have only a partial time series
        • Predict for countries with no data
    • Predictive validity
      • We can construct four types of predictive validity test to validate our model
      • The basic idea:
        • Sample 20% of the data, depending on what type of test you want to conduct
          • Randomly sample 20% of country-years with data
          • Randomly sample 20% of countries with data
          • Hold out the first 20% of years of data for all countries
          • Hold out the last 20% of years of data for all countries
    • Predictive validity
      • We can construct four types of predictive validity test to validate our model
      • The basic idea:
        • Sample 20% of the data, depending on what type of test you want to conduct
        • Estimate the model in the remaining 80% of data
        • Using the model from step (2), predict into the 20% hold-out sample
        • Calculate metrics of fit to determine how well the model did predicting the observed data in the 20% hold-out sample
    • Predictive validity
      • We repeat steps 1-3 30 times for the test of gaps in time and countries with no data, to make sure our results are not an artifact of a given random sample
        • Sample 20% of the data, depending on what type of test you want to conduct
        • Estimate the model in the remaining 80% of data
        • Using the model from step (2), predict into the 20% hold-out sample
        • Calculate metrics of fit to determine how well the model did predicting the observed data in the 20% hold-out sample
    • Predictive validity results: comparing the linear and spatio-temporal models 20% of Countries Regression Root Mean SE* Root Median SE Mean RE** Median RE Linear 214.84 27.00 0.604 0.417 Spatio-Temporal 189.27 25.34 0.521 0.357 First 20% of Country Years Regression Root Mean SE Root Median SE Mean RE Median RE Linear 208.28 22.04 0.702 0.437 Spatio-Temporal 129.32 11.92 0.392 0.199 Last 20% of Country Years Regression Root Mean SE Root Median SE Mean RE Median RE Linear 158.86 13.23 0.538 0.421 Spatio-Temporal 104.08 7.46 0.284 0.213 Random 20% of Country Years Regression Root Mean SE Root Median SE Mean RE Median RE Linear 215.44 24.22 0.619 0.419 Spatio-Temporal 125.34 10.36 0.286 0.165 * SE = Squared Error ** RE = Relative Error
    • Outline
      • Predictive validity
      • Uncertainty
      • Comparisons with new WHO model
    • Uncertainty
      • Uncertainty is the “life preserver” for any researcher!
      • While uncertainty intervals are sometimes ignored by policy-makers, they are crucial when interpreting results
      • Identifying and incorporating all relevant types of uncertainty into uncertainty intervals in an empirical way is crucial
    • What is the objective of uncertainty measurement? This line is the true, underlying risk of maternal death in a sample country, or the “expected value”
    • What is the objective of uncertainty measurement? But we don’t observe that expected value; we observe particular data points
    • What is the objective of uncertainty measurement? We want our uncertainty bounds to contain the expected value 95% of the time
    • What are the sources of uncertainty?
      • Sampling uncertainty
      • Non-sampling uncertainty
      • Parameter uncertainty
        • From the linear model
        • From the spatial-temporal local regressions
      • Remaining systematic variation
    • Uncertainty: source 1
        • Sampling uncertainty
        • Any data source will have some degree of associated stochastic sampling error, which must be reflected in any estimates of uncertainty
        • We capture this uncertainty by drawing from a binomial distribution with the observed maternal cause fraction as p and the number of trials ( n ) as the total number of observed deaths
        • We simulate 100 datasets by drawing from these distributions, and use these to propagate the sampling uncertainty through the modeling process
    • Sampling uncertainty
    • Uncertainty: sources 2 and 3
      • Parameter uncertainty
      • The application of a statistical model yields uncertainty in the parameter estimates of the model
        • You don’t just get an estimate of the β : you get a β ± a measure of uncertainty
        • Here we have two stages of parameter uncertainty
          • From the linear model
          • From the spatial-temporal local regressions
    • Simulating for parameter uncertainty
      • For each of the 100 datasets generated:
        • Estimate the linear model
        • Make five draws from the variance-covariance matrix of the regression β s
        • Estimate the spatial-temporal model for each of these draws from the linear model
        • Make five draws from the variance-covariance matrix of each of the local regressions
    • Parameter uncertainty: a simple example Here’s one potential model Here’s another potential model Parameter uncertainty takes into account the different models that could potentially fit the data
    • Uncertainty: source 4
      • The fourth source we want to capture is the remaining systematic variation that our model does not explain
        • i.e. Education, fertility, etc and spatio-temporal relatedness do not explain all variation in maternal mortality
      • However, we cannot estimate the systematic variation directly; the remaining variation consists of three parts
        • Systematic variation
        • Stochastic variation
        • Non-sampling variation
    • The leftover variation Non-sampling error Systematic error, but we don’t observe the true value This difference could be partially stochastic error, partially non-sampling error and partially non-sampling error
    • Uncertainty: source 4
      • We can separate out the stochastic variation from the systematic and non-sampling variation using simulation
      • But we have no way to separate out the systematic and non-sampling variation, so to be conservative, we include both
        • This will dramatically overestimate our uncertainty as non-sampling variation is quite large
    • Summarizing uncertainty
    • Outline
      • Predictive validity
      • Uncertainty
      • Comparisons with new WHO model
    • The recent WHO estimates (2010): input data
      • The study divides countries into categories defined by the type of data available in that country
        • Group A: Civil registration characterized as complete (63 countries)
        • Group B: Other types of data available (85 countries)
        • Group C: No national data available (24 countries)
    • The recent WHO estimates (2010): input data
      • Group A : Civil registration characterized as complete (63 countries total – none of the workshop countries)
        • Requirements for inclusion:
          • Earliest year of data before 1996, latest year after 2002
          • Data available for more than half the range of years available
          • Estimated completeness at more than 85% for all years
          • Deaths to ICD-10 R codes did not exceed 20%
        • Inflated by a factor of 1.5, unless country-specific adjustments were available
          • Based on reports in 15 countries; reported misclassification ranges from 1.08 (Uzbekistan) to 3.2 (El Salvador)
        • Maternal and all-cause deaths of unknown age redistributed proportionally over the age range
        • VR collapsed into 5 year time periods
    • The recent WHO estimates (2010): input data
        • Group B : Other types of data available (85 countries, including all workshop countries)
        • Sisterhood data
          • Assumed fraction of pregnancy-related deaths is understated, up-adjusted by a factor of 1.1
        • Deaths in the HH (including Indian SRS)
          • Adjusted upward by a factor of 1.1
        • Other “Special studies” (confidential enquires, “RAMOS”)
          • Adjusted upward by a factor of 1.1
      • Group C : No data available (24 countries)
    • Other WHO adjustments
      • AIDS-related mortality
      • Pregnancy related vs. maternal deaths
    • WHO AIDS adjustment
      • Wanted the dependent variable in the regression model to reflect non-AIDS-related maternal deaths only
      • Used unpublished UNAIDS tables on the proportion of total deaths of women aged 15-49 due to AIDS
      • Assume the fraction of AIDS deaths that occur during pregnancy that should be counted as maternal deaths, non-AIDS related maternal deaths depending on data source:
        • 0.1 for pregnancy-related data points
        • 0.5 for maternal data points
      • Use this non-AIDS-related PMDF as the dependent variable in the regression model
    • WHO Pregnancy-related adjustment
      • Distinction between:
        • Pregnancy-related mortality (all deaths occurring during pregnancy up to 42 days after – including incidental deaths)
        • Maternal mortality (death related to pregnancy, childbirth or puerperium, both direct and indirect causes)
      • Adjust the input non-AIDS PMDF from data sources identifying pregnancy-related mortality:
        • By a factor of 0.85 for most of the world
        • By a factor of 0.9 in Sub-Saharan Africa
        • Based on data from 8 countries
    • WHO and partners regression-based approach
      • Construct a database of 484 observations (680 total, but exclude 196)
      • Use a model to predict maternal mortality for the 109 countries in Group B (non-VR data) and Group C (no data)
    • WHO regression approach
      • Dependent variable: ln(non-AIDS PMDF)
      • Offset: ln(1- a ) where a is the proportion of all AIDS deaths among women aged 15-49 in the population
      • Covariates:
        • ln(GDP per capita): most data from the WB
        • ln(general fertility rate): UNPD
        • Coverage of skilled attendant at birth (UNICEF database, filled in using a logit model with time as the only covariate)
      • Multi-level regression model with random effects for country and region
      • Predicted values for 5-year intervals centered around 1990, 1995, 2000, 2005 and 2008
    • WHO counts of all-cause deaths for maternal age women
      • All-cause counts of deaths very different from IHME estimates
    • All-cause death counts comparison
      • WHO vs. UNPD
        • UNPD estimates only available for five year blocks of time (1995, 2000, 2005)
    • WHO: AIDS-related maternal deaths
      • Given that the dependent variable was non-AIDS PMDF, after estimation, must estimate contribution of AIDS to maternal mortality, and add this back in
        • Move from non-AIDS PMDF to total PMDF
      • Assume that half of the estimated number of AIDS deaths that occur during pregnancy should be counted as maternal deaths
      • Assume the relative risk of dying from AIDS for a pregnant versus non-pregnant woman is 0.4
    • IHME and the recent UN estimates   IHME UN (H4) Data Sources 2651 2142
        • Vital Statistics
      2186 2010
        • Surveys
      204 819**
        • Census
      46 19
        • Verbal Autopsy
      215 113 Scope of Study    
        • Time series
      1980-2008 1990-2008
        • Countries
      181 172 Correction    
        • Misclassification
      Country specific Correction factor 1.5 (63 countries)
        • Completeness
      Country specific UN estimates Number of female deaths (15–49) Rajaratnam, 2010 WHO lifetables Estimate based on Model for all countries 118 model & 63 correction factor Model Linear + Space-time Multilevel Dependent variable MM rate (ln) by age group Fraction of MM (log) all ages Treatment of HIV Model-based Estimated deaths separately Covariates    
        • GDP
      yes yes
        • Education
      yes no
        • TFR
      yes yes
        • HIV
      yes no
        • Health services
      Neonatal mort SBA Model Validation yes no Uncertainty yes yes