Validating maternal mortality estimates November 2, 2010 Christopher J.L. Murray Institute Director
Outline <ul><li>Predictive validity </li></ul><ul><li>Uncertainty </li></ul><ul><li>Comparisons with new WHO model </li></ul>
Validation of our model approach <ul><li>Given the range of options of the modeling strategy it is essential to objectivel...
Validation <ul><li>To validate our model, we need something to compare the model’s output to </li></ul><ul><li>Ideally, we...
What do we care about? <ul><li>We care that our model can: </li></ul><ul><ul><li>Predict for  gaps in time   </li></ul></u...
Predictive validity <ul><li>We can construct four types of predictive validity test to validate our model </li></ul><ul><l...
Predictive validity <ul><li>We can construct four types of predictive validity test to validate our model </li></ul><ul><l...
Predictive validity <ul><li>We repeat steps 1-3 30 times for the test of gaps in time and countries with no data, to make ...
Predictive validity results: comparing the linear and spatio-temporal models 20% of Countries Regression Root Mean SE* Roo...
Outline <ul><li>Predictive validity </li></ul><ul><li>Uncertainty </li></ul><ul><li>Comparisons with new WHO model </li></ul>
Uncertainty <ul><li>Uncertainty is the “life preserver” for any researcher! </li></ul><ul><li>While uncertainty intervals ...
What is the objective of uncertainty measurement? This line is the true, underlying risk of maternal death in a sample cou...
What is the objective of uncertainty measurement? But we don’t observe that expected value; we observe particular data poi...
What is the objective of uncertainty measurement? We want our uncertainty bounds to contain the expected value 95% of the ...
What are the sources of uncertainty? <ul><li>Sampling uncertainty </li></ul><ul><li>Non-sampling uncertainty </li></ul><ul...
Uncertainty: source 1 <ul><ul><li>Sampling uncertainty  </li></ul></ul><ul><ul><li>Any data source will have some degree o...
Sampling uncertainty
Uncertainty: sources 2 and 3 <ul><li>Parameter uncertainty </li></ul><ul><li>The application of a statistical model yields...
Simulating for parameter uncertainty <ul><li>For each of the 100 datasets generated: </li></ul><ul><ul><li>Estimate the  l...
Parameter uncertainty: a simple example Here’s one potential model Here’s another potential model Parameter uncertainty ta...
Uncertainty: source 4 <ul><li>The fourth source we want to capture is the remaining systematic variation that our model do...
The leftover variation Non-sampling error Systematic error, but we don’t observe the true value This difference could be p...
Uncertainty: source 4 <ul><li>We can separate out the stochastic variation from the systematic and non-sampling variation ...
Summarizing uncertainty
Outline <ul><li>Predictive validity </li></ul><ul><li>Uncertainty </li></ul><ul><li>Comparisons with new WHO model </li></ul>
The recent WHO estimates (2010): input data <ul><li>The study divides countries into categories defined by the type of dat...
The recent WHO estimates (2010): input data <ul><li>Group A : Civil registration characterized as complete (63 countries t...
The recent WHO estimates (2010): input data <ul><ul><li>Group B : Other types of data available (85 countries, including a...
Other WHO adjustments <ul><li>AIDS-related mortality </li></ul><ul><li>Pregnancy related vs. maternal deaths </li></ul>
WHO AIDS adjustment <ul><li>Wanted the dependent variable in the regression model to reflect non-AIDS-related maternal dea...
WHO Pregnancy-related adjustment <ul><li>Distinction between: </li></ul><ul><ul><li>Pregnancy-related mortality (all death...
WHO and partners regression-based approach <ul><li>Construct a database of 484 observations (680 total, but exclude 196)  ...
WHO regression approach <ul><li>Dependent variable: ln(non-AIDS PMDF) </li></ul><ul><li>Offset: ln(1- a ) where a is the p...
WHO counts of all-cause deaths for maternal age women <ul><li>All-cause counts of deaths very different from IHME estimate...
All-cause death counts comparison <ul><li>WHO vs. UNPD </li></ul><ul><ul><li>UNPD estimates only available for five year b...
WHO: AIDS-related maternal deaths <ul><li>Given that the dependent variable was non-AIDS PMDF, after estimation, must esti...
IHME and the recent UN estimates   IHME UN (H4) Data Sources 2651 2142 <ul><ul><li>Vital Statistics </li></ul></ul>2186 20...
Upcoming SlideShare
Loading in...5
×

maternal mortality sri lanka validating maternal mortality estimates_murray_110210_ihme_1210

657

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
657
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

maternal mortality sri lanka validating maternal mortality estimates_murray_110210_ihme_1210

  1. 1. Validating maternal mortality estimates November 2, 2010 Christopher J.L. Murray Institute Director
  2. 2. Outline <ul><li>Predictive validity </li></ul><ul><li>Uncertainty </li></ul><ul><li>Comparisons with new WHO model </li></ul>
  3. 3. Validation of our model approach <ul><li>Given the range of options of the modeling strategy it is essential to objectively evaluate model performance </li></ul><ul><li>We want an empirical answer to questions like: </li></ul><ul><ul><li>Are these the right covariates to include in the first stage? </li></ul></ul><ul><ul><li>Are these the right transformations of the covariates? </li></ul></ul><ul><ul><li>Does the spatial-temporal stage improve model performance? If so, by how much? </li></ul></ul>
  4. 4. Validation <ul><li>To validate our model, we need something to compare the model’s output to </li></ul><ul><li>Ideally, we would have the “truth” to compare the model to, but we just have observed data points, not the true underlying risk of maternal death </li></ul><ul><li>Instead, we can “hold back” some of the observed data and then see how well our model, fit to the remaining data, does in predicting the held back data points </li></ul>
  5. 5. What do we care about? <ul><li>We care that our model can: </li></ul><ul><ul><li>Predict for gaps in time </li></ul></ul><ul><ul><ul><li>For country-years that are missing in the middle of the time series </li></ul></ul></ul><ul><ul><li>Predict out of time (i.e. forecast and backcast) </li></ul></ul><ul><ul><ul><li>For countries where we have only a partial time series </li></ul></ul></ul><ul><ul><li>Predict for countries with no data </li></ul></ul>
  6. 6. Predictive validity <ul><li>We can construct four types of predictive validity test to validate our model </li></ul><ul><li>The basic idea: </li></ul><ul><ul><li>Sample 20% of the data, depending on what type of test you want to conduct </li></ul></ul><ul><ul><ul><li>Randomly sample 20% of country-years with data </li></ul></ul></ul><ul><ul><ul><li>Randomly sample 20% of countries with data </li></ul></ul></ul><ul><ul><ul><li>Hold out the first 20% of years of data for all countries </li></ul></ul></ul><ul><ul><ul><li>Hold out the last 20% of years of data for all countries </li></ul></ul></ul>
  7. 7. Predictive validity <ul><li>We can construct four types of predictive validity test to validate our model </li></ul><ul><li>The basic idea: </li></ul><ul><ul><li>Sample 20% of the data, depending on what type of test you want to conduct </li></ul></ul><ul><ul><li>Estimate the model in the remaining 80% of data </li></ul></ul><ul><ul><li>Using the model from step (2), predict into the 20% hold-out sample </li></ul></ul><ul><ul><li>Calculate metrics of fit to determine how well the model did predicting the observed data in the 20% hold-out sample </li></ul></ul>
  8. 8. Predictive validity <ul><li>We repeat steps 1-3 30 times for the test of gaps in time and countries with no data, to make sure our results are not an artifact of a given random sample </li></ul><ul><ul><li>Sample 20% of the data, depending on what type of test you want to conduct </li></ul></ul><ul><ul><li>Estimate the model in the remaining 80% of data </li></ul></ul><ul><ul><li>Using the model from step (2), predict into the 20% hold-out sample </li></ul></ul><ul><ul><li>Calculate metrics of fit to determine how well the model did predicting the observed data in the 20% hold-out sample </li></ul></ul>
  9. 9. Predictive validity results: comparing the linear and spatio-temporal models 20% of Countries Regression Root Mean SE* Root Median SE Mean RE** Median RE Linear 214.84 27.00 0.604 0.417 Spatio-Temporal 189.27 25.34 0.521 0.357 First 20% of Country Years Regression Root Mean SE Root Median SE Mean RE Median RE Linear 208.28 22.04 0.702 0.437 Spatio-Temporal 129.32 11.92 0.392 0.199 Last 20% of Country Years Regression Root Mean SE Root Median SE Mean RE Median RE Linear 158.86 13.23 0.538 0.421 Spatio-Temporal 104.08 7.46 0.284 0.213 Random 20% of Country Years Regression Root Mean SE Root Median SE Mean RE Median RE Linear 215.44 24.22 0.619 0.419 Spatio-Temporal 125.34 10.36 0.286 0.165 * SE = Squared Error ** RE = Relative Error
  10. 10. Outline <ul><li>Predictive validity </li></ul><ul><li>Uncertainty </li></ul><ul><li>Comparisons with new WHO model </li></ul>
  11. 11. Uncertainty <ul><li>Uncertainty is the “life preserver” for any researcher! </li></ul><ul><li>While uncertainty intervals are sometimes ignored by policy-makers, they are crucial when interpreting results </li></ul><ul><li>Identifying and incorporating all relevant types of uncertainty into uncertainty intervals in an empirical way is crucial </li></ul>
  12. 12. What is the objective of uncertainty measurement? This line is the true, underlying risk of maternal death in a sample country, or the “expected value”
  13. 13. What is the objective of uncertainty measurement? But we don’t observe that expected value; we observe particular data points
  14. 14. What is the objective of uncertainty measurement? We want our uncertainty bounds to contain the expected value 95% of the time
  15. 15. What are the sources of uncertainty? <ul><li>Sampling uncertainty </li></ul><ul><li>Non-sampling uncertainty </li></ul><ul><li>Parameter uncertainty </li></ul><ul><ul><li>From the linear model </li></ul></ul><ul><ul><li>From the spatial-temporal local regressions </li></ul></ul><ul><li>Remaining systematic variation </li></ul>
  16. 16. Uncertainty: source 1 <ul><ul><li>Sampling uncertainty </li></ul></ul><ul><ul><li>Any data source will have some degree of associated stochastic sampling error, which must be reflected in any estimates of uncertainty </li></ul></ul><ul><ul><li>We capture this uncertainty by drawing from a binomial distribution with the observed maternal cause fraction as p and the number of trials ( n ) as the total number of observed deaths </li></ul></ul><ul><ul><li>We simulate 100 datasets by drawing from these distributions, and use these to propagate the sampling uncertainty through the modeling process </li></ul></ul>
  17. 17. Sampling uncertainty
  18. 18. Uncertainty: sources 2 and 3 <ul><li>Parameter uncertainty </li></ul><ul><li>The application of a statistical model yields uncertainty in the parameter estimates of the model </li></ul><ul><ul><li>You don’t just get an estimate of the β : you get a β ± a measure of uncertainty </li></ul></ul><ul><ul><li>Here we have two stages of parameter uncertainty </li></ul></ul><ul><ul><ul><li>From the linear model </li></ul></ul></ul><ul><ul><ul><li>From the spatial-temporal local regressions </li></ul></ul></ul>
  19. 19. Simulating for parameter uncertainty <ul><li>For each of the 100 datasets generated: </li></ul><ul><ul><li>Estimate the linear model </li></ul></ul><ul><ul><li>Make five draws from the variance-covariance matrix of the regression β s </li></ul></ul><ul><ul><li>Estimate the spatial-temporal model for each of these draws from the linear model </li></ul></ul><ul><ul><li>Make five draws from the variance-covariance matrix of each of the local regressions </li></ul></ul>
  20. 20. Parameter uncertainty: a simple example Here’s one potential model Here’s another potential model Parameter uncertainty takes into account the different models that could potentially fit the data
  21. 21. Uncertainty: source 4 <ul><li>The fourth source we want to capture is the remaining systematic variation that our model does not explain </li></ul><ul><ul><li>i.e. Education, fertility, etc and spatio-temporal relatedness do not explain all variation in maternal mortality </li></ul></ul><ul><li>However, we cannot estimate the systematic variation directly; the remaining variation consists of three parts </li></ul><ul><ul><li>Systematic variation </li></ul></ul><ul><ul><li>Stochastic variation </li></ul></ul><ul><ul><li>Non-sampling variation </li></ul></ul>
  22. 22. The leftover variation Non-sampling error Systematic error, but we don’t observe the true value This difference could be partially stochastic error, partially non-sampling error and partially non-sampling error
  23. 23. Uncertainty: source 4 <ul><li>We can separate out the stochastic variation from the systematic and non-sampling variation using simulation </li></ul><ul><li>But we have no way to separate out the systematic and non-sampling variation, so to be conservative, we include both </li></ul><ul><ul><li>This will dramatically overestimate our uncertainty as non-sampling variation is quite large </li></ul></ul>
  24. 24. Summarizing uncertainty
  25. 25. Outline <ul><li>Predictive validity </li></ul><ul><li>Uncertainty </li></ul><ul><li>Comparisons with new WHO model </li></ul>
  26. 26. The recent WHO estimates (2010): input data <ul><li>The study divides countries into categories defined by the type of data available in that country </li></ul><ul><ul><li>Group A: Civil registration characterized as complete (63 countries) </li></ul></ul><ul><ul><li>Group B: Other types of data available (85 countries) </li></ul></ul><ul><ul><li>Group C: No national data available (24 countries) </li></ul></ul>
  27. 27. The recent WHO estimates (2010): input data <ul><li>Group A : Civil registration characterized as complete (63 countries total – none of the workshop countries) </li></ul><ul><ul><li>Requirements for inclusion: </li></ul></ul><ul><ul><ul><li>Earliest year of data before 1996, latest year after 2002 </li></ul></ul></ul><ul><ul><ul><li>Data available for more than half the range of years available </li></ul></ul></ul><ul><ul><ul><li>Estimated completeness at more than 85% for all years </li></ul></ul></ul><ul><ul><ul><li>Deaths to ICD-10 R codes did not exceed 20% </li></ul></ul></ul><ul><ul><li>Inflated by a factor of 1.5, unless country-specific adjustments were available </li></ul></ul><ul><ul><ul><li>Based on reports in 15 countries; reported misclassification ranges from 1.08 (Uzbekistan) to 3.2 (El Salvador) </li></ul></ul></ul><ul><ul><li>Maternal and all-cause deaths of unknown age redistributed proportionally over the age range </li></ul></ul><ul><ul><li>VR collapsed into 5 year time periods </li></ul></ul>
  28. 28. The recent WHO estimates (2010): input data <ul><ul><li>Group B : Other types of data available (85 countries, including all workshop countries) </li></ul></ul><ul><ul><li>Sisterhood data </li></ul></ul><ul><ul><ul><li>Assumed fraction of pregnancy-related deaths is understated, up-adjusted by a factor of 1.1 </li></ul></ul></ul><ul><ul><li>Deaths in the HH (including Indian SRS) </li></ul></ul><ul><ul><ul><li>Adjusted upward by a factor of 1.1 </li></ul></ul></ul><ul><ul><li>Other “Special studies” (confidential enquires, “RAMOS”) </li></ul></ul><ul><ul><ul><li>Adjusted upward by a factor of 1.1 </li></ul></ul></ul><ul><li>Group C : No data available (24 countries) </li></ul>
  29. 29. Other WHO adjustments <ul><li>AIDS-related mortality </li></ul><ul><li>Pregnancy related vs. maternal deaths </li></ul>
  30. 30. WHO AIDS adjustment <ul><li>Wanted the dependent variable in the regression model to reflect non-AIDS-related maternal deaths only </li></ul><ul><li>Used unpublished UNAIDS tables on the proportion of total deaths of women aged 15-49 due to AIDS </li></ul><ul><li>Assume the fraction of AIDS deaths that occur during pregnancy that should be counted as maternal deaths, non-AIDS related maternal deaths depending on data source: </li></ul><ul><ul><li>0.1 for pregnancy-related data points </li></ul></ul><ul><ul><li>0.5 for maternal data points </li></ul></ul><ul><li>Use this non-AIDS-related PMDF as the dependent variable in the regression model </li></ul>
  31. 31. WHO Pregnancy-related adjustment <ul><li>Distinction between: </li></ul><ul><ul><li>Pregnancy-related mortality (all deaths occurring during pregnancy up to 42 days after – including incidental deaths) </li></ul></ul><ul><ul><li>Maternal mortality (death related to pregnancy, childbirth or puerperium, both direct and indirect causes) </li></ul></ul><ul><li>Adjust the input non-AIDS PMDF from data sources identifying pregnancy-related mortality: </li></ul><ul><ul><li>By a factor of 0.85 for most of the world </li></ul></ul><ul><ul><li>By a factor of 0.9 in Sub-Saharan Africa </li></ul></ul><ul><ul><li>Based on data from 8 countries </li></ul></ul>
  32. 32. WHO and partners regression-based approach <ul><li>Construct a database of 484 observations (680 total, but exclude 196) </li></ul><ul><li>Use a model to predict maternal mortality for the 109 countries in Group B (non-VR data) and Group C (no data) </li></ul>
  33. 33. WHO regression approach <ul><li>Dependent variable: ln(non-AIDS PMDF) </li></ul><ul><li>Offset: ln(1- a ) where a is the proportion of all AIDS deaths among women aged 15-49 in the population </li></ul><ul><li>Covariates: </li></ul><ul><ul><li>ln(GDP per capita): most data from the WB </li></ul></ul><ul><ul><li>ln(general fertility rate): UNPD </li></ul></ul><ul><ul><li>Coverage of skilled attendant at birth (UNICEF database, filled in using a logit model with time as the only covariate) </li></ul></ul><ul><li>Multi-level regression model with random effects for country and region </li></ul><ul><li>Predicted values for 5-year intervals centered around 1990, 1995, 2000, 2005 and 2008 </li></ul>
  34. 34. WHO counts of all-cause deaths for maternal age women <ul><li>All-cause counts of deaths very different from IHME estimates </li></ul>
  35. 35.
  36. 36.
  37. 37.
  38. 38. All-cause death counts comparison <ul><li>WHO vs. UNPD </li></ul><ul><ul><li>UNPD estimates only available for five year blocks of time (1995, 2000, 2005) </li></ul></ul>
  39. 39.
  40. 40.
  41. 41.
  42. 42. WHO: AIDS-related maternal deaths <ul><li>Given that the dependent variable was non-AIDS PMDF, after estimation, must estimate contribution of AIDS to maternal mortality, and add this back in </li></ul><ul><ul><li>Move from non-AIDS PMDF to total PMDF </li></ul></ul><ul><li>Assume that half of the estimated number of AIDS deaths that occur during pregnancy should be counted as maternal deaths </li></ul><ul><li>Assume the relative risk of dying from AIDS for a pregnant versus non-pregnant woman is 0.4 </li></ul>
  43. 43. IHME and the recent UN estimates   IHME UN (H4) Data Sources 2651 2142 <ul><ul><li>Vital Statistics </li></ul></ul>2186 2010 <ul><ul><li>Surveys </li></ul></ul>204 819** <ul><ul><li>Census </li></ul></ul>46 19 <ul><ul><li>Verbal Autopsy </li></ul></ul>215 113 Scope of Study     <ul><ul><li>Time series </li></ul></ul>1980-2008 1990-2008 <ul><ul><li>Countries </li></ul></ul>181 172 Correction     <ul><ul><li>Misclassification </li></ul></ul>Country specific Correction factor 1.5 (63 countries) <ul><ul><li>Completeness </li></ul></ul>Country specific UN estimates Number of female deaths (15–49) Rajaratnam, 2010 WHO lifetables Estimate based on Model for all countries 118 model & 63 correction factor Model Linear + Space-time Multilevel Dependent variable MM rate (ln) by age group Fraction of MM (log) all ages Treatment of HIV Model-based Estimated deaths separately Covariates     <ul><ul><li>GDP </li></ul></ul>yes yes <ul><ul><li>Education </li></ul></ul>yes no <ul><ul><li>TFR </li></ul></ul>yes yes <ul><ul><li>HIV </li></ul></ul>yes no <ul><ul><li>Health services </li></ul></ul>Neonatal mort SBA Model Validation yes no Uncertainty yes yes
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×