Why different methods generate different
numbers: case study from Ethiopia
John Butterworth, IRC International Water and Sanitation Centre
Key points
• Provider (scheme inventory) and user (household survey) data
are, obviously, fundamentally different
• Ministry of Water and Energy collect one, Central Statistical Agency the
other, but the National WASH Inventory is an exception
• Neither method good or bad, but both have strengths and weaknesses
• Neither are well related to the published targets
• Parallel monitoring processes at global, national, and local levels likely
to persist: may even be more complex post-2015
• Investment in multi-stakeholder reconciliation processes vital
New data
points
available
2013
Focus on JMP estimates
• Data, data, data:
– the most recent data hasn’t always been available and trend
lines are sensitive
– accuracy and representativeness of underlying household
survey data
• Definition of ‘improved water facilities’
• Errors
• Transparency in methodology and calculations
Focus on MoWE estimates pre-2010
• Estimates based on design capacity: outputs rather than outcomes
• Occasional inventories, new schemes added
• Simple, reliable, low cost: provides useful data for frontline service
delivery
• Multiplication factors the Achilles heel: under- and over-estimates
• Overestimates generated by inclusion of non-functional
schemes, clustering
Post-2010 and the National WASH Inventory
• Standard nationwide survey
• Uniquely collected provider and user data
• Beyond water points and households: institutions included
• Estimate made of users within quantity/distance norms and total
users
• Data approved, to be validated, made available and put to use
Where are we? Somewhere in the 40s
• JMP (2011) – 39%
• DHS(2011) – 42%
• MoWE NWI household survey – 45%
• MoWE NWI scheme inventory – 49%
• Bearing in mind water quality issues and risks of slippage, the rural
water supply challenge is bigger than previously acknowledged
Top priorities: what is most important?
• Transparency: methods, access to data
• Put data to use
• Update and maintain the NWI
• Strengthen links with Central Statistical Agency
• Promote monitoring as a research topic
• Investing in communications and multi-stakeholder reconciliation
processes
Key points
• Provider (scheme inventory) and user (household survey) data
are, obviously, fundamentally different
• Ministry of Water and Energy collect one, Central Statistical Agency the
other, but the National WASH Inventory is an exception
• Neither method good or bad, but both have strengths and weaknesses
• Neither are well related to the published targets
• Parallel monitoring processes at global, national, and local levels likely
to persist: may even be more complex post-2015
• Investment in multi-stakeholder reconciliation processes vital

Why different methods generate different numbers: Case study from Ethiopia

  • 1.
    Why different methodsgenerate different numbers: case study from Ethiopia John Butterworth, IRC International Water and Sanitation Centre
  • 3.
    Key points • Provider(scheme inventory) and user (household survey) data are, obviously, fundamentally different • Ministry of Water and Energy collect one, Central Statistical Agency the other, but the National WASH Inventory is an exception • Neither method good or bad, but both have strengths and weaknesses • Neither are well related to the published targets • Parallel monitoring processes at global, national, and local levels likely to persist: may even be more complex post-2015 • Investment in multi-stakeholder reconciliation processes vital
  • 4.
  • 5.
    Focus on JMPestimates • Data, data, data: – the most recent data hasn’t always been available and trend lines are sensitive – accuracy and representativeness of underlying household survey data • Definition of ‘improved water facilities’ • Errors • Transparency in methodology and calculations
  • 6.
    Focus on MoWEestimates pre-2010 • Estimates based on design capacity: outputs rather than outcomes • Occasional inventories, new schemes added • Simple, reliable, low cost: provides useful data for frontline service delivery • Multiplication factors the Achilles heel: under- and over-estimates • Overestimates generated by inclusion of non-functional schemes, clustering
  • 7.
    Post-2010 and theNational WASH Inventory • Standard nationwide survey • Uniquely collected provider and user data • Beyond water points and households: institutions included • Estimate made of users within quantity/distance norms and total users • Data approved, to be validated, made available and put to use
  • 8.
    Where are we?Somewhere in the 40s • JMP (2011) – 39% • DHS(2011) – 42% • MoWE NWI household survey – 45% • MoWE NWI scheme inventory – 49% • Bearing in mind water quality issues and risks of slippage, the rural water supply challenge is bigger than previously acknowledged
  • 9.
    Top priorities: whatis most important? • Transparency: methods, access to data • Put data to use • Update and maintain the NWI • Strengthen links with Central Statistical Agency • Promote monitoring as a research topic • Investing in communications and multi-stakeholder reconciliation processes
  • 10.
    Key points • Provider(scheme inventory) and user (household survey) data are, obviously, fundamentally different • Ministry of Water and Energy collect one, Central Statistical Agency the other, but the National WASH Inventory is an exception • Neither method good or bad, but both have strengths and weaknesses • Neither are well related to the published targets • Parallel monitoring processes at global, national, and local levels likely to persist: may even be more complex post-2015 • Investment in multi-stakeholder reconciliation processes vital

Editor's Notes

  • #3 Acknowledgements: 2 previous seminars during the first phase of RiPPLE (2010 and 2011), a DFID supported research partnership. The MoWE and the NWI team.
  • #4 Acknowledgements: 2 previous seminars during the first phase of RiPPLE (2010 and 2011), a DFID supported research partnership. The MoWE and the NWI team.
  • #5 Figure shows a compilation of two different sets of rural water data.1. MoWE provider data, the officially reported data in the sector used for planning.2. JMP estimates, widely reported globally, based upon household surveys undertaken by the Central Statistical AgencyAround 2000, these different datasets started to diverge significantly, with a difference of over 30 percentage points by 2010. Huge accelerated progress by the sector reflecting the extra investments being made? Or were only the figures accelerating?More recently, JMP results have been significantly revised upwards (to 39% for 2011) while the National WASH Inventory has lead to revision downwards of MoWE results (to 49%). So a 10% difference.
  • #6 Facilities are in use, can include family wells if adequately protectedStatistical representation at national level: Disaggregation does not extend to local level
  • #11 Acknowledgements: 2 previous seminars during the first phase of RiPPLE (2010 and 2011), a DFID supported research partnership. The MoWE and the NWI team.