Data quality in remote monitoring (Mona Fetouh, UNOIOS)
Data Quality in Remote Monitoring A comparative analysis of experiences in Somalia and Eastern BurmaMona Fetouh (Co-Author and Presenter), Christian Balslev-Olesen (Co-Author), and Volker Hüls (Co- Author)
Remote Monitoring Humanitarian space restricted in many situations; risks have increased Both delivery and monitoring of programmes affected Remote monitoring and cross-border approaches required in a number of countries in recent years Increased reliance on local actors Reduced ability to collect and verify information Often results in compromises on data quantity & quality Although remote management is increasingly common, evidence on best practice is still emerging
Comparable Contexts Somalia Classic remote management situation Difficulty of access due to insecurity, heavy reliance on national civil society (NGOs, communities) for aid delivery (still largely managed from neighbouring Kenya) Risk to external monitors; monitoring relies heavily on local partners – risk of bias Eastern Burma Hard to impossible to access from capital due to government restrictions and long-running conflicts between ethnic groups in the East and the government Heavy reliance on civil society (LNGOs, communities) for aid delivery that is managed from neighbouring Thailand Little to no access for INGO staff, including national staff Risk to external monitors, monitoring relies heavily on local partners – risk of bias
Different Data Environments (1) Somalia: Main concern is availability of data. Little opportunity to collect data regularly, even for local partners Local partners have varied capacity to produce quality data, severe education gap results in overall low staff capacity Focus of strengthening objective monitoring was through investment in independent systems (―third party verification‖).
Different Data Environments (2) Eastern Burma: Main concern is management and quality of data Data are abundant, both monitoring data and surveys Local partner capacity is varied but good – strong ethnic and professional exchange with Thai border area; better access to education Information remains in technical silos; local NGOs are largely confined to ethnic areas; access opportunities are often limited to a particular sector. Few mechanisms for independent verification/ triangulation of data. Focus of strengthening objective monitoring was on the quality and verification of information Access is improving due to ceasefire agreements
Comparative Analysis Similar contexts require different approaches Presentation with details on Third party verification in Somalia Quality assurance of monitoring information in Eastern Burma Experiences presented are based on of work of UNICEF (Somalia) and International Rescue Committee and the Border Consortium (Thailand/Burma)
Somalia – Third Party Verification (1) First Level: Information from partners and networks Implementing Partner Reports are main source of primary performance data Reviewed against: Previous track record of partner / confidence level Specific concerns about partner performance Reporting complete and realistic? Specific issues flagged in or apparent from requiring follow-up Comparison to occasional information from staff contacts in the filed (email, telephone)
Somalia – Third Party Verification (2) Second Level: Third Party Verification Flagged issues are scheduled for third party monitoring 3rd Party Systems use field monitors that are not affiliated with any implementing partner often outside of the aid business And therefore can move more freely with less risk are not as qualified to judge details of implementation used mostly for verification of easily obtainable information Third party monitors, are ‗blind‘ tasked to avoid fabrication of reports Were successfully tasked to track leakage of relief goods into markets, including quantities and pricing.
Somalia – Third Party Verification (3) Third Level: Follow up on concerns from Level 1 and 2 Third party information is kept confidential and assessed for the risk level of a particular performance issue. Depending on risk level, issues are taken up with the partner: without revealing source e.g. dedicated open monitoring mission at next opportunity Main reasons for staggering: First level flags issues, but is in itself not sufficient for reliable data Second level is costly, and can be targeted to only flagged issues Third level is costly and not timely, and should only be used with knowledge of problems Low-level third party networks have worked well in Somalia for other purposes, e.g. for food price monitoring
Eastern Burma – StrengtheningMonitoring Quality (1) Correlating data in geographical information systems Main limitation to data correlation / triangulation is sector-based systems (Health, Education, Relief information management systems) Sector IMS are basis for cross-sectoral GIS solution Platform maps service delivery data of all sectors to village location Allows analysis of performance data from all sectors per location Allows sectors to engage in cross-monitoring and data sharing
Eastern Burma – StrengtheningMonitoring Quality (2) Regular surveys are expanded in scope and feed into the information system Key strength of the Eastern Burma programmes is history of conducting regular (sector) surveys. All partners are now supporting the expansion of the scope and the coverage of these surveys. Example: Annual Poverty Survey
Eastern Burma – StrengtheningMonitoring Quality (3) Increased linkages between implementing partners, and improved M&E capacity Effortsin recent years to connect local organizations across sector and ethnic group Discussion on cross-monitoring Comprehensive M&E training for local groups Improved community feedback/village monitoring (also used by Oxfam in Somalia and Tearfund in Afghanistan)
Eastern Burma – StrengtheningMonitoring Quality (4) Post-facto review of health centre logbooks and patient files Simple but innovative example: Can be done remotely and is not time sensitive Reveals substantial information about quality of support and services Initiated by the IRC, and now being expanded to whole sector Variations conceivable for other sectors
Eastern Burma – StrengtheningMonitoring Quality (5) Increased use of photographic and video evidence of implementation Where possible, local NGOs use photography video to document their work. Use of video started with success by one local NGO Provides better representation e.g. of trainings and public awareness activities.
Lessons learned, and looking towardsthe future Similar contexts, different data environments Good examples of when similar contexts warrant different approaches Both contexts are changing rapidly – more access in Somalia, political reforms in Burma Improved monitoring systems instill long-term effects to adapt to these changes—stronger information, increased local capacity Changes in Somalia may make Eastern Burma lessons applicable in near future Experience in Somalia valuable for similar situations elsewhere, e.g. Syria