Monitoring-fit for purpose?
•Download as PPTX, PDF•
1 like•852 views
By Rachel Norman, PhD researcher and Dr Richard Franceys, senior lecturer. Prepared for the Monitoring sustainable WASH service delivery symposium, Addis Ababa, Ethiopia, 9-11 April 2013.
Recommended
More Related Content
Similar to Monitoring-fit for purpose?
Similar to Monitoring-fit for purpose? (20)
More from IRC
More from IRC (20)
Recently uploaded
Recently uploaded (20)
Monitoring-fit for purpose?
- 1. Monitoring – Fit for purpose? Rachel Norman, PhD Researcher & Dr Richard Franceys, Senior Lecturer 1
- 2. The Research Inquiry WASH Sector Service Delivery WHAT IS CONSIDERED AN APPROPRIATE LEVEL OF M&E TO ENSURE AN ANSWER OF YES TO THE FOLLOWING QUESTIONS? Is it there? Is it working continuously? Is it used by all? Is it having an impact? Where does the responsibility lie in Does an appropriate level of M&E Is the implementation of M&E terms of defining what is appropriate? currently exist in the sector? ensuring improved service delivery? WHAT WHY HOW COST USE 2
- 3. Research Overview Research Aim Objectives To understand better how much WASH sector stakeholders 1. To identify the evolution of M&E. are investing in monitoring and evaluation and how the data 2. To examine the conceptual framework of Cost of M&E. generated is being used. 3. To examine budgeted and absolute costs of M&E at global, national and programme level. 4. To explore the underlying purpose and use of M&E. 5. To examine the conceptual framework of VfM and Working Hypothesis identify whether there is a core set of indicators. Monitoring and evaluation in the WASH sector in lower- 6. To understand the potential for harmonisation and income countries is not value for money …… because it is not alignment with the SDGs and HR frameworks. used. Case Study Strategy – Global / Kenya / Uganda Mixed method design whereby qualitative data is used to validate quantitative data collection, analysis and visualisation 3
- 4. Data collection, analysis and visualisation LITERATURE REVIEWS DOCUMENT & DATA INTERVIEWS SURVEYS ARCHIVE REVIEWS • Approx. 90 stakeholders identified • Approx. 900 academic articles sourced and reviewed • Approx.1000 (Global, Kenya, Uganda), documents & data archives sourced and analysed: • Approx. 90 interviews carried out. 4
- 5. What are monitoring and evaluating? Literature shows: • 1960s: Coverage data • 1980s: Minimum Evaluation Procedures • 2000s: Guidelines and Toolkits providing hundreds of indicators to choose to monitor 5
- 6. What are monitoring and evaluating? Country level data analysis shows: • Increasing number of indicators being reported over time. 6
- 7. Where are the indicators coming from? Kenya: Uganda: National: NIMES & KJAS National: NIMES & JAF & Baraza Sector: SWaP, JAR, Regulator Sector: SWaP, JSR, JTR Programmes and Projects: Organisations, institutions, agencies National and International public, private and civil society organisations 7
- 8. Clustered data •Is it there? Service •Is it working? •Is it being used by the right (Consumer) people? •Is it having an impact? Service Provider •Regional & Local (Regional & Local) • Approximately 50% of Sector •National indicator entries relate to (National) Service level data • Coverage is still the most frequently reported indicator • Functionality reporting is limited 8
- 9. Why are we monitoring and evaluating? KENYA UGANDA • ~ 50% of data records • < 50% of data records reported a purpose reported a purpose 9
- 10. How is M&E carried out? Theory • Academic research • 644,000 articles of which 3% related to WASH • 79 different tools methods and approaches of which 57% related to WASH such as: • Randomised controlled trials • Cost-benefit analysis • Benchmarking • Needs assessments 10
- 11. How is M&E carried out? (cont.) Practically WASH M&E in pictures? • Practical application • Driven by accountability requirements • Lack of understanding of the options 11
- 12. What is the cost of M&E? • M&E…. ‘are expensive activities yet there has been little analysis of whether they provide value for money’ (Maddock, 1993) • Information systems….. ‘because of the lack of published research in this area, no established methodology could readily be adapted to this study…’ (Rommelmann et al, 2003) • Multiplicity of M&E systems: ‘they also have a cost in terms of the muddle that it creates…’ (Anonymous, 2011) 12
- 13. What is the current spending on M&E? • Possible spending of sector budget on M&E - Kenya • MWI Sector Actual / Budget figures are sourced from 2011 Environment Water & Irrigation Sector Report 2010 • Possible spending of sector budget on M&E - Uganda • WSS sub-sector figures are sourced from MWE (2010) SPR and MWE (2011) SPR 13
- 14. Is information actually USED to improve service delivery? Very limited reported/documented information on use: Used for internal and external accountability Used for internal management decisions Stakeholder interviews suggest use: • Through learning and knowledge/information sharing • Policy formulation and to inform new planning • Performance monitoring. 14
- 15. Analysis of stakeholder interviews on Use Kenya Uganda 15
- 16. Discussion • Who, What and Why? • Common goal and yet multiplicity of indicators • Internally or externally driven or supported? • The definition crisis causing duplication • Who, How and Why? • Capacity and capability is an issue • Research and development is an opportunity • Roles and responsibilities are a requisite 16
- 17. Discussion (cont.) • What Cost • Accountability and yet no accountability • Too much or too little – difficult to quantify? • Embedded or extracted? • How Used • Limited resources – a repetitive explanation • Used but no audit trail • Lead time on use 17
- 18. Interim Conclusions • What is considered an appropriate level of M&E? • Where does the responsibility lie in terms of defining what is appropriate? • Does an appropriate level of M&E currently exist in the WASH sector? • Is the implementation of M&E ensuring improved service delivery? 18
Editor's Notes
- Context:Global water and sanitation targets have been set and re-set over the last, at least 50 years – the aim to save lives and achieve economic growth. And, in order to determine progress in the WASH sector, there is a need for an appropriate level of M&E to be able to answer the questions ‘has the service been constructed?’, ‘is it working continuously?’, ‘is it being used?’ (by the right people – all) and ‘is it having the desired impact?’.But what is considered an appropriate level of M&E, what are the key components that make M&E effective and how does that translate to reality on the ground? Just some of the questions which led me to establishing the aim: to understand better cost and use of M&E in the WASH sector in lower-income countries. It became clear, early on, that to examine cost and use, I also need to understand the what, why and how.
- In terms of research methodology I am following a case study strategy, with a mixed method design whereby qualitative data is used to capture multiple perspectives and support or challenge reported data. The project has evolved since 2010 and in response to the increasing dialogue in terms of post-2015 MDGs – SDGs and the ‘progressive realisation of HR’, an additional objective added – to examine the potential for the harmonisation and alignment of M&E with the SDGs.Descriptive statistics, data reduction techniques, coding, clustering, counting, content analysis.Graphs, charts, matrices, and vignettes.
- So in terms of the data... through different sampling techniques I am analysing data from approximately 90 stakeholders covering public, private and civil society organisations from individuals, programmes, sector, national and global levelsThe literature review comprises approximately 900 academic articles Approximately 1000 M&E related document and data records relating to Global, Kenya and Uganda have been collected of which between 40 % and 50% contain information on the What, Why and How whereas cost and use is much lower.To date approximately 90 key informant and semi-structured interviews have also been carried out across the case studies whilst the surveys are scheduled for later this year and will be used to support existing quantitative and qualitative data.
- The literature highlights since the 1960s donor activity and funding has increased year on year resulting in the increased complexity of the global aid architecture, including what we should be monitoring.It also highlights that back in the 1960s the what was more simply focused on coverage data and it wasn’t really until the 1980s with the introduction of the MEP, where M&E considered, the ‘core’ questions of is it functioning, is it used and is it having impact – but what happened to the MEP? From the 2000s at a global level we have the opportunity, with frameworks such as the Country Status Overview, to report against something like 108 indicators, or can we can chose from development partner indicator handbooks (2011 AfDB Handbook on infrastructure statistics – primary watsan 50 – several hundred / 2011 EU toolkit). Even as we continue consultations and move towards the SDGs beneath the 4 targets, as recently proposed, there are a potential 44 indicators. Note – reference in one of the sessions yesterday, to another tool, acknowledge the potential of hundreds of indicators...
- In practical terms and taking a look at the two case studies the data analysis shows an increasing number of indicator entries per data record being reported over time.
- And whilst Kenya has its 9 core indicators reported against for the sector by the regulator, and Uganda has its 11 golden indicators, indicating a focused monitoring approach, however, acknowledgement needs to be given to two aspects:1st – some of these indicators are ratios / composite indicators and are made up of more than one data point – and 2nd – this is not the only sector related monitoring process on-going in the country. Other public sector related organisations are also collecting data, relevant to their own organisational objectives as is the private sector and civil society organisation and individual projects and programmes also collect data. As we have heard repeatedly over the course of the last few days different levels have different purposes and different data collection needs.Combining the data extracted from the documents and data records and using a simple filter tool on Excel to see the number of different indicators being reported against. For Kenya: 60% of 1404 indicators were reported against once indicating a variety of data being reported against.For Uganda: 18% of the 5644 indicators were reported against once indicating a certain level of consistency.In both cases as the frequency of entries increased, the number of identical indicators being reported against reduced. However, through visual inspection taking entries at random, similarities emerged and so I decided to apply another analysis method - clustering.
- By clustering the data, into Service, Service Provider and Sector level indicators, the results show that in both cases coverage is still the most frequently reported indicator, and reporting functionality remains low as does reporting use and impact, particularly in the case of Uganda. As we consider the ‘what’ moving forward and recognising the common goal of sustainable services, we need to consider if and how data collection activities can be aligned and whether there is a core set of recurring indicators that can be applied to ensure M&E is fit for purpose?Is it simply an emphasis change at service levelthat is needed, for example from coverage to functionality or to water quality and safety or equity, or is the shift needed at service provider or sector level, or even a combination? and what is really influencing the purpose of M&E which is leading to the setting of theseindicators?Notes: Kenya impact 11/27 = Health; Others include: Economic; Service delivery; ConflictUganda impact 21/24 = Health
- When we consider Purpose – the why, we are referring to ‘Intended use’.The literature review identified 3 different ways to categorise purpose to which I used to analyse the information on intended use as extracted from the data records.Casley & Kumar (1998) – description, explanation, prediction IPDET (2007) – ethical, managerial, decisional motivationalCotton & Bartam (2008); Gosling & Edwards (2003) - levelsInterviews supported the review of data records that description and explanation are the two predominant reasons why M&E is undertaken. The findings also suggest that purpose is a combination of both internal (such as management decision making) and external (depicted as external conditionality and accountability) rather than one or the other. As an acknowledgement, it is important to note that just because the ‘Why’ is not reported or documented, does not mean that there is not a purpose behind M&E. That said, I have also received feedback that suggests otherwise.Does this provide evidence of a link between the why and what and high number of coverage indicators
- In theory, again through the literature review one fairly comprehensive academic database shows the limited research being carried out in terms of M&E in the WASH sector and yet a large number of tools, methods and approaches covered. Grey literature also whilst, providing some toolkits, there does not appear to be any comprehensive review or cost benefit analysis of these 79 M&E tools, methods and approaches (to which, I think I can probably add a few more to the list!) available as far as I have been able to find let alone specific to the sector.Another aspect to consider is that over the last 50 years of development decades, there has been a continuing emphasis on supporting national M&E systems however, only in more recent years under the label of governance, have development partners been directly funding national and sector level M&E.
- In practical terms, interim findingsIndicate/Support the theory that there is a lack of access to information, confidence and understanding of the options Highlight a link between purpose and how particularly in terms of financial accountability toolsAs well as identify the externally driven introduction of new and innovative technology by development partners in an attempt to ease and support the data management processHowever, is there sufficient information and analysis of the benefits of the various tools, methods and approaches, to understand whether the How of M&E is fit for purpose?
- The references on the slide highlight the continued & unanswered question of cost, dating back some 20+ years. Only very recently – and increasingly in the last few months - has the discourse of cost of M&E in the WASH sector been reported through forums like this and the collation of material by RWSN/WaterAid on water point mappingThe research analysis carried out so far highlight a variation of cost typologies being used to report cost, such as percentage figures over base rates (1%-5%), through to descriptive characteristics (staffing salaries, travel and accommodation, equipment), and ranking formats (low, medium, high costing). Whilst being quite different frameworks there is a common thread that pulls these together: the ad-hoc and unsubstantiated nature of the budgeting and costing process. This sporadic nature is perceived as a consequence of a number of possible factors but includes the absence of a rigorous, evidence based understanding or definition of what is deemed an appropriate cost of M&E.
- As an example, if we were to apply the proposed percentage figures of 1, through to 5% to the sector and sub-sector budgets the variance is quite considerable so how should we determine what is reasonable and appropriate. If we go one step further and apply the suggestion that the average cost of an evaluation is approximately 400k, the number of evaluations that could have been carried out, in Uganda, for 2011, range between 2 and 8 depending on whether 1% or 5% of sector budget has been allocated – are two evaluations per year considered reasonable for the sub-sector? Maybe it is, although, I would also like to highlight that this calculation is done so against an average cost of an evaluation alone and does not take into account the cost of monitoring for the year. As we know continuous monitoring is considered a pre-requisite for an effective and efficient evaluation.
- Now to use – the research is examining whether data and information generated by M&E activities really Used and whether is it having an impact on Service delivery?In short, qualitative data analysis suggests M&E information is used however, finding an audit trail of such use and linking it back to a report is hard to find.
- Putting to one side the To Be Confirmed, in the case of Kenya, as the analysis remains a little ‘fuzzy’, the findings for Uganda shows a wide response in terms of actual use and barriers to use in Uganda.
- So in summary when considering who, what & why, Despite having a common goal there is an increasing number of indicators being reported against and there is also evidence to suggest thatnot all of the data being collected is being analysed or reported.And yet there seems to be an interest to establish and agree on a core set of recurring data points to reduce the burden but that this interest needs to be internally and externally demand driven and supported.When considering the who, how & why,Recognising the difference in capacity and capability of M&E, further R&D on the ‘how’ would provide an opportunity to enhance M&E effectiveness But it also needs to go hand in hand with a clear understanding of roles and responsibilities, perhaps also having its foundations in national legislation and regulation.
- The cost component is an interesting one –There is a consensus of agreement that conditionality, transparency, and audits are primarily focused on financial accountability which in turn is valued and yet, there is little if any accountability of the cost of M&E. Some say too much is spent across the levels of M&E particularly when the data and information generated is either not analysed, or used effectively….. Others say not enough is spent particularly when the budgets for M&E are not increasing at the same rate as implementation budgets So, how can we determine what is too much and what is too little when we have such a limited evidence base?Here in lies the difficulty - in quantifying the cost of M&E – what typology do you use to frame cost? How to extract out specific budget lines when monitoring (and evaluation) is a process, but then is a % figure really appropriate either?Linked to cost is use, as insufficient resource is directly and in-directly stated as a reason for limited use. The long lead time from data collection to use is also stated as a reason for limited use – and whilst this latter issue may be being addressed through the introduction of new technologies this is only one part of the process – data acquisition you still need to analyse and use the data… have we learned enough about previous technology advancement or are we at risk of simply adding to the increasing number of tools, methods and approaches?
- So, when we consider what is an appropriate level of M&E the evidence suggests that each of six components Who, What, Why, How, Cost and Use, needs to be considered as the sum of all the parts spanning the entire M&E process.There seems to be an indication……….that we don’t yet have the balance right in terms of roles and responsibilities for monitoring within the sector. Whilst some of this imbalance may be due to variation in governance capability and capacity whilst other factors such as the complexity around the integrated nature of service delivery are also seen as a barrier.Without a rigorous evidence base how can we determine whether M&E is appropriate in the WASH sector today?Despite having some examples where M&E information is leading to improved service delivery, our knowledge is seemingly a little ad-hoc.
- So when we talk about whether something is fit for purpose, we are considering “the extent to which data produced by a measurement process enables a user to make technically (& administratively) correct decisions for a stated purpose” (Thompson & Ramsey, 1995)Perhaps to move toward M&E being fit for purpose we need:A minimum set of procedures: so that instead of looking in isolation at whether we are capturing everything we should be monitoring and adding to the list of what ‘should be monitored’, or instead of developing another tool to improve data acquisition because the advancement of technology means that we can…. we should aim to understand better what is ‘sufficient’ in all aspects of the process to answer the question, or solve the problem being considered.A data quality standard: There are no international standards for M&E for the WASH sector.A data accreditation process: We know need to strengthen data quality and that the world is a place with different governance capacities and capabilities, which also extends to information governance capacities and capabilities.In achieving this, our awareness of the extent to which M&E is contributing to saving lives and achieving economic growth is ultimately better understood. In turn this would also enable a better understanding of the extent to which M&E is providing value for money across the sector and where harmonisation and alignment can, could or should be improved or confidently applied.