Systemic M&E discussion paper, version 2 - 9 Oct 2012


Published on

Note: the results of this discussion are available at:

This is the first version of the paper that we will use to promote debate, reflection and progress around the systemic M&E initiative. The initiative’s main objective is to promote a rethink of how we measure our impacts on market systems and their evolution towards more inclusion, productivity and efficiency (i.e. how do we know that the markets systems we work with are actually going to continue reducing poverty and protecting the environment even after we have left the scene).

The paper is a live document and it is intended to evolve with the conversations that donors, academic researchers, and practitioners working in inclusive market development and finance/microfinance development. Most of these conversations will take place in MaFI, in USAID’s Microlinks (23-25 Oct, 2012) and the SEEP 2012 Annual Conference. Your comments and questions are welcome (please use the comments box here).

The systemic M&E is one of the concrete solutions proposed by the MaFI-festo ( to make international development cooperation more facilitation-friendly, and therefore, more cost-effective.

Published in: Self Improvement
1 Comment
  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Systemic M&E discussion paper, version 2 - 9 Oct 2012

  1. 1. Monitoring and measuring change in market systems -rethinking the current paradigm- (Discussion paper1, rev. 22 - 9 Oct 2012)1. IntroductionAs is widely experienced, markets are not static, but in constant flux. The changes are notpredictable and often surprising, even after thorough market analyses and strategic planning.Our interventions turn out results that were either unforeseen or unanticipated. This type ofbehavior creates specific challenges for monitoring and evaluation of developmentinterventions.Changes in markets must be monitored and measured using approaches that embrace theirdynamic and unpredictable nature. Advances in complexity sciences can provide usefulprinciples and concepts to build the foundations of more effective and relevant market andfinancial system evaluations.This paper is intended to promote online and in-person discussions that will bring togetherpractitioners and experts to explore two broad issues: the obstacles and challenges that field practitioners face when trying to monitor and measure changes in market systems using the current linear, top-down, rigid, mechanistic paradigm the principles and guidelines that can bring donors and practitioners together to build systemic M&E frameworks and tools that produce not only relevant evidence about their impacts on markets systems, but also appropriate and timely information for field practitioners to help them navigate the fast changing and unpredictable landscape of market systems.2. Main obstacles and challenges with the current M&E paradigmBased on various online and face-to-face discussions with market development practitioners,three broad issues that encapsulate their major concerns about current M&E practices havebeen identified. While these issues, as described below, paint a rather dire picture of thecurrent M&E paradigm, practitioners do acknowledge that there have been positive andimportant debates and initiatives that are creating fertile spaces for further innovations andimprovements. However, some of such debates and initiatives are still based on a paradigm1 This paper is one of the main inputs to the systemic M&E initiative promoted by SEEP’s MaFI (The MarketFacilitation Initiative). Systemic M&E is one of the solutions proposed in the MaFI-festo, which is a documentthat attempts to synthesize the most pressing issues that MaFI members consider critical and urgent to make theinternational cooperation development system more “facilitation- and complexity-friendly” and, ultimately,more cost-effective. Note: there are still aspects of the MaFI-festo where not all MaFI members agree.This paper was written by Marcus Jenal and Lucho Osorio-Cortes using inputs from MaFI members. It isintended to promote discussion, reflection and debate between development practitioners, donors, policy-makersand researchers. The paper will evolve with the on-line and in-person discussions that will take place in MaFI,USAID’s Microlinks and the SEEP Annual Conference (see the plenary here).The ideas in this paper do not necessarily reflect the opinions of The SEEP Network or fhi360.2 Changed top-down accountability for upward accountability and minor typos corrected. 1
  2. 2. that relies onsimplistic and static worldviews rather than embracing complexityto then buildbetter models to make sense, transform and measure reality.2.1. Issue One: Excessive focus on our direct effects on the poorDonors, parliaments and eventually taxpayers ask development projects to prove their directimpacts on the poor: “how many schools or hospitals did WE build; how many cows or bagsof improved seeds did WE delivered; how many training workshops did WE host and howmany people attended; etc”. The list of the inputs and services that “we” –the developmentagents, can deliver is long.Measuring direct impacts on the “poor” is possible when using a direct delivery approachbecause the relationship between the provider and the receiver is straightforward and easilyverifiable; consequently, it is possible to attribute changes at the level of the “poor” to aproject intervention. Accountability is, thus,translated into an act of keeping track of theresources delivered into the system and demonstrating their positive and direct effect onpoverty indicatorssuch as income and employment levels.Detailed planningand standardized result measurement frameworks are believed to helpdevelopment projects to plan their impact and measure it afterwards.M&E approaches,procedures, and tools focus on collecting information on numbers of beneficiaries andchanges at beneficiary level. Attribution of the changes to project interventions is done usingextrapolations based on alternative scenarios without the project.2.2. Issue Two: Excessive focusonextraction of information and accountabilityto thedonorsThe belief in upward accountability has survived because it gives NGOs, donors, parliamentsand taxpayers a comforting sense of control over the process and certainty about theoutcomes of their development initiatives.This is needed by the whole development“industry” to survive in a demanding political economy exacerbated by global recession.In the reality of the field, practitioners are dealing with highly complex and dynamiceconomic, social and cultural systems and trying to navigate fast-changing and in many casesunpredictable landscapes of opportunities and risks.Instead of supporting project implementation, M&E procedures catering for upwardaccountabilitybecome an additional burden of data collection –most of which is meaningless,irrelevant or outdated, tied to predefined and rigid work plans and log-frames, and decoupledfrom realities and needs on the ground.The focus on changes at beneficiary level has incentivized project managers to directlyintervene for the poor, mostly ignoring the need for changes in the wider system to make theintervention sustainable and scalable. Deliverables and impact targets are defined during theplanning phase incorporated into planning tools and contractually fixed by the donors and theimplementers. There are strong incentives for the implementers to try to control the evolutionof the system to make it fit within the agreed plans, instead of promoting adaptivemanagement that “flows” with and leverages the energies of the system. 2
  3. 3. 2.3. Issue Three: Sustainability understood as longevity of our legacyThe understanding of sustainability as the permanence of what development agents do orprovide to the poor drives the efforts of the current M&E system.The goal is to prove thatpositive changes in poverty indicators will persist beyond the project and by virtue of theproject’s deliverables. The dynamic and complex nature of market systems is, however,incompatible with this approach: prices change, industries strive or collapse, droughtsdevastate crops, new technologies or business models destroy jobs, etc. Too much focus onthe permanence or longevity of what development projectsprovide is distracting donors andimplementers alike from building and measuring the ability of the market system actors tocreate their own solutions towithstand current and future shocks, and adapt to changes thatare very difficult to foresee.Rather than small adaptations to current M&E practices, these issues ask for a radical rethinkthe current M&E paradigm, in particular in the field of inclusive market development wherecomplexity plays a crucial role. Indeed, the thinking that constitutes not only the foundationsof M&E but also of project planning and design has to shift from a mechanistic view on theeconomy and society to an understanding that socio-economic systems are complex anddynamic networks of many different, interconnected and interdependent actors.3. Proposed principles for a systemic M&E frameworkThe following are five principles that are based on complexity theory and on the premise thatsystemic M&E approaches, frameworks, tools and incentives have to be designed and used tomeasure "systemic change".Systemic change is thereby defined as transformations in the structure or dynamics of asystem that leads to impacts on large numbers of people, either in their material conditions orin their behavior.These principles are intended to shed some light on the question of "how would a systemicM&E framework look in practice" and to promote debate and convergence between donors,researchers and practitioners in the field of inclusive market development about how to buildfully-operational and user-friendly systemic M&E frameworks.3.1. In systemic interventions, all beneficiaries are indirect beneficiariesIn systemic interventionsall beneficiaries are indirect from the point of view of the projectbecause their context, relationships and possibilities to access tangible and intangible assetsare affected by changes driven by the market system itself, not by the project.The relatively few market actors with whom a projectengages directly are in factcollaborators. A project enables collaboratorsto test new ideas, demonstrate benefits,mobilize large numbers of other market actors, etc.Collaborators actually work with thefacilitators to transform the structures and dynamics of their market system. 3
  4. 4. If the project wants to achieve impact at scale, it is not feasible for the facilitators to workdirectly with the vast numbers of marginalized actors that constitute the project targetpopulations. Therefore, projects have to aspire to systemic change, reaching targetpopulations indirectly.As a consequence, the focus of accountability has to shift from counting direct beneficiariesand assessingdirect impacts at beneficiary level to a broader view of changesin the structuresand dynamics of the market system with indirect effects on the target populations.Recommendation: shift the accountability focus from counting direct beneficiaries toproving contribution to systemic change and estimating systemic (indirect) beneficiaries.3.2. The deeper the systemic change, the larger and longer lasting the effectsCurrent M&E practices prescribe the use of indicators such as beneficiaries incomes ornumber of new jobs created. These types of indicators detect changes in the stocks and flowsof the system (e.g. how much money farmers have in their pocket, how many new jobs werecreated last month or how much milk a group of farmers sells in one day). These indicatorscan be useful to detect changes in the system but these are changes that take place at the mostsuperficial levels of the system and can therefore be produced with relative ease by thepresence and investments of development agents.Superficial changes in the system can also expose market actors to a high risk of relapsing totheir earlier (pre-project) statesor even worse states such as dependency on donor funding orconflicts fueled by project subsidies.Focusing the attention on these superficialchangescreates a false illusion of success that can easily mislead donors when decidingwhere, how much and when to invest, and practitioners when deciding how and when to act,what and how much to subsidize, and when to exit.It is very important to pay more attention to deeper and more structural changes in the marketsystem, such as the creation of new networks, associations, or business models; increasedaccess to information; shifts in power dynamics; collaboration around jointly agreedobjectives, etc. Information about what goes on at deeper levels of the market system can beused to determine whether the system is changing its trajectory towards a horizon of moreinclusion, productivity and efficiency, and whether it will stick to its new course beyond thelife of the project.Recommendation: put more emphasis on changes at deeper levels of the system3.3. The facilitator does not change the market system; the actors with whom thefacilitatorcollaboratesdoSystemic change affects large numbers of people. The extent of the change and its impactsare shaped by the structure of the system (e.g. legislation, infrastructure or networks) and thecapacity of its actors to use the system’s resources. Deep changes in the system are causednot by the project, but by decisions taken by a large number of actors who are beyond thereach and control of the facilitator. As a consequence, the further away an observed impact isfrom the facilitator’s interventions (i.e. the more links in the chain of events that led tochangesin thesituation or behavior of the actors),the more difficult it becomes to attribute theimpact to the interventions. Systemic M&E frameworks should surrender any attempts to 4
  5. 5. attribute impact at the end of the chain to a specific intervention and focus on the effect of thefacilitator on the groups of immediate collaborators that actually drives the transformations oftheir own market system. These groups of collaborators can be called precursor networks.Precursor networks are any type of informal or formal network or group of actors that createsspaces and routines for market actors to learn from one another; align objectives, strategiesand activities; pool and leverage resources; experiment with new ideas and disseminate thesuccessful ones throughout the system, etc. Precursor networks can be informal farmergroups, cooperatives, chambers of commerce, lead firms, multi-stakeholder platforms,grassroots associations, etc.Three basic types of precursor networks can be proposed: networks that empower marketactors to engage with others, networks that bring different market actors together to transformthe system, and networks that disseminate knowledge throughout the system.A better understanding of how these networks can be created, sustained or improved and howthey drive change throughout the market system will allow development agents to improvetheir interventions and to establish more robust causal links between the actions of facilitatorsand the behavior of these networks (which in turn would allow to infer the facilitators’contributions to improvements in the wider system).Recommendation: assess the effects of the facilitators on the collaborators and infersystemic impact from the performance of the precursor networks.3.4. Complex systems behave in extreme and unpredictable ways and are sensitive to thepresence of the projectThe highly dynamic and unpredictable nature of complex systems undermines theeffectiveness of tools and approaches that are currently used in market development, such aslong-term strategic analysis and planning, log-frames, or fixed indicators and goals. Thesetools and approaches are based on a world of relative stability and predictability, andrepeatability of causes and effects.In order to deal with the dynamism and unpredictability of market systems that we find inreality, new approaches need to be built on the principles of variation, experimentation andadaptation.To implement these approaches, successful projects also require skillfulfacilitators who can sense and adapt to conditions on the ground and respond appropriatelyand swiftly to the actions, intentions and fears of market actors –some of which are generatedby the presence of the project itself.The facilitators’ responsiveness is not only determined by their skills and knowledge but alsoby the organizational environment they belong and are accountable to: their project team,their organization, their professional associations and networks, the donors, and, of course,the M&E frameworks, tools and procedures that the facilitators are required to use.The systemic M&E frameworks required by the field of inclusive market development shouldnot only provide evidence of impact to the key development agents, but also fast, reliable andrelevant information to the facilitators to help them navigate the “systemic jungle”effectively, for example: taking advantage of unforeseen opportunities, minimizing conflicts,leveraging the resources of public and private market actors, minimizing operational costs, 5
  6. 6. contributing to deeper impacts, etc. Systemic M&E is needed not just for “proving” but alsofor “improving”.Recommendation:invest in M&E frameworks, procedures and incentives that, besidesproving the effectiveness of the project, enable facilitators on the ground to adapt, learnand collaborate effectively.3.5. Adaptability is the ultimate manifestation of systemic sustainability:As mentioned above, one key aspect of systemic M&E frameworks is the focus on how thefacilitators are influencing the performance of precursor networks and how these networksare moving the system towards a more desirable horizon. Additionally, it is important toassess if the system is building the necessary conditions to avoid or minimize future shocksand benefit from new trends whilst staying inclusive, productive and efficient.Predicting how a market system will react to a shock is practically impossible.It is, however,possible to assess if a system is becoming more or less adaptable –and therefore moresustainable.A better understanding of how a market system builds and maintain resilience andadaptability would provide important information to build systemic M&E frameworks thatpay attention to the structures, dynamics and parameters that would allow developmentagents to assess trends towards more or less sustainability.Recommendation: do more research about the components and drivers of resilience andadaptability of different types of market systems (sectors and subsectors) in differentcontexts and design M&E frameworks to detect these changes. 6