Note: the results of this discussion are available at: http://www.slideshare.net/marketfacil/systemic-mand-e-synthesis-31jan2013 This is the first version of the paper that we will use to promote debate, reflection and progress around the systemic M&E initiative. The initiative’s main objective is to promote a rethink of how we measure our impacts on market systems and their evolution towards more inclusion, productivity and efficiency (i.e. how do we know that the markets systems we work with are actually going to continue reducing poverty and protecting the environment even after we have left the scene). The paper is a live document and it is intended to evolve with the conversations that donors, academic researchers, and practitioners working in inclusive market development and finance/microfinance development. Most of these conversations will take place in MaFI, in USAID’s Microlinks (23-25 Oct, 2012) and the SEEP 2012 Annual Conference. Your comments and questions are welcome (please use the comments box here). The systemic M&E is one of the concrete solutions proposed by the MaFI-festo (http://slidesha.re/mafifesto2) to make international development cooperation more facilitation-friendly, and therefore, more cost-effective.

1. Monitoring and measuring change in market systems
-rethinking the current paradigm-
(Discussion paper1, rev. 22 - 9 Oct 2012)
1. Introduction
As is widely experienced, markets are not static, but in constant flux. The changes are not
predictable and often surprising, even after thorough market analyses and strategic planning.
Our interventions turn out results that were either unforeseen or unanticipated. This type of
behavior creates specific challenges for monitoring and evaluation of development
interventions.
Changes in markets must be monitored and measured using approaches that embrace their
dynamic and unpredictable nature. Advances in complexity sciences can provide useful
principles and concepts to build the foundations of more effective and relevant market and
financial system evaluations.
This paper is intended to promote online and in-person discussions that will bring together
practitioners and experts to explore two broad issues:
the obstacles and challenges that field practitioners face when trying to monitor and
measure changes in market systems using the current linear, top-down, rigid,
mechanistic paradigm
the principles and guidelines that can bring donors and practitioners together to build
systemic M&E frameworks and tools that produce not only relevant evidence about
their impacts on markets systems, but also appropriate and timely information for
field practitioners to help them navigate the fast changing and unpredictable
landscape of market systems.
2. Main obstacles and challenges with the current M&E paradigm
Based on various online and face-to-face discussions with market development practitioners,
three broad issues that encapsulate their major concerns about current M&E practices have
been identified. While these issues, as described below, paint a rather dire picture of the
current M&E paradigm, practitioners do acknowledge that there have been positive and
important debates and initiatives that are creating fertile spaces for further innovations and
improvements. However, some of such debates and initiatives are still based on a paradigm
1
This paper is one of the main inputs to the systemic M&E initiative promoted by SEEP’s MaFI (The Market
Facilitation Initiative). Systemic M&E is one of the solutions proposed in the MaFI-festo, which is a document
that attempts to synthesize the most pressing issues that MaFI members consider critical and urgent to make the
international cooperation development system more “facilitation- and complexity-friendly” and, ultimately,
more cost-effective. Note: there are still aspects of the MaFI-festo where not all MaFI members agree.
This paper was written by Marcus Jenal and Lucho Osorio-Cortes using inputs from MaFI members. It is
intended to promote discussion, reflection and debate between development practitioners, donors, policy-makers
and researchers. The paper will evolve with the on-line and in-person discussions that will take place in MaFI,
USAID’s Microlinks and the SEEP Annual Conference (see the plenary here).
The ideas in this paper do not necessarily reflect the opinions of The SEEP Network or fhi360.
2
Changed top-down accountability for upward accountability and minor typos corrected.
1

2. that relies onsimplistic and static worldviews rather than embracing complexityto then build
better models to make sense, transform and measure reality.
2.1. Issue One: Excessive focus on our direct effects on the poor
Donors, parliaments and eventually taxpayers ask development projects to prove their direct
impacts on the poor: “how many schools or hospitals did WE build; how many cows or bags
of improved seeds did WE delivered; how many training workshops did WE host and how
many people attended; etc”. The list of the inputs and services that “we” –the development
agents, can deliver is long.
Measuring direct impacts on the “poor” is possible when using a direct delivery approach
because the relationship between the provider and the receiver is straightforward and easily
verifiable; consequently, it is possible to attribute changes at the level of the “poor” to a
project intervention. Accountability is, thus,translated into an act of keeping track of the
resources delivered into the system and demonstrating their positive and direct effect on
poverty indicatorssuch as income and employment levels.
Detailed planningand standardized result measurement frameworks are believed to help
development projects to plan their impact and measure it afterwards.M&E approaches,
procedures, and tools focus on collecting information on numbers of beneficiaries and
changes at beneficiary level. Attribution of the changes to project interventions is done using
extrapolations based on alternative scenarios without the project.
2.2. Issue Two: Excessive focusonextraction of information and accountabilityto the
donors
The belief in upward accountability has survived because it gives NGOs, donors, parliaments
and taxpayers a comforting sense of control over the process and certainty about the
outcomes of their development initiatives.This is needed by the whole development
“industry” to survive in a demanding political economy exacerbated by global recession.
In the reality of the field, practitioners are dealing with highly complex and dynamic
economic, social and cultural systems and trying to navigate fast-changing and in many cases
unpredictable landscapes of opportunities and risks.
Instead of supporting project implementation, M&E procedures catering for upward
accountabilitybecome an additional burden of data collection –most of which is meaningless,
irrelevant or outdated, tied to predefined and rigid work plans and log-frames, and decoupled
from realities and needs on the ground.
The focus on changes at beneficiary level has incentivized project managers to directly
intervene for the poor, mostly ignoring the need for changes in the wider system to make the
intervention sustainable and scalable. Deliverables and impact targets are defined during the
planning phase incorporated into planning tools and contractually fixed by the donors and the
implementers. There are strong incentives for the implementers to try to control the evolution
of the system to make it fit within the agreed plans, instead of promoting adaptive
management that “flows” with and leverages the energies of the system.
2

3. 2.3. Issue Three: Sustainability understood as longevity of our legacy
The understanding of sustainability as the permanence of what development agents do or
provide to the poor drives the efforts of the current M&E system.The goal is to prove that
positive changes in poverty indicators will persist beyond the project and by virtue of the
project’s deliverables. The dynamic and complex nature of market systems is, however,
incompatible with this approach: prices change, industries strive or collapse, droughts
devastate crops, new technologies or business models destroy jobs, etc. Too much focus on
the permanence or longevity of what development projectsprovide is distracting donors and
implementers alike from building and measuring the ability of the market system actors to
create their own solutions towithstand current and future shocks, and adapt to changes that
are very difficult to foresee.
Rather than small adaptations to current M&E practices, these issues ask for a radical rethink
the current M&E paradigm, in particular in the field of inclusive market development where
complexity plays a crucial role. Indeed, the thinking that constitutes not only the foundations
of M&E but also of project planning and design has to shift from a mechanistic view on the
economy and society to an understanding that socio-economic systems are complex and
dynamic networks of many different, interconnected and interdependent actors.
3. Proposed principles for a systemic M&E framework
The following are five principles that are based on complexity theory and on the premise that
systemic M&E approaches, frameworks, tools and incentives have to be designed and used to
measure "systemic change".
Systemic change is thereby defined as transformations in the structure or dynamics of a
system that leads to impacts on large numbers of people, either in their material conditions or
in their behavior.
These principles are intended to shed some light on the question of "how would a systemic
M&E framework look in practice" and to promote debate and convergence between donors,
researchers and practitioners in the field of inclusive market development about how to build
fully-operational and user-friendly systemic M&E frameworks.
3.1. In systemic interventions, all beneficiaries are indirect beneficiaries
In systemic interventionsall beneficiaries are indirect from the point of view of the project
because their context, relationships and possibilities to access tangible and intangible assets
are affected by changes driven by the market system itself, not by the project.
The relatively few market actors with whom a projectengages directly are in fact
collaborators. A project enables collaboratorsto test new ideas, demonstrate benefits,
mobilize large numbers of other market actors, etc.Collaborators actually work with the
facilitators to transform the structures and dynamics of their market system.
3

4. If the project wants to achieve impact at scale, it is not feasible for the facilitators to work
directly with the vast numbers of marginalized actors that constitute the project target
populations. Therefore, projects have to aspire to systemic change, reaching target
populations indirectly.
As a consequence, the focus of accountability has to shift from counting direct beneficiaries
and assessingdirect impacts at beneficiary level to a broader view of changesin the structures
and dynamics of the market system with indirect effects on the target populations.
Recommendation: shift the accountability focus from counting direct beneficiaries to
proving contribution to systemic change and estimating systemic (indirect) beneficiaries.
3.2. The deeper the systemic change, the larger and longer lasting the effects
Current M&E practices prescribe the use of indicators such as beneficiaries' incomes or
number of new jobs created. These types of indicators detect changes in the stocks and flows
of the system (e.g. how much money farmers have in their pocket, how many new jobs were
created last month or how much milk a group of farmers sells in one day). These indicators
can be useful to detect changes in the system but these are changes that take place at the most
superficial levels of the system and can therefore be produced with relative ease by the
presence and investments of development agents.
Superficial changes in the system can also expose market actors to a high risk of relapsing to
their earlier (pre-project) statesor even worse states such as dependency on donor funding or
conflicts fueled by project subsidies.Focusing the attention on these superficial
changescreates a false illusion of success that can easily mislead donors when deciding
where, how much and when to invest, and practitioners when deciding how and when to act,
what and how much to subsidize, and when to exit.
It is very important to pay more attention to deeper and more structural changes in the market
system, such as the creation of new networks, associations, or business models; increased
access to information; shifts in power dynamics; collaboration around jointly agreed
objectives, etc. Information about what goes on at deeper levels of the market system can be
used to determine whether the system is changing its trajectory towards a horizon of more
inclusion, productivity and efficiency, and whether it will stick to its new course beyond the
life of the project.
Recommendation: put more emphasis on changes at deeper levels of the system
3.3. The facilitator does not change the market system; the actors with whom the
facilitatorcollaboratesdo
Systemic change affects large numbers of people. The extent of the change and its impacts
are shaped by the structure of the system (e.g. legislation, infrastructure or networks) and the
capacity of its actors to use the system’s resources. Deep changes in the system are caused
not by the project, but by decisions taken by a large number of actors who are beyond the
reach and control of the facilitator. As a consequence, the further away an observed impact is
from the facilitator’s interventions (i.e. the more links in the chain of events that led to
changesin thesituation or behavior of the actors),the more difficult it becomes to attribute the
impact to the interventions. Systemic M&E frameworks should surrender any attempts to
4

5. attribute impact at the end of the chain to a specific intervention and focus on the effect of the
facilitator on the groups of immediate collaborators that actually drives the transformations of
their own market system. These groups of collaborators can be called precursor networks.
Precursor networks are any type of informal or formal network or group of actors that creates
spaces and routines for market actors to learn from one another; align objectives, strategies
and activities; pool and leverage resources; experiment with new ideas and disseminate the
successful ones throughout the system, etc. Precursor networks can be informal farmer
groups, cooperatives, chambers of commerce, lead firms, multi-stakeholder platforms,
grassroots associations, etc.
Three basic types of precursor networks can be proposed: networks that empower market
actors to engage with others, networks that bring different market actors together to transform
the system, and networks that disseminate knowledge throughout the system.
A better understanding of how these networks can be created, sustained or improved and how
they drive change throughout the market system will allow development agents to improve
their interventions and to establish more robust causal links between the actions of facilitators
and the behavior of these networks (which in turn would allow to infer the facilitators’
contributions to improvements in the wider system).
Recommendation: assess the effects of the facilitators on the collaborators and infer
systemic impact from the performance of the precursor networks.
3.4. Complex systems behave in extreme and unpredictable ways and are sensitive to the
presence of the project
The highly dynamic and unpredictable nature of complex systems undermines the
effectiveness of tools and approaches that are currently used in market development, such as
long-term strategic analysis and planning, log-frames, or fixed indicators and goals. These
tools and approaches are based on a world of relative stability and predictability, and
repeatability of causes and effects.
In order to deal with the dynamism and unpredictability of market systems that we find in
reality, new approaches need to be built on the principles of variation, experimentation and
adaptation.To implement these approaches, successful projects also require skillful
facilitators who can sense and adapt to conditions on the ground and respond appropriately
and swiftly to the actions, intentions and fears of market actors –some of which are generated
by the presence of the project itself.
The facilitators’ responsiveness is not only determined by their skills and knowledge but also
by the organizational environment they belong and are accountable to: their project team,
their organization, their professional associations and networks, the donors, and, of course,
the M&E frameworks, tools and procedures that the facilitators are required to use.
The systemic M&E frameworks required by the field of inclusive market development should
not only provide evidence of impact to the key development agents, but also fast, reliable and
relevant information to the facilitators to help them navigate the “systemic jungle”
effectively, for example: taking advantage of unforeseen opportunities, minimizing conflicts,
leveraging the resources of public and private market actors, minimizing operational costs,
5

6. contributing to deeper impacts, etc. Systemic M&E is needed not just for “proving” but also
for “improving”.
Recommendation:invest in M&E frameworks, procedures and incentives that, besides
proving the effectiveness of the project, enable facilitators on the ground to adapt, learn
and collaborate effectively.
3.5. Adaptability is the ultimate manifestation of systemic sustainability:
As mentioned above, one key aspect of systemic M&E frameworks is the focus on how the
facilitators are influencing the performance of precursor networks and how these networks
are moving the system towards a more desirable horizon. Additionally, it is important to
assess if the system is building the necessary conditions to avoid or minimize future shocks
and benefit from new trends whilst staying inclusive, productive and efficient.
Predicting how a market system will react to a shock is practically impossible.It is, however,
possible to assess if a system is becoming more or less adaptable –and therefore more
sustainable.A better understanding of how a market system builds and maintain resilience and
adaptability would provide important information to build systemic M&E frameworks that
pay attention to the structures, dynamics and parameters that would allow development
agents to assess trends towards more or less sustainability.
Recommendation: do more research about the components and drivers of resilience and
adaptability of different types of market systems (sectors and subsectors) in different
contexts and design M&E frameworks to detect these changes.
6