Participatory M & E Framework for MAMTA_HIMC

1. Participatory M&E Framework
for MAMTA HIMC
Diego Peña
Tejveer Singh
Juanita Duque
Aiste Kaftaniukaite
Deepti Gaur Mukerjee
MAMTA
Health Institute for Mother and Child

2. • Introduction to MAMTA
• Objectives, scope and problem statement
• Methodology
• Situational Analysis
• Key findings
• Proposed framework
• Recommendations
• Benefits and implications
• Critical reflections and conclusion
Overview

3. • Introduction to MAMTA
• Objectives, scope and problem statement
• Methodology
• Situational Analysis
• Key findings
• Proposed framework
• Recommendations
• Benefits and implications
• Critical reflections and conclusion
Overview

4. Empower the underserved and marginalized individuals and
communities through gender sensitive participatory processes,
for achieving optimal and sustainable health and development.
Mission
Approach
Intervention
Areas
Women &
Child Health
Stakeholders
Community
Mobilization
Capacity
Building
Networking Advocacy
EmpowermentReproductive Health &
Rights of Adolescents &
Youth
Community
and CBOs
Health Officials Public
Representatives
Donors
Introduction to MAMTA

5. • Introduction to MAMTA
• Objectives, scope and problem statement
• Methodology
• Situational Analysis
• Key findings
• Proposed framework
• Recommendations
• Benefits and implications
• Critical reflections and conclusion
Overview

6. The project has 2 main objectives:
o To develop a system of outcome measures in maternal care which are able to
measure MAMTA’s social impact (important to understand its impact and to
justify it for its donors); and
o How can MAMTA use this impact measurement for learning and innovation
of its structures, processes, and so on? What processes and mechanisms can
MAMTA install in its organization to benefit from this information in an
optimal way (in terms of improving its business model and in turn its social
impact)?
Focus on 3 projects:
o The 10 k club - A club for the health & development of the poorest
o Improvement of Reproductive & Child Health through a participatory approach
in Urban Poor scenario – Tata Delhi
o Strengthening capacities of district public health services towards improved
reproductive health choices for young married couples - Mac Arthur Project
Objectives and Scope

7. MAMTA has a difficulty in measuring its long-
term outcomes as well as in empowering the
communities to continue the M&E once
MAMTA’s intervention has finished.
Problem Statement

8. • Introduction to MAMTA
• Objectives, scope and problem statement
• Methodology
• Situational Analysis
• Key findings
• Proposed framework
• Recommendations
• Benefits and implications
• Critical reflections and conclusion
Overview

9. Initial information
gathering: LSE and
MAMTA
Literature and existing framework review
Call with MAMTA
–
Kick-off and
scope definition
Interviews with
stakeholders:
Internal and external
Analysis – Building
outcome framework
Recommendations
and implications on
Business Model
Information analysis
KeyFindings
Analysis – Designing
Framework of
Community-based M&E
process
Methodology
• Qualitative
• Case Study

10. • Introduction to MAMTA
• Objectives, scope and problem statement
• Methodology
• Situational Analysis
• Key findings
• Proposed framework
• Recommendations
• Benefits and implications
• Critical reflections and conclusion
Overview

11. • Implementation Strategy – community is the key
• Expansionary strategic plan - to widen footprint across India,
Nepal and Bangladesh
• Existing evaluation framework
• M&E Challenge - measuring long-term outcomes/ impacts
Project-based
MIS based on Government data &
Community Workers
Shift in focus from output to outcome
measurement
Situational Analysis

12. • Survey of 82 respondents by MAMTA – step towards capturing
outcomes and perceived social impact
• MCH – improved access to healthcare facilities, better service delivery,
better observance of health practices, improved reproductive health
behavior
• Social Empowerment – positive impact on gender relations, domestic
violence & alcoholism; on education of girl child, access to government
schemes (social insurance), financial inclusion, self-employment
training, community ‘voice’ in interaction with political leaders
• Demands from community – additional Government facilities, need for
hand-holding support
• Corroboration by interviews with Community Workers and MAMTA
functionaries
Impact well beyond areas of project intervention
Situational Analysis – Survey Insights

13. • Introduction to MAMTA
• Objectives, scope and problem statement
• Methodology
• Situational Analysis
• Key findings
• Proposed framework
• Recommendations
• Benefits and implications
• Critical reflections and conclusion
Overview

14. LITERATURE REVIEW
1. Sen’s (1999) capability
approach
2. Empowerment
• OECD (2012)
• Kabeer (1999)
3. Outcome measurements
and community monitoring
in Health Sector – inSCALE
UCL et al (2011)
INTERVIEWS AND DATA COLLECTION
1. Dr. Mahasweta Satpati
and Dr. Archana – MAMTA
2. Community Workers –
MAMTA
3. Dr. S.K. Mishra –
SeniorTechnical Advisor for
M&Es
4. Mr. Prasanth
Subrahmanian, Senior
Consultant, Ministry of
Health & FW, GoI
FRAMEWORK ANALYSIS
1. United Way of America –
Measuring Program
Outcomes (1998)
2. Participatory M&E –
Estrella & Gaventa (1998)
inspired by R. Chambers
3. A methodological
approach and framework
for sustainability
assessment - Sarriot et al.
(2004)
Key Findings: Inputs

15. LITERATURE REVIEW
1. Definition of empowerment: “[…]process by which those who
have been denied the ability to make strategic life choices
acquire such an ability” (Kabeer 1999, 435)
2. Importance of participatory processes in both implementation
and M&E (OECD, Sen)
Empowerment cannot be externally controlled: “…there is no
real escape from the evaluative need for public discussion” (Sen
1999)
3. Measuring empowerment through 3 dimensions: Resources,
Agency, Achievements (Kabeer 1999)
Key Findings: Insights

16. INTERVIEWS AND DATA COLLECTION
1. MAMTA has been managing its M&E System for over 2 decades
2. MAMTA has an ability to map outputs and outcomes and to
adjust to donors’ requirements of M&Es
3. Donors are looking for community empowerment and for the
results of long-term impact evaluations
4. The Gov. of India calls for a more complementary action on the
part of NGOs
5. Main challenge consists in measuring long-term impact and
in empowering communities to continue the M&E system once
MAMTA has finished the intervention
Key Findings: Insights

17. FRAMEWORKS ANALYSED
1. Participatory M&E as a useful tool to measure long-term impacts in
communities where intervention has finished
2. Also key in empowering communities from the beginning of the
intervention and after the intervention has ended
3. Allows learning while doing (in this case for both for MAMTA and
beneficiaries)
4. Ideal for attracting donors and for achieving greater impact
Key Findings: Insights

18. • Introduction to MAMTA
• Objectives, scope and problem statement
• Methodology
• Situational Analysis
• Key findings
• Proposed framework
• Recommendations
• Benefits and implications
• Critical reflections and conclusion
Overview

19. MAMTA identifies
target community
Formation of
Community-based
organization (CBO)
1 2
Project implementation with support of CBOs:
-Tata Delhi
-10 K Club
-MacArthur
Monitoring and Evaluation (M&E) from different
sources: Direct, CBO, government
3
4
-----------------------------------------------------------------
End of MAMTA’s
intervention
• Difficulty in
measuring
MAMTA’s long-
term impact
• Community is not
necessarily
empowered to
continue the M&E
of Mother and
Child Health
(MCH) indicators
• Local
governments
have difficulty
finding this data
How can these
problems be
addressed?
Adaptation of Framework by Sarriot et al. 2004
MAMTA’s intervention model
Status quo

20. MAMTA
identifies
target
community
Formation of
Community-based
organization (CBO)
Project implementation with support of CBOs:
-Tata Delhi
-10 K Club
-MacArthur
Monitoring and Evaluation (M&E) from
different sources: Direct, CBO, government
--------------------------------------------------------
Beginning of MAMTA’s
handover
• Community-based
organizations are
empowered to
continue M&E process
• They are encouraged to
support other CBOs
and share knowledge
• Provide MAMTA and
local Govs. with long
term outcome
measurements
• CBO network for M&E
data sharing, technical
support, transfer of
knowledge
Participatory M&E
Proposed framework of community-
based M&E process

21. Type of Outcome Quality of Indicator Measurement Method
Initial: first benefits of the
intervention
• Meaningfulness – Low
• Feasibility – High
Direct measurement by
quantifying MAMTA’s activities
Intermediate: changes in
behaviour due to
knowledge, skills, attitudes
• Meaningfulness – Medium
• Feasibility – Medium
Measurement through surveys,
interviews and focus groups
Long-term: changes his/her
condition or status
• Meaningfulness – High
• Feasibility – Low
Measurements through
participatory methods
+Levelofparticipation-
Inputs: Resources
a program uses to
achieve program
objectives
Activities: What a
program does
with its inputs
Outputs: Products
of program's
activities
Outcomes:
Benefits for
participants during
or after the
program
Community-
owned M&E
System
Adapted from United Way 1996. Measuring Program Outcomes: A practical approach
1.Initial
2.Intermediate
3.Long Term
Sequence of outcome measurement

22. Envision outcome
goals
Establish how the
outcomes are to be
reported
Finalise with
review of all
outcomes,
indicators, and
responsibilities and
decide on a revision
period
- What levels are
expected to be
achieved and when
- Analysis of results
- Blackboard visible to
everyone
- Brochure
- Ask the participants
who is willing
- Agree on frequency of
measurement
Point out possible
costs
There are various available tools: Semi-structured group discussions - village workshops - field transects - historical
profiles - ranking - mapping - diagramming - household interviewing - field surveys - Timelines / trends - seasonal
calendars - trend analysis - Personal diaries - oral histories - pictorial diaries - voting ...
Who will be
responsible for
each outcome?
Identify
participants
Manage
expectations
Ask participants
what they want to
contribute in
Recognise priorities
for M&E (outcomes
)
Determine how the
outcomes can be
measured
(indicators)
- Who should be
involved?
- Are they keen on
participating?
- Both positive and
negative
- Intended and
unintended
- Final expected outcomes
- Outcomes required to
achieve final outcomes
- Who will benefit?
- Who will be affected?
- Construct links between
each outcome
- Data source
- Collection method
- Is it already measured
elsewhere
- Can it be included in
current
measurements?
- Factors that can
influence the indicator
Adapted from Marisol, Estrella and John Gaventa. 1998. “Who Counts Reality? Participatory monitoring and Evaluation: A Literature Review.”
Detail of Proposed CBM&E Framework

23. • Introduction to MAMTA
• Objectives, scope and problem statement
• Methodology
• Situational Analysis
• Key findings
• Proposed framework
• Recommendations
• Benefits and implications
• Critical reflections and conclusion
Overview

24. • Paradigm - creating social capital through impact of MAMTA’s interventions
• Community-based M&E - formally institutionalize into MAMTA projects with shared
definition of outcomes/ impact between NGO & community, eg.
- Women’s ability to make informed choices
- Participation in decision-making at family & community level
- Greater ‘voice’ – confidence to access services and articulate needs
• Dual Approach – Joint M&E using participatory methods and Mixed Methods Study
(qualitative and surveys) during project cycle; CBM&E in post-project phase
Continuum of Responsibility for CBM&E
-----------------------------------------------------------------------------
Externally-led CBM&E Joint CBM&E Internally-led CBM&E
(adapted from Estrella & Gaventa 1998)
• Institutionalization of community networks - networking CBOs in projects (through
web portal) for experience-sharing and post-project ‘support’ – to be initiated on
pilot basis
• Social Return on Investment - through avoided costs to society using quantitative and
qualitative indicators (e.g. reduced pregnancy complications, reduction of fertility rate) –
possible negative outcomes
Recommendations

25. • Introduction to MAMTA
• Objectives, scope and problem statement
• Methodology
• Situational Analysis
• Key findings
• Proposed framework
• Recommendations
• Benefits and implications
• Critical reflections and conclusion
Overview

26. Benefits
• Participatory methodologies
can improve sustainability and
impact of MAMTA’s
interventions
• Empowerment as the major
impact of MAMTA’s
intervention
• Ability to measure long-term
outcomes even in post-project
phase
• Enables MAMTA to widen
footprint of its work
• Facilitate increase in donor
interest
Implications
• Strategic repositioning of
MAMTA’s approach more as a
Mother NGO
• Strengthening M&E set-up with
agreement on choice of suitable
methods
• Structuring CBM&E workshops
using tools suggested and
building on MAMTA experience
• Formation of CBO network and
redefinition of post-project role
• Semi-annual or annual reports
with long-term outcomes
measured through CBO’s
Benefits and Managerial Implications

27. • Introduction to MAMTA
• Objectives, scope and problem statement
• Methodology
• Situational Analysis
• Key findings
• Proposed framework
• Recommendations
• Benefits and implications
• Critical reflections and conclusion
Overview

28. Challenges in Community-based M&E
- Power relations
- Methodological issues
- Resource intensive
- Institutionalization
Strengths and opportunities
- Strong organizational capacity & ethos
- Sustainability of intervention
- In sync with donor priorities
- The way to go!
MAMTA is well-placed to make this strategic transition
Critical Reflections and Conclusion

29. Thank You