Information Note prepared for the Bangkok KM/KS write-shop (IFAD) about Scampis MSC experience

1. Information note:
“RBM KS in IFAD Rural
Development Context”
Bangkok, 3-7 December
 Cecilia Ruberto,
M&E consultant, IFAD Hq Policy and Technical
Advisory division, Rome, Italy. Mail:
c.ruberto@ifad.org and
ceciliaruberto@gmail.com Skype: ceci1982lia
 Nrusingh Charan Pradhan,
Project Coordinator, International
Development Enterprises India, Plot No. Local
Shopping Centre, Sector-12, Dwarka, New
Delhi-110075 (India)
nrusingh.p@ide-india.org
ncp.bbsr@gmail.com

2. Contents
Abstract ................................................................................................................................................ 2
1.What is Most significant Change? ...................................................................................................... 2
2. The SCAMPIS MSC technique ............................................................................................................ 3
Stories collections ............................................................................................................................. 3
The analysis implemented at various levels ...................................................................................... 4
Lessons Learned ................................................................................................................................ 5
3. Experience of Most Significant Change (MSC) Methodology under the SCAMPIS Project in Odisha. 5
Staff Training & Kids Training ............................................................................................................ 5
Kids/ Youth Training .......................................................................................................................... 6
Stories collection: methods and results ............................................................................................ 6
Second level analysis: methods and results (how it has been done, main results and lessons
learned about the methodology ....................................................................................................... 7
Overall Lessons learned MSC Scampis India ..................................................................................... 7
Web references .................................................................................................................................... 8
Bibliography .......................................................................................................................................... 8
List of Acronyms
SCAMPIS Scaling up of Micro Irrigation Systems
MIS Micro Irrigation Systems
LOF Liquid Organic Fertilizers
STP Surface Treadle Pump
BK Bucket Kit
DK Drum Kit
IFAD International Fund for Agriculture Development
MSC Most Significant Change
NGO Non Government Organisation
RBM Resource Based Management
KM Knowledge management
KS Knowledge Sharing
M&E Monitoring & Evaluation
OTELP Odisha Tribal Empowerment and Livelihoods programme
BKK Information Note – Scampis MSC experience 1

3. Abstract
In 2012 the Scampis project1 introduced an innovative M&E methodology to complement the
quantitative data collection. The project, implemented in India, Guatemala and Madagascar
experimented in the first 2 countries the Most Significant Change methodology.
The main objective in using MSC was to maximize the possibilities to learn from the project, from the
beneficiaries and the staff involved and to discover hidden factors/dynamics that are the cause of
the project success or not. Factors invisible to quantitative, pre-established M&E tools.
Many are the lessons learned, problems solved and problems still open in the adaptation and
application of the MSC method, but overall we can say that in both India and Guatemala it has been
appreciated very much and also from the Hq. point of view it has been very important to solve sense
making issues and create connection among different (and far) stakeholders.
The experience has been documented in an informal blog: www.scampisblog.blogspot.com . Some
work should still be done to capitalize the experience, for sure the same learning from it for further
innovation and use in new context is part of the capitalization process, as something will not be
really part of our background until we do not experiment it practically. For the advantage of the
Bangkok write-shop event a synthetic overview of what has been the Scampis MSC experience with
the exemplification of the India case study is offered. The Guatemala experience is available here.
1.What is Most significant Change?
The most significant change (MSC) technique is a form of participatory monitoring and
evaluation. It is participatory because many project stakeholders are involved both in deciding
the sorts of change to be recorded (collection of significant change (SC) stories) and in analysing
the data. It is a form of monitoring because it occurs throughout the program cycle and provides
information to help people manage the program.
It contributes to evaluation because it provides data on impact and outcomes that can be used
to help assess the performance of the program as a whole. And it contributes in strengthening
organization learning and communication through the analysis process, when the staff comes
together to discuss about the perceived MSC or the not perceived (Answering to the question:
“what went wrong?”)
Why collecting stories?
• People tell stories naturally - indigenous
• Stories can deal with complexity and context
• People remember stories
• Stories can carry hard messages /undiscussables
• But stories not known for accuracy/truth
Why the analysis process
• Analysis for discussions/communication
• For learning: elaborate information and transform it in practical lessons
• For improving our work/our organization work: increase efficetiveness in our activities
• To create bridges among far places and different cultures/experiences
• To understand numbers
• To discover new important element difficoultly perceived by normal M&E tools (especially if
qualitative)
1
http://www.ifad.org/english/water/scampis/
BKK Information Note – Scampis MSC experience 2

4. Why MSC is relevant for us?
To complete qualitatively the quantitative M&E analysis
• To improve planning (having a better understanding of what is going on in the
field)
• To help explain a point to a farmer (can be used as KM material)
• To recruit new participants
• To help explain a point to another member of staff
2. The SCAMPIS MSC technique
In our case, the SCAMPIS project, the technique was extremely relevant to complete the
participatory M&E process started in 2010 2. The methodology was adopted by SCAMPIS in
Guatemala and India, developed in a participatory way with IFAD Scampis M&E responsible. The tool
consisted in the implementation of 4 steps:
• Stories collection
• Field level analysis and selection / second field level analysis and selection
• National level Analysis (Hq) (only analysis no more selection)
• Cross-countries analysis
The methodology have been reshaped and modified to better match: specific needs and context,
limited availability of resources (money, time).
The main goal for Scampis MSC were: i) understanding M&E numbers, ii) knowledge and learn about
(unexpected) farmers perception of the project/technology, iii) fill the gaps between field reality and
higher top management or donor, iv) involve all staff in a coherent process of discussion and
reflection of the work conducted.
Stories collections3
The methodology has been reshaped from the original MSC (Davis and Dart) in order to adapt to
specific goal and context.
Collected by kids (age between 13-16 years) who will contemporaneously, as real reporters, film,
photograph and ask questions to the project beneficiaries. A training for the kids was provided at the
beginning of the day to break the ice, practice the use of the technologies and understand the
reason why of the activity as well the questions for the adults.
Why Kids or young people?? They are curious, they can easily learn how to use the technology, they
will let the interviewed feel comfortable (elder or senior people are not advisable as they would
influence to much the interview). They can at the best preserve the “ethic” of the MSC stories
collection that is: DO NOT INFLUENCE THE ANSWERS, LET COME OUT THE REAL STORY OF THE
INTERVIEWED. 4 Finally the see how important they are for their community!
2
Video on Scampis M&E and MSC http://www.youtube.com/watch?v=LbjJ4If9yUI
3
Guatemala photos here: http://www.youtube.com/playlist?list=PLA547A54E873204EB&feature=addto ; and videos:
https://picasaweb.google.com/CeciliaRuberto/ScampisGuatemala2012
4
Additional information on how kids have been trained and involved in the stories collection, the central role they played,
opportunities, pitfalls and lessons learned will be shared separately.
BKK Information Note – Scampis MSC experience 3

5. Figure 1 Stories collection India Figure 2 Stories collection Guatemala
The analysis implemented at various levels
The analysis at various level was conducted using brain storming modality reflecting on each story
form 4 different point of view:
A. Who is the interviewed and what I understood of his specific life experience (general)
B. MSC perceived and shared
C. Difficulties and risks perceived
D. New ideas (hidden stories)
After each analysis an overview of the findings have been discussed within the group. During the
first stories analysis, the staff selected the “best”10 MSC stories, and during the second analysis
among this 10 stories 6 have been chosen as most representatives. The videos were translated and
subtitled.5 After the national analysis a cross-country analysis have been implemented (Guatemala
national staff analysed the Indian video and vice versa).
Finally the various analysis plus the stories collected have been collected at IFAD Hq. level for final
evaluation and connection to M&E data. Below a representation of the main learning from MSC.
Figure 3 Scampis MSC - findings
5
Guatemala MSC video playlist: http://www.youtube.com/playlist?list=PLmkP-_uSr1KqUMlmTRPu6crS5FDBsWAGx India
MSC video playlist: http://www.youtube.com/playlist?list=PL166815510C378424
BKK Information Note – Scampis MSC experience 4

6. Lessons Learned
Limits of the analysis process were mostly due to time constrains and end of the activities in the
field, no longer staff employed and involved in the project):
- Cross-country analysis only at national level and not local level (which would have been
extremely useful).
- IFAD Hq analysis has not been implemented yet (not sure if will be done..)
- Storage of information: even if the MSC community suggested various different tools for
appropriate storage and elaboration of the info accumulated (text, photos, videos) there has
not been time to learn to use them, so has been used a simple excel programme.
The strength of the tool adopted showed strengthens on the major areas of monitoring, evaluation,
learning and communicating
- Organization development/learning: improve dialogue, trust and democratic discussion.
Moreover it can fill the gaps between top management and field reality
- Project implementation: Activities implementation & Strategy adaptation: if expected results
are not satisfied, the staff will understand it quick and can modify the approach
- M&E: Understanding and contextualize quantitative M&E (it is compatible)
- Effectiveness: Involve project beneficiaries in improving project effectiveness.
- Communication: Improve internal and external communication
- Sustainability: it can be easily implemented and managed by communities and staff (low
costs).
- Evaluation: It can be used for ex-post evaluation
3. Experience of Most Significant Change (MSC) Methodology under
the SCAMPIS Project in Odisha.6
Staff Training & Kids Training
The project facilitators like 2 Senior Marketing Officers, 5 Business Associates, Project Coordinator
and Chief General Manager participated in a two day state level training programme on
“Introduction of the new concept of Most Significant Change (MSC). The trainings was attended by
all the stakeholders namely, NGOs, Government agencies under the SCAMPIS programme in May-
2012 at Bhubaneswar
The Scampis IFAD M&E responsible implemented a
training to develop skills and knowledge on the
concept during the training. Basically we learnt
about the technique which was completely new and
innovative in nature. The process of adoption was
purely participatory where the local youths/kids and
users would be the main stakeholders in executing
the process in the operational project villages.
During training programme, the IFAD officials Figure 4 Youths training, India
discussed with all the participants on M&E and MSC
similarities and differences, the principles of MSC, importance of getting the stories and also did the
6
See authors references in the bibliography section. Additional info on the Scampis India MSC activities and
results: http://www.scampisblog.blogspot.com/p/scampis-india.html .
Training event info: http://scampisblog.blogspot.com/2012/11/most-significant-change-training.html
BKK Information Note – Scampis MSC experience 5

7. practical exercise on story collection, practical exercise in analysing stories and the role of various
stakeholders.
Accordingly we identified two blocks namely Mohana & Raigad of Gajapati district & three blocks
Pottangi, Nandpur & Dasmanthpur of Koraput district for execution of the MSC tools.
Kids/ Youth Training
After completion of the training we proceeded to the identified operational blocks and in the first
step we organised the village meetings to identify user farmers and youth/kids who could be trained
to conduct the interviews. Their identification & training was critical as they were to take the lead in
executing Most Significant Change (MSC) activities. Subsequently 5 training programmes for the
village youths/kids was conducted by the trained Senior Marketing Officers, Business Associates of
the project and also trained staff of local NGOs in 5 blocks. There are few questionnaires developed
by the facilitators and put up during interaction in order to support the narration of the story of the
users and understand the individual Most Significant Change:
I) Could you tell me your name?
II) Where are you from?
III) How many are in your family? How many sons and daughters do you have?
IV) When did you start using the SCAMPIS technology? Where did you see it for the first
time? What did you think of it at the beginning? What technology do you use?
V) Do you cultivate and take care of the plot alone? Someone helps you? Who?
VI) What did you harvested since you started/ How much?
VII) Do you used to cultivate vegetables before having technologies? If answer yes- Do
you see any difference? If answer No-Why?
VIII) Is the main source of income for your family? Have you sold part of the vegetables
produced with MIS? What did you do with the amount obtained from the selling?
IX) How was before the project and now, which are the most significant change you
perceive? Why those changes are important?
The team conducted interactions with the user farmers and shared about the MSC and its purpose.
As a result many farmers volunteered to participate and share their leanings, degree of mobilisation,
success in adopting the technologies and the benefits they accrued. Based on this, plan for the kids
was made.
The youths/Kids were trained in different aspects including conducting interviews, making videos,
taking still photography, developing story writing & also in observing and documenting the process.
At the end of training, each youth/Kid was asked if they had understood their roles & responsibilities
well & any doubts were clarified. In all 24 youth were trained and out of these 18 were boys & 6
were girls. Those Youths/Kids learned about:
 Interaction with project beneficiaries  Videography
 Questionnaires  Still Photography
 Observation  Documentation of success stories
 Field Visits  Analysis of the stories
 Focus Group Discussion
Stories collection: methods and results
How it has been done, how many interviews and the results of the interviews. Highlight difficulties
and ideas for improving it.
BKK Information Note – Scampis MSC experience 6

8. After the youth were trained they started conducting the interviews. A total of 24 kids- 18 boys and
6 girls were trained and they conducted the interview.
Based on the farmer feedback and numbers distributed across the identified locations, the
youths/kids were spread out. 24 kids covered 13 villages in Koraput and Gajapati district from 18-25
May’ 2012 and they met with and interviewed 25 user farmers (Male-23 & Female-2). This was
followed by organising a one day farmers meeting & training for the youths/Kids in each of the
selected villages.
Feedback from the participants (MSC Facilitators) on MSC: “I was very much excited about the
training & facilitating interviews with the user farmers. I learnt many things out of the training and
was enthusiastic how to carry out my responsibilities like making of videos, taking still photographs
and interaction with the users etc. After training, I started to think myself how I will do it…what I will
say etc. But when I actually interacted with the farmers I was very surprised to hear them talk so
much. I am personally inspired by the task entrusted to me carried out up to my satisfaction. This
training helped me to learn more about M&E. It helped me in understanding the communities and
impact of the development interventions. I was also delighted to hear with the farmers who have
been benefited out of the project their increased income and how they feel a sense of social dignity.
Once all the interviews were completed and videos recorded, these were sent for translation into
English. In the meantime, during IDEI’s monthly SCAMPIS staff meeting, the team invited people
from other NGOs, Odisha Tribal Empowerment and Livelihoods Programme (OTELP), Business
Associates (IDEI) and Senior Marketing Officers of SCAMPIS (IDEI) who had attended state level
training at Bhubaneswar to analyze all stories and to select 10 best stories (1st level) on MSC
collected by the youths/Kids. This exercise was successfully completed and 10 stories of the total 25
were selected for 2nd level analysis.
Second level analysis: methods and results (how it has been done, main
results and lessons learned about the methodology
Soon after the 2nd level analysis was undertaken. In the second level analysis senior representatives
from NGOs, Odisha Tribal Empowerment and Livelihoods Programme (OTELP), Senior Marketing
Officers and Project Coordinator of SCAMPIS were invited to analyze selected 10 stories and to
select best 6 stories on MSC which would be again analyzed and to select best 3 stories at state level
by the next level representatives of IDEI, and IFAD Delhi team.
Keeping the guidelines in mind, the second level analysis involved watching the videos together (all
representatives present). Everybody was handed over the MSC analysis sheet and each filled their
comments on it. At the end of watching all the videos, six from the ten were selected and six analysis
sheets (one for each video) was finalized and submitted.
Everybody liked the idea of involving the local youth, since it is there habitation they understand the
local environment better and they would not miss the nuances so this reduces the loss of
information.
Overall Lessons learned MSC Scampis India
Pitfalls, opportunities, potentiality of the MSC tool for monitoring, for evaluation (at various level) for
strengthening the organization, limitations of the tool.
The overall experience of participating in the MSC training and conducting it was very encouraging
for all involved. First key learning, which we now look as an opportunity is engaging with the local
youth. At first when this was shared we were bit sceptical. We had not involved the local youth in an
BKK Information Note – Scampis MSC experience 7

9. evaluation process before like this so the anticipation. However, as we met with the local youth and
shared with them about the training and the following work they need to undertake, needless to say
they were very excited and were keen learners.
Secondly, MSC is a good technique that has helped us capture the qualitative aspects/ impact of the
work. It has helped us go beyond numbers and also actually helped us better analyse the numbers
and make cross analysis. Example- if a certain number was achieved (as reported) and in the MSC
farmers shared how they used the income it validated the impact. It helped us get a better insight
into the ‘real’ impact of our work.
While it has many strengths, however I feel that it would be useful to always use it as in addition to a
tool that measures quantitative value of programme success.
Web references
Scampis blog www.scampisblog.blogspot.com
Youtube (video material) www.youtube.com/ceciliaruberto
PicasA https://picasaweb.google.com/CeciliaRuberto
Scampis IFAD webpage http://www.ifad.org/english/water/scampis/
M&E news (Rick Davies’ Blog) http://mande.co.uk/special-issues/most-significant-change-msc/
Bibliography
The “Most Significant Change”(MSC) Technique – A Guide to Its Use by Rick Davies and Jess Dart
(April 2005). Funded by CARE International, UK
The Most Significant Change Technique (MSC)Power Point document - Dr Jessica Dart, Clear Horizon
The Scampis “Most Significant Change”(MSC) Technique (May 2012) – Cecilia Ruberto, IFAD
MSC Experience in India, Information Note, November 2012.
• Sri Nrusingh Charan Pradhan, Designation: Branch Manager, IDE-India, Mailing Address: A/7,
Nilakantha Nagar, Nayapalli, Unit-8, Bhubaneswar-751012, Odisha, India, Email Id:
nrusingh.p@ide-india.org
• Sri Debendra Kumar Samal, Senior Marketing Officer, IDE-India, mail: debendra@ide-
india.org
• Sri Tapan Kumar Pattanayak, Chief General Manager, IDE-India, mail: tapan@ide-india.org
BKK Information Note – Scampis MSC experience 8