The Internal Learning System, Assessing Impact While Empowering Internal Learning
Self Help Group Promotion, PRADAN. Photo source: Helzi NoponenTHE INTERNALLEARNING SYSTEMASSESSING IMPACT WHILE EMPOWERINGINTERNAL LEARNINGAnna Kiel Martin American Evaluators Association National Conference, MinneapolisKatrina Mitchell October 26, 2012
WHAT IS ILS?• SIMPLE• Quantitative• Evaluation +• Non-extractive• Highly participatory• Pictorial/accessible• Customized• Embedded in the program• Multi-year
HOW DID ILS DEVELOP?Helzi Noponen in Bihar, 2006. Photo credit: Valerie Arendt
EXERCISE:HOW DOES THIS WORK?You have been given a sample page.Identify what the purpose of the page is and how you wouldinteract with it as a participant.How would you use it to assess impact or empower learning?
WHAT’S IN THE ILS TOOLBOX?“Women own the ILS book. It is their record.Though illiterate, they can “read” it. No onehas ever before suggested they can do suchwork. They take the responsibility seriously.It is like a green light for them to dare to thinkabout achieving a better life.Each indicator picture has an impliedprogram value attached to it.Women absorb this.It gives permission or space to discusssensitive empowerment and wider socialissues and the courage to attempt change”Helzi Noponen
HOW DO YOU KNOW YOURCURRENT SITUATION? General Health Care PracticesEvaluation formats: • Yes/no questions • Quantity questions • Multiple choice questions Woman goes to doctor when needed. Safe water practices • Ranking and/or scaling questions Knows oral re-hydration therapy. Washes hands after latrine and before food handling. Woman herself uses mosquito net. Hygiene around menses
HOW DO YOU IMAGINEA BETTER FUTURE?Learning formats: “Suicide is Not a Solution” • Bad Scene / Good Scenes • Cautionary Cartoons / Illustrated Folk Tales • Dream Scenes • Panorama Scenes • 6 panel Stories (photo novella, picture parade, comic strip) ?
HOW WILL YOU CREATEA BETTER FUTURE? Plan for Meeting Strategic InterestsPlanning formats:• Problem Sorting Exercises think discuss husband family help friend help get comfort• Goal Setting Exercises• Priority Choosing Exercises other action get service• Planning Formats – budgeting meet officials work with men trees, loan repayment plans write officials research elite help campaign 0 1 2 3 cluster help SHG help other SHG help protest action exposure visit speak out training
HOW DO YOU WEAVE ITTOGETHER? PlanningLearning Evaluation
EXERCISE:HOW DO YOU…You are going to a pot-luck and have to bring a dish.What mental steps do you go through from the invitation toarriving at the right place, at the right time, with the right food toshare?
PARTICIPANT:HOW DOES ILS CHANGE BEHAVIOR? 1. Collect 2. Assess 3. Analyze 5 Tasks 4. Plan or alter 5. Document, share, reinforce
PARTICIPANT: HOW DO I USE THISTO IMPROVE MY SITUATION? Dalit women in Varanasi displaying their job cards Photo source: Leena Patel/UN Women (seeking permission)
HOW DOES A PARTICIPANTINTERACT WITH ILS?• 5 tasks of ILS• Ownership of diary/ data – participatory• Census – they all have one• Community-building, decreases isolation, empowers (story telling)
WHO INTERACTS WITH ILS? PARTICIPANT PROGRAM STAFF EVALUATOR AGENCY/ORGANIZATIONProgram participants with diaries/workbooks PRADAN, Photo source: Helzi Noponen Field promoter with PRADAN, Photo source: PRADANHelzi Noponen in Bihar, 2006. Photo credit: Valerie Arendt Image source: PRADAN website
HOW DOES A PROGRAM MANAGERINTERACT WITH ILS?• Facilitation in a group setting• Reveals conditions/ behaviors which aren’t readily visible• Highlights patterns• Demands radical accountability• Encourages reflective practice and innovation
HOW DOES AN EVALUATORINTERACT WITH ILS?• How to (steps to ILS): • Design • Implementation • Measurement• Commitment to participatory techniques• Effective creation and implementation mirrors the tool; lots of learning and adapting
HOW DOES THE AGENCY INTERACTWITH ILS?• Impact assessment• Can quantify hard-to-measure social change outcomes• Use with outside stakeholders for funding and advocacy• Commitment to participatory techniques, accountability and innovation/learning
HOW WOULD YOU USE IT? EARLY CHILDHOOD HEALTH CARE BUSINESS DEVELOPMENT HOMELESS / YOUTHStock photo Maternal / neo natal health care. Photo courtesy of Neighborhood HouseRecent immigrants. Photo courtesy of Neighborhood House Stock photo
IS ILS RIGHT FOR MYPROJECT / ORGANIZATION?YES NOI want QUANTITATIVE data I need QUALITATIVE dataLearning is important I just need to assessParticipant input is important Summative onlyFormative and summative Outcome onlyProcess and outcome data Short term or one-off evaluationLongitudinal, long-termHighly customized USE “TRADITIONAL” M&E or EXTRACTIVE PIA
HOW WOULD YOU CONSTRUCTSOMETHING LIKE THIS… ….as it was originally meant to be used?
WHAT SHOULD I CONSIDER WHENDESIGNING AN ILS DIARY?Research design • Identify your needs and analysis plan before you startSurvey design • Both formative and summative • Types of questions • Level of detail • Relevant to all participantsDiary design • Pictorialize – lots of considerations! • Create modules • Start simple. Build up to complexity and risk.
HOW DO YOU TURN PICTURES ANDPENCIL LINES INTO DATA? PARTICIPANT DIARY INTERVIEWER DIARYThe semi-extractive Land Utilization Land Utilization 99=not farmer (skip page)interview event: Grows crop? 0=No; 1=Yes number of yearly harvests? average yield? base mid point end point• Side-by-side number of harvests # grain harvests optimum yield base mid point end point• Sharing base mid end grain yield 5 optimum yield base mid point end point• Transparency base mid end # pulses harvests base mid point end point above 75% yield pulses yield 4• Verification base mid point end point above 75% yield # oilseeds harvest• Human Subjects 50% - 75% yield base mid end base mid point end point Protocol oilseeds yield 3 base mid point end point 50% - 75% yield # vegetables harvest base mid end base mid point end point 2 25% - 50% yield vegetables yield 25% - 50% yield base mid point end point # fruits harvest Include sugarcane base mid point end point 1 base mid end Less than 25% yield Less than 25% yield fruits yield 20 20
WHAT ARE ITS STRENGTHS?Traditional M&E Participatory Impact AssessmentExtractive METHOD EmpoweringDonors/policy AUDIENCE ClientsJudging PURPOSE LearningExternal ORIENTATION InternalFrom Simonowitz, 2001
COPYRIGHT, ATTRIBUTIONAND USAGE A note on copyright, attribution, and appropriate usage.The Internal Learning System was developed by Dr. Helzi Noponen in partnership with Ford Foundation India and Imp-Act, and implemented with a number of organizations including PRADAN, the Handloom Weavers Development Society (HLWDS), ASA, the High-range Plantation Workers Development Society (HPWDS), PLAN Honduras, and others.Many of the illustrations contained in this presentation come from diaries/workbooks created by Dr. Noponen for these organizations. Parts of this presentation were adapted from presentations and materials created by Dr. Noponen. Anna and Katrina are currently working with Dr. Noponen’s family and colleagues to preserve and extend the legacy of ILS. Please consult with us for attribution and usage information prior to distributing or re-using these materials.
Muhammad Yunus, Nobel Prize Winner, with Helzi Noponen, 2008. Photo source: Gary Smaby HELZI NOPONEN, 1953-2012 Innovative mind, open heart. helzinoponen.wordpress.com
THANK YOU! KATRINA MITCHELL Research and Strategy for Good email@example.com 612-269-2891 ANNA MARTIN Thinking Evaluation firstname.lastname@example.org 612-805-6460 internallearningsystem.com
Building a Diary | Components The diaries are composed of a series of themed modules. The modules together form a development curriculum that is a logical whole. Each module utilizes a variety of different approaches, or components, for meeting program goals and evaluation. The three types of components are for learning, planning and evaluation. Every module does not have all three, but most include at least some evaluation exercises. Usually a learning exercise introduces the module. Learning Planning Evaluation Heavily facilitated, typically in a group setting, by Planning tools allow for us to take on greater Some of the evaluation is for impact assessment the program officer or ﬁeld agent. These exercises complexity in our lives than if we try to keep it all in (proving) perspective, many of them help the make up the “curriculum” of the program; they are our heads. The planning exercises in the diary participant to assess their own life and circum- messages from the program/agency to its constitu- open up a whole new world for participants. For stances in a meaningful way. Many have never ents about the intervention. They tend to be many, it is the ﬁrst time they have purposefully thought in these terms before. Simply writing problem-solving or strengths-based. The purpose created and acted on a plan. These plans help down and quantifying thier life and experience is to inspire participants on how life could be, or participants move from reactive/crisis orientation to can provoke new ways of doing things, new ways how a problem could turn out well. They are not a proactive/future orientation. of thinking. prescriptive but are conversation starters. It is important that the participant takes an active • Yes/no questions • Bad Scene / Good Scenes role in their own planning. The diary is a tool to • Quantity questions • Cautionary Cartoons / Illustrated Folk Tales identify, prioritize and then strategize about the • Multiple choice questions • Dream Scene problems and possibilities in their life before taking • Ranking and/or scaling questions • Panorama Scenes (e.g. Gender Benders) action. • 6 panel Stories (photo novella, picture parade, comic strip) Planning components also allow for accountability -- both the participant to themself and the partici- pant to the group, vice versa. • Problem Sorting Exercises • Goal Setting Exercises • Priority Choosing Exercises • Planning Formats Image source: Helzi Noponen Image source: Helzi Noponen ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell.ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com
Building a Diary | 5 Tasks of ILS The core of the ILS strategy is to engage the participants in a ﬁve-step process that carries over and informs how they will think about their life. We all do these things, but the diary makes this process explicit, in order to empower the participants with choice. 1. Collecting Data Participants own the ILS diary. It is their record. The diary promotes reﬂection on their current situation, in a struc- tured and relevant format. Each indicator picture has an implied program value attached to it. The process of carefully marking and keeping the diary acts as a green light for them to dare to think about achieving a better life. 2. Assessing Data / Change Participants, though illiterate, can “read” their own diary and that of others. Individually, and in the context of a group, they gain an understanding of their changing life and livelihood situation. Patterns emerge, the ability to set goals, prioritize problems and solutions and plan for the future are all facilitated through the use of the diary. 3. Analysing Causes of Change / Troubleshoot Once a goal is set, or a pattern emerges, participants use their data and assessment of the situation to identify potential causes and reasons. Asking “why?” and pursuing an answer calls forth the wisdom and skills of each participant, empowering them to step into a position of primary actor in their lives. 4. Plan or Alter Strategies, Training With an understanding of the forces at play, participants make intentional changes to their strategies and behavior for achieving their goals. They begin to track and assess data again, to understand the impact of this new plan of action—gaining greater self-awareness, conﬁ- dence and better results in the process. 5. Documenting, sharing and reinforcing values Sharing between participants can be a wonderful source of inspiration, support and power; discus- sion is an oft over-looked learning tool. The diary acts as a testimony to individual experiences, which are often duplicated between group members. These recorded similarities give permission to discuss sensitive wider social issues and inequitable structures and the courage to attempt change. Collective action to summon needed resources and services is bolstered by the diaries’ written record. The source for all images on this page: Helzi Noponen ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell.ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com
Why ILS? | Going beyond proving While ILS can help “prove” program effectiveness, and can be used for internal accountability, its real strength lies in its ability to support improving. It is the learning and planning components of ILS coupled together with evaluation that make it a powerful tool which is integral to the program design. Program Impact Assessment Objectives PROVING IMPROVING / LEARNING Strengthen Self Proving to Stakeholder advocacy program Relationships participants and staff EMPOWERMENT Planning Proving to funders + organizational Staff Participant leadership learning learning Reﬂection + Assessment “Traditional” Impact Assessment Participatory Impact Assessment (ILS, and others) Traditional impact assessment has an external focus (to the donors and Participatory Impact Assessment (PIA) is internally focused, aimed at improving policy makers) and a more quantitative and “objective” approach. It tends to outcomes through the empowerment of the program staff and participants. PIA falls be extractive, the information gathered is primarily used outside of the along a spectrum of participatory methods used to gather data from extractive formats community and little, if any, feedback is provided to the community in terms to fully engaging participants in evaluating their own progress and the program.2 While they can understand. The purpose of the assessment is to prove effective- some focus on the pictorial aspect of ILS as a way to gather information from a ness and efficiency rather than to empower the community, participants or low-literacy population, the true strength of ILS is in its abiltity to empower learning and program staff to learn and improve.1 inspire action.3 1. Marisol Estrella and John Gaventa, “Who Counts Reality? Participatory Monitoring and Evaluation: A Literature Review”,as summarized in Anton Simonawitz, “Making impact assessment more participatory” June 2000 2. Anton Simonawitz, “Making impact assessment more participatory” June 2000 3. For a discussion of how ILS differs from PRA techniques using pictures see Noponen, H. (2002) The Internal Learning System—a tool for micro-ﬁnance and livelihoods interventions. Development Bulletin 57: 106–110. ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell.ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com
Building a diary | How to ILS (the basics) ILS is not just about creating a pictorial diary. It is based on well-grounded research design and starts with the creation of a simple survey instrument using indicators derived from thorough background research on the community and the program. Initial research is often done with a participatory approach, though a survey could be designed in collaboration with program staff who have sufficient ﬁeld expertise. In both cases it is critical to do at least one ﬁeld test before implementation. In order to make a diary that is interesting and relevant to all participants, it should contain very few skip patterns. The survey should be designed with questions that can be answered pictorially such as status or yes/no, simple quantities, multiple choice and scale ratings. Each indicator should relate to a learning component. Do not include survey questions that are only of interest to program staff or for external proving. CREATING A DIARY PUTTING IT TO USE Train program staff Conduct background research to develop appropriate indicators and determine barriers to success Print and mark diaries with ID and date Distribute census wide Draft a simple survey instrument From identiﬁed barriers and program including demographics and key goals conceptualize learning and indicators of program success planning components For baseline make ﬁrst marks in red Pictorialize the survey (Hire an artist to create drawings) ANALYSIS & REPORTING Likely, issues arise with: Develop research design Draft layout and assemble learning, planning and evaluation components a. Layout of a question on a page causes into modules that form a logical whole. confusion Train interviewers! b. Individual pictures are not understood at all, Train 1-2 staff in facilitation or interpreted differently across the group Select sample based on design c. A particular question or learning component (panel or cross-sectional) causes discomfort, stone-walling or conﬂict FIELD TEST!! from the group Semi-extractive side-by-side d. Particular subject matter or issues were interview event. Participatory! repeat Debrief / sharing with overlooked as necessary participants and staff e. One of the choices of a multiple choice Data coding and entry question is incorrect or missing Revise* f. Facilitation of a particular learning component Analysis should occur differently or at a different time. Create facilitation guide Report to all stakeholders ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell.ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com