Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Workshop proceedings of "Identifying contextualized indicators to measure SDGs"

395 views

Published on

Keeping social impact management at the centre, 4th Wheel Social Impact is committed to strengthening social programs in India by improving the way they are designed, implemented, monitored and evaluated. The organization believes the integration of data, technology and partnerships will enable the achievement of the Sustainable Development Goals (SDGs).

The workshop focussed on Theory of Change, Indicator Development, SDG linkages of indicators.

This workshop was supported by Swedish Institute.

Published in: Government & Nonprofit
  • Be the first to comment

Workshop proceedings of "Identifying contextualized indicators to measure SDGs"

  1. 1. IDENTIFYING CONTEXTUALISED INDICATORS TO MEASURE IN INDIA SDGS SHOWCASING BEST PRACTICES ON INDICATOR IDENTIFICATION AND IMPACT EVALUATION 15th December, 2018 Powai, Mumbai
  2. 2. Introduction Who we are Keeping social impact management at the centre, 4th Wheel Social Impact is committed to strengthening social programs in India by improving the way they are designed, implemented, monitored and evaluated. The organization believes the integration of data, technology and partnerships will enable the achievement of the Sustainable Development Goals (SDGs). the workshop. The Swedish Institute is a public agency that encourages interest and trust in Sweden around the world. They operate in the fields of culture, education, science and business by seeking to strengthen ties and promote development. To encourage dialogues around SDGs, Swedish Institute supported Scope and Purpose of the Workshop Theory on the data revolution, indicator identification and designing Theory of Change (ToC) for programs Application of theory and practice learnt through group work Practical examples on ToC development in different social contexts The workshop was spread across three components – Agenda 1. THEORY 1.1 1.2 1.3 2.1 2.2 2.3 3.1 3.2 3.3 The Data Revolution Indicator Development Theory of Change Saathi EcoSolutions Magic Bus Medha Skilling Services Develop TOCs Identify Indicators Linkage to SDG Targets 2. PRACTICE 3. APPLICATION 1 2 3
  3. 3. Establish the need for collecting high quality data at the local level to track progress being made towards achievement of the global goals Share best practices in identifying and defining indicators to measure key elements of a program Improve capacities of stakeholders on designing theories of change (ToCs) with well-established causal pathways (external factors, assumptions, program design) Develop a list of contextualised indicators based on the global indicator framework by the Inter- Agency and Expert Group on SDG indicators (IAEG-SDGs) What we aimed to achieve The UN World Data Forum advocates the important role data and statistics will play in the implementation of the 2030 Agenda for Sustainable Development. It is believed that shared goals and metrics among stakeholders will provide a coherent narrative for action, mobilize all actors involved in a particular area, galvanize the community to develop clear strategies for implementation, raise the levels of available finance and spur innovations and technologies needed to implement them. Only 33% of the workshop participants had developed ToCs for their organisations while 29% stated that they face challenges in indicator development. Limited experiences of participants in ToC and indicator development substantiated the need for the workshop to contribute to the global goals at the national level. The workshop facilitated this macro-level goal by bringing together development practitioners from across India involved in planning, implementing, monitoring and evaluating social programs in India. Why Sectors of focus Rationale of the workshop 1 2 3 4
  4. 4. Workshop proceedings 1. Contributing to the data revolution ““Timely and usable data is critical for informed decision-making, monitoring of progress and evaluation of outcomes, especially in the context of the post- 2015 development 1 agenda.” The workshop began with a discussion on the need and benefits of collecting and producing high quality data to contribute to the data revolution. Techniques ranging from geo-referencing and satellite imagery to collect big data as well as using smartphones and tablets for surveys at the local level were explored. Management buy-in is essential to ensure regular data collection and using of data to manage program impact Resource allocation for comprehensive and regular data collection is essential Collection of reliable data requires equipped and capacitated enumerators Structured and defined program goals and outcomes are important to collect usable and multi- functional data Key takeaways Overcoming budget and time constraints in collecting good data Rationalise your data needs and accordingly modify the evaluation design Look for reliable secondary data sources Use technology to reduce the time and resources spent in data entry and cleaning Select a sample size suitable to your time and budget 1 The UN Secretary General's Independent Expert Advisory Group on a Data Revolution for Sustainable Development Using technology to ensure quality and reliable data Collecting data that is usable for measuring and tracking program goals Train your data collection team to ensure that they understand the value of data collection and how they are contributing to broader program goals Have a well-trained and prepared team member(s) to monitor your data collection process Ensure that all data points fit into at least one of the outcome indicators in your ToC Ensure a multi-stakeholder approach in data collection to establish triangulation and validity of data Questions and discussions 1 2 3 4
  5. 5. ““An indicator is a variable that measures key elements of a program. Indicators serve as a scientific and researched method to highlight problems, identify trends, and contribute to the process of priority-setting, policy formulation and evaluating 2 and monitoring progress.” 2. Identifying the right indicators Types of indicators Key components of defining indicators Best practices in indicator identification A well-defined and measurable indicator should include four components – the population/stakeholder for whom change is being measured and tracked, the target for change expected to be brought about, the timeline in which change will be brought about, and the threshold/lower limit to consider impact outcomes as 'good enough’ Indicators should be SMART by focusing on the 4 Ws – what, where, who, when and 1 H – how much All stakeholders – their realities and views - should be taken into account while developing indicators Select the most suitable indicators that capture the essence of the program to ensure cost and time effectiveness Involve subject experts to validate indicators Ensure that all indicators relate to the program targets Participants were oriented on three key aspects of indicator development – Key takeaways 2 Health in Sustainable Development Planning: The Role of Indicators, World Health Organisation (2002) 1 2 3 1 2 4 5 6 3 Parameters for defining targets and thresholds Keeping information from existing databases, define your targets and thresholds to suit the context of your program's reach and scale Refer to reliable sources of existing data and literature to gauge already existing frameworks and previous research Ideal timeline for testing and defining indicators Program design, its scale and reach, management buy- in and funder requirements are aspects that should shape indicator development and timelines There is no universal timeline for testing and defining indicators Ensuring involvement of all stakeholders All stakeholders representative should be consulted at different stages of project planning, design and implementation Multi-stakeholder views and opinions should be included with a focus on intersectionality Questions and discussions
  6. 6. ““The first step in designing a right-fit data collection strategy is to create a solid theory of change. A theory of change is a clear visual map that represents how a program will make an impact on the world. It illustrates what goes into a program, what gets done, and how the 3 world is expected to change as a result.” A ToC should start with the program goal and then sequentially move to the outcomes and required pre-conditions to reach the program goal (backward mapping/back-casting) Define indicators that will measure progress for every expected outcome A ToC should not remain as just a linear chart/graphic. It should tell a story about how your program will progress Assumptions should be put in at each layer of the ToC to account for external factors that can impact program success Key takeaways 3 Theory of Change: Laying the Foundation for Right-Fit Data Collection, Innovations for Poverty Action (2016) 3. Developing a comprehensive ToC Developing a comprehensive and all- encompassing ToC Theory of Change vs Log-frame Analysis (LFA) - the difference Do not look at ToCs as a linear representation. Make multiple connections between and across different outcomes Build in as many assumptions as possible. They are foundational for program successes and shortcomings Design the ToC keeping in mind contextual realities of your program area and beneficiaries Be flexible and ready to adapt your ToC across the program cycle ToCs start with the goal and then progress downwards i.e. backward mapping/back-casting as against LFAs which start with the entry point of the program and then progress upwards to the goal ToCs, through assumptions, take into account external influencers. LFAs do not leave scope for assumptions Interventions (inputs) are featured in an LFA. In ToC, the focus in only on outcomes and goals 1 3 2 4 Questions and discussions
  7. 7. ““Assumptions are a critical aspect of any ToC. They can determine the potential shortcomings and successes of your program.” ““A ToC cannot be looked at as a linear representation of a program. While this may be possible in theory, during on-ground implementation there can be multiple connections and relations between pre- conditions.” ““A strong and comprehensive ToC must have a multi-stakeholder perspective. Along with outcomes and pre- conditions, assumptions must carefully account for the voices of all stakeholders. “ CONTEXTUAL EXAMPLES Theory of Change –
  8. 8. Participants developed their own organisation’s Theory of Change They identified indicators and created an indicator list Participants mapped their indicators to the IAEG-SDG indicator list Participants collated indicators within the group The three groups presented their indicator list Practical application of workshop learnings - GROUP ACTIVITY Based on their interest, participants were divided into three groups EDUCATION HEALTH SKILLING Flow of practical exercises and group activities 1 2 3
  9. 9. DEVELOPING A THEORY OF CHANGE PRESENTING AND SHARING IDENTIFYING AND COLLATING INDICATORS During this practical exercise, participants tackled thought- provoking challenges as they attempted the backward mapping method instead of the straightforward approach used in logic model frameworks. They contended in distinguishing between activities and outcomes. Participants also faced difficulties in choosing a single theme for their TOCs. This was because the selected domains – health, education and skilling – are vast and in many situations even a single program covered multiple areas within the domain. Participants selected a single program of their organization and outlined its goal. Based on the established goal, they added essential pre-conditions, in the form of outcomes, which the initiative would need to fulfill, in order to achieve their goal. Participants then came up with an extensive indicator list which would enable them to measure their outcomes. They also referenced a comprehensive list of SDG indicators and linked their indicators to co-relating SDGs. Afterwards, participants shared their program's TOC and indicator list within the group and in-depth discussions resulted in an extensive list of all outcome indicators within the themes of education, health and skilling. Using various materials, the three groups designed an indicator overview and each group presented their indicator overview. Programs for which ToCs were developed spanned the education cycle of a student – from early childhood education to higher education. The indicators defined for corresponding outcomes had a multi-stakeholder approach. Apart from students, educators/teachers and the community, especially parents, were considered. Indicators were listed that could potentially measure program impact from the lens of different stakeholders. Lastly, different indicators related to infrastructure at educational institutions were also listed. Education The group working on health looked at this broad domain from different lenses - mental health, menstrual health, water and its impact on community health, gender and its underlying implications for good health. Keeping in mind this broad range of sub-themes, the group defined indicators for each of these emerging themes. This was presented in terms of indicators that should be prioritised across these themes while designing a ToC and indicator framework. The group working in the area of skill development traced indicators across the life cycle of a skill development program – from mobilisation to job placements and retention. Further, the group came up with indicators on how skill development can improve the socio-economic status and overall standard of living for trainees and their families. Indicators were also listed on how skill development improves individual capacities in terms of aspirations and control on their own life decisions. Health Skilling
  10. 10. Feedback and insights Gained insights into development of ToC Will apply workshop learning Felt that the workshop was relevant to the development sector Gained insights into development of indicators Perception Scores 81.71% 91.42% 89.14% 84% “ “The orientation to ToCs was beneficial – specifically the causal pathway and how it is a better tool than LFA for program planning, design, implementation and evaluation” “The inputs on indicator development, specifically on integrating them with the ToC outcomes was highly beneficial.” “Data revolution was a great concept. It could serve as a good base for future workshops, specifically the different statistical and mathematical techniques to enable a data revolution.”
  11. 11. Participating organisations ampUz Atul Foundation Bansidha & Ila Panda Foundation Antarang Foundation Apnalaya Arvind Foundation Baker Tilly DHC Dobility India (SurveyCTO) Grameen Capital India Pvt. Ltd. IIT Kanpur Dream a Dream Godrej IDOBRO Media and Marketing services Private Limited Barefoot Edu Foundation INREM Foundation Cognify Labs Impact India Foundation Indian Metals & Ferro Alloys Ltd Learning Links Foundation Medha Omkar Foundation One planet rating Piramal Water Private Ltd. Rizwan Adatia Foundation Saathi EcoInnovations JPPL Magic Bus One Future Collective Sangli Mission Society Setco Foundation WNS Toybank - The Opentree Foundation United Sisters Foundation YUVA UnLtd India
  12. 12. +91 22 62214191 +91 98258 85044 www.the4thwheel.com

×