Successfully reported this slideshow.

2010 10 22_slides_nbo wkshp_revision 4_vk

881 views

Published on

  • Be the first to comment

2010 10 22_slides_nbo wkshp_revision 4_vk

  1. 1. Monitoring and Evaluation Workshop for Grantees City, Country Dates
  2. 2. Session 1 Introduction to the Workshop Adapted from: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  3. 3. • Introductions • What do we have in common? • Expectations and Learning Objectives • Agenda
  4. 4. Workshop Objectives By the end of this workshop, participants will be able to: • Differentiate between monitoring and evaluation • Write/revise goals and smart outcomes for your VAW program • Demonstrate an understanding how to develop an appropriate M & E plan for your program • Identify criteria for indicator selection and information sources for VAW indicators, and select appropriate indicators for your program • Discuss factors to consider when choosing an evaluation design • Demonstrate an understanding of the different methods used to monitor and evaluate programs Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  5. 5. Session 2 What is M & E? What is the Purpose of M &E? Adapted from: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  6. 6. Monitoring versus Evaluation MONITORING = Tracking changes in program performance over time EVALUATION = Assessing whether objectives have been met Assessing extent to which program is responsible for observed changes Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  7. 7. Illustration of Program Monitoring Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009 Program start Program end TIME-> Program indicator
  8. 8. Illustration of Program Impact Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009 Program start Program end TIME-> Change in Program Outcome With program Without program Program impact
  9. 9. Discussion: Is it Monitoring or Evaluation? • The Ministry of Women’s Affairs wants to know if programs carried out in Province A are reducing the prevalence of intimate partner violence (IPV). • The UN Trust Fund wants to know how many villages in Region B have been reached with anti-VAW messages by your program this year. Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  10. 10. The Purpose of M & E The purpose of monitoring and evaluation is to measure program effectiveness efficiency and to support evidence- based decision making. Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  11. 11. Is it monitoring? Is it evaluation? • Were resources made available to the program in the quantities and at the times specified by the program plan? • Were the program activities carried out as planned? • Which program activities were more effective and which were less effective? • Did the expected changes occur? How much change occurred? • Can improved outcomes be attributed to program efforts? • Did the target population benefit from the program and at what cost? Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009 Discussion: Monitoring and Evaluation Questions
  12. 12. M&E Across Program Life Cycle Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009 ASSESSMENT What is the nature of the problem? EVALUATION How Do I know that the strategy is working? How do I judge if the intervention is making a difference? STRATEGIC PLANNING What primary objectives should my program pursue to address this problem? IMPLEMENTATION/MONITORING How do I know the activities are being implemented as designed? How much does implementation vary from site to site? How can be program become more efficient or effective? DESIGN What strategy, interventions and approaches should my program use to achieve these priorities? 3 2 1 3 4 5
  13. 13. What is an M&E Plan? An M&E Plan is a document that describes a system which links strategic information obtained from various data collection systems to decisions that will improve programs.
  14. 14. Functions of an M&E Plan • Describes how the program is going to measure achievements (ensure Accountability) • Document consensus (encourage Transparency and Responsibility) • Guide M&E implementation (Standardization and Coordination) • Preserve Institutional Memory M&E plan is a living document and needs to be adjusted when necessary.
  15. 15. Elements of M&E Plan 1. Brief project description 2. Purpose(s) of M&E plan 3. Brief history of M&E plan development 4. Evaluation framework 5. Indicator system 6. Information system (Data sources) 7. Dissemination and utilization plan 8. Adjustments to M&E plan
  16. 16. Group Activity: M&E Plans • Do you have M&E plans (or something similar)? • What are the components of the plans? What sections do the plans include? • Look at Handout 1: Template for M&E plan • What is missing from your programs’ M&E plans? • What additional components do your programs’ M&E plans include?
  17. 17. Session 3 Evidenced-based Programming for VAW
  18. 18. What do we know about violence against women? • Around the world, at least one out of three women is beaten, coerced into sex or otherwise abused during her lifetime • Women are most at risk at home and from men they know, usually a family member or spouse
  19. 19. Definition “Any act of gender-based violence that results in, or is likely to result in, physical, sexual, or psychological harm or suffering for women, including threats of such acts, coercion, or arbitrary deprivations of liberty, whether occurring in public or private life.” United Nations General Assembly, 1993
  20. 20. Violence Against Women Includes: • Physical violence • Sexual violence • Emotional violence • Economic violence
  21. 21. Violence Against Women Includes: • Violence that takes place in the family – domestic abuse – incest • Violence that occurs in the community – sexual assault – sexual harassment at work and at school • Violence that is organized or tolerated by governments
  22. 22. Violence against women begins early... • female infanticide • neglect, malnutrition • child sexual abuse • trafficking • female genital mutilation/cutting • child marriage
  23. 23. And continues throughout the life cycle… • physical and sexual partner abuse • dowry and “honor” related violence • coerced sex (within marriage or outside) • forced prostitution & trafficking • sexual harassment at work and at school • rape as a weapon of war
  24. 24. Physical and sexual violence is extremely common in women’s lives Prevalence of physical or sexual violence against women by anyone (partners and others), since age 15 years 0 20 40 60 80 100 B angladesh city B angladesh province B razilcity B razilprovince Ethiopia province Japan city N am ibia cityPeru city Peru provinceSam oa Serbia and M onteneg.. Thailand city Thailand province U nited R epublic ofT... U nited R epublic ofT... percentage Source: WHO Multi-country Study on Women’s Health and Domestic Violence against Women, 2005
  25. 25. 91 82 74 65 63 62 59 59 54 48 47 47 41 37 15 4 8 12 15 18 16 18 18 20 29 35 19 28 23 30 5 10 14 20 19 22 23 23 26 23 18 34 31 40 55 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Ethiopia province B angladesh province Thailand province Thailand city U .R ep.Tanzania province B angladesh city Japan city B razil province Serbia & M ontenegro city N am ibia city Peru province U .R ep.Tanzania city Peru city B razil city Sam oa % partner only partner and non-partner non-partner only Women’s greatest risk of violence is from a partner Source: WHO Multi-country Study on Women’s Health and Domestic Violence against Women, 2005
  26. 26. Domestic violence is prevalent in all countries, but the levels can very greatly among settings Japan and Serbia: 4% current violence Bangladesh, provinces in Ethiopia, Peru and Tanzania: 30% to 54% current violence
  27. 27. Prevalence of forced sexual initiation 48% 40% 37% 29% 28% 21% 19% 9% 7% Female adolescents reporting forced sexual initiation, as % of those having had sex (population-based surveys 1993-99) Source: WHO’s World Report on Violence and Health, 2002
  28. 28. Violence against women is a violation of human rights and a significant development challenge • UN Declaration on Violence against Women (1993) • UN Human Rights Summit (Vienna, 1993) • International Conference on Population and Development (Cairo, 1994) • Fourth World Conference on Women (1995) • World Health Assembly resolution on violence as a public health priority (1996) • UN Secretary General’s In-depth Study on VAW (2006)
  29. 29. Violence increases women’s risk for …  Fatal outcomes  homicide  suicide  maternal deaths  HIV/AIDS-related deaths  Non-fatal outcomes  physical  mental  high risk behaviors  reproductive health For example:  unwanted pregnancy  chronic pain syndromes  injury  depression  alcohol/drug use  STDs/HIV  gynecological disorders  low birth weight
  30. 30. Violence against women in the region Group Exercise • What are the most common forms of violence against women and girls in the region? • What are the main barriers to addressing them?
  31. 31. Key principles for evidence- based programming on VAW
  32. 32. Guiding Principles in GBV Programming The underlying principle is “Do no harm” • Respect survivors’ safety and autonomy • Ensure relevance and appropriateness of interventions to local setting • Use a human rights perspective
  33. 33. Promising practices for addressing VAW • Increasing access to justice • Providing support to survivors of violence • Prevention of VAW
  34. 34. Violence occurs at many levels…and therefore must be addressed at many levels Society Community Families/ household Individual woman/girlSociety Laws that sanction VAW and protect survivors National plans on VAW Media campaigns to change gender norms Address VAW in national institutions (judiciary, education, welfare, etc.) Community based networks for referrals and prevention Multi-sector, integrated services for survivors of VAW Community mobilization to change norms Support for organizations that protect rights of women Engage men as partners in prevention VAW Work with youth (boys and girls) to promote gender equity Girls’ education, Opportunities for economic empowerment, Information on civil/political rights Increased mobility and autonomy
  35. 35. Improved policies and programs – National Commissions and Plans against Violence – More services for victims of violence (women’s police stations, NGO women’s services) – Policies and protocols for education and other sectors
  36. 36. Promising practices in strengthening women’s access to justice • Improved laws on domestic and sexual violence • Police procedures that require mandatory arrest of abusers or “no drop” policies • Protective/restraining orders that can be easily accessed by women • Specialized police stations for women • Promoting coordination with traditional justice system
  37. 37. Lessons learned from the justice system • Train all justice personnel (judges, police, forensic doctors, clerks, etc.); address attitudes as well as skills • Human and financial resources • Multi-sectoral coordination • Public education on how to use the laws • Monitor implementation of the law to see if reporting of violence and detention of abusers increases
  38. 38. Promising practices to increase support for survivors • Women’s safe houses or shelters • Crisis centers (legal, medical, psychological support – usually ngos.) • One- Stop centers (all services in one place – usually a hospital or police station)
  39. 39. Lessons learned from programs to increase access to services • Services should be integrated (with different types of support in one place) and coordinated through referral networks • Specialized centers are mostly in urban centers and may be too costly for rural areas • Community support networks can provide safe haven and support for survivors of violence in rural areas
  40. 40. Prevention of violence is key! • Community mobilization • Mass media / Communication for social change programs • Engaging men and boys in violence prevention • Women’s economic empowerment
  41. 41. Communication, or “edutainment” programs can help prevent violence • Model new behaviors and ways of thinking • Create an enabling environment for increased communication about gender, sexuality • Create synergies and enhance interpersonal communication • It is possible to document and measure impact (GEM Scale)
  42. 42. Lessons learned for effective communications campaigns • “One-off” campaigns are less effective than ongoing programs • Multiple strategies and techniques and permanent saturation of alternative messages should be used • Importance of working with journalists to change the way VAW is addressed in main stream media
  43. 43. Raising Voices and CEDOVIP in Uganda Promising practices for community mobilization
  44. 44. Promising practices in engaging men and boys as partners in violence reduction • MenEngage.org
  45. 45. Promising practices in women’s empowerment… •Information and education •Technologies and services •Economic assets and income •Social support and networks •Leadership and political participation
  46. 46. Image Program in South Africa Microfinancing and training on violence, together with community mobilization activities reduced domestic violence by 50% in intervention group over 2 years
  47. 47. Addressing VAW in schools • Sexual harassment and violence are widespread in schools, (both peers and teachers) and an obstacle to girls educational advancement • Schools teach values around gender, conflict resolution, sexual and reproductive health
  48. 48. 48 Lesson Learned in Addressing Violence in Education programs • Anti violence and gender norms need to be integrated into school curriculum, including family life skills • Address sexual violence in schools through sexual harassment policies and codes of conduct
  49. 49. We need to invest in evaluation! • We need to know what works and what doesn’t • More rigorous designs and rethink success indicators • National level indicators to monitor overall trends
  50. 50. References • Barker, G., C. Ricardo, et al. (2007). Engaging Men and Boys in Gender Equality: Evidence from Programme Interventions. Geneva, World Health Organiztion: 70. • Ellsberg, M. and L. Heise (2005). Researching violence against women: A practical guide for researchers and activists. Geneva, Switzerland, Program for Appropriate Technology in Health, World Health Organization, . • Morrison, A., M. Ellsberg, et al. (2007). "Addressing Gender- Based Violence: A Critical Review of Interventions." The World Bank Research Observer 22(1): 25-51. • Michau, L. and D. Naker (2003). Mobilising Communities to Prevent Domestic Violence. Kampala, Uganda, Raising Voices. • Velzeboer, M., M. Ellsberg, et al. (2003). Violence against Women: The Health Sector Responds. Washington, DC, Pan American Health Organization, PATH. • World Health Organization (2005). WHO Multi-Country Study on Women's Health and Domestic Violence Against Women: Initial Results on Prevalence, Health Outcomes and Women's Responses. Geneva, Switzerland, WHO.
  51. 51. Session 4 Program Goals and Objectives Adapted from: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  52. 52. Program Goals A goal is a broad statement of a desired, long-term outcome of a program. Goals represent general big picture statements of desired results. • Examples: – To improve the lives of women who experience gender- based violence – To decrease injury and mortality due to gender-based violence – To eliminate gender-based violence Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  53. 53. Program Objectives (Outcomes) • Objectives: statements of desired, specific, realistic and measurable program results • SMART – Specific: identifies concrete events or actions that will take place – Measurable: quantifies the amount of resources, activity, or change to be expended and achieved – Appropriate: logically relates to the overall problem statement and desired effects of the program – Realistic: Provides a realistic dimension that can be achieved with the available resources and plans for implementation – Time-bound: specifies a time within which the objective will be achieved Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009: (GAP 2003) NOTE: On UN Trust Fund logframes the “objective” is the “outcome”.
  54. 54. How to Write Program Objectives (Outcomes) To [action] The (what) [specify what knowledge, attitudes, skills, behaviors] Among (who) [specific population or segment] From – to [from baseline to desired level] or by [ x percent ] or to [specific level By (when) [time frame] Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  55. 55. Discussion: Is it a Goal or an Objective (Outcome)? Is it a goal or an objective? If an objective, is it SMART? Why or Why not? • To reduce violence against women • To increase the % of men and women in beneficiary population who believe that violence is not an acceptable way of dealing with conflict from 40% in 2002 to 80% by 2007 • To increase the number of domestic violence clients recommended for counseling who utilize counseling services by 30% Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  56. 56. Group Activity In your groups: • Choose two statements from the list on the handout • Decide whether the statement is a goal or an outcome • If a goal, is it properly stated? Why or why not? If an outcome, is it SMART? Why or why not? • If an objective (outcome) is not SMART, rewrite it in order to make it SMART.
  57. 57. Session 5 Conceptual Frameworks, Logic Models and Logical Frameworks Adapted from: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  58. 58. Conceptual Frameworks Diagram that identifies and illustrates the relationships between all relevant systemic, organizational, individual, or other salient factors that may influence program/project operation and the successful achievement of program or project goals. M&E Purpose: • To show where program fits into wider context • To clarify assumptions about causal relationships • To show how program components will operate to influence outcomes • To guide identification of indicators • To guide impact analysis (causal pathways) Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  59. 59. Illustrative Conceptual Model for IPV Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009 Society Community Relationship Individual Perpetrator •Norms •Laws •Notions of masculinity •Gender roles •Poverty •Unemployment •Family isolation •Community acceptance of violence •Marital conflict •Male control of wealth •Male control of decision-making in family •Being male •Witnessing marital violence as child •Being abused as child •Absent or rejecting father •Alcohol use Source: Heise 1998 Society
  60. 60. Logic Models Logic models are diagrams connecting program inputs to activities, outputs, outcome and impact as they relate to a specific problem or situation. Logic models show what resources your program will need to accomplish its goals, what your program will do, and what it hopes to achieve, emphasizing the links between all these things. Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  61. 61. Logic Model Components • Input – Resources used in a program (e.g., money, staff, curricula, and materials) • Activity– Actions taken or work performed through which inputs such as funds, technical assistance, and other types of resources are mobilized to produce specific outputs • Output – Direct results of program activities (e.g., people trained; materials distributed; couples counseled, etc.) • Outcome – Program results at population level (e.g., knowledge, attitudes, behavior, coverage) • Impact – Long-term change in morbidity, mortality, and fertility Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009 and UNAIDS MERG, 2008
  62. 62. Logic Model Shows Intended Results 6months – 1 year After implementation Inputs Activities (work plans) Outputs (Services) Outcomes Goal immediate immediate 5 years or more2 to 5 years efficiency effectiveness indicators Programme Level Global Level Source: UNIFEM RBM Curriculum: Module 3a & 3b: The Results Chain; and The World Bank Global HIV/AIDS Programme and UNAIDS 62
  63. 63. If-Then Logic – Cause and Effect Activity Output Outcome Goal then If If If if if if then then Input if then
  64. 64. 64 INPUT •Human and financial resources to develop training materials & implement training program ACTIVITIES •Develop GBV training curriculum •Conduct TOT workshops •Conduct GBV training for providers OUTCOME •Increased awareness of GBV •Increased disclosure of GBV •Increased knowledge and utilization of GBV services Illustrative Logic Model for Provider Training Program Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009 IMPACT •Improved well-being and safety of GBV survivors OUTPUT •Providers trained in GBV •Improved provider attitudes toward GBV •Increased provider ability to identify, counsel, care for, and refer GBV survivors
  65. 65. Role of Logic Model • Program planning – Helps one think through program strategy – where you are & where you want to be • Program management – Helps one track and monitor operations to better manage results • Communication – Shows stakeholders at a glance what program is doing and what it expects to achieve Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  66. 66. Role of Logic Model • Consensus building – Promotes common understanding about what program is, how it works and what it is trying to achieve • Fundraising – Can structure and streamline grant writing • Monitoring and evaluation – Provides thorough understanding of what resources program has to work with, what it is doing, and what it hopes to achieve. Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  67. 67. 67 Group Activity: Identifying Logic Model Components • Return to earlier groups • Each of the scenarios in handout #3 corresponds to one of the five components in the logic model: – Input, activity, output, outcome, and impact. • In your groups, decide which component the scenario illustrates. • Discuss in plenary
  68. 68. Logical Frameworks A logical framework (LogFrame) is a tool for strategic planning, and management of a program/project. It is presented as a table and includes the most pertinent information about the project in a consise, logical and systematic manner.
  69. 69. Logical frameworks Logical frameworks answer the following questions in a standardized way: • What does the project aim to achieve? • How will it be achieved • What is needed to make the project a success? • How will progress and potential problems be assessed?
  70. 70. Logical frameworks • Summarize what the project plans to achieve and why • Summarize key hypotheses • Summarize the results and products that were monitored and evaluated
  71. 71. UNIFEM Definitions Term Definition Example Goal The higher-order objective to which a development intervention is intended to contribute. The change is long- term and achieved through collective efforts of many partners. Morbidity and mortality are reduced among VAW/G survivors Outcome Intended or achieved short and medium term effects of an intervention’s outputs. Outcomes represent changes in development conditions which occur between the completion of outputs and the achievement of impact. At least 75% of VAW/G survivors who received appropriate care Output The products, services, and capacities resulting from completion of activities. 50 service providers trained to care for and refer VAW/G survivors who demonstrate increased skills Activity Actions taken or work performed through which inputs are mobilised to produce specific outputs. Training of 50 service providers to care for and refer VAW/G survivors Source: UNIFEM (2005). Essential Guide to RBM; and UNIFEM RBM Curriculum: Module 3a & 3b: The Results Chain 71
  72. 72. Project Team Group Work: Developing a Logic Model/Framework Part 1: • Work in project teams • Develop/revise a logic model for at least one of your program outcomes • Questions to address: – Have you expressed your outcomes in terms of change? – Do activities, outputs, and outcomes relate to each other logically (the if-then relationship)? – Does your program have adequate resources to implement the activities and achieve the desired outcomes? – Have you included all of the major activities needed to implement your porgram and achieve the expected outcome(s) – Would the activities listed enable someone who is unfamiliar with your program to understand its scope?
  73. 73. Project Team Group Work: Developing a Logic Model continued Part 2: Compare the logic model to the logical frameworks you have developed for your UN Trust Fund program – What is the same? What is different? How can creating a logic model help when developing a logical framework? What additional information is in the logical framework?
  74. 74. Session 6 Choosing Appropriate Indicators for Different Types of Interventions Adapted from: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  75. 75. What is an indicator? An indicator is a specific, observable and measurable characteristic that can be used to show changes or progress a program is making toward achieving a specific outcome. Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  76. 76. Characteristics of Good Indicators • Valid: accurate measure of a behavior, practice or task • Reliable: consistently measurable in the same way by different observers • Precise: operationally defined in clear terms • Measurable: quantifiable using available tools and methods • Timely: provides a measurement at time intervals relevant and appropriate in terms of program goals and activities • Programmatically important: linked to the program or to achieving the program objectives that are needed for impact Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  77. 77. Indicators do not specify a particular level of achievement -- the words “improved”, “increased”, or “decreased” do not belong in an indicator. Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  78. 78. Common Indicator Metrics • Counts – # of legal and service organizations for VAW/G survivors – # of women and children using VAW/G social welfare services • Calculations: percentages, rates, ratios – % of sites referring survivors of GBV with at least one service provider trained to care for and refer survivors – % of women aged 15-49 who have ever experienced physical violence from an intimate partner • Index, composite measures – Gender-equitable men (GEM) scale – Sexual relationship power scale • Thresholds – Presence, absence; pre-determined level or standard Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  79. 79. Proxy Indicators • Use a "proxy" indicator when you cannot directly measure a result • Result: Decreased trafficking of women and girls from country X. – Direct Indicator: # of women and girls moved across the border against their will or under false pretences per year – Proxy Indicator: # of women and girls reported missing / lost by families in country x – Proxy Indicator: # of arrests of persons operating trafficking rings in country x Source: Source: UNIFEM RBM Curriculum: Module 5: Indicators
  80. 80. Common Challenges in Indicator Selection • Choosing an indicator that program activities cannot affect • Choosing an indicator that is too vague • Indicators that do not currently exist and cannot realistically be collected • Selecting an indicator that does not accurately represent the desired outcome • Too many indicators Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  81. 81. How Many Indicators Are Enough? • No more than one or two indicators per key activity or result – (e.g., training, BCC) – (ideally, from different data sources) • Use a mix of data collection methods/source • Use indicators that track change over the life of the project Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  82. 82. Factors to Consider When Selecting Indicators • Logic/link to framework • Programmatic needs/information for decision making • Data availability • Resources – HR capacity, data collection forms, $$$ • External requirements (government, donor, headquarters) • Standardized indicators (if available) Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  83. 83. Group Activity: Selecting Program Indicators • Return to groups from previous activities • Select indicators that your group might use to measure progress towards your program goals and outcomes with the following considerations: – To what extent are the indicators logically linked to your logframe? – Do the proposed indicators share the characteristics of good indicators? – How will the indicators be used for program decision-making and for what decisions? – Are data available to measure the indicators? – Can the indicators be realistically collected given available resources? Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  84. 84. Where to Find Standardized Indicators for GBV Programs Bloom, S. 2008. Violence Against Women and Girls: A Compendium of Monitoring and Evaluation Indicators. Ms.08-30. Chapel Hill, NC: Carolina Population Center, the University of North Carolina and Chapel Hill. http://www.cpc.unc.edu/measure/publications/pdf/ms-08-30.pdf UN Division for the Advancement of Women (UNDAW), UN Economic Commission for Europe (UNECE) and UN Statistical Division. 2008. Indicators to measure violence against women. Report of the Expert Group Meeting, 8 to 10 October 2007, Geneva, Switzerland; UN Human Rights Council. 2008. Report of the Special Rapporteur on violence against women, its causes and consequences by Yakin Ertürk. Indicators on violence against women and State response. 25 February 2008. Friends of the Chair of the United Nations Statistical Commission on the indicators on violence against women by the Secretary General. 11 September 2008. Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  85. 85. Session 7 Indicators for VAW programs Adapted from: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  86. 86. Illustrative Indicators: Community Mobilization • Proportion of people who would assist a woman being beaten by her husband or partner • Proportion of people who say that wife beating is an acceptable way for husbands to discipline their wives • Proportion of people who agree that rape can take place between a man and woman who are married • Proportion of youth-serving organizations that include training for beneficiaries on sexual and physical VAW/G Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  87. 87. Illustrative Indicators: Behavior Change Communication • Proportion of people who have been exposed to VAW/G prevention messages • Proportion of girls who say that they would be willing to report any experience of unwanted sexual activity • Proportion of girls that feel able to say no to sexual activity • Proportion of individuals who know any of the legal sanctions for VAW/G Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  88. 88. Illustrative Indicators: Service Delivery • Proportion of rape survivors who received comprehensive care • Proportion of law enforcement units following a nationally established protocol for VAW/G complaints • Proportion of women who know of a local organization that provides legal aid to VAW/G survivors • Number of women and children using VAW/G social welfare services Source: Bloom, S. 2008. Violence Against Women and Girls: A Compendium of Monitoring and Evaluation Indicators.
  89. 89. Illustrative Indicators: Law and Public Policy • Number of law enforcement professionals trained to respond to incidents of VAW/G according to an established protocol • Number of VAW/G complaints reported to the police • Proportion of VAW/G cases that were investigated by the police Source: Violence Against Women and Girls: A Compendium of Monitoring and Evaluation Indicators. Ms.08-30. 2008
  90. 90. Activity: Matching indicators to components in the logic model • Return to small groups from previous activities • Look at the indicators that your group is given by the facilitator and determine what component of the logic model each indicator fall into. • Discussion: Which indicators fall where and why?
  91. 91. Measuring Indicators Indicator 1: # Number of teacher training programs that include sexual and physical VAW/G in their Curriculums “trained” refers to attending every day of a five-day training course on VAW/G Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  92. 92. Measuring Indicators Indicator 2: % Proportion of rape survivors who received comprehensive care Numerator: Number of rape survivors who received the comprehensive package of care as defined by international and national guidelines. Denominator: Record review: Total number of rape survivors who presented at each facility under the program’s domain. Exit interview: Total number of women interviewed. Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  93. 93. Measuring Composite Indicators • Example: How to measure, “attitudes toward VAW” ? • What question do you ask? • How do you know any one question is a good representation of general attitude? • How do you scale each question (agree/disagree; strongly disagree to strongly agree; yes/no)? • How do you combine answers to the questions to come up with one indicator? • What value or weight do you assign to any one question on GBV attitude in the survey? Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  94. 94. Example of Measuring Attitudes Toward Wife Beating in the DHS • Sometimes a man is annoyed or angered by things that his wife does. In your opinion, is a man justified in hitting or beating his wife in the following situations: – If she goes out without telling him? – If she neglects the children? – If she argues with him? – If she refuses to have sex with him? – If she burns the food? Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  95. 95. Project Team Activity: Specifying Indicator Metrics • Work in your project teams • Look at the indicators that your group has selected to measure progress towards your goals and objectives • Define the metrics for each indicator Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  96. 96. Session 8 Quantitative Evaluation Designs
  97. 97. Guiding Questions for Choosing an Evaluation Design • What question does your program need to answer? • What do you want to measure (indicators)? • How sure do you want to be? (VERY SURE = GOOD ) – What is the cost of making a mistake (low, medium, high)? • When do you need the results? (FAST) • How much are you willing to pay? (CHEAP) • Has the program already started? Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  98. 98. When monitoring and evaluating VAW programs, the confidentiality, privacy, safety, and well-being of VAW survivors must be top priority at all times. Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  99. 99. Causality Requirements • A (program) precedes B (outcome) • B (outcome) is present only when A (program) is present • We can rule out all other possible causes of B (outcome).
  100. 100. The Basic Experimental Principle • The intervention is the only difference between two groups • This is achieved by random assignment
  101. 101. An Experimental Design Experimental group Control group O1 X O2 O3 O4 RA
  102. 102. Factors that May Lead Us to Make Invalid Conclusions • Dropout: There may be loss to follow-up. • Instrumentation effects: Occur when a questionnaire is changed between pretest and posttest. • Testing effects: Occur because study participants remember questions that were asked of them at pretest and perform better at posttest because they are familiar with the questions.
  103. 103. A Second Experimental Design Experimental group Control group X O2 O4 RA
  104. 104. A Non-Experimental Design Experimental group O1 X O2 Time In this method of evaluation, only people who are participating in the program get the pre- and post-test Steps 1. Pre-test everyone in the program. 2. Deliver the intervention. 3. Post-test the same individuals.
  105. 105. A Non-Experimental Design- Weaknesses • Does not provide any information about what kinds of results might have occurred without the program and is the weakest in terms of scientific rigor • History effects: These occur when extraneous events (events that occur outside the study) influence outcomes that are measured by a study.
  106. 106. A Second Non-Experimental Design: Time Series Experimental group O1 O2 O3 X O4 O5 O6 Time Data are collected (typically using a survey) multiple times - before, during, and after a program
  107. 107. Time Series Design Steps: 1. Select a program outcome measure that can be used repeatedly. 2. Decide who will be in the experimental group. Will it be the same group of people measured many times, or successive groups of different people? 3. Collect at least three measurements prior to the intervention and made at regular intervals. 4. Check the implementation of the intervention. 5. Continue to collect measurements at least through the duration of the program.
  108. 108. A Quasi-Experimental Design Experimental group O1 X O2 Time O3 O4Comparison group --------------------------------- Two groups which are similar, but which were not formed by random assignment, are measured both before and after one of the groups gets the program intervention
  109. 109. Summary Features of Different Study Designs True experiment Quasi-experiment Non-experimental Partial coverage/ new programs Control group Strongest design Most expensive Partial coverage/ new programs Comparison group Weaker than experimental design Less expensive Full coverage programs -- Weakest design Least expensive
  110. 110. Summary Features of Different Study Designs I. Non-experimental (One-Group, Post-Only) II. Non-experimental (One-Group, Pre- and Post- Program) IMPLEMENT PROGRAM ASSESS TARGET GROUP AFTER PROGRAM ASSESS TARGET GROUP BEFORE PROGRAM IMPLEMENT PROGRAM ASSESS TARGET GROUP AFTER PROGRAM
  111. 111. Summary Features of Different Study Designs III. Experimental (Pre- and Post-Program with Control Group) RANDOMLY ASSIGN PEOPLE FROM THE SAME TARGET POPULATION TO GROUP A OR GROUP B TARGET GROUP A CONTROL GROUP B ASSESS TARGET GROUP A IMPLEMENT PROGRAM WITH TARGET GROUP A ASSESS TARGET GROUP A ASSESS CONTROL GROUP B ASSESS CONTROL GROUP B
  112. 112. Summary Features of Different Study Designs IV. Quasi-Experimental (Pre- and Post-Program with Comparison Group but not randomized) ASSESS TARGER GROUP BEFORE PROGRAM IMPLEMENT PROGRAM ASSESS TARGET GROUP AFTER PROGRAM ASSESS COMPARISON GROUP BEFORE PROGRAM ASSESS COMPARISON GROUP AFTER PROGRAM
  113. 113. Summary Features of Different Study Designs-ctd • The different designs vary in their capacity to produce information that allows you to link program outcomes to program activities. • The more confident you want to be about making these connections, the more rigorous the design and costly the evaluation. • Your evaluator will help determine which design will maximize your program's resources and answer your team's evaluation questions with the greatest degree of certainty.
  114. 114. Important issues to consider when selecting a design • Complex evaluation designs are most costly, but allow for greater confidence in a study's findings • Complex evaluation designs are more difficult to implement, and so require higher levels of expertise in research methods and analysis • Be prepared to encounter stakeholder resistance to the use of comparison or control groups, such as a parent wondering why one group of individuals will not receive a potentially beneficial intervention • No evaluation design is immune to threats to its validity; there is a long list of possible complications associated with any evaluation study. However, your evaluator will help you maximize the quality of your evaluation study
  115. 115. Activity: Choosing a study design Return to your small groups from previous activities. Take 20 minutes to discuss the following questions using the examples below: 1. What outcome indicators would you use? 2. What study design would you choose ? Example 1: A family planning service wishes to determine if carrying out a screening and referral program for violence will lead to more women being able to successfully use family planning methods. Example 2: What study design would you recommend to test the hypothesis that women who are offered counseling services for violence during family planning counseling sessions are more likely to use family planning than women are not offered services?
  116. 116. Evaluation Questions • Are the services available? • Are services accessible? • Is the quality of services provided adequate? • Are the services being used? • Is the target population being reached? • Were there improvements in patterns of violence or behaviors?
  117. 117. Adequacy Assessment • Adequacy studies only describe if a condition is met or not – Typically addresses provision, utilization or coverage aspects. No need for control, pre/post data in such cases • Hypothesis tested: Are expected levels achieved? – Can also answer questions of impact (magnitude of change) provided pre/post data is available • Hypothesis tested: Difference is equal or greater than expected
  118. 118. Features of Adequacy Assessment • Simplest (and cheapest) of evaluation models, as it does not try to control for external effects. Data needed only for outcomes • If only input or output results are needed, then the lack of controls is not a problem • When measuring impact, however, it is not possible to infer that the change is due to the program due to lack of controls. • Also, if there is no change, it will not be possible to say whether the lack of change is due to program inefficiency or if the program has impeded a further deterioration
  119. 119. Adequacy Assessment Inferences • Are objectives being met? – Compares program performance with previously established adequacy criteria – No control group – 2+ measurements to assess adequacy of change over time • Provision, utilization, coverage – Are activities being performed as planned? • Impact – Are observed changes in of expected direction and magnitude? • Cross-sectional or longitudinal Source: Habicht, Victora and Vaughan (1999)
  120. 120. Group Discussion • What are the advantages of adequacy evaluations? • What are the limitations of adequacy evaluations? • If an adequacy evaluation shows a lack of change in indicators, how can this be interpreted? • Which of the study designs discussed earlier can be used for adequacy evaluations?
  121. 121. References • Adamchak S et al. (2000). A Guide to Monitoring and Evaluating Adolescent Reproductive Health Programs. Focus on Young Adults, Tool Series 5. Washington D.C.: Focus on Young Adults. • Fisher A et al. (2002). Designing HIV/AIDS Intervention Studies. An Operations Research Handbook. New York: The Population Council. • Habicht JP et al. (1999) Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. International Journal of Epidemiology, 28: 10-18. • Rossi P et al. (1999). Evaluation. A systematic Approach. Thousand Oaks: Sage Publications.
  122. 122. Session 9 Quantitative Methods and Tools Adapted from: Module II M&E GBV Prevention and Mitigation Programs, June 2009 Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  123. 123. Types of Information Sources • Quantitative data – Useful for tracking trends accurately and highlighting differences • Qualitative data – Useful for understanding the context in which the trends/differences occur and to interpret quantitative data accurately (e.g. focus groups; in-depth interviews) Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  124. 124. Quantitative Information Sources • Population-based surveys – Cross-sectional surveys • DHS (domestic violence module); VAW survey • Facility surveys – Client exit interviews; provider surveys; observing service delivery • Program statistics • Crime statistics (of poor quality in many LDCs) • Behavioral surveillance systems Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009
  125. 125. Group Activity: Advantages and Disadvantages of Quantitative Data Sources • Return to groups from previous activities • You will be given one of the data sources from the previous slide • Discuss: – What are the advantages of using the sources of information? – What are the disadvantages of using this source of information? – Have your programs used this type of information before? In what way?
  126. 126. Triangulating Data Sources: Soul City Source: Module II M&E GBV Prevention and Mitigation Programs, June 2009 Individual 1. National Survey Community Society 2. Sentinel Site Studies 5. Media Monitoring and Analysis 4. NNVAW Partnership Study 6. Cost – Effectiveness Study 3. (National) Qualitative Impact Assessment
  127. 127. Ethical and safety issues Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005 • Ethical Review Process • Prioritise women’s safety • Protect confidentiality: essential to ensure women’s safety and data quality • Selection and specialised training and on-going support for research team • Take actions to reduce any possible distress to participants (referrals, information). • Ensure findings are properly interpreted and used
  128. 128. Issues that influence how violence is measured • How to define the study population? • How violence is defined • How the interview is performed Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  129. 129. Defining a study population • Cutoff ages • Marital experience Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  130. 130. Effect of “study population” on reported prevalence estimates in Nicaragua Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005 All women 15-49 Ever married women 15-49 Currentl y married women 15-49 Formally married women 15-49 Current physical violence 20% 27% 30% 17%
  131. 131. Defining Violence • Who defines: the researcher or the respondent? • Time frame • Multiple perpetrators • Frequency • Types and severity of violence included in the definition Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  132. 132. Factors that Affect Disclosure • How the questions are phrased • Number of opportunities to disclose • Context in which questions are asked • Characteristics and skill of interviewers • Social stigma attached to issue
  133. 133. Suggestions for Measuring Domestic Violence • Define the study population broadly • Use behaviorally specific questions • Give multiple opportunities to disclose • Specify discrete time frames (last year, ever) • Cue respondent to different contexts and offenders
  134. 134. Quantitative Tools: Standardized Questionnaire • Developing the conceptual framework • Operationalizing the main variables • Formulating questions • Formatting the questionnaire • Translating the instrument • Pre-testing the instrument
  135. 135. Developing a conceptual Framework • What information do you need? • How will you measure this information? Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  136. 136. Formulating Questions • Closed questions • Open questions Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  137. 137. Formatting the Questionnaire • Sequence of questions – Early questions should be easy and pleasant to answer – Place difficult or sensitive questions toward the end – Don’t assume you can predict what will be considered sensitive if you are working in another culture • Appropriate answer scales – Be careful when using complex scales across different populations and cultures • Appropriate skip patterns – Make sure skip patterns are correctly placed Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  138. 138. Translating the Instrument • Accurate and precise translation • Involve activists or service providers in translating the test instrument Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  139. 139. Pre-testing the Instrument • Try the questionnaire on women who do not live in the research area, but who are similar to the women you are planning to study • Try the test on women of different ages and backgrounds, including women who are known to be experiencing different forms of abuse • Pre-test translated versions Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  140. 140. Session 10 From Research to Action Adapted from: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  141. 141. How to use evaluation findings for advocacy • Outreach to key constituencies • Matching the message to the audience • Sharing findings with the community • Reaching beyond your borders Source: Researching Violence Against Women: A Training Course for Researchers and Activists, 2005
  142. 142. Let’s create harmony in intimate relationships. Thailand
  143. 143. Silence for the Sake of Harmony, Central Java, Indonesia
  144. 144. Candies in Hell … After the beatings he would court me and buy my clothes, but my grandmother would say to me, child, what are you going to with candies in hell?
  145. 145. MOMMY, YOU LOOK LIKE A MONSTER… “When he would beat me, my daughters would get involved in the fight. Then he would throw them around in his fury, and this hurt me more than when he beat me. …and once, I was recovering after he had beaten me, and my daughter came up to me and said “Mommy, you look like a monster.” And she began to cry, and what really hurt me wasn’t so much the blows, it was her sobbing and the bitterness that she was feeling… Survivor of violence from Nicaragua 49 23 10 5 0 10 20 30 40 50 60 Children Husband's family Husband's friends Wife's friends % of women who report witnesses to violence against them
  146. 146. From Insight to Action • What experiences have you had in presenting evaluation results to different stakeholders? • What are strategies that have been most successful?

×