• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
2010 ALLIES Learning Exchange: Sheherazade Hirji- Evaluation

2010 ALLIES Learning Exchange: Sheherazade Hirji- Evaluation






Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    2010 ALLIES Learning Exchange: Sheherazade Hirji- Evaluation 2010 ALLIES Learning Exchange: Sheherazade Hirji- Evaluation Presentation Transcript

    • AGENDA • Introductions • Current context • Why evaluate? • Evaluation 101 • Evolution in evaluation thinking • What works • Developing an Evaluation Plan • Evaluation in action – participant activity
    • INTRODUCTIONS • Participant introductions – Name, affiliation – What you hope to get out of this session
    • CURRENT CONTEXT “There are a number of interesting social value calculations in use throughout the sector. We also learned about their limitations in terms of data quality and comparability. We now appreciate that while each method generates actionable information for its own users, no one approach has yet emerged as the single best method. In fact, to some extent, it was the discipline and rigor of application that is the most important common ingredient among the methods. Each of the practitioners acknowledged the importance of their calculation model forcing them to make their assumptions explicit and transparent. It is only once the assumptions are laid bare, that a true debate about the merits of a program, strategy or grant relative to costs can fully be vetted and debated, even if not fully known with precision. “ Gates Foundation Cover Letter on the Report on Measuring and/or estimating Social Value Creation, December 2008
    • WHY EVALUATE AT ALL? • Accountability • Assessing impact - what difference did you make? • Learning: what works, what doesn’t • Building capacity • “The difference between what we do and what we are capable of doing would suffice to solve most of the world's problems.” Mahatma Gandhi • Sharing/transferring knowledge
    • EVALUATION 101 Distinction: 1. Monitoring = Accountability • Were funds used as agreed? • Did the grantee do what they said they would? • Mid-grant adjustments 2. Evaluation = Impact • What changed as a result of the funding? • What was learned about the issue/intervention? • So what? • Transferring/disseminating knowledge
    • EVALUATION 101 •Inter-related levels in evaluation: • The organization: vision, mission, mandate, capacity • The program: impact of funding •The issue: what’s different, needs to change
    • EVOLUTIONARY THINKING IN EVALUATION Assumptions Reality • Purpose is to prove • Focus is to improve • It’s about the grantee • Involves many stakeholders • Happens at the end • Starts as soon as program is conceived • Measures everything • Select indicators to help critical decisions • Looking for attribution • Satisfied with contribution and learning • Done by experts, pre-determined • Participatory, evolving process assumptions • Rigorous thinking • Rigorous methodology • Specific to organization/program age/stage • One size fits all • Also looks for impact and learning • Focus is accountability, measurement • Use learning and knowledge transfer to • Internally focused influence and inform decision-making, policy
    • EVALUATION AS A LEARNING TOOL • Starts with organizational strategy, mission, goals • Integrated into operational and planning cycle • Anchor and amplify it in existing activities • Allocate time and resources • Encourage “evaluative thinking”
    • EVALUATION AS A LEARNING TOOL • Mutual accountability • Creates space for conversation • Increases transparency, trust • Meets broader public agenda by sharing what we learn • Great social impact: what works/does not work • Increases efficiency and effectiveness
    • DEVELOPING AN EVALUATION PLAN • Develop Logic Model/Theory of Change • Evaluation work plan • Identify information required: quantitative output • Qualitative impact • Increase awareness/knowledge • Change attitudes • Change behaviours • Increase skill levels • Improve individual status • Improve community status • How/by whom will these be measured? • Approach – participatory, developmental, formative, summative? • Resources required (human, financial, technical)
    • DEVELOPING AN EVALUATION PLAN • Limit your evaluation plan to the actual population/community served and scope of activity • Avoid things outside your control (systemic barriers, regulations, for example) unless you intend to address these as part of the program and be accountable for changing them • Under-promise and over-deliver • Complex grants need external evaluation help. Cost can range from 5%-15% of total project costs • Provide staff/board the results and learn about what worked and could be improved in future programming. • Use the results to report to your funders
    • ALLIES GENERIC COMMUNITY LOGIC MODEL Key components                   Major Activities           Evidence process          Contributing factors     Project Outcomes  Ultimate  Impact is working                      to success   Increase in numbers  Assess and adapt  Adaptation of  All sectors are  of skilled  core model to  model to meet  represented  immigrants  reflect local  local conditions  and engaged in  employed in their  Research and baseline Improve‐ conditions. is accepted by  the process. field. data about local ment in  immigration and stakeholders. local hiring  employment. and  Increased number  systems  New ways of  of employers  related to  doing things  committed to  immigrant  Adapt or develop  emerge through  improving hiring  employmen There is  guides and other  reflection and  practices. t significant parti‐ resources for use  learning. cipation of  by local  employers. employers. Understanding of Evidence of  Increased   capacity  community strengths, financial  of employers to  assets and processes support for  learn, plan and  related to immigrant longer term  implement  employment (current Establish action  sustainability. strategies. programs, resources, Programs are  group comprised  leaders, etc.). developed and  of all relevant  implemented. sectors  to  Guides and  develop and  Increased  Increased  resources are  implement  community   economic  well used by  solutions. capacity to learn,  and social  employers and  plan and implement  capital in  others. Opportunity for  strategies. local  engagement of all local  community. sectors to address  There are  Local project  systems and barriers  current and  has ‘brand’  Create  potential  recognition.  Increased public  related to improving  opportunities to  leaders in the  Media  awareness and  appropriate  increase local  community who  coverage/  support for  employment for skilled  awareness and  are available  public events  appropriate hiring  immigrants. support for the  and interested  reinforce  of skilled  issue. in this issue. message.   immigrants.
    • EVALUATION IN ACTION • Applying what you have heard • Group exercise
    • WHAT WORKS • Clarity in purpose and audiences • Clarity on theory of change, assumptions • Supporting evaluation framework • Selecting a few indicators that help assess progress in each area • Focus on contribution, rather than attribution • Balance quantitative and qualitative • No stories without numbers and no numbers without stories • Share learning, celebrate successes • Tell the story as it unfolds, periodically tie themes together
    • RESOURCES • W.K.Kellogg Foundation Logic Model Worksheet www.wkkf.org/knowledge-center/resources/2004/05/Logic-Model- Worksheet.aspx • Developmental evaluation: The J.W.McConnell Family Foundation www.mcconnellfoundation.ca/en/resources • Community Builder’s Approach to Theory of Change http://tamarackcommunity.ca/g3s4_7.html • Measuring and/or estimating social value creation www.gatesfoundation.org/learning/Pages/december-2008-measuring- estimating-social-value-creation-report-summary.aspx