Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2016 and 2017 Return on Investment, Development Impact

This presentation follows on our previous work from measuring the impact and return on investment of social, community, enterprise development programs. This presentation provides evidence of our work, our methodology and the impact that we measure of development practices. Our impact assessment methodology was developed for Africa, by Africa and is aimed at practitioners from both the investment and development fraternity.

  • Login to see the comments

2016 and 2017 Return on Investment, Development Impact

  1. 1. Impact Assessments: Lessons Learnt 2015/2016 REANA ROSSOUW: NEXT GENERATION CONSULTANTS
  2. 2. Who we are:  Next Generation Consultants helps organisations to become more sustainable and have greater positive impact on the economy, society and the environment.  In the community/social investment and development sectors - we provide consulting and advisory, research and engagement, training and facilitation, impact assessment and due diligence services.  We have developed the Investment Impact Index™ - a methodology that measures the impact and return on investment of social/community and enterprise development investments.  We are recognised as industry/subject experts and thought leaders within the sustainable/social/ community development sectors. 2016/05/03Next Generation Consultants 2 Please see: community-development AND
  3. 3. Background:  Our vision  To continuously contribute to increased socio economic impact and ensure enhanced shared value and sustainability for all shareholders on the continent  Our Context  We have developed the Investment Impact Index (III)™ in 2009.  Since then we have assessed R3 billion worth of social/community and enterprise development investments. This included the assessment of more than 600 programs across 15 focus areas/investment portfolio’s.  As an outcome of our work we have developed an indicator library with more than 5 000 indicators. We have also identified 15 dimensions of impact and more than 15 dimensions of return.  Evidence  This presentation is an update and representation of our work, case studies and evidence of impact and return for major funders in Africa. In particular – to assist the industry with developing indicators to measure impact and return. 2016/05/03Next Generation Consultants 3
  4. 4. Why measure impact?  To improve and consider a range of stakeholder opinions, communication and reporting  To plan/revise a program/strategy  To strengthen/inform future work  To learn/confirm whether original objectives were achieved  To learn/understand/quantify and qualify outcomes, impact and return  To consider all input resources collectively  To provide an opportunity to not only confirm impact (change) but also to understand what was done right/wrong  It provides a holistic overview of all investments – collectively – and individually  To make resource allocation decisions  To share organisational and development practices within the field  To share best practices/lessons learnt  To learn about implementation  In support of advocacy/policy recommendations  To contribute to the development of a body of knowledge in the field/sector  To implement/determine sustainability scaling/replication strategies  To strengthen public policy 2016/05/03Next Generation Consultants 4
  5. 5. Guiding Principles of our work:  Impact means impact  The goal is to understand what changes as a result of investment from donors in communities as beneficiaries and recipients of interventions  The impact is shared  The goal is to understand who is impacted along the value chain – including donors, intermediaries and beneficiaries  Impact includes and involve all stakeholders  Analysis must be comprehensive. Instead of cherry picking something that’s working and leaving out what is not, the analysis should include all aspects of impact and those impacted  Results must be transparent  Companies should report to their investors, and investors should aggregate and report results. What is left out should be stated. Assumptions and sources should be stated. It should be possible for a third party to replicate the analysis based on the documentation of it and get the same result.  Context matters  It is harder to create a stable job in a rural area than in a city. The qualitative and quantitative context should be provided to inform the impact as well as an understanding of how much of the problem may exist or remain. 01/08/2013Next Generation Consultants 5
  6. 6. How the model works: (1) 01/08/2013Next Generation Consultants 6 HowWhatWhoImpact Community Impact Teachers Improved capacity Improved morale Short term Learners Improved pass rates Social Schools Improved enrolment Empowered Government Leverage of resources Cost Savings
  7. 7. How the model works (2) 2016/05/03Next Generation Consultants 7 • Strategic, operational and programmatic impact • Impact and Return per program • Impact and Return per portfolio • Impact and Return on total investment Investor (ROI) • Organisational Impact Intermediary (Impact) • Individual/Community Impact • Thematic Impact Beneficiary (Impact
  8. 8. Key attributes of the III™:  Measures value to both society and the business  The model builds on existing measures of value, complementing these with the broader impacts of business on society – and the social value created through the process – as required by current reporting frameworks  Backward and forward looking  The model can be applied looking backwards (past investments/programs) to understand the value generated and looking forward (future possible impact) to inform strategy and project/investment related decisions  Flexible: The model can be applied at multiple levels – for:  1. Singular programs/interventions  2. Specific projects in a particular portfolio (i.e. education, health, etc.)  3. Impacts in a country or region or sector (i.e. local development, national or mining)  4. Within a portfolio or across the entire investment portfolio (total investment portfolio)  5. Equally it can be applied in social/community/socio economic or enterprise development contexts 2016/05/03Next Generation Consultants 8
  9. 9. Key attributes continue:  A balanced understanding of impact  By covering all the key aspects of impact – the model provides a holistic and balanced view of value creation – not just positive impact but also negative impact, trade-off’s, causality and attribution  Consistent information  By analysing quantitative and qualitative data – through comparing, synthesizing and triangulating data – over time and between different strategic objectives – by involving stakeholders – a balanced and consistent view of impact can be built – agreed too and confirmed by all stakeholders  Comparable information  By equalising all impact and return (to the value of 1) – it provides for comparison across different types of impacts – which provides value no matter the type or size of investment/input resources  Produces decision ready/useful information  It provides a strengthened basis for decision-making, (for all stakeholders) and provides timely and reliable data that employs estimates, assumptions and attribution that are fit for purpose to make better informed decisions and engage stakeholders in meaningful discussions  Focuses on material impacts  One size does not fit all. The framework enables funders to select their focus and impact as well as return 2016/05/03Next Generation Consultants 9
  10. 10. State of current Practice: THE FOLLOWING RESEARCH DATA IS THE OUTCOME OF OUR OWN RESEARCH DURING IMPACT ASSESSMENTS 2016/05/03Next Generation Consultants 10
  11. 11. The state of M & E practices: 2016/05/03Next Generation Consultants 11 90% of all funders and intermediaries measure their work through quantitative assessment 75% of all funders and intermediaries claim they have the capacity to measure their work 58% of all funders and intermediaries have a framework to measure their work 24 % of all funders and intermediaries have a data evaluation framework 25% of all funders and intermediaries have an evaluation plan 5% of all funders collect qualitative date
  12. 12. Barriers to evaluation: 2016/05/03Next Generation Consultants 12 Limited staff time (71%) Insufficient financial resources (61%) Funders: Funders asking for the wrong data (32%) Funders and intermediaries: Don’t know where to find expertise/independent evaluators (21%) Limited staff expertise (43%) Intermediaries: Low on list of priorities (Fundraising, financial management and communications) are more important (20%)
  13. 13. The issues experienced during program impact assessments: - There is no evidence that selected programs have conducted feasibility, efficiency, viability, sustainability or cost benefit analysis There is no evidence that selection criteria are being applied to the suitability of selected programs for envisaged impact + All programs can be monitored and evaluated All programs fulfil some social need Next Generation Consultants 13
  14. 14. The issues experienced during focus area impact assessments: 2016/05/03Next Generation Consultants 14 - On average each funder in SA has 4 focus areas Generally – each focus area are seen as completely separate investment areas Focus areas are aligned (after the fact) to popular development contexts (NDP/SDG’s) There is no evidence in SA that focus areas were chosen based on community stakeholder input + There is some evidence that focus areas are based on future business priorities (i.e. education) There is some evidence that selection criteria for programs is based on focus area selection (indicators to measure progress in a specific context) – mostly quantitative There is little evidence that focus areas support the overall vision, mission, strategic objectives and intent of investment strategies There is very little evidence that focus areas are integrated to leverage and extend impact envisaged
  15. 15. Overall - Limited Impact is evident:  Limited understanding of the often complex local contexts  Insufficient participation and ownership by local stakeholders/beneficiaries  A perception of giving rather than investment  Detachment from core business strategy and competencies  Responding to local requests in an ad hoc manner without clear focus  Lack of sufficient planning, scoping, clear objectives, outcomes  Lack of transparency and clear criteria  Lack of due diligence, budget oversight, insufficient management processes  Insufficient focus on sustainability  Lack of both professionalism and skills  No exit or handover strategy  Overemphasis on infrastructure and under- emphasis on other development areas such as skills/capacity building  Failure to measure and communicate results either during or after project completion  And most importantly, the lack of delivery and fulfilment of promises and commitments by funders 2016/05/03Next Generation Consultants 15
  16. 16. Key challenges: Trade Off’s  Negative impact vs benefits  Loss of income from current livelihoods because of delayed implementation  Restricted local economic development because of lack of support from government  Encouraged competition – increased prices – community activism/jealousy  Increase water/energy consumption or increased carbon emissions Attribution  What would have happened anyway?  What about policy/structural/unforeseen changes/challenges  What other factors could contribute to the impact – i.e. other funders, policy changes, etc. 2016/05/03Next Generation Consultants 16
  17. 17. Aspects of Impact LEARNING FROM OUR EXPERIENCE 2016/05/03Next Generation Consultants 17
  18. 18. A different perspective of the Impact value chain: Our Model - III™ 2016/05/03 18  There are many views and perspectives of impact – not just input, activities, outputs and outcomes: ImpactDimension Economic/Social/ Socio-Economic/ Environmental Direct/Indirect Positive/Negative/ Combined Intended/ Unintended Short/Medium/Long Term Perceived/ Empowered/ Pre-emptive & post impact Significant/Residual/ Capital impact ImpactPortfolio Education & Bursaries Health and quality of life Environment and climate change Safety & Security Welfare & Community Agriculture and Food Security Economic, Enterprise and Social Development Skills Development and Job Creation Sports Development ScopeofImpact Project/Program Focus Area and investment portfolio Signature, cause related and Flagship Geographic (region – local/national) Demographic (girls/ boys/ women/ disabled) Stakeholder based (value chain – intermediaries, learners, teachers, government departments, other funders, etc.) Boundaryofimpact Stakeholders (direct/indirect) Funders (primary/secondary) Partners (intermediaries) Time (1/3/5 year) Depth / weighted (related to strategic objectives/ outcomes) Reach (primary/ secondary/ tertiary - value chain)
  19. 19. Impact over time: Short Term Short term impacts or “quick wins” are important for projects as they build trust, credibility, and local support. They also maximise the value to the stakeholders very quickly. But when project lag and expectations are not met, then impact is diminished. Example: Food gardens/feeding schemes provide immediate relieve to/for malnutrition, food access, food security – but is not sustainable over the long term. Additionally if resources for water or seeds or access to markets are not factored in – food gardens have a short life span. Medium Term There does not appear to be a longer term approach for business partnerships or opportunities beyond the initial funding phase. Most programs require additional funding specifically for capacity building in order to ensure long term impact as well as sustainability – a clear oversight in current program funding cycles. Example: A science/maths program may yield increased pass rates within 12 months, but may not affect increased university access, or subject/career choices. In particular infrastructure programs require additional resources i.e. operational expenses or capacity building (maintenance) to move along the value chain. Long Term Very few programs have access to long term funding, therefore measuring long term impact becomes impossible. 80% of all programs are only funded between 12 and 24 months, 15% of programs are funded up to 36 months, but less than 5% of all programs are funded for 60 months (5 years) meaning that impact of each and every program is diminished over time. Example: An ECD program may yield results in the form of increased school enrolment but evidence needs to be provided of improved literacy and numeracy skills or school readiness to ensure long term sustainability – if not followed by a program in primary education – as such it will be unsustainable 2016/05/17Next Generation Consultants 19
  20. 20. Types of impact: Positive Positive impact is seen as additional to direct/intended impact. It is therefore surprising that so few proposals and subsequent programs consider ‘additional’ impact and this indicates the limited focus of particular interventions - i.e. it is only designed to achieve one specific outcome. Example: A food garden may yield additional positive impacts such as improved concentration/ ability, increased school attendance, increased motivation/positive behaviour, decrease in absenteeism, increased subject knowledge – but in general these types of programs only measures the number of people affected by the garden. Negative We present two models to clients – one where negative is measured as part of the total impact scenario – and one where we subtract the negative impact from total impact. We found that negative impact plays out on two levels 1) as a result of actions from the funder i.e. delayed payments and 2) as a direct/ indirect/ unintended consequence of the program – i.e. highlighting oversight within program design aspects. Example: Providing a computer lab/Building a soccer field, but not considering security, where the water will come from or materials for maintenance or increased costs for electricity/software programs generally leads to negative impact. Combined / Cumulative This impact aspect reflects an opportunity to expand impact and in our experience is mostly linked to program implementation and design aspects. Example: A food garden not only increases food security, but improved nutrition which is linked to increased productivity or quality of life or improved school attendance. Aggregation and/or interaction of impacts within a system are defined from the perspective of the stakeholders experiencing them and should therefore be considered and accounted for. 2016/05/17Next Generation Consultants 20
  21. 21. Types of impact: Direct  These are impacts that can be directly attributed to the implementation and therefore outputs and outcomes of a program.  Example:  The objective was to increase literacy/numeracy/subject knowledge or technical skills.  If the strategic objective is met and evidence is provided then there is direct or impact – i.e. stakeholders were tested and based on the results pass rates increased. Indirect  Indirect impact is very often linked to unclear focus areas, unclear development outcomes, unclear accountability/ responsibility, lack of research, lack of engagement, lack of impact which renders the programs of little value for any stakeholder groups or that resulted in ‘accidental’ impact.  Example:  The fact that classes are presented on Saturdays requires additional resources, food, transport, security, staff and other additional costs, for both the funder and intermediary sides – therefore attendance of classes can drop and dropout rates are high – which leads to indirect impact which is negative or unconsidered impact. 2016/05/17Next Generation Consultants 21
  22. 22. Types of impact: Intended  This aspect refers to the intended (direct) (stated/strategic objectives/outcomes) of a specific intervention. The lack of indicators to measure direct and intended impact is a serious issue, which could mean, that there was no shared value distributed. If there is little evidence of impact it is an indication that the programs funded were bad choices, or the objectives and desired/intended outcomes were not clearly defined.  Example:  To improve the pass rates in maths (intended outcome) but without specifying the minimum pass rate (80% of pupils must achieve a minimum of 70% and 60% must continue with the subject choice for the next grade – or no more than 10% drop out rate during a 3 year cycle) should confirm the intended impact objectives. Unintended  No community program is intended or designed/implemented to have unintended impacts as this would mean that not enough planning or research or engagement has been conducted. This implies that there is disconnect between strategy/objectives, project management and execution.  Example:  Whilst providing a food garden – the garden yielded little or no production because of drought/lack of water/knowledge/skills. This program then requires additional resources as unintended impact diminishes the intended impact. 2016/05/17Next Generation Consultants 22
  23. 23. Types of impact: Perceived Impact A potential/perceived impact rather than an actual impact. This is about how people (stakeholders) feel about the impact and how they behave generally, thus perception is a reality to them. Example: Specifically used if impact is an anomaly – only mentioned once by a particular stakeholder group, or evidenced by few stakeholders or cannot be confirmed by other stakeholders. Because of increased skills/knowledge, salaries increased which improved the living conditions of a family and increased family sustainability. Empower(ed) Impact Is the enhancement of the assets and capabilities of individuals or groups to engage and influence institutions, and to increase the accountability of institutions. Example: Capacity building for stakeholder organisations/groups – now access social security services Strengthening legal status of stakeholder groups/organisations – now increased fundraising/ marketing/attracting new donors Stakeholder authority to manage funds, hire and fire workers, supervise work and procedure materials – increased effectiveness of organisations Support for new and spontaneous initiatives by stakeholders – now help others to become more empowered Pre/Post Impact Depending on the lifecycle or life stage of a project, pre-emptive assessments can be made that will indicate post impact assessment impact. This focuses on likely impacts of a planned intervention – i.e. has not happened yet. Example: A program can be assessed to determine likely/significant impact and develop indicators to measure such impact in the future. 2016/05/17Next Generation Consultants 23
  24. 24. Types of impact: Significant Impact Focuses on intended outcomes – i.e. prioritisation of outcomes to be considered. Impacts are assessed for their significance according to predetermined criteria. Example: For instance – if job creation was the intended/direct outcome – the significance of the impact would refer to: Direct/indirect/full- time/part-time or even decent jobs created – not temporary jobs or below living wage categories. Residual Impact Impact that reflects negative impact and will continue to contribute to negative impact without mitigation/correction. Example: The intention was to create jobs, but now there is the realisation that the intervention requires substantial skills development and then certification to ensure a qualification before a job can be secured. Therefore impact envisaged was not achieved, rather residual impact can be achieved through significant changes. Capitals Impact Typically this could include: • Financial - (income, security, wealth, credit, investment, savings) • Social (leadership, networks, relationships, trust, reciprocity) • Environmental/natural capital – (landscape, soil, land ownership, water, energy) • Human – (self-esteem, worthiness, social cohesion) • Intellectual – (community ownership, community assets, community contribution) • Manufactured / production – (products, services, crafts, indigenous products) • Also considered are political impact, institutional impact, infrastructure impact, cultural or spiritual impact – (language, traditions, rituals). Example: Social cohesion improved (racial discrimination decreased). 2016/05/17Next Generation Consultants 24
  25. 25. Aspects of Return on Investment LEARNING FROM OUR EXPERIENCE 2016/05/03Next Generation Consultants 25
  26. 26. Business value of determining ROI: Knowledge Deep understanding of value and impact as well as risks Comparative data – industry/sector Insights into and across impact dimensions Insights into stakeholder groups affected Action New or enhanced business decisions, practices and behaviours Develop new products/services/markets Changes policies, strategies and practices to increase impact and return Report in a more credible, integrated and useful way Results Improved performance – profitability/competitiveness Reduce potential risks – community activism/licencing Cost Savings – of court cases/mitigation of risks Enhanced stakeholder relationships Improved licence to operate conditions Improved trust and transparency 2016/05/03Next Generation Consultants 26 Please see:
  27. 27. Return on investment impact: Strategic Aspects Support of corporate values and strategies Support of sustainability strategy/programs Support of future growth, development and market access Investor / Shareholder Aspects Share price not affected when industry or sector are targeted by activists Rated as industry leader in Sustainability Indices Increased investment from socially responsible investment funds Inclusion and high ratings in awards programs Reputation Aspects Recognition/awards Media coverage Increased brand awareness 2016/05/03Next Generation Consultants 27
  28. 28. ROI Impact: Profit Aspects Sales generated from programs Value of new products and services generated from CI/CSI programs Increased worker productivity Increased share price (e.g. from attention of socially-screened investment funds) Environmental Aspects Costs mitigated from rehabilitation Costs saved from waste management/recycling Carbon emissions sequestrated Costs of fines Sector Specific Aspects Financial Sector • Economic trends and demographics and expanding workforce needs • Increasing regulatory activity (e.g. CRA, PRI, CRESA, JSE, investment screening) • Increasing equality/disparity between haves/have-nots – financial inclusivity/Gini Co- efficient • Globalization strategies • Opportunities to brand company through community involvement Mining • Intensity of opposition - Previous negative incidents • Regulators’ sensitivities - Compatibility with existing development • Reputation of company - Level of community involvement • Involvement of external advocates 2016/05/03Next Generation Consultants 28
  29. 29. ROI Impact: Stakeholder Aspects Increased community/government awareness/positive relationships/stakeholder relations Decreased complaints/grievances/ activism/strikes/boycotts/negative press coverage Cost savings/avoidance Prevention of operational stoppages/delays Reducing/decreasing legal costs/law suits Support for market entry/expansion plans Savings Aspects Tax rebates received from philanthropic/ charity/social/community contributions Saved costs of free advertising space received from media coverage of the CI/CSI programs Legal fees averted (includes legal department staff time and projected billable hours from contracted firms) Savings Aspects Cont Crisis PR efforts averted (includes PR staff time and projected billable hours from contracted firms) Costs of avoided down-time from failure to receive building approval, work stoppages, etc. Reduced employee recruitment costs, reduced turnover costs, and/or reduced absenteeism Reduced employee training costs (e.g., through community service learning initiatives) Reduced customer turnover Other staff management hours saved 2016/05/03Next Generation Consultants 29
  30. 30. ROI Impact: Customer Aspects Surveys indicating improved customer perceptions and impacts on shopping decisions Sales leads generated in specific geographic or demographic markets Development/increased sales of specific products/services in targeted geographic or demographic markets Annual brand tracking surveys indicating higher scores Collaboration/participation/co-design of new product/service development Greater participation/involvement/ contribution in community investment and development programs Increased brand awareness Increased customer acquisition/retention Operational Aspects Mitigation of operational risks (health/environment/safety) Support and enhancement of business operational requirements (integration, skills development, etc.) Compliance Aspects B-BBEEE Licence to operate SLP Mandate/Strategy DMR/King III/ICMM/IPIECA Approval rates/new explorations/extensions Rehabilitation Drop in complaints/grievances Global Compliance 2016/05/03 30
  31. 31. ROI Impact: Employee Aspects  Positive response to utilizing volunteerism for professional development/skills development and team building  Employee surveys demonstrating that volunteer activities contribute to leadership development  Voted one of the best companies to work for  Surveys showing increased employee morale from participation and increased numbers of employee volunteers, volunteer hours, and the number of company-sponsored volunteer projects  Satisfaction surveys indicating positive impact and anecdotal evidence  Employee training programs designed to use volunteers and products with most donations  Employees learning to use products to that they are more equipped to sell/market them  CSI/CI projects used for team building or during orientation/induction or other training  Recruitment from communities where CSI/CI projects are run  Internal surveys showing an increase in employee pride, morale and commitment as a result of employee involvement in volunteer activities Social Aspects  Improvement of quality of life  Community job creation / empowerment  Improved stakeholder relations within the community  Poverty reduction 2016/05/03 31
  32. 32. In Conclusion OUR NEW ROADMAP 2016/05/03Next Generation Consultants 32
  33. 33. What our clients say: Funders It provides validation of investment decisions Opportunities for increased partnerships and collaboration Contributes to better financial, project and risk management/reporting Contributes to learning, capacity building and better results (impact) The outcome of the process informs sustainability and integrated reports The detailed stakeholder engagement process provide insight never documented or previously considered in evaluations The impact assessment process not only provides guidance for future strategies and programs, but identify areas requiring attention, confirms whether the needs of beneficiaries are met, it monitors relationships, the lessons learnt provide detailed actions of issues that needs to be addressed and improved, and it informs future best practice Intermediaries We feel comfortable with the transparency of the process The process have added value to our own work – especially M&E and reporting practices The processes have increased our effectiveness and own performance; increased our learning and knowledge; built internal capacity; and increased our credibility We believe we were assured independently by someone who can verify our claims – it validated our own beliefs We have learnt the value of qualitative indicators, to consider impact more broadly and we are now more convinced of the actual value of our program It ensured increased funding for both programmes, internal capacity and increased our own sustainability Beneficiaries We had an opportunity to talk without being judged – we could be honest We learnt to document our own work and the contribution we made We feel we are being trusted, being heard and someone asks our opinion We had an opportunity to share and learn 2016/05/17 33 Competitors Transparent process Credible and verifiable process considering all stakeholders input Contributes to more efficient and integrated strategies, policies, programs Contributes to industry capacity building
  34. 34. The next level of impact assessments: 2016/05/17Next Generation Consultants 34 Static Impact No movement – no change Changed Impact Increased or decreased impact Sustained Impact Impact validated and confirmed over time
  35. 35. Direction of impact: Adverse or beneficial impact  Do some stakeholders benefit more than others  Are there trade offs between potential negative and positive impact  Is the impact sustainable or time bound  Will impact escalate or diminish over time Impact manageability  What will be required in future to maintain the impact  How resilient is the impact  Is mitigation required to enhance the impact  How could impact be accelerated  How could vulnerability be managed with adverse impact 2016/05/03Next Generation Consultants 35
  36. 36. Final Thoughts:  It is our opinion that the ability to quantify and qualify impact must receive much greater attention in future. It is the core of developmental work as the basic assumption is that:  1) Resources are applied  2) Activities are conducted  3) Qualitative change and impact and outcomes are the ultimate expectation that will lead to change in a specific social context  The development fundamental principle – also referred to a ‘theory of change’ is the cornerstone of social/community development. Therefore being able to identify what changes in a developmental context is the primary reason for doing community investment and development in the first place.  The fact that both funders and intermediaries have difficulty identifying qualitative impacts indicates a lack of not only:  1) Understanding developmental principles  2) Contextualising developmental outcomes  3) Quantify and qualify developmental impacts as a result (outcome) of their own (designed and implemented) intervention  The reason why qualitative impact is the most complex aspect to measure; is quite simply, because both intermediaries and CSI/SED program managers/practitioners are at a loss of HOW to develop and identify indicators to measure such change 2016/05/03 36
  37. 37. 2016/05/03Next Generation Consultants 37 Beyond Evaluations: Key Question What do we want to know? Criteria for impact/ value of impact What matters? Standards and Definitions What would indicate impact? Can we define the impact and envisaged / required Information How will we know? What evidence do we need/ have? Method How will we determine impact or gather evidence? What level of engagement will be required? Analysis What impact was achieved? What does the evidence show? How can it be confirmed and collaborated? What tools will we use? What skills do we need to draw conclusions? Synthesis and Triangulation So what? Do we share the results/ outcomes What would have happened anyway? Decision Now what?