PRESENTATION for Aash Mishra UN DESA Water Energy SDGs Interlinkages Evaluation Report Jun 2022

1. EVALUATION REPORT
Sustainable Water and Energy Solutions: Achieving Water and Energy Sustainability and other SDGs in support
of the implementation of the 2030 Agenda for Sustainable Development
Evaluation Findings Presentation to U.N. DESA, Aashish Mishra, aash.mishra@nyu.edu, June 2022
1

2. Approach for the Evaluation
Theory of Change
Contribution Analysis
 Review of Theory of Change
 Analysis of Logical Framework
 Evaluation of Outcomes and Results
 Evidence of Contribution to Impacts
 Triangulation of Findings with Stakeholder
Survey Questionnaires
 Consolidated analysis of above
2

3. Evaluation Framework
Framework for Evaluation
Evidence Triangulation
1. Relevance ≈ Design & Implementation
2. Effectiveness ≈ Implementation & Results
3. Efficiency ≈ Results
4. Coherence ≈ Knowledge & Partnerships
5. Sustainability ≈ Medium- to Long-term
6. Gender & Human Rights ≈ Embedded in
ALL PROGRAM PHASES
7. Lessons Learnt ≈ Sequencing of Causal
Impact, ie. (looking back) from Design to Impact
3

4. Methodology (continued)
Procedures and Instruments for Data Analysis
a. The Evaluation Framework was designed to capture Evaluation Criteria and its Sub-Questions with
related percentage point weights for scoring.
b. The Data Collection templates were designed to capture the primary data.
c. Key project documents were reviewed including the Project Agreement Document, Project
Website, 2018 to 2021 Annual Progress Reports, 2018 to 2022 Annual Workplans, 2020 & 2021
Annual Reports, Case Study Collection and Third to Fifth Steering Committee Minutes.
d. Primary data was collected, compiled and sorted from these key project documents.
e. Primary data captured was triangulated across multiple sources to validate its accuracy.
f. The Evaluation Framework’s Evaluation Criteria were scored based on the available primary data
and other data from UN DESA.
g. The draft report was outlined in order to quickly ascertain whether further data analysis were
required.
h. The evaluation report was drafted based on the above-mentioned.
i. Stakeholder Survey Questionnaires administered to a cluster of UN DESA, Itaipu Binacional and
Civil Society representatives, in order to further triangulate preliminary data analysis.
4

5. Documents Reviewed (for Evaluation)
a. UN DESA and Itaipu Binacional Project Agreement
b. Global Knowledge Platform Project Website
c. 2018 to 2022 Annual Workplans
d. 2018 to 2021 Annual Progress Reports
e. 2019 to 2021 Annual Reports
f. SWESN Case Studies Collection
g. 2018 to 2021 Budget and Expenditure Financial Statements
h. Third to Fifth Steering Committee Minutes
i. Stakeholder Survey Questionnaires (Administered) Results
5

6. Evaluation Criterion Findings
6

7. 1. Relevance = Acceptable (11/15)
1.1 Relevant in “building a Global (Multistakeholder) Partnership and Network”
 # of Partnerships Established & Operationalized increased steadily from 1 (Baseline) to 23 (2021)
1.2 Relevant in “building capacity of Govt. Institutions and (relevant) stakeholders”
 # of Stakeholders Reached increased exponentially from 165 (2018) to 2,000 (2021)
1.3 Relevant in its “activities conducted to enhance Knowledge and Best Practices”
 # of Case Studies, Best Practices & Reports Compiled increased significantly from 2 (2018) to 152 (2021)
1.4 Less than Relevant in “activities to enhance Global Advocacy and Outreach”
 # of Advocacy and Outreach Activities had dropped steadily from 2018 to 2021
7

8. 2. Effectiveness = Acceptable (15/25)
2.1 Effective in “achieving its Expected Objectives”
 Indicators of Achievement increased annually as compared to Baseline
2.2 Effective in its “Strategy for the implementation of Project Activities”
 Well-defined Logical Framework except for a surprising lack of Empirical Targets for Results
2.3 Less than Effective in “consistent strategy and activities with its objectives”
 Contrary to Project Agreement, did not undertake Program Reviews and Biannual Evaluations
2.4 Effective in “project strategy, processes, meetings, events, reports & activities
strengthening an Integrated Approach to SDGs 6 and 7 in Developing Nations”
 i.e., Covid-19 (Webinar) Trainings reinforced the Integrated Approach towards Water, Energy and Climate SDGs
8

9. 2. Effectiveness (continued)
2.5 Highly Effective in its “combination of project events, activities, trainings and
papers for achieving its Expected Objectives”
 Synergistic Combination of project activities, trainings and technical papers that increased its ratios annually
2.6 Effective in “approaches to engage stakeholders and disseminate information”
 i.e., 2022 Technical Papers on SWES led by SWESN Members with UN DESA support
2.7 Highly Effective in “collaboration between project partners and stakeholders
during implementation viz its degree and results”
 More than a dozen Trainings and Webinars conducted that built the technical capacities of its stakeholders
2.8 Less than Effective in “reasonable time parameters for its activities & schedules”
 Postponements due to both foreseeable & unforeseeable reasons, such as the change in Itaipu Leadership and Covid Pandemic
2.9 Less than Effective “Indicators of Achievement for its progress”
 Indicators failed to provide Empirical Targets, thus there was no basis of comparison as to whether its progress was on-track 9

10. 3. Efficiency = More than Acceptable (12/15)
3.1 Less than Efficient in “delivering activities as per set timelines”
 Several Postponements in Key Project Deliverables, ie. First Global Gathering on SWES delayed from 2020 to 2022
3.2 Less than Efficient in “overcoming operational challenges ie. Covid”
 Three Key Deliverables Postponed from 2020 to 2022: First Global Gathering, LAC Regional Meeting, First World Report
3.3 Efficient in “its Governance and Management Structures”
 Although Could be Simplified if project donor shifted to more of an independent oversight and advisory role
3.4 Efficient in “improving stakeholder communication & collaboration
 Several Water-Energy Interlinkages Dissemination Events: National Capacity-building, World Water Forum, COP24, HLPF & UNGA
3.5 Efficient in “its Enabling Factors”
 Demonstrative and Transformative Nature of this cross-sector SDGs partnership project
3.6 Efficient for “UN DESA to continue to deliver this type of project"
 Discussed Later in Conclusions and Recommendations Sections
10

11. 4. Gender, Human Rights & Disability
Issues = Acceptable (12/15)
4.1 Somewhat Acceptable in “Mainstreaming Gender Equality and Women’s Empowerment”
 Document Review showed little evidence of GEWS being mainstreamed into both design & implementation of activities
 However, to its merit, Balanced Gender Ratio in Steering & Operational Committees and Itaipu SDG 5 Case Study
4.2 Somewhat Acceptable in “Mainstreaming Human Rights and Disability Perspectives”
 Document Review showed no concrete evidence of such issues being explicitly addressed
 However, in Hindsight, project does address fundamental human rights issues of pro-poor services delivery
Moreover, in light of Stakeholder Survey Findings, Scoring should be increased Acceptable
11

12. 5. Coherence = More than Acceptable (15/15)
5.1 Highly complementary and coordinated with other work undertaken by the
participating entities, as well as other UN and non-UN agencies and stakeholders
• UN-Energy’s SDG 7 Objectives for Partnership, Advocacy & Communications complementary with project’s work
• UN Water’s SDG 6 Integrated Approach for review, knowledge & learning extremely coherent to project’s work
5.2 Numerous Lessons were Generated from Project’s Strengths & Challenges that
should guide UN DESA’s future engagements in supporting SDGs implementation
 Discussed Later in Recommendations and Lessons-learnt Sections
12

13. 6. Sustainability = More than Acceptable (12/15)
6.1 Likely Sustainable for replication of project’s activities for future needs of SDG 6 & 7 issues
 Case Studies Portfolio, Best Practices, Webinars & Knowledge Platform can be deepened & replicated for similar project’s needs
6.2 There are some Limitations or Barriers for continuing this type of project in the future
 More comprehensive project design and preparation work needed before it proceeds to implementation
 Securing diverse public and private funding has great challenges, as such projects only yield longer-term visible results to sell
6.3 Likely Sustainable to continue leveraging intergovt. processes like HLPF for SWES partnerships
 Only if UN DESA continues to leverage this role as the convener and implementor for the SWESN program
6.4 Likely Sustainable for providing incentives to continue moving forward its objectives
 Contingencies in-place for SWESN’s Continuation Post-2022, in the well-developed outline of the “2022-2026 SWESN Workplan”
6.5 Likely Sustainable for contributing to institutional and policy changes or follow-up
 SWESN 2022-26 Workplan’s endorsed activities through 2026 will most likely contribute to institutional changes, policy changes,
or sustained follow-on support of SDG6 and SDG 7 and its nexus in developing countries 13

14. Stakeholder Survey Questionnaires:
Key Findings
14

15. Project’s Responsiveness to
SWESN Members?
 1/3 of Respondents stated that SWESN was not responsive to
evolving needs of its members
 2/3 of Respondents stated that SWESN was responsive
 The former could diminish the Scoring for (2) Effectiveness and
(6) Sustainability
15

16.  UN DESA Respondent made an important point:
“Promoting Access to Water and Energy are key elements of human rights, in order to
achieve greater equality, equity & sustainability (of these basic services)”
 1/3 Respondents found this area weak in design & execution
 More than 1/2 found this area to be strong to very strong
 Thus, the Evaluation Score should be raised as it didn’t consider
the valuable feedback provided by the project stakeholders
Gender, Human Rights &
Disability Issues?
16

17. SWESN Sustainable
After 2022 Extension?
 1/3 of Respondents stated that SWESN was unsustainable
without (i) U.N.-led Secretariat or (ii) Additional Donor Support
 2/3 of Respondents stated that SWESN was sustainable
particularly due to its 2023-2026 Action Planning
17

18. Satisfaction with Itaipu
& UN DESA Partnership?
 85% of Respondents stated that they were satisfied or very
satisfied with Itaipu and UN DESA Partnership
 15% of Respondents stated that they were neutral
18

19.  Global Knowledge Platform: 15% Disagree, 85% Agreed
 Case Studies: 15% Neutral, 85% Agreed
 Webinars: 15% Neutral, 85% Agreed
 Capacity-building Activities: 50% Disagree or Neutral
 Technical Reports: 15% Neutral, 85% Agreed
 High-level Events and Conferences: 15% Neutral, 85% Agreed
Thus, Capacity-building Activities and SWESN Platform could be strengthened
Extent Project Outputs
were Highly Effective?
19

20.  50% of Respondents stated that Project Outcomes, Outputs and
Stakeholder Inclusion should be more clearly defined and
articulated to the SWESN
 50% of Respondents stated that Lack of In-Person Activities due to
Covid or Funding was detrimental to the objectives of the program
 50% of Respondents stated that planning of activities needs to be
improved as either off-track or delayed
Feedback for
Improvement?
20

21. Conclusions
21

22. Conclusions Part I
a) The project was Transformative in-nature and Demonstrative in-effect
 As a unique cross-sector SDGs platform that relevantly, effectively, efficiently and coherently addressed the
urgent and challenging problems of the Water, Energy and Climate-related SDGs interlinkages
 During COVID, SWESN supported transformative pathways for universal access to water and energy services
b) UN DESA demonstrated its comparative advantage & value-added for this work
 vis-à-vis facilitating and leveraging multilateral, intergovernmental processes and events in support of the
advocacy, implementation, and cross-learning of the SDGs
c) Extraordinary Circumstances of the Covid Pandemic led to some lagging results
d) “2022-2026 Action Plan” fosters foundations of success for SWESN after end-date
22

23. Conclusions Part II
f) Unfortunately, there were a number of challenges that the project faced.
 Project should have undertaken program reviews and biannual evaluations, in order to incorporate necessary mid-
course corrections to the implementation strategy, well before the project is near to completion.
 Project was relatively weak to coherently and effectively integrate different, interrelated SDGs dimensions of water,
energy and climate thematic areas with the remaining 14 SDGs
 Project was perhaps overly ambitious in its inclusion of Gender-, Human Rights- and Disability-issues in both its
operational- and technical-aspects of its work.
g) If UN DESA continues its partnership with Itaipu Binacional…
 this type of project will no doubt continue to leverage Itaipu’s field-based expertise with UN DESA’s
intergovernmental processes and events for advancement of the SDGs-agenda
23

24. Recommendations
24

25. Recommendations - Part I
 How this or similar projects can be improved?
 Leveraging & strengthening multi-dimensionality of SDGs viz SDG 6/7 interlinkages with all SDGs
 Gender, Human Rights & Disability-issues more deeply enshrined in technical work
 Governance processes could be simplified if donor had more of independent oversight & advisory role
 reduce operational transaction costs & time and instill more clear-cut accountability for results
 More systematic efforts for financial outreach from a diverse range of public-private donors
25

26. Recommendations - Part II
 Should UN DESA continue to deliver this type of project?
 Indeed, DESA should continue to lead, anchor & implement similar projects (SDGs partnerships)
 More comprehensive project preparation activities should be undertaken during project design
 ie. identifying the design and membership of the SDGs partnership and platform in early stages
 Undertake systematic & periodic program reviews/evaluations, in order to incorporate necessary
mid-course corrections to the implementation strategy, well before the project is near to completion.
 Should mitigate vs. haphazard financial outreach efforts & mobilize resources from public-private
 For this type of program, the financing model should be improved through partnering and leveraging
more financing-savvy foundations, universities, non-profits and corporations.
26

27. Recommendations - Part III
FINAL RECOMMENDATIONS
 The project should be continued and scaled-up after the completion of its 2022 end-date
to maximize its full results and potential of the ambitious agenda
 UN DESA should continue its leadership role as the SDGs program implementation and
convening authority for this or similar work
 There would be considerable merit in UN DESA continuing its unique partnership with
Itaipu, in order to leverage:
 Itaipu’s field-based expertise & knowledge of water-energy SDGs interlink. in developing nation contexts
 UN DESA’s convening role for strengthening SDGs partnerships and networks through its high-level
processes and events such as HLPF and COP 27

28. Lessons Learnt (for Similar Projects)
28

29. Lessons Learnt (for Similar Projects)
1) Similar projects could immensely learn from this project’s transformative response to COVID pandemic.
2) In this tumultuous environment, similar partnership-based SDGs projects need to be flexible in approach &
modus-operandi thru having contingencies for both virtual & in-person activities to accomplish objectives.
3) Similar projects should always undertake more comprehensive project preparation activities during the
initial design stage, including necessary scoping studies for the design of the SDGs partnership platform.
4) Similar projects should always embed systematic and periodic program reviews and midterm evaluations,
in order to recommend and incorporate necessary mid-course corrections to the implementation strategy.
5) Securing funding through both public and private sector resources is challenging for such projects that have
long-term visible results. The financing model could be improved through partnering and leveraging with
more financing-savvy foundations, universities, NGOs and corporations that are more experienced in
financing SDGs-implementation programs for developing nations.
29

30. Thank you
Aashish Mishra, Evaluation Expert, U.N. Department of Economic & Social Affairs
cell: +1 646-407-9622, aash.mishra@nyu.edu, https://www.linkedin.com/in/aashmishra
30