New Challenges for the Evaluation Community: timely responses in times of urgency 1) The document discusses new challenges for the evaluation community given the urgent need for action on issues like climate change and biodiversity loss. 2) It argues that evaluation needs to move beyond just examining whether past policies worked and instead support systemic change and transitions to sustainability. 3) Evaluation should be integrated into the policy process and consider long time frames given the slow pace of transitions and risk of lock-in to unsustainable paths.

1. New Challenges for the Evaluation Community:
timely responses in times of urgency
Hans Bruyninckx | European Environmental Evaluators Network | 4 November 2020

2. Context of ‘environmental’ evaluation

3. 2020s: a pivotal decade
“Europe stands at a critical juncture in 2020.
Its leaders have opportunities to shape future
developments that will not be available to their
successors. The coming decade will therefore be
of decisive importance in determining Europe’s
opportunities in the 21st century.”
SOER 2020

4. SOER2020
1. IPCC report on global
warming of 1.5°C
3. International Resource
Panel global outlook 2019
2. IPBES global report on biodiversity
and ecosystem services
Global context: unprecedented challenges, improved knowledge
•Urgent action needed
•Irreversibilities
•Tipping points
•Interconnected

5. A context of risk and uncertainty

6. Global response: Sustainable Development Goals
- 1992….2015….2030
- Reasons for failure?
- What needs to change?

7. A new policy context: European Green Deal
• First climate-neutral continent
• Biodiversity Strategy 2030
• New Circular Economy Action Plan
• Zero pollution strategy
• Farm to fork strategy
• Just transition
• Sustainable European Investment Plan
• Future ready economy – new industrial strategy

8. The emergence of transformative policy since 2014
Change has come very rapidly*
2014: EU 7th EAP, Horizon 2020
2015: UN Agenda 2030/SDGs, Paris Agreement
2015-2018: EU strategies: energy, mobility, circular, low-carbon, etc.
2019: European Green Deal, Digital Agenda
2020: Multiple EGD packages; 8th EAP; Horizon Europe
* But late considering 50 years of signals from science (Club of Rome; Brundlandt;
Rio 92; GEOs; IPCC; …; a vast body of knowledge of (un)sustainability )

9. The (changing) nature and role of evaluation?
What is the impact of this changing context on ‘environmental’ evaluation?
• What is the goal of the evaluation?
• What is being evaluated?
• What is the place of the evaluation in the policy process?
• When does it take place?
• Forward looking evaluation?

10. The goal of evaluation

11. Traditional approach
- Did (a part of) the policy work?
- Revision of legislation
- Changes to a policy instrument
- Changes to the target
- Reviewing the timing
- Adjusting the budget
- Impact on the environment
- Political legitimacy

12. Evaluation for system innovation and transitions
Source: based on Geels & Schot (2007)
Small networks of actors support innovation on the basis of expectations and visions.
Learning and experiments take place
Niches
Landscape
Regime
Time
Markets and
consumer preferences
Science
Industry
Culture
Policy
Technology
New configuration breaks through, taking advantage
of ‘windows of opportunity’. Adjustments occur in
regime
The regime is dynamically stable
Landscape developments put pressure on existing regime
- System dynamics!
- Complexity
- Guidance
- Options
- Corrective action
- Learning

13. What is being evaluated?

14. Traditional approach
- Legislation by legislation (revision)
- Institutional focus
- Ex post: efficiency, effectiveness focus (not impact)
- Ex ante: (economic) impact assessments
- Often rather detached from environmental and climate monitoring

15. Catalysing systemic change
• Recognise fundamental drivers
and system interlinkages
• Adopt transformative policy
frameworks
• Fill crucial policy gaps:
• Food
• Land and soil
• Chemicals
• Leverage the power of cities,
businesses and communities
for society-wide action

16. Environmental policy
integration and policy
coherence
Policy needs to consider
environmental, economic,
social and governance
dimension and their
synergies and trade-offs
SOER 2020
“
”
© Istvan Kerekes, Sustainably Yours/EEA

17. Policy integration largely unsuccessful: transport
EU GHG emissions in the transport sector, 1990-2017 (1990 = 100%)
International aviation
International navigation
All transport
National total GHG emissions
Other UNFCCC sectors

18. • Unsustainable agriculture still
main threat to biodiversity and
natural capital in Europe
• Pollution of soil, water, air and
food
• Over-exploitation of natural
resources
• Greening of the CAP shown to
be ineffective
Policy integration largely unsuccessful: agriculture

19. Example: climate mitigation vs air pollution
• Synergies: decarbonisation of transport
also reduces air pollution
• Trade-offs: promoting diesel vehicles and
biomass increases air pollution
Ensuring policy alignment and coherence

20. Investing in sustainability, not dead-end streets

21. Social vulnerability and just transitions
Source: EEA, 2019, unequal exposures, unequal impacts
GDP per capita

22. Forward looking evaluation?

23. Beyond the rear view mirror
Source: based on Geels & Schot (2007)
Small networks of actors support innovation on the basis of expectations and visions.
Learning and experiments take place
Niches
Landscape
Regime
Time
Markets and
consumer preferences
Science
Industry
Culture
Policy
Technology
New configuration breaks through, taking advantage
of ‘windows of opportunity’. Adjustments occur in
regime
The regime is dynamically stable
Landscape developments put pressure on existing regime

24. Systemic change is disruptive: the ‘x-curve’
Source: Loorbach et al .
Optimisation
Destabilisation
Breakdown
Phase outExperimentation
Acceleration
Institutionalisation
Stabilisation
Conflicts
Speeding up
Scaling up

25. Investments can be “locked-in” into
unsustainable consumption and
production patterns for decades.
Lock-in risks

26. Evaluation in a context of long term change
Source: EURO-CORDEX (Jacob et al., 2014)
Precipitation
(projected change for 2071–2100)

27. Time as critical element?
Source: based on Geels & Schot (2007)
Small networks of actors support innovation on the basis of expectations and visions.
Learning and experiments take place
Niches
Landscape
Regime
Time
Markets and
consumer preferences
Science
Industry
Culture
Policy
Technology
New configuration breaks through, taking advantage
of ‘windows of opportunity’. Adjustments occur in
regime
The regime is dynamically stable
Landscape developments put pressure on existing regime
Non-problematic
Under-specified
Under-defined

28. Speeding-up in addition to scaling -up
• Seen as a consequence of scaling-up
• Thus, not an autonomous factor

29. Dangers of underestimating time
• Irreversibility
• Lower quality outcomes
• Rewarding laggards: countries and actors
• Focus on the wrong time dimension?
• Timing of reporting
• Timing of implemention
• Time dimension at appropriate level of analysis?

30. Taking time more seriously
• Explicit understanding of time dimension
• Developing approaches to understand the link between time
dimensions and the DPSIR
• Developing the necessary methodologies
• Learning and developing necessary skills, ICT systems

31. Place of the evaluation in the policy process?

32. From exogenous to systemic integration of evaluative practices
• Part of transition pathways
• Iterations of evaluation
• Part of learning toolbox
• Stronger link to monitoring and reporting efforts
• Mandate for evaluators

33. Possible elements for the future
Foresight and
evaluation?
Evaluation and
Resilience
Post-normal
Evaluation

34. Conclusions
• Context is changing fundamentally
• Reflection about evaluation in this new context
necessary
• Investing in innovation
• Investing in evaluation as integral to the transition
process