Evaluating Behaviour Change

IEA DSM Implementing Agreement
Subtasks of Task XXIV
Task 24
‘Subtask 3- monitoring and
evaluating ‘behaviour’ change
Dr Sea Rotmann,
Graz Task 24 workshop, October 14, 2014

SubtaSskusb otf aTsaksks XXIV
5- Social Media Expert platform
1- Helicopter
view of models,
frameworks,
contexts, case
studies and
evaluation
metrics
2-
In depth
analysis in
areas of
greatest need
(buildings,
transport,
SMEs, smart
metering)
3-
Evaluation tool
for
stakeholders
4-
Country-specific
recommen-dations,
to do’s
and not to do’s

SubtaSskusb otf aTsaksks XXIV
5- Social Media Expert platform
1- Helicopter
view of models,
frameworks,
contexts, case
studies and
evaluation
metrics
2-
In depth
analysis in
areas of
greatest need
(buildings,
transport,
SMEs, smart
metering)
3-
Evaluation tool
for
stakeholders
4-
Country-specific
recommen-dations,
to do’s
and not to do’s
3-
Evaluation tool for
stakeholders

subtask III -
Subtask I - Pre mHiesleic fooprt Tera sOkv XerXvIiVew
evaluation
 WHAT IS A SUCCESSFUL LONG-TERM
BEHAVIOUR CHANGE OUTCOME TO YOU?
3

Subtask I - Pre mHiesleic fooprt Tera sOkv XerXvIiVew Subtask III: Outputs
indicators19 such as number of installed installations or KWh saved potentially are not even
a real proxy; minor savings might involve most intensive behaviour changes whilst major
savings might have been the result of a relatively isolated behaviour change, e.g. buying
and installing a new heating system or LED lighting. 20
x A last point is that if only modelled savings are calculated, and real savings are not meeting
these calculations, the uptake and acceptance of the involved technologies, e.g. passive
houses, or services such as energy performance contracting will face serious problems
(Batey, Mourik and Garcia 2013).
x See below an illustrative picture that demonstrates quite clearly why a proxy such as
savings or KWh reduction is unable to explain the why and how of behaviour change.
4

- Individual evaluation and monitoring metrics for each domain in
the Subtask I Monster/Wiki, plus separate report
4

- An overview and some recommendations on monitoring and
evaluation can be found in Subtask III report ‘Did you behave as
we designed you intdoicat?or’s19 such as number of installed installations or KWh saved potentially are not even
4

- An overview and some recommendations on monitoring and
evaluation can be found in Subtask III report ‘Did you behave as
we designed you to?’
- There will also x be A last point a methodological is that if only modelled savings are calculated, review and real savings based are not meeting
on ‘Beyond
kWh’ which will feed houses, or services such as energy performance contracting will face serious problems
(Batey, Mourik into and Garcia Subtask 2013).
IX
4

subtask III -
evaluation metrics
Subtask I - Pre mHiesleic fooprt Tera sOkv XerXvIiVew
Conventional monitoring of smart metering success More systemic monitoring of smart metering
5
success
x !"#$%&'()''*#+&,'#%,%&*'+!-'(&')%%-$+./'
0!,%&)+.%*'0!*,+11%-'
x 1(+-'*20),'3%&'.10%!,''
x !"#$%&'()',0#%*'.10%!,*'1((/%-'+,',2%'
)%%-$+./'3&(40-%-'
x +..%3,+!.%'+!-'+,,0,"-%*',(5+&-*'*#+&,'
#%,%&*'
x 61%.,&0.0,7'.(!*"#3,0(!'(4%&'+'7%+&'
x 1%4%1'()',%.2!(1(87'+))0!0,7'.(!.%&!0!8',2%'
"*%'()',2%',%.2!0.+1')%%-$+./'%9"03#%!,'
x +11'()',2%'0**"%*'10*,%-'1%),:'+!-',2(*%'
#%!,0(!%-'"!-%&'*7*,%#0.'&%,&()0,,0!8'
#(!0,(&0!8'31"*;
x <%&*(!+1'#(,04+,0(!',('3+&,0.03+,%'0!',2%'
.(#3%,0,0(!'
x =.,"+1'%!%&87>&%1+,%-'$%2+40("&*'
x ?%.%!,'3"&.2+*%*'0!'%!%&87',%.2!(1(80%*'
@10/%'%!%&87'%))0.0%!,'$(01%&*:'!%5'
50!-(5*:',%.AB'
x C2%'0!)(&#+,0(!'1%4%1'(!'%!%&87'%))0.0%!.7'
+!-'&%!%5+$1%'%!%&87'*("&.%*'
x D("&.%*'(!'0!)(&#+,0(!'(!'%!%&87'0**"%*'
x =,,0,"-%*'(!'%!%&87'+!-'.10#+,%'
3&(,%.,0(!'0**"%*'
x 6*,0#+,0(!'()',2%'1%4%1'()'(5!'%!%&87'.(*,*'
x $"01-0!8'()'.+3+.0,7:''
x .&%+,0(!'()'%!8+8%#%!,'
x ."*,(#%&'*%!,0#%!,:''
x 3+&,0.03+,0(!'0!'(,2%&'%!%&87'%))0.0%!.7'
3&(8&+#*'
x )%%10!8'()'.(!,&(1'@(4%&'%!%&87'$011*:',2%'
2(#%:'%!%&87B'
x 1%4%1'()'"!%#31(7#%!,:''
x 1%4%1'()'0110,%&+.7'
x E!,%&!%,'3%!%,&+,0(!'&+,%'
'

Life seemed easy…
What is it?
• Monitoring: measuring progress and achievements
and production of planned outputs
• Evaluation: structured process of assessing success in
meeting goals and reflect on learnings. Explicitly
places a value judgement on the data and information
gathered in an intervention
6

Life seemed easy…
What is it?
Why do it the way we do now?
Establish effect of policies
Assess need for improvements
Assessing value for money
Contribution to evidence base for effectiveness of
behavioral interventions at population level
6

Life seemed easy…
What is it?
Why do it the way we do now?
Establish effect of policies
Assess need for improvements
Assessing value for money
Contribution to evidence base for effectiveness of
behavioral interventions at population level
How to do it…….???
6

It’s getting challenging…
7

• Evaluation team often not included in design
7

• Often not even part of the programme…
7

• Evaluation usually is only a snapshot at end or just after
7

• Often insufficient benchmarking
7

• Not longitudinal, sustainability/rebound often not assessed
7

• No insight in formation of networks supporting lasting
change
7

change
• Mismatch between needs of project managers s/h it is
aimed at
7

change
aimed at
• Large-scale M&E of actual behaviour too costly
7

change
aimed at
• Modeling or self-reported (at best)
7

change
aimed at
• ‘proxies’, such as savings or even better: cost
7
effectiveness

change
aimed at
7
effectiveness
• Proxies = NOT actual behaviour change, only about value
for money etc

change
aimed at
7
effectiveness
• Proxies = NOT actual behaviour change, only about value
for money etc
• No participatory process or feedback loops in the
traditional M&E

To make life more difficult..
8

We increasingly value interventions that are:
• tailored,
• multidisciplinary,
• varied interventions,
• qualitative and iterative,
• systemic,
• and have outcomes beyond the duration of project and beyond energy
8

• tailored,
• systemic,
And at the same time we judge the ‘behaviour’ of policymakers
who demand for simple, focused, quantitative and up-scaled
evaluations defining success in efficiency and effectiveness
terms.
8

• tailored,
• systemic,
terms.
But how could M&E look like that is:
8

• tailored,
• systemic,
terms.
But how could M&E look like that is:
Relevant to end-users, ‘cost effective’, doable, lasting actual behavioral
change, formation of networks, focusing on alignment, and processes
underpinning that change?
8

What now?
• No unified way of designing and M&E interventions
• Different disciplinary approaches have different methods
and foci of M&E, all pertinent to what they aim
9

How do different behavioural
models/disciplines evaluate?
1. Economic theory: individuals’ behaviours are seen as (semi-)
rational decisions that are made through cost-benefit calculations.
II. Psychological theory: the individual also takes a central
role; however, it is increasingly acknowledged that this individual
also operates as part of a collective e.g. by imitating the behaviour
of important others. Many psychological approaches view decision
making of an individual as a mental calculation aimed at making
choices; these calculations are informed by both emotion and cold
calculus.
III. Sociological theory: they put more emphasis on the
importance of the social nature of energy use and to the abilities of
people to participate in change in ways that fit their own contexts
and concerns. The central focus is on social practices, individuals
move into the background.
10

How do different behavioural
models/disciplines evaluate?
Intervention goals and evaluation methodologies commonly used in interventions underpinned by
the three disciplines discussed above are shown in the table below (this is not an extensive list, it
is aimed at highlighting foci and differences).
Goals 14 Methodologies Remarks (e.g. about causal
11
relationships)
Economic perspectives
Outputs
Cost-efficiency and
effectiveness
Units, and proxies e.g.
number of participants,
home insulated,
technologies installed, KWh
saved etc.
Labels
Modelling
Surveys
Experiments
Randomised control trials
Presence of cause Æ effect
relationship.
Aim is to meet a priori set goals
Monitoring and evaluation often
only for duration of
implementation, no longer term
Psychological perspectives
Outputs
Cost-efficiency and
effectiveness
Behavioural changes
Surveys
self-reported behavioural
changes
structured interviews
randomised control trials
Surveys to identify behavioural
determinants like motivations,
attitudes, etc.
Cause-effect relationships:
Effect on individuals of a
particular incentive, via e.g.
awareness, attitude, behaviour.
Interfering variables like social
context often not taken into
account
Sociological perspectives
Outputs and Outcomes
Cost-efficiency and
effectiveness
Learning about what works,
when, where, who, how
(long) and why
Learning about
interdependencies
Learning about co-shaping
and reshaping
User accounts
Time diaries
Cultural probes
In-depth open interviews
Analysis of fit of interventions
with daily life
measuring real, not modelled
energy consumption
Context & mechanism/conditions
produce an outcome.
Direct cause-effect relationships
hard to establish because of
interdependencies that cannot
be analysed separately.
14 We will also insert a column on the underlying processes - how does an intervention work, admittedly typically at the individual
level (what changed in people's understanding, motivations, attitudes)!

What now?
• Perhaps more fruitful to focus on learning processes*?
1. Single loop = instrumental, focused on short-term
learning about effectiveness in meeting goals/
outcome focused
12

What now?
outcome focused
2. Double loop = process oriented, focused on the
how and why, long-term learning
12

What now?
outcome focused
2. Double loop = process oriented, focused on the
how and why, long-term learning
*Based on work by Prof Chrys Argyris, Psychological and Organisational Development
12

Single vs double-loop
learning
Single-loop learning involves connecting a strategy for action with a result. Eg, if an action we take yields
results that are different to what we expected, through single-loop learning, we will observe the results,
automatically take in feedback, and try a different approach. This cyclical process of applying a new strategy
to achieve an expected or desired outcome may occur several times and we may never succeed. Running out of
strategies may push us to re-evaluate the deeper governing variables that make us behave the ways we do. Re-evaluating
and reframing our goals, values and beliefs is a more complex way of processing information and
involves a more sophisticated way of engaging with an experience. This is called double-loop learning and
looks at consequences from a wider perspective.
13
2.

Single vs double-loop
learning
14

Way forward?
M and E of single-loop learning doable to undertake and fine for low-hanging
fruit and non-habitual change. Sees behavioural change interventions
more or less as linear cause and effect relationships (A+B=C:
Intervention A targeted on group B will cause the intended Change C)
15

Way forward?
A change of focus amongst policymakers and funders towards allowing
experimentation with more systemic and messy real life interventions
that do not provide easily quantifiable and scalable information is a big
transition. It demands amongst others that policymakers appreciate that
these systemic interventions cannot be evaluated in terms of cause and
effect, but are the outcome of a complex process.
15

Way forward?
Double-loop learning much more difficult but more relevant to our aims…? We
15
want to focus on:

Way forward?
15
want to focus on:
• Interaction

Way forward?
want to focus on:
• Interaction
• Participation quality
15

Way forward?
want to focus on:
• Interaction
• Learning by doing and doing by learning
15

Way forward?
want to focus on:
• Interaction
• Aligning
15

Way forward?
want to focus on:
• Interaction
• Aligning
• Iteration
15

Way forward?
want to focus on:
• Interaction
• Aligning
• Iteration
• Can or should one central body do this?
15

Way forward?
want to focus on:
• Interaction
• Aligning
• Iteration
• Can or should one central body do this?
• Or do we need user generated content? A decentralised collective
15
participatory M&E?

change both the contents and context of the intervention. It will change the way how stakeholders
frame problems, solutions and their own role. Double-loop learning is seen as a process in which
learning is an important Way precondition for systematic forward?
transitions to take place.
Indicators that focus on double-loop learning can be used to evaluate DSM interventions and to
see whether they contribute to long-term, broader and more lasting changes (Breukers at al.
2009). In the table below single- and double-loop learning and their main indicators are shown.
Learning/evaluation Type of measurement/evaluation
Single-loop learning Efficiency indicators:
- Cost-effectiveness
- Goals reached (within given time and allocated budget)
Effectiveness indicators:
- Reaching the intended goals
- Lowering the total energy consumption
Double-loop learning Process indicators:
- Realizing a network of the intermediary filled with a heterogeneous set
of actors
- Interaction and participation by the target group (so that they can learn
about their own behaviour and consequences for energy consumption)
- Interaction and participation with a diverse set of stakeholders since
the design phase
- Learning as an explicit aim of the intervention
- Record new lessons for future interventions
- Making use of lessons that are learned during previous interventions
perspectives of intermediaries before and after a intervention
changes in assumptions, norms and beliefs
Content indicators:
- Alignment of the expectations of the stakeholder
- Learned lessons during the intervention are translated into
(re)designs.
- Improving the capacity of own or similar organizations to perform
successful DSM interventions
16
29 refs
- Creation of new networks and institutions that support the newly
formed behaviour and its outcomes
- Lasting changes (behavioural change)
Table 2: Indicators for evaluating successful learning processes (Breukers et al, 2009)

How to evaluate different levels
This applies to both habitoual fan d bonee-off hor oane-vshoit boehauviourr. ?See the figure below for an
overview of the types of behaviour interventions can target:
Figure 1: behaviour spectrum, retrieved from Breukers & Mourik 2013
We differentiate between one-shot behaviours that are performed rarely and consciously e.g.
investing in energy efficiency improvements. Habitual behaviour is more frequent, e.g. the
showering, changing the settings of the thermostat. Lasting changes in namely habitual behaviour
will continuously lead to energy savings. According to Breukers et al (2009), in this definition of
effectiveness, an energy DSM intervention is highly effective when it has reached its goals and/or
has had a positive effect on reducing the total energy consumption and when it has led to lasting
behavioural change and energy savings in the target group. Evaluating this lasting effectiveness
is, however, a major challenge, as will be discussed in the next section.
Efficiency is usually measured in 17
terms of cost-effectiveness, which compares the inputs and
outputs of a DSM intervention. These cost-effectiveness calculations can be made from various
Effectiveness is based on changing habitual behaviours which
will lead to ongoing energy savings. This is very difficult to
undertake. Efficiency is usually measured in terms of cost-effectiveness,
which compares the inputs and outputs of a DSM
intervention.

Some conclusions
• more negotiable and flexible practice of monitoring with a
mix of both quantitative and qualitative indicators
18

Some conclusions
• become smart about identifying end users to work with,
and approach these selected end users with more qualitative
methods to understand the, where, when, whom, how and
why
18

Some conclusions
why
• methods can be interviews, house tours, diary exercises
and unobtrusive health and eg temperature monitoring. This
can help cluster different behaviour types that can explain
variations between end users
18

Some conclusions
why
• methods can be interviews, house tours, diary exercises
and unobtrusive health and eg temperature monitoring. This
can help cluster different behaviour types that can explain
variations between end users
• don’t be afraid to tell stories and anecdotes. Perceptions of
success can be more important than actual measures of kWh
savings...
18

Storytelling to evaluate impact?
An effective way to also report on the learning process is to focus explicitly on
the learning stories which are in essence a process of co-design and dialogue and
retrace replicable elements in these learning stories to allow for a more
successful delivery of comprehensive EE DSM interventions (Moezzi and Janda
2014). Storytelling is an effective dialogue and evaluation tool, it allows for
multiple perspectives and creates a deeper appreciation for the fact that there is
not one truth. It allows to move beyond the presented and pretended objectivity
of a more quantitative approach. It not only allows for different morals to be
discussed, it almost demands it, we are all aware of the almost inherited right of
stories to have multiple interpretations depending on the reader, so instead of
either accepting or opposing a story, readers are encouraged to try to
understand a story and its multiple interpretations. Through the telling of stories
the listeners and presenters learn, also about negative and unintended
consequences. But they also learn to experience bad experiences as learning and
turning points in a story, with the aim to do better next time.
19

Or: how to evaluate the impact of
storytelling?
20

Evaluating Behaviour Change

More Related Content

Similar to Evaluating Behaviour Change

More from SEA - Sustainable Energy Advice Ltd

Recently uploaded

Evaluating Behaviour Change