Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluating Behaviour Change

613 views

Published on

This is a presentation held by IEA DSM Task 24 Operating Agent, Dr Sea Rotmann in Graz, October 13, 2014. It presents some of the main findings of Dr Ruth Mourik's Subtask 3 report 'Did you behave as we designed you to?'.

Published in: Science
  • Be the first to comment

  • Be the first to like this

Evaluating Behaviour Change

  1. 1. IEA DSM Implementing Agreement Subtasks of Task XXIV Task 24 ‘Subtask 3- monitoring and evaluating ‘behaviour’ change Dr Sea Rotmann, Graz Task 24 workshop, October 14, 2014
  2. 2. SubtaSskusb otf aTsaksks XXIV 5- Social Media Expert platform 1- Helicopter view of models, frameworks, contexts, case studies and evaluation metrics 2- In depth analysis in areas of greatest need (buildings, transport, SMEs, smart metering) 3- Evaluation tool for stakeholders 4- Country-specific recommen-dations, to do’s and not to do’s
  3. 3. SubtaSskusb otf aTsaksks XXIV 5- Social Media Expert platform 1- Helicopter view of models, frameworks, contexts, case studies and evaluation metrics 2- In depth analysis in areas of greatest need (buildings, transport, SMEs, smart metering) 3- Evaluation tool for stakeholders 4- Country-specific recommen-dations, to do’s and not to do’s 3- Evaluation tool for stakeholders
  4. 4. subtask III - Subtask I - Pre mHiesleic fooprt Tera sOkv XerXvIiVew evaluation  WHAT IS A SUCCESSFUL LONG-TERM BEHAVIOUR CHANGE OUTCOME TO YOU? 3
  5. 5. Subtask I - Pre mHiesleic fooprt Tera sOkv XerXvIiVew Subtask III: Outputs indicators19 such as number of installed installations or KWh saved potentially are not even a real proxy; minor savings might involve most intensive behaviour changes whilst major savings might have been the result of a relatively isolated behaviour change, e.g. buying and installing a new heating system or LED lighting. 20 x A last point is that if only modelled savings are calculated, and real savings are not meeting these calculations, the uptake and acceptance of the involved technologies, e.g. passive houses, or services such as energy performance contracting will face serious problems (Batey, Mourik and Garcia 2013). x See below an illustrative picture that demonstrates quite clearly why a proxy such as savings or KWh reduction is unable to explain the why and how of behaviour change. 4
  6. 6. Subtask I - Pre mHiesleic fooprt Tera sOkv XerXvIiVew Subtask III: Outputs indicators19 such as number of installed installations or KWh saved potentially are not even a real proxy; minor savings might involve most intensive behaviour changes whilst major savings might have been the result of a relatively isolated behaviour change, e.g. buying and installing a new heating system or LED lighting. 20 x A last point is that if only modelled savings are calculated, and real savings are not meeting these calculations, the uptake and acceptance of the involved technologies, e.g. passive houses, or services such as energy performance contracting will face serious problems (Batey, Mourik and Garcia 2013). x See below an illustrative picture that demonstrates quite clearly why a proxy such as savings or KWh reduction is unable to explain the why and how of behaviour change. 4
  7. 7. Subtask I - Pre mHiesleic fooprt Tera sOkv XerXvIiVew Subtask III: Outputs - Individual evaluation and monitoring metrics for each domain in the Subtask I Monster/Wiki, plus separate report indicators19 such as number of installed installations or KWh saved potentially are not even a real proxy; minor savings might involve most intensive behaviour changes whilst major savings might have been the result of a relatively isolated behaviour change, e.g. buying and installing a new heating system or LED lighting. 20 x A last point is that if only modelled savings are calculated, and real savings are not meeting these calculations, the uptake and acceptance of the involved technologies, e.g. passive houses, or services such as energy performance contracting will face serious problems (Batey, Mourik and Garcia 2013). x See below an illustrative picture that demonstrates quite clearly why a proxy such as savings or KWh reduction is unable to explain the why and how of behaviour change. 4
  8. 8. Subtask I - Pre mHiesleic fooprt Tera sOkv XerXvIiVew Subtask III: Outputs - Individual evaluation and monitoring metrics for each domain in the Subtask I Monster/Wiki, plus separate report - An overview and some recommendations on monitoring and evaluation can be found in Subtask III report ‘Did you behave as we designed you intdoicat?or’s19 such as number of installed installations or KWh saved potentially are not even a real proxy; minor savings might involve most intensive behaviour changes whilst major savings might have been the result of a relatively isolated behaviour change, e.g. buying and installing a new heating system or LED lighting. 20 x A last point is that if only modelled savings are calculated, and real savings are not meeting these calculations, the uptake and acceptance of the involved technologies, e.g. passive houses, or services such as energy performance contracting will face serious problems (Batey, Mourik and Garcia 2013). x See below an illustrative picture that demonstrates quite clearly why a proxy such as savings or KWh reduction is unable to explain the why and how of behaviour change. 4
  9. 9. Subtask I - Pre mHiesleic fooprt Tera sOkv XerXvIiVew Subtask III: Outputs - Individual evaluation and monitoring metrics for each domain in the Subtask I Monster/Wiki, plus separate report - An overview and some recommendations on monitoring and evaluation can be found in Subtask III report ‘Did you behave as we designed you to?’ indicators19 such as number of installed installations or KWh saved potentially are not even a real proxy; minor savings might involve most intensive behaviour changes whilst major savings might have been the result of a relatively isolated behaviour change, e.g. buying and installing a new heating system or LED lighting. 20 - There will also x be A last point a methodological is that if only modelled savings are calculated, review and real savings based are not meeting these calculations, the uptake and acceptance of the involved technologies, e.g. passive on ‘Beyond kWh’ which will feed houses, or services such as energy performance contracting will face serious problems (Batey, Mourik into and Garcia Subtask 2013). IX x See below an illustrative picture that demonstrates quite clearly why a proxy such as savings or KWh reduction is unable to explain the why and how of behaviour change. 4
  10. 10. subtask III - evaluation metrics Subtask I - Pre mHiesleic fooprt Tera sOkv XerXvIiVew Conventional monitoring of smart metering success More systemic monitoring of smart metering 5 success x !"#$%&'()''*#+&,'#%,%&*'+!-'(&')%%-$+./' 0!,%&)+.%*'0!*,+11%-' x 1(+-'*20),'3%&'.10%!,'' x !"#$%&'()',0#%*'.10%!,*'1((/%-'+,',2%' )%%-$+./'3&(40-%-' x +..%3,+!.%'+!-'+,,0,"-%*',(5+&-*'*#+&,' #%,%&*' x 61%.,&0.0,7'.(!*"#3,0(!'(4%&'+'7%+&' x 1%4%1'()',%.2!(1(87'+))0!0,7'.(!.%&!0!8',2%' "*%'()',2%',%.2!0.+1')%%-$+./'%9"03#%!,' x +11'()',2%'0**"%*'10*,%-'1%),:'+!-',2(*%' #%!,0(!%-'"!-%&'*7*,%#0.'&%,&()0,,0!8' #(!0,(&0!8'31"*; x <%&*(!+1'#(,04+,0(!',('3+&,0.03+,%'0!',2%' .(#3%,0,0(!' x =.,"+1'%!%&87>&%1+,%-'$%2+40("&*' x ?%.%!,'3"&.2+*%*'0!'%!%&87',%.2!(1(80%*' @10/%'%!%&87'%))0.0%!,'$(01%&*:'!%5' 50!-(5*:',%.AB' x C2%'0!)(&#+,0(!'1%4%1'(!'%!%&87'%))0.0%!.7' +!-'&%!%5+$1%'%!%&87'*("&.%*' x D("&.%*'(!'0!)(&#+,0(!'(!'%!%&87'0**"%*' x =,,0,"-%*'(!'%!%&87'+!-'.10#+,%' 3&(,%.,0(!'0**"%*' x 6*,0#+,0(!'()',2%'1%4%1'()'(5!'%!%&87'.(*,*' x $"01-0!8'()'.+3+.0,7:'' x .&%+,0(!'()'%!8+8%#%!,' x ."*,(#%&'*%!,0#%!,:'' x 3+&,0.03+,0(!'0!'(,2%&'%!%&87'%))0.0%!.7' 3&(8&+#*' x )%%10!8'()'.(!,&(1'@(4%&'%!%&87'$011*:',2%' 2(#%:'%!%&87B' x 1%4%1'()'"!%#31(7#%!,:'' x 1%4%1'()'0110,%&+.7' x E!,%&!%,'3%!%,&+,0(!'&+,%' '
  11. 11. Life seemed easy… 6
  12. 12. Life seemed easy… What is it? • Monitoring: measuring progress and achievements and production of planned outputs • Evaluation: structured process of assessing success in meeting goals and reflect on learnings. Explicitly places a value judgement on the data and information gathered in an intervention 6
  13. 13. Life seemed easy… What is it? • Monitoring: measuring progress and achievements and production of planned outputs • Evaluation: structured process of assessing success in meeting goals and reflect on learnings. Explicitly places a value judgement on the data and information gathered in an intervention Why do it the way we do now? Establish effect of policies Assess need for improvements Assessing value for money Contribution to evidence base for effectiveness of behavioral interventions at population level 6
  14. 14. Life seemed easy… What is it? • Monitoring: measuring progress and achievements and production of planned outputs • Evaluation: structured process of assessing success in meeting goals and reflect on learnings. Explicitly places a value judgement on the data and information gathered in an intervention Why do it the way we do now? Establish effect of policies Assess need for improvements Assessing value for money Contribution to evidence base for effectiveness of behavioral interventions at population level How to do it…….??? 6
  15. 15. It’s getting challenging… 7
  16. 16. It’s getting challenging… • Evaluation team often not included in design 7
  17. 17. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… 7
  18. 18. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… • Evaluation usually is only a snapshot at end or just after 7
  19. 19. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… • Evaluation usually is only a snapshot at end or just after • Often insufficient benchmarking 7
  20. 20. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… • Evaluation usually is only a snapshot at end or just after • Often insufficient benchmarking • Not longitudinal, sustainability/rebound often not assessed 7
  21. 21. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… • Evaluation usually is only a snapshot at end or just after • Often insufficient benchmarking • Not longitudinal, sustainability/rebound often not assessed • No insight in formation of networks supporting lasting change 7
  22. 22. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… • Evaluation usually is only a snapshot at end or just after • Often insufficient benchmarking • Not longitudinal, sustainability/rebound often not assessed • No insight in formation of networks supporting lasting change • Mismatch between needs of project managers s/h it is aimed at 7
  23. 23. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… • Evaluation usually is only a snapshot at end or just after • Often insufficient benchmarking • Not longitudinal, sustainability/rebound often not assessed • No insight in formation of networks supporting lasting change • Mismatch between needs of project managers s/h it is aimed at • Large-scale M&E of actual behaviour too costly 7
  24. 24. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… • Evaluation usually is only a snapshot at end or just after • Often insufficient benchmarking • Not longitudinal, sustainability/rebound often not assessed • No insight in formation of networks supporting lasting change • Mismatch between needs of project managers s/h it is aimed at • Large-scale M&E of actual behaviour too costly • Modeling or self-reported (at best) 7
  25. 25. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… • Evaluation usually is only a snapshot at end or just after • Often insufficient benchmarking • Not longitudinal, sustainability/rebound often not assessed • No insight in formation of networks supporting lasting change • Mismatch between needs of project managers s/h it is aimed at • Large-scale M&E of actual behaviour too costly • Modeling or self-reported (at best) • ‘proxies’, such as savings or even better: cost 7 effectiveness
  26. 26. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… • Evaluation usually is only a snapshot at end or just after • Often insufficient benchmarking • Not longitudinal, sustainability/rebound often not assessed • No insight in formation of networks supporting lasting change • Mismatch between needs of project managers s/h it is aimed at • Large-scale M&E of actual behaviour too costly • Modeling or self-reported (at best) • ‘proxies’, such as savings or even better: cost 7 effectiveness • Proxies = NOT actual behaviour change, only about value for money etc
  27. 27. It’s getting challenging… • Evaluation team often not included in design • Often not even part of the programme… • Evaluation usually is only a snapshot at end or just after • Often insufficient benchmarking • Not longitudinal, sustainability/rebound often not assessed • No insight in formation of networks supporting lasting change • Mismatch between needs of project managers s/h it is aimed at • Large-scale M&E of actual behaviour too costly • Modeling or self-reported (at best) • ‘proxies’, such as savings or even better: cost 7 effectiveness • Proxies = NOT actual behaviour change, only about value for money etc • No participatory process or feedback loops in the traditional M&E
  28. 28. To make life more difficult.. 8
  29. 29. To make life more difficult.. We increasingly value interventions that are: • tailored, • multidisciplinary, • varied interventions, • qualitative and iterative, • systemic, • and have outcomes beyond the duration of project and beyond energy 8
  30. 30. To make life more difficult.. We increasingly value interventions that are: • tailored, • multidisciplinary, • varied interventions, • qualitative and iterative, • systemic, • and have outcomes beyond the duration of project and beyond energy And at the same time we judge the ‘behaviour’ of policymakers who demand for simple, focused, quantitative and up-scaled evaluations defining success in efficiency and effectiveness terms. 8
  31. 31. To make life more difficult.. We increasingly value interventions that are: • tailored, • multidisciplinary, • varied interventions, • qualitative and iterative, • systemic, • and have outcomes beyond the duration of project and beyond energy And at the same time we judge the ‘behaviour’ of policymakers who demand for simple, focused, quantitative and up-scaled evaluations defining success in efficiency and effectiveness terms. But how could M&E look like that is: 8
  32. 32. To make life more difficult.. We increasingly value interventions that are: • tailored, • multidisciplinary, • varied interventions, • qualitative and iterative, • systemic, • and have outcomes beyond the duration of project and beyond energy And at the same time we judge the ‘behaviour’ of policymakers who demand for simple, focused, quantitative and up-scaled evaluations defining success in efficiency and effectiveness terms. But how could M&E look like that is: Relevant to end-users, ‘cost effective’, doable, lasting actual behavioral change, formation of networks, focusing on alignment, and processes underpinning that change? 8
  33. 33. What now? 9
  34. 34. What now? • No unified way of designing and M&E interventions • Different disciplinary approaches have different methods and foci of M&E, all pertinent to what they aim 9
  35. 35. How do different behavioural models/disciplines evaluate? 1. Economic theory: individuals’ behaviours are seen as (semi-) rational decisions that are made through cost-benefit calculations. II. Psychological theory: the individual also takes a central role; however, it is increasingly acknowledged that this individual also operates as part of a collective e.g. by imitating the behaviour of important others. Many psychological approaches view decision making of an individual as a mental calculation aimed at making choices; these calculations are informed by both emotion and cold calculus. III. Sociological theory: they put more emphasis on the importance of the social nature of energy use and to the abilities of people to participate in change in ways that fit their own contexts and concerns. The central focus is on social practices, individuals move into the background. 10
  36. 36. How do different behavioural models/disciplines evaluate? Intervention goals and evaluation methodologies commonly used in interventions underpinned by the three disciplines discussed above are shown in the table below (this is not an extensive list, it is aimed at highlighting foci and differences). Goals 14 Methodologies Remarks (e.g. about causal 11 relationships) Economic perspectives Outputs Cost-efficiency and effectiveness Units, and proxies e.g. number of participants, home insulated, technologies installed, KWh saved etc. Labels Modelling Surveys Experiments Randomised control trials Presence of cause Æ effect relationship. Aim is to meet a priori set goals Monitoring and evaluation often only for duration of implementation, no longer term Psychological perspectives Outputs Cost-efficiency and effectiveness Behavioural changes Surveys self-reported behavioural changes structured interviews randomised control trials Surveys to identify behavioural determinants like motivations, attitudes, etc. Cause-effect relationships: Effect on individuals of a particular incentive, via e.g. awareness, attitude, behaviour. Interfering variables like social context often not taken into account Sociological perspectives Outputs and Outcomes Cost-efficiency and effectiveness Learning about what works, when, where, who, how (long) and why Learning about interdependencies Learning about co-shaping and reshaping User accounts Time diaries Cultural probes In-depth open interviews Analysis of fit of interventions with daily life measuring real, not modelled energy consumption Context & mechanism/conditions produce an outcome. Direct cause-effect relationships hard to establish because of interdependencies that cannot be analysed separately. 14 We will also insert a column on the underlying processes - how does an intervention work, admittedly typically at the individual level (what changed in people's understanding, motivations, attitudes)!
  37. 37. What now? 12
  38. 38. What now? • Perhaps more fruitful to focus on learning processes*? 1. Single loop = instrumental, focused on short-term learning about effectiveness in meeting goals/ outcome focused 12
  39. 39. What now? • Perhaps more fruitful to focus on learning processes*? 1. Single loop = instrumental, focused on short-term learning about effectiveness in meeting goals/ outcome focused 2. Double loop = process oriented, focused on the how and why, long-term learning 12
  40. 40. What now? • Perhaps more fruitful to focus on learning processes*? 1. Single loop = instrumental, focused on short-term learning about effectiveness in meeting goals/ outcome focused 2. Double loop = process oriented, focused on the how and why, long-term learning *Based on work by Prof Chrys Argyris, Psychological and Organisational Development 12
  41. 41. Single vs double-loop learning Single-loop learning involves connecting a strategy for action with a result. Eg, if an action we take yields results that are different to what we expected, through single-loop learning, we will observe the results, automatically take in feedback, and try a different approach. This cyclical process of applying a new strategy to achieve an expected or desired outcome may occur several times and we may never succeed. Running out of strategies may push us to re-evaluate the deeper governing variables that make us behave the ways we do. Re-evaluating and reframing our goals, values and beliefs is a more complex way of processing information and involves a more sophisticated way of engaging with an experience. This is called double-loop learning and looks at consequences from a wider perspective. 13 2.
  42. 42. Single vs double-loop learning 14
  43. 43. Single vs double-loop learning 14
  44. 44. Single vs double-loop learning 14
  45. 45. Single vs double-loop learning 14
  46. 46. Single vs double-loop learning 14
  47. 47. Single vs double-loop learning 14
  48. 48. Single vs double-loop learning 14
  49. 49. Single vs double-loop learning 14
  50. 50. Way forward? 15
  51. 51. Way forward? M and E of single-loop learning doable to undertake and fine for low-hanging fruit and non-habitual change. Sees behavioural change interventions more or less as linear cause and effect relationships (A+B=C: Intervention A targeted on group B will cause the intended Change C) 15
  52. 52. Way forward? M and E of single-loop learning doable to undertake and fine for low-hanging fruit and non-habitual change. Sees behavioural change interventions more or less as linear cause and effect relationships (A+B=C: Intervention A targeted on group B will cause the intended Change C) A change of focus amongst policymakers and funders towards allowing experimentation with more systemic and messy real life interventions that do not provide easily quantifiable and scalable information is a big transition. It demands amongst others that policymakers appreciate that these systemic interventions cannot be evaluated in terms of cause and effect, but are the outcome of a complex process. 15
  53. 53. Way forward? M and E of single-loop learning doable to undertake and fine for low-hanging fruit and non-habitual change. Sees behavioural change interventions more or less as linear cause and effect relationships (A+B=C: Intervention A targeted on group B will cause the intended Change C) A change of focus amongst policymakers and funders towards allowing experimentation with more systemic and messy real life interventions that do not provide easily quantifiable and scalable information is a big transition. It demands amongst others that policymakers appreciate that these systemic interventions cannot be evaluated in terms of cause and effect, but are the outcome of a complex process. Double-loop learning much more difficult but more relevant to our aims…? We 15 want to focus on:
  54. 54. Way forward? M and E of single-loop learning doable to undertake and fine for low-hanging fruit and non-habitual change. Sees behavioural change interventions more or less as linear cause and effect relationships (A+B=C: Intervention A targeted on group B will cause the intended Change C) A change of focus amongst policymakers and funders towards allowing experimentation with more systemic and messy real life interventions that do not provide easily quantifiable and scalable information is a big transition. It demands amongst others that policymakers appreciate that these systemic interventions cannot be evaluated in terms of cause and effect, but are the outcome of a complex process. Double-loop learning much more difficult but more relevant to our aims…? We 15 want to focus on: • Interaction
  55. 55. Way forward? M and E of single-loop learning doable to undertake and fine for low-hanging fruit and non-habitual change. Sees behavioural change interventions more or less as linear cause and effect relationships (A+B=C: Intervention A targeted on group B will cause the intended Change C) A change of focus amongst policymakers and funders towards allowing experimentation with more systemic and messy real life interventions that do not provide easily quantifiable and scalable information is a big transition. It demands amongst others that policymakers appreciate that these systemic interventions cannot be evaluated in terms of cause and effect, but are the outcome of a complex process. Double-loop learning much more difficult but more relevant to our aims…? We want to focus on: • Interaction • Participation quality 15
  56. 56. Way forward? M and E of single-loop learning doable to undertake and fine for low-hanging fruit and non-habitual change. Sees behavioural change interventions more or less as linear cause and effect relationships (A+B=C: Intervention A targeted on group B will cause the intended Change C) A change of focus amongst policymakers and funders towards allowing experimentation with more systemic and messy real life interventions that do not provide easily quantifiable and scalable information is a big transition. It demands amongst others that policymakers appreciate that these systemic interventions cannot be evaluated in terms of cause and effect, but are the outcome of a complex process. Double-loop learning much more difficult but more relevant to our aims…? We want to focus on: • Interaction • Participation quality • Learning by doing and doing by learning 15
  57. 57. Way forward? M and E of single-loop learning doable to undertake and fine for low-hanging fruit and non-habitual change. Sees behavioural change interventions more or less as linear cause and effect relationships (A+B=C: Intervention A targeted on group B will cause the intended Change C) A change of focus amongst policymakers and funders towards allowing experimentation with more systemic and messy real life interventions that do not provide easily quantifiable and scalable information is a big transition. It demands amongst others that policymakers appreciate that these systemic interventions cannot be evaluated in terms of cause and effect, but are the outcome of a complex process. Double-loop learning much more difficult but more relevant to our aims…? We want to focus on: • Interaction • Participation quality • Learning by doing and doing by learning • Aligning 15
  58. 58. Way forward? M and E of single-loop learning doable to undertake and fine for low-hanging fruit and non-habitual change. Sees behavioural change interventions more or less as linear cause and effect relationships (A+B=C: Intervention A targeted on group B will cause the intended Change C) A change of focus amongst policymakers and funders towards allowing experimentation with more systemic and messy real life interventions that do not provide easily quantifiable and scalable information is a big transition. It demands amongst others that policymakers appreciate that these systemic interventions cannot be evaluated in terms of cause and effect, but are the outcome of a complex process. Double-loop learning much more difficult but more relevant to our aims…? We want to focus on: • Interaction • Participation quality • Learning by doing and doing by learning • Aligning • Iteration 15
  59. 59. Way forward? M and E of single-loop learning doable to undertake and fine for low-hanging fruit and non-habitual change. Sees behavioural change interventions more or less as linear cause and effect relationships (A+B=C: Intervention A targeted on group B will cause the intended Change C) A change of focus amongst policymakers and funders towards allowing experimentation with more systemic and messy real life interventions that do not provide easily quantifiable and scalable information is a big transition. It demands amongst others that policymakers appreciate that these systemic interventions cannot be evaluated in terms of cause and effect, but are the outcome of a complex process. Double-loop learning much more difficult but more relevant to our aims…? We want to focus on: • Interaction • Participation quality • Learning by doing and doing by learning • Aligning • Iteration • Can or should one central body do this? 15
  60. 60. Way forward? M and E of single-loop learning doable to undertake and fine for low-hanging fruit and non-habitual change. Sees behavioural change interventions more or less as linear cause and effect relationships (A+B=C: Intervention A targeted on group B will cause the intended Change C) A change of focus amongst policymakers and funders towards allowing experimentation with more systemic and messy real life interventions that do not provide easily quantifiable and scalable information is a big transition. It demands amongst others that policymakers appreciate that these systemic interventions cannot be evaluated in terms of cause and effect, but are the outcome of a complex process. Double-loop learning much more difficult but more relevant to our aims…? We want to focus on: • Interaction • Participation quality • Learning by doing and doing by learning • Aligning • Iteration • Can or should one central body do this? • Or do we need user generated content? A decentralised collective 15 participatory M&E?
  61. 61. change both the contents and context of the intervention. It will change the way how stakeholders frame problems, solutions and their own role. Double-loop learning is seen as a process in which learning is an important Way precondition for systematic forward? transitions to take place. Indicators that focus on double-loop learning can be used to evaluate DSM interventions and to see whether they contribute to long-term, broader and more lasting changes (Breukers at al. 2009). In the table below single- and double-loop learning and their main indicators are shown. Learning/evaluation Type of measurement/evaluation Single-loop learning Efficiency indicators: - Cost-effectiveness - Goals reached (within given time and allocated budget) Effectiveness indicators: - Reaching the intended goals - Lowering the total energy consumption Double-loop learning Process indicators: - Realizing a network of the intermediary filled with a heterogeneous set of actors - Interaction and participation by the target group (so that they can learn about their own behaviour and consequences for energy consumption) - Interaction and participation with a diverse set of stakeholders since the design phase - Learning as an explicit aim of the intervention - Record new lessons for future interventions - Making use of lessons that are learned during previous interventions perspectives of intermediaries before and after a intervention changes in assumptions, norms and beliefs Content indicators: - Alignment of the expectations of the stakeholder - Learned lessons during the intervention are translated into (re)designs. - Improving the capacity of own or similar organizations to perform successful DSM interventions 16 29 refs - Creation of new networks and institutions that support the newly formed behaviour and its outcomes - Lasting changes (behavioural change) Table 2: Indicators for evaluating successful learning processes (Breukers et al, 2009)
  62. 62. How to evaluate different levels This applies to both habitoual fan d bonee-off hor oane-vshoit boehauviourr. ?See the figure below for an overview of the types of behaviour interventions can target: Figure 1: behaviour spectrum, retrieved from Breukers & Mourik 2013 We differentiate between one-shot behaviours that are performed rarely and consciously e.g. investing in energy efficiency improvements. Habitual behaviour is more frequent, e.g. the showering, changing the settings of the thermostat. Lasting changes in namely habitual behaviour will continuously lead to energy savings. According to Breukers et al (2009), in this definition of effectiveness, an energy DSM intervention is highly effective when it has reached its goals and/or has had a positive effect on reducing the total energy consumption and when it has led to lasting behavioural change and energy savings in the target group. Evaluating this lasting effectiveness is, however, a major challenge, as will be discussed in the next section. Efficiency is usually measured in 17 terms of cost-effectiveness, which compares the inputs and outputs of a DSM intervention. These cost-effectiveness calculations can be made from various Effectiveness is based on changing habitual behaviours which will lead to ongoing energy savings. This is very difficult to undertake. Efficiency is usually measured in terms of cost-effectiveness, which compares the inputs and outputs of a DSM intervention.
  63. 63. Some conclusions 18
  64. 64. Some conclusions • more negotiable and flexible practice of monitoring with a mix of both quantitative and qualitative indicators 18
  65. 65. Some conclusions • more negotiable and flexible practice of monitoring with a mix of both quantitative and qualitative indicators • become smart about identifying end users to work with, and approach these selected end users with more qualitative methods to understand the, where, when, whom, how and why 18
  66. 66. Some conclusions • more negotiable and flexible practice of monitoring with a mix of both quantitative and qualitative indicators • become smart about identifying end users to work with, and approach these selected end users with more qualitative methods to understand the, where, when, whom, how and why • methods can be interviews, house tours, diary exercises and unobtrusive health and eg temperature monitoring. This can help cluster different behaviour types that can explain variations between end users 18
  67. 67. Some conclusions • more negotiable and flexible practice of monitoring with a mix of both quantitative and qualitative indicators • become smart about identifying end users to work with, and approach these selected end users with more qualitative methods to understand the, where, when, whom, how and why • methods can be interviews, house tours, diary exercises and unobtrusive health and eg temperature monitoring. This can help cluster different behaviour types that can explain variations between end users • don’t be afraid to tell stories and anecdotes. Perceptions of success can be more important than actual measures of kWh savings... 18
  68. 68. Storytelling to evaluate impact? An effective way to also report on the learning process is to focus explicitly on the learning stories which are in essence a process of co-design and dialogue and retrace replicable elements in these learning stories to allow for a more successful delivery of comprehensive EE DSM interventions (Moezzi and Janda 2014). Storytelling is an effective dialogue and evaluation tool, it allows for multiple perspectives and creates a deeper appreciation for the fact that there is not one truth. It allows to move beyond the presented and pretended objectivity of a more quantitative approach. It not only allows for different morals to be discussed, it almost demands it, we are all aware of the almost inherited right of stories to have multiple interpretations depending on the reader, so instead of either accepting or opposing a story, readers are encouraged to try to understand a story and its multiple interpretations. Through the telling of stories the listeners and presenters learn, also about negative and unintended consequences. But they also learn to experience bad experiences as learning and turning points in a story, with the aim to do better next time. 19
  69. 69. Or: how to evaluate the impact of storytelling? 20

×