Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Exploring Deep Savings: A Toolkit for Assessing Behavior-Based Energy Interventions

376 views

Published on

While research assessing behavior-based energy interventions shows great promise, results vary widely and much is still unknown about the specific variables that impact program effectiveness. As utilities and regulatory agencies focus more attention on behavior-based energy interventions, it becomes critical to ensure that evaluations of such programs are rigorous and accurate. While the metric used to measure whether these various programs work (kWh) is fairly standard and easy to compare between studies, the metrics used to measure how and for whom they work have been left to individual researchers and evaluators. Standardization of assessment methods is common in related fields such as education and psychology, but has yet to take hold in energy program evaluation. This paper argues for a more systematic and comprehensive approach to the evaluation of behavior-based energy interventions, and describes a preliminary toolkit that is currently being developed and validated in conjunction with the International Energy Agency Demand Side Management Programme (IEA-DSM) Task 24 on Behavior Change as well as two large investor-owned utilities. Our approach is informed by theories and empirical research on behavior change as well as a content analysis of 85 behavior-based energy interventions. It includes questions on: context (demographics), user experience (ease of use, engagement), material culture (what people have), energy practices (what people do), and beliefs around energy use (what people think). Sample items for each construct and suggestions for implementation are presented. Broad use of such an instrument can improve and aggregate our overall knowledge across the countless additional studies expected to be conducted in the coming years.

Published in: Environment
  • Be the first to comment

Exploring Deep Savings: A Toolkit for Assessing Behavior-Based Energy Interventions

  1. 1. Exploring Deep Savings: A Toolkit for Assessing Behavior-Based Energy Interventions Beth Karlin Transformational Media Lab University of California, Irvine Papers listed here. Contact me for details: bkarlin@uci.edu.
  2. 2. Is Feedback Effective?  100+ studies conducted since 1976  Total n = 256,536 (mean 119/study)  Mean r-effect size = .1174 (p < .001)  Average energy savings: 9% Significant variability in effects (from negative effects to over 20% savings) Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.
  3. 3. Is Feedback Effective? It depends… Moderators identified in meta-analysis • Study population (WHO?) • Study duration (HOW LONG?) • Frequency of feedback (HOW OFTEN?) • Feedback medium (WHAT TYPE?) • Disaggregation (WHAT LEVEL?) • Comparison (WHAT MESSAGE?) Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.
  4. 4. Methodological Limitations 1. Not naturalistic  Participants generally recruited to participate  May be different from “active adopters” 2. Not comparative  Most studies tests one type of feedback (vs. control)  Very few studies isolating or combining variables 3. Not testing mediation  DV is energy use, but studies rarely test possible mediators to explain effectiveness Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.
  5. 5. Methodological Limitations 1. Not naturalistic  Participants generally recruited to participate  May be different from “active adopters” 2. Not comparative  Most studies tests one type of feedback (vs. control)  Very few studies isolating or combining variables 3. Not testing mediation  DV is energy use, but studies rarely test possible mediators to explain effectiveness Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.
  6. 6. Will they come? If you build it,
  7. 7. Program x Outcome y Does it work? Does program x lead to outcome y?
  8. 8. Program x Outcome y Questions remain… What is the program? What is going on here? How do we measure outcomes? Does program x lead to outcome y?
  9. 9. Program x Outcome y A theoretical approach Hypothesis / Theory Clearly defined and operationalized Metrics tested for reliability & validity Does program x lead to outcome y? How and for whom does program x lead to outcome y?
  10. 10. How and For Whom?
  11. 11. How and for Whom?
  12. 12. Toolkits in other fields
  13. 13. Toolkits in other fields
  14. 14. Question Bias
  15. 15. Question Bias
  16. 16. Question Bias
  17. 17. Psychometrics • Theory and technique of measurement: knowledge, abilities, attitudes, traits • Construction and validation of instruments: questionnaires, tests, assessments
  18. 18. Psychometrics Psychometric Properties 1.Factor Structure 2. Reliability 3. Criterion Validity 4. Sensitivity
  19. 19. Our Project Initial concept paper IEA DSM Task 24 Methods review Report for IEA Karlin et al., 2015 Toolkit Pilot testing Member country testing Consultation Feedback SCE Psychometric testing 2013 201620152014 PG&E Field testing (scheduled)
  20. 20. Methodological Review Karlin, Ford, Wu, & Nasser. (2015). What Do We Know About What We Know? A Review of Behaviour-Based Energy Efficiency Data Collection. IEA-DSM Task 24 Subtask 3 Report.
  21. 21. Karlin, Ford, Wu, & Nasser. (2015). What Do We Know About What We Know? A Review of Behaviour-Based Energy Efficiency Data Collection. IEA-DSM Task 24 Subtask 3 Report. Data Collection Methods Used
  22. 22. Instruments Provided? Karlin, Ford, Wu, & Nasser. (2015). What Do We Know About What We Know? A Review of Behaviour-Based Energy Efficiency Data Collection. IEA-DSM Task 24 Subtask 3 Report.
  23. 23. Toolkit Development
  24. 24. Energy Cultures Frameworks Material culture Energy practices Norms and aspiration s Have DoThink
  25. 25. Toolkit Development Heating devices Energy sources Insulation House structure Have
  26. 26. Toolkit Development Social expectations and Environmental concern Expected warmth levels Maintaining traditions Think
  27. 27. Toolkit Development Turning on heater Putting on jersey Maintaining heating technologies Drawing curtainsDo
  28. 28. Toolkit Development Have DoThink Have DoThink Energy Culture at start Energy Culture at end Have DoThink Have DoThink Energy Culture at start Energy Culture at end Intervention Natural changes User experience Technology interaction ControlGroupTreatmentGroup
  29. 29. Toolkit Development Have DoThink Have DoThink Energy Culture at start Energy Culture at end Have DoThink Have DoThink Energy Culture at start Energy Culture at end Intervention Natural changes User experience Technology interaction ControlGroupTreatmentGroup
  30. 30. Next Steps 1. Psychometric testing (SCE) 2. Local field testing (PG&E / SCE) 3. Member country review 4. Global field testing 5. Wide scale adoption? 
  31. 31. Beth Karlin Transformational Media Lab University of California, Irvine bkarlin@uci.edu Thank you! (comments and suggestions welcome)

×