1.
Exploring Deep Savings:
A Toolkit for Assessing Behavior-Based Energy
Interventions
Beth Karlin
Transformational Media Lab
University of California, Irvine
Papers listed here. Contact me for details: bkarlin@uci.edu.

2.
Is Feedback Effective?
 100+ studies conducted since 1976
 Total n = 256,536 (mean 119/study)
 Mean r-effect size = .1174 (p < .001)
 Average energy savings: 9%
Significant variability in effects
(from negative effects to over 20% savings)
Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy
Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.

3.
Is Feedback Effective?
It depends…
Moderators identified in meta-analysis
• Study population (WHO?)
• Study duration (HOW LONG?)
• Frequency of feedback (HOW OFTEN?)
• Feedback medium (WHAT TYPE?)
• Disaggregation (WHAT LEVEL?)
• Comparison (WHAT MESSAGE?)
Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy
Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.

4.
Methodological Limitations
1. Not naturalistic
 Participants generally recruited to participate
 May be different from “active adopters”
2. Not comparative
 Most studies tests one type of feedback (vs. control)
 Very few studies isolating or combining variables
3. Not testing mediation
 DV is energy use, but studies rarely test possible
mediators to explain effectiveness
Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy
Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.

5.
Methodological Limitations
1. Not naturalistic
 Participants generally recruited to participate
 May be different from “active adopters”
2. Not comparative
 Most studies tests one type of feedback (vs. control)
 Very few studies isolating or combining variables
3. Not testing mediation
 DV is energy use, but studies rarely test possible
mediators to explain effectiveness
Karlin, Ford & Zinger. (In Press). The Effects of Feedback on Energy
Conservation: A Preliminary Theory and Meta-Analysis. Psychological Bulletin.

6.
Will they come?
If you build it,

7.
Program x Outcome
y
Does it work?
Does program x lead to outcome y?

8.
Program x Outcome
y
Questions remain…
What is the
program?
What is going on
here?
How do we measure
outcomes?
Does program x lead to outcome y?

9.
Program x Outcome
y
A theoretical approach
Hypothesis /
Theory
Clearly defined and
operationalized
Metrics tested for
reliability & validity
Does program x lead to outcome y?
How and for whom does program x lead to outcome y?

10.
How and For Whom?

11.
How and for Whom?

12.
Toolkits in other fields

13.
Toolkits in other fields

14.
Question Bias

15.
Question Bias

16.
Question Bias

17.
Psychometrics
• Theory and technique of measurement:
knowledge, abilities, attitudes, traits
• Construction and validation of instruments:
questionnaires, tests, assessments

18.
Psychometrics
Psychometric Properties
1.Factor Structure
2. Reliability
3. Criterion Validity
4. Sensitivity

19.
Our Project
Initial
concept
paper
IEA DSM Task 24
Methods
review
Report for IEA
Karlin et al., 2015
Toolkit
Pilot
testing
Member
country
testing
Consultation
Feedback
SCE
Psychometric
testing
2013 201620152014
PG&E
Field testing
(scheduled)

20.
Methodological Review
Karlin, Ford, Wu, & Nasser. (2015). What Do We Know About What We
Know? A Review of Behaviour-Based Energy Efficiency Data Collection.
IEA-DSM Task 24 Subtask 3 Report.

21.
Karlin, Ford, Wu, & Nasser. (2015). What Do We Know About What We
Know? A Review of Behaviour-Based Energy Efficiency Data Collection.
IEA-DSM Task 24 Subtask 3 Report.
Data Collection Methods Used

22.
Instruments Provided?
Karlin, Ford, Wu, & Nasser. (2015). What Do We Know About What We
Know? A Review of Behaviour-Based Energy Efficiency Data Collection.
IEA-DSM Task 24 Subtask 3 Report.

23.
Toolkit Development

24.
Energy Cultures Frameworks
Material
culture
Energy
practices
Norms
and
aspiration
s
Have
DoThink

25.
Toolkit Development
Heating devices Energy sources
Insulation House structure
Have

26.
Toolkit Development
Social
expectations and
Environmental
concern
Expected
warmth levels
Maintaining
traditions
Think

27.
Toolkit Development
Turning on
heater
Putting on jersey
Maintaining heating
technologies
Drawing
curtainsDo

28.
Toolkit Development
Have
DoThink
Have
DoThink
Energy Culture at start Energy Culture at end
Have
DoThink
Have
DoThink
Energy Culture at start Energy Culture at end
Intervention
Natural changes
User experience
Technology interaction
ControlGroupTreatmentGroup

29.
Toolkit Development
Have
DoThink
Have
DoThink
Energy Culture at start Energy Culture at end
Have
DoThink
Have
DoThink
Energy Culture at start Energy Culture at end
Intervention
Natural changes
User experience
Technology interaction
ControlGroupTreatmentGroup

30.
Next Steps
1. Psychometric testing (SCE)
2. Local field testing (PG&E / SCE)
3. Member country review
4. Global field testing
5. Wide scale adoption? 

31.
Beth Karlin
Transformational Media Lab
University of California, Irvine
bkarlin@uci.edu
Thank you!
(comments and suggestions welcome)