The document discusses the Usability Perception Scale (UPscale) developed to evaluate eco-feedback displays, emphasizing the impact of technology and new media on environmental behaviors. It highlights the effectiveness of feedback in promoting energy savings while addressing methodological limitations in existing studies. Additionally, it outlines the scale's psychometric properties, factor structures, and the importance of addressing unique needs for eco-feedback systems.

1. The Usability Perception Scale (UPscale):
A Measure for Evaluating Feedback Displays
Beth Karlin
Transformational Media Lab
bkarlin@uci.edu
Rebecca Ford
Center for Sustainability
rebecca.ford@otago.ac.nz

2. Underlying Assumptions
1. Technology and new media are changing how people
interact with our natural, built, and social worlds.
B. Karlin

3. Underlying Assumptions
1. Technology and new media are changing how people
interact with our natural, built, and social worlds.
2. There are potential opportunities to leverage these
changes for pro-social / pro-environmental benefit.
B. Karlin

4. Underlying Assumptions
1. Technology and new media are changing how people
interact with our natural, built, and social worlds.
2. There are potential opportunities to leverage these
changes for pro-social / pro-environmental benefit
3. A psychological approach provides a theoretical base
and empirical methodology to study this potential.
B. Karlin

5. Transformational Media Lab
Mission:
Our lab studies how technology and new media are (and can be)
used to transform individuals, communities, and systems.
Documentary
Film
Campaigns
Home Energy
Management
B. Karlin

6. Energy Feedback
“Information about the result of a process or
action that can be used in modification or
control of a process or system”
Oxford English Dictionary
B. Karlin

7. Energy Feedback
1888

8. Energy Feedback
— Average frequency: monthly
(approx. 12 data points/year)
— Average frequency: hourly
(approx 8,760 data points/year)
B. Karlin

9. Small changes, big impacts
Energy usage tells its own story...
Power Consumption (Watts)
$9.24 $5.28 Savings: $3.96
43%
And the computer
is still plugged in…
B. Karlin
(uci@home project)

10. Appliance Disaggregation
(up to 6.3 trillion data points/year)
blu-ray netflix
streaming
200 microsecond sampling
(uci@home project) B. Karlin

11. Savings Add Up
“Household actions can provide a behavioral wedge to rapidly
reduce carbon emissions …”
• 5-12% reduction in 5 years
• 9-22% reduction in 10 years
“…without waiting for new technologies or regulations or
changing household lifestyle.”
Dietz, Gardner, Gilligan, Stern, & Vandenbergh (2009)
B. Karlin

12. Over 200 devices on the market
(Karlin, Ford, & Squiers, in press) B. Karlin

13. Public and Private Interest
What are we
missing?

14. Feedback is effective…
— 100+ studies conducted since 1976
— Reviews found average 10% savings
— Mean r-effect size = .1174 (p < .001)
• Significant variability in effects
(from negative effects to over 20% savings)
Darby, 2006; Ehrhardt-Martinez et al., 2010;
Fischer, 2008; Karlin & Zinger, under review B. Karlin

15. Feedback i✗s c an be effective…
— 100+ studies conducted since 1976
— Reviews found average 10% savings
— Mean r-effect size = .1174 (p < .001)
• Significant variability in effects
(from negative effects to over 20% savings)
Da rby, 2006; Ehrhardt-Martinez et al., 2010;
Fischer, 2008; Karlin & Zinger, in preparation
B. Karlin

16. Feedback i✗s c an be effective…
It
depends.
.
.
Ehrhardt-­‐Martinez,
Laitner,
&
Donnely.,
2010
10%
5%
15%
2%
20%
average
savings

17. ✗ can be effective…
Feedback is
It depends…
Moderators identified in meta-analysis
• Study population (WHO?)
• Study duration (HOW LONG?)
• Frequency of feedback (HOW OFTEN?)
• Feedback medium (WHAT TYPE?)
• Disaggregation (WHAT LEVEL?)
• Comparison (WHAT MESSAGE?)
Karlin & Zinger, in preparation B. Karlin

18. Methodological Limitations
1. Not naturalistic
— Participants generally recruited to participate
— May be different from “active adopters”
2. Not comparative
— Most studies tests one type of feedback (vs. control)
— Very few studies isolating or combining variables
3. Not testing mediation
— DV is energy use, but studies rarely test possible
mediators to explain effectiveness
B. Karlin, 2013

19. Methodological Limitations
— Not naturalistic
— Participants generally recruited to participate
— May be different from “active adopters”
— Not comparative
— Most studies tests one type of feedback (vs. control)
— Very few studies isolating or combining variables
— Not testing mediation
— DV is energy use, but studies rarely test possible
mediators to explain effectiveness
B. Karlin, 2013

20. Simple causal model
What is going on here?
Program x
Outcome y
Does program x lead to outcome y?
B. Karlin
What is the program?
How do we measure outcomes?
How and for whom does program x lead to outcome y?

21. How and For Whom?
B. Karlin

22. Beyond kWh Model
B. Karlin
Experience

23. Usability
B. Karlin

24. Psychometrics
• Theory and technique of measurement:
knowledge, abilities, attitudes, traits
• Construction and validation of instruments:
questionnaires, tests, assessments.
B. Karlin

25. Psychometrics
B. Karlin
Psychometric Properties
1. Factor Structure
2. Reliability
3. Criterion Validity
4. Sensitivity

26. System Usability Scale
Identified Factors:
1. System usability
2. Learnability
(Brooke, 1986) B. Karlin

27. Other scales
ASQ
1. User satisfaction
SUMI
1. Affect
2. Efficiency
3. Learnability
4. Helpfulness
5. Control
PSSUQ
1. System usefulness
2. Information quality
QUIS 3. Interface quality
1. Overall reaction
2. Learning
3. Terminology
4. Information flow
5. System output
6. System characteristics UMUX
1. Efficiency
2. Effectiveness
3. Satisfaction

28. Identified Limitations
1. Designed primarily to evaluate products or
systems rather than info-visualizations
2. Assessed with metrics primarily associated
w/ease of use (e.g., learnability) & efficiency.
Less focus on continued engagement.
B. Karlin

29. Identified Needs
1. Address the unique needs of eco-feedback
displays (as opposed to systems or products)
2. Incorporate validated sub-scales for ease of
use and engagement
B. Karlin

30. UPscale (Usability Perception)
B. Karlin

31. Testing UPscale
B. Karlin
• Online survey (Mechanical Turk)
• 1103 people
• Part of larger study, testing framing
messages and info-visualization

32. Results
B. Karlin
1. Factor Structure
2. Reliability
3. Criterion Validity
4. Sensitivity

33. Results
B. Karlin
1. Factor Structure
2. Reliability
3. Criterion Validity
4. Sensitivity

34. Results
B. Karlin
1. Factor Structure
2. Reliability
3. Criterion Validity
4. Sensitivity
Overall scale (α=.85)
Ease of use (α=.84)
Engagement (α=.83)

35. Results
B. Karlin
1. Factor Structure
2. Reliability
3. Criterion Validity
B4e.h Saveinosriatli vinittye ntion (p<.001)
• overall scale (r=.536)
• ease of use (r=.213)
• engagement (r=.685)

36. Results
B. Karlin
1. Factor Structure
2. Reliability
3. Criterion Validity
4. Sensitivity
Image Type.
• Full scale (F=3.616, p=.001)
• Ease of use subscale (F=6.411, p<.001)
• Engagement subscale (F=1.744, p=.095).
Demographic Variables.
• Full Scale: Age, Environmentalism
• Engagement: Gender, age, environmentalism, income
• Ease of Use: None

37. UPscale (Usability Perception)
B. Karlin

38. Closing Thoughts
“If you do not know how to ask the right question,
you know nothing.”
– Edward Deming
Thank You!
Beth Karlin
Transformational Media Lab
bkarlin@uci.edu
Rebecca Ford
Center for Sustainability
rebecca.ford@otago.ac.nz

39. A theoretical approach
Program x
Outcome y
Hypothesis / Theory
Clearly defined and
operationalized
Metrics tested for
reliability & validity
How and for whom does program x lead to outcome y?
B. Karlin, 2013