Measuring users' experience - or, the memory of them?

Measuring users’ experiences
or, the memory of them?

Trajectory reminders EmoSnaps Footprint tracker

Evangelos Karapanos

Newcastle, 8 March 2011

Nuno
Nunes Vassilis
Kostakos Monchu
Chen

Laura Rodríguez Gonçalo
Gouveia Néstor
Catano

• 20
faculty Pedro
Campos Paulo
Sampaio Eduardo
Fermé

– 14
countries,
8
languages

• Areas: Larry
ConstanHne Jos
van
Leeuwen Barbara
Pizzileo

– 11
CS,
2
physics/electronics,

2
psychology,
2
architecture,

2
design,
2
art,
2
other
Ian
Oakley Luis
Gomes Ron
Salden

Leonel
Nóbrega ValenHna
Nisi Evangelos
Karapanos

David
Aveiro Luis
Gomes Yoram
Chisik

MSc HCI & Entertainment Technology

My
background

BSc Physics / microelectronics, U Patras, Greece (2004)
Thesis: Model based design and evaluation of walk-up-and-use interfaces (HCI
Group, ECE department)

MSc HCI / UCL Interaction Centre, UK (2005)
Thesis: User acceptance of nomadic user interfaces (Philips Research,
Eindhoven)

PhD HCI / TU Eindhoven, NL (2010)
Title: Quantifying diversity in user experience

Some
of
my
current
work

(that
I
will
not
discuss
today)

Socially translucent eco-feedback technologies
How do eco-feedback technologies:
a) raise mutual awareness of family members’ consumption behaviors
b) induce feelings of accountability on individuals regarding their consumption behaviors.
!

Citizen participation on the go
How can we motivate citizen participation through mobile technologies?
•Public transit: The role of psychological empowerment: self-efﬁcacy, sense of community,
and causal importance
•Inclucity: The role of visual and location cues on users’ ability to reconstruct the context,
and form an empathic understanding of the experience of disabled individuals

Location-aware narratives: Does locality matter?
Does the coupling between physical and virtual space result to increased immersion in
the narrative world?

Outline
User experience over time
What makes for positive experiences in the long run?

iScale - longitudinal data through memory
Can an online survey tool assist users in recollecting their experiences
with a product?

Technology Assisted Reconstruction
Can mobile sensors assist participants in reconstructing their daily
experiences and whereabouts?

Soft Reliability

48% of returned products are not attributed to a
violation of product speciﬁcations

problems rooted early
in (concept) design phase

failure to truly incorporate it in one’s life

User
experience
over
Hme
An
exploratory
study

How
do
users
form
overall
evalua/ve
judgments

about
interac/ve
products?

Hassenzahl,
2004

Karapanos, E., Hassenzahl, M., & Martens, J.-B. (2008). User experience over time. CHI ’08 extended abstracts on Human factors in
computing systems (pp. 3561-3566). Florence, Italy: ACM.

User
experience
over
Hme
An
exploratory
study

Items
close
together
are
highly
correlated.
Lines
represent
clusters.

Karapanos, E., Hassenzahl, M., & Martens, J.-B. (2008). User experience over time. CHI ’08 extended abstracts on Human factors in
computing systems (pp. 3561-3566). Florence, Italy: ACM.

What makes for positive experiences in the long run?
Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2009). User experience over time: an initial framework. In CHI '09:
Proceedings of the 27th international conference on Human factors in computing systems (pp. 729-738). New York, NY, USA: ACM.

Now
think
of
the
three
experiences
that
were
for
you
personally
most

sa.sfying
or
unsa.sfying
experiences
of
today.
Please,
use
your

own
feeling
or
a
deﬁni.on
of
what
“sa.sfying”
and

“unsa.sfying
experience”
means.
Take
a
couple
of
minutes
to
be

sure
to
come
up
with
three
most
crucial
experiences;
you
may
also
want
to
write

them
down
for
yourself.
We
want
you
to
be
open
as
to
which
experiences
to

report.


Cross-‐sec8onal Repeated
sampling

“Longitudinal”
paradigms
in
HCI
Longitudinal Retrospec8ve

Karapanos, E., Martens, J.-B., Hassenzahl, M. (2009) Reconstructing Experiences through Sketching. Arxiv preprint, arXiv:0912.5343.

Remembering
is
an
act
of
reconstruc.on

rather
than
reproduc.on
Barlea
(1932)

How do we recall experiences?

• Validity? i.e. do memories reﬂect what we really
experienced?

• Reliability? i.e. in a second trial, will we recall
the same experiences?

Can we assist people in recalling - more
reliably - their experiences with a product?

How do we recall emotional
experiences?

Two schools of thought
- The Constructive approach
- The Value-Account approach

The Constructive approach
The emotional experience can neither be Design principles
stored nor retrieved. It is reconstructed 1. Feed-forward sketching, as each recalled event
from recalled contextual details will cue more events, eventually resulting to richer
episodic memories from which to infer emotions
Robinson & Clore (2002)
2. Concurrency between sketching and
reporting, as reporting will positively contribute
Type of Knowledge Source of information Type of Self-Report towards the recall of episodic cues

Experiential Online emotion, e.g.
Episodic
Knowledge Experience Sampling

Retrospective, e.g.
Episodic Episodic memory
Day Reconstruction

Situation-speciﬁc
Semantic Exit questionnaires
belief

Identity-related
belief

The Value-Account approach
People may recall an overall evaluation of an Design principles
event even when they fail to recall contextual
1. Top-down sketching (i.e., split completed line in
details - “I like it but I don’t know why” parts), as participants are expected to have direct
(Betsch et al., 2001) access to this value-charged information
2. Non-concurrency between sketching and
reporting, as reporting (i.e., thinking about concrete
This type of memory, Value-Account: episodic details) might hinder or bias the recall of
value-charged information
- is more accessible than episodic memory
- can be used to cue the reconstruction
from episodic memory
- is better retained over time

Constructive Value-Account Control (no-graphing)

Constructive iScale, but not the Value-Account, performed better than
control condition

•More experience reports
•With more details (references to temporal information, discrete events)
•Higher test-retest consistency of time estimation (i.e., when did
an experience take place)
•Higher test-retest consistency of graphed patterns (over Value-
Account) 24

Technology Assisted Reconstruction

Trajectory reminders Emosnaps Footprint tracker

Newcastle, 8 March 2011

Experience Sampling Method
What variables do ES studies measure?
We distinguish below between self-reported measures of
behavior and experience (see Table 1). This distinction is
sampling method. Two of these provided no justifications
for their choice. Analyzing the remaining 19 papers resulted
to a total of 11 reasons for choosing alternative methods to
relevant as their reconstruction follows a different process ESM (see table 2).
whereas behavioral information may be directly accessible
through episodic memory while experiential information Table 2. Reasons for not selecting the Experience Sampling
Prompts at random, or computationally
has to be further inferred from recalled episodic cues [63]. Method along with frequency of occurence (No of papers).

to self-report on ongoing
estimated times,measures of of ES studies eliciting or both.
Table 1. Number
behavior, experience,
self-reported Reason No

behaviors and experiences. Disrupts the activity 6
Type of measures that studies elicit No
Imposes high burden to participants 3
– Where areSelf-reported measures of behavior
you? 5
Requires high effort from researchers 3
– What are you doing? of experience
Self-reported measures 22
– How far isSelf-reported measures phone?& experience
your mobile of behavior 18
Inappropriate for eliciting rich qualitative data 3

– How do you feel? Misses rare and brief events 3
Behavioral measures related most frequently to the activity
a few relevant venues would still miss a substantial number method while following a user-initiated diary approach.
of studies. We instead used the ACM Guide towas engaged with prior to of the remaining 49 user should be reported
(n=18) that the participant Computing The analysis being The studies is being in control of when, what and 2
Literature querying for the term “experience duration (n=2) (e.g. [35]), the
interrupted (e.g. [30]), its sampling” below. how often to report
No of papers referring to
without constraining to particular venues. This query
participant’s current physical location (n=15) length, sampling frequency, and response rate
returned 284 papers, published in more than fifty venues. Study
(e.g. [17])
Experience Sampling
and the social context (n=10), e.g. the number or nature of of the studies had a duration of several
The majority (80%) Limits sample size 2
60
relationship of people that are in close proximity or
days up to one month with 14 studies (34%) lasting
participate with in a conversation (e.g. [34]). Other seven days (see figure 3). Only two
between four and Depends on participants’ ability to articulate 2
45 measures of behavior related to mode studies had a duration of more than a month.
of transit [25], ongoing experience
participants’ current physical engagement [18] and mode of
15
convrersation (e.g. f2f, fixed/mobile phone etc.) [32].
30 Poses privacy concerns 2
Experiential measures related to:
15
10
• Attitudes towards behaviors or events (n=15) such as Limits number of measured variables 1
0 being interrupted (e.g. [55]), disclosing information to
2001 2003 2005 2007 2009 2011being video recorded (e.g.
relevant others (e.g. [17]), or Technology limitations 1
5
[58]).
Figure 2. 243 papers referring to experience sampling over a

Day Reconstruction Method
Kahneman et al. (2004)

Can a retrospective method help participants in
recalling more accurately their experiences?

ry attempt to Type of Knowledge Source of information Type of Self-Report

ntation of the
an unfamiliar Episodic
Experiential
Knowledge
Online emotion, e.g.
Experience Sampling
called stories
tails, altering
in applying Retrospective, e.g.
Episodic Episodic memory
riginal story. Day Reconstruction

h repeated
Situation-speciﬁc
econstruction belief

e support. At
ion between
Identity-related
hile episodic Semantic
belief
Exit questionnaires
om the past,
lar event but Robinson & Clore (2002)

Technology
Assisted
ReconstrucHon
Can mobile sensors assist participants in reconstructing their daily experiences and
whereabouts?

Trajectory reminders in location-based preferences
Do trajectory reminders (locations visited before and after) increase the test-retest
reliability of the reconstruction process?

Emosnaps - inferring emotion from self-face pics
Can self-face pictures assist in recalling momentary emotions?
If so, is it through a recognition or a reconstruction process?

Footprint tracker
How do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls
made or received) assist in reconstructing daily behaviors and experiences?

Control condition With trajectory reminders

! !

When recalling with trajectory reminders, participants were more consistent over two repeated recalls
separated by 1 week

Measuring
emoHons
in
mobile
contexts
Self-Cam ’06 Wearable EMG interface ’10
Teeters, Kaliouby & Picard Gruebler & Suzuki

Can we develop a tool that is truly unobtrusive to daily
life and can be employed in long-term ﬁeld studies?

Emosnaps - inferring emotion from self-face pics
Can self-face pictures assist in recalling momentary emotions?
If so, is it through a recognition or a reconstruction process?

Experience Sampling (Ground truth)
78% of pictures
could be used for
inferring emotions

Time-Day Photo-day Photo-week
control condition Emotion reconstruction Emotion recognition

1. Sensecam

2. Location logging
3. Context logging
(SMS/calls made or received)

Thank you
Evangelos Karapanos
ekarapanos.com

User experience over time
What makes for positive experiences
in the long run?

iScale
Can an online survey tool assist users in
recollecting their experiences with a product?

Tech. Assist. Reconstruction
Can mobile sensors assist participants in
reconstructing their daily experiences and
whereabouts?

Measuring users' experience - or, the memory of them?

Recommended

Recommended

More Related Content

Similar to Measuring users' experience - or, the memory of them?

Similar to Measuring users' experience - or, the memory of them? (20)

More from Evan Karapanos

More from Evan Karapanos (6)

Recently uploaded

Recently uploaded (20)

Measuring users' experience - or, the memory of them?