Download as PDF, PPTX
![Experience Sampling Method
What variables do ES studies measure?
We distinguish below between self-reported measures of
behavior and experience (see Table 1). This distinction is
sampling method. Two of these provided no justifications
for their choice. Analyzing the remaining 19 papers resulted
to a total of 11 reasons for choosing alternative methods to
relevant as their reconstruction follows a different process ESM (see table 2).
whereas behavioral information may be directly accessible
through episodic memory while experiential information Table 2. Reasons for not selecting the Experience Sampling
Prompts at random, or computationally
has to be further inferred from recalled episodic cues [63]. Method along with frequency of occurence (No of papers).
to self-report on ongoing
estimated times,measures of of ES studies eliciting or both.
Table 1. Number
behavior, experience,
self-reported Reason No
behaviors and experiences. Disrupts the activity 6
Type of measures that studies elicit No
Imposes high burden to participants 3
– Where areSelf-reported measures of behavior
you? 5
Requires high effort from researchers 3
– What are you doing? of experience
Self-reported measures 22
– How far isSelf-reported measures phone?& experience
your mobile of behavior 18
Inappropriate for eliciting rich qualitative data 3
– How do you feel? Misses rare and brief events 3
Behavioral measures related most frequently to the activity
a few relevant venues would still miss a substantial number method while following a user-initiated diary approach.
of studies. We instead used the ACM Guide towas engaged with prior to of the remaining 49 user should be reported
(n=18) that the participant Computing The analysis being The studies is being in control of when, what and 2
Literature querying for the term “experience duration (n=2) (e.g. [35]), the
interrupted (e.g. [30]), its sampling” below. how often to report
No of papers referring to
without constraining to particular venues. This query
participant’s current physical location (n=15) length, sampling frequency, and response rate
returned 284 papers, published in more than fifty venues. Study
(e.g. [17])
Experience Sampling
and the social context (n=10), e.g. the number or nature of of the studies had a duration of several
The majority (80%) Limits sample size 2
60
relationship of people that are in close proximity or
days up to one month with 14 studies (34%) lasting
participate with in a conversation (e.g. [34]). Other seven days (see figure 3). Only two
between four and Depends on participants’ ability to articulate 2
45 measures of behavior related to mode studies had a duration of more than a month.
of transit [25], ongoing experience
participants’ current physical engagement [18] and mode of
15
convrersation (e.g. f2f, fixed/mobile phone etc.) [32].
30 Poses privacy concerns 2
Experiential measures related to:
15
10
• Attitudes towards behaviors or events (n=15) such as Limits number of measured variables 1
0 being interrupted (e.g. [55]), disclosing information to
2001 2003 2005 2007 2009 2011being video recorded (e.g.
relevant others (e.g. [17]), or Technology limitations 1
5
[58]).
Figure 2. 243 papers referring to experience sampling over a](https://image.slidesharecdn.com/newcastle2012march8-120310134622-phpapp01/75/Measuring-users-experience-or-the-memory-of-them-26-2048.jpg)
Uploaded byEvan Karapanos
Measuring users' experience - or, the memory of them?
This document discusses several studies related to measuring and reconstructing user experiences over time. It first describes research on how users form overall evaluations of interactive products over time. It then examines whether an online survey tool called iScale can help users recollect their experiences with a product by assisting in the reconstruction of memories. Finally, it explores whether mobile sensors and technology can help participants reconstruct their daily experiences and locations through tools like trajectory reminders, photos of emotional expressions, and a footprint tracker using visual, location and context cues.
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Measuring users' experience - or, the memory of them?
- 1. Measuring users’ experiences or, the memory of them? Trajectory reminders EmoSnaps Footprint tracker Evangelos Karapanos Newcastle, 8 March 2011
- 3. Nuno Nunes Vassilis Kostakos Monchu Chen Laura Rodríguez Gonçalo Gouveia Néstor Catano • 20 faculty Pedro Campos Paulo Sampaio Eduardo Fermé – 14 countries, 8 languages • Areas: Larry ConstanHne Jos van Leeuwen Barbara Pizzileo – 11 CS, 2 physics/electronics, 2 psychology, 2 architecture, 2 design, 2 art, 2 other Ian Oakley Luis Gomes Ron Salden Leonel Nóbrega ValenHna Nisi Evangelos Karapanos David Aveiro Luis Gomes Yoram Chisik
- 4. MSc HCI &Entertainment Technology
- 5.
- 6. My background BSc Physics/ microelectronics, U Patras, Greece (2004) Thesis: Model based design and evaluation of walk-up-and-use interfaces (HCI Group, ECE department) MSc HCI / UCL Interaction Centre, UK (2005) Thesis: User acceptance of nomadic user interfaces (Philips Research, Eindhoven) PhD HCI / TU Eindhoven, NL (2010) Title: Quantifying diversity in user experience
- 7. Some of my current work (that I will not discuss today) Socially translucent eco-feedback technologies How do eco-feedback technologies: a) raise mutual awareness of family members’ consumption behaviors b) induce feelings of accountability on individuals regarding their consumption behaviors. ! Citizen participation on the go How can we motivate citizen participation through mobile technologies? •Public transit: The role of psychological empowerment: self-efficacy, sense of community, and causal importance •Inclucity: The role of visual and location cues on users’ ability to reconstruct the context, and form an empathic understanding of the experience of disabled individuals Location-aware narratives: Does locality matter? Does the coupling between physical and virtual space result to increased immersion in the narrative world?
- 8. Outline User experience overtime What makes for positive experiences in the long run? iScale - longitudinal data through memory Can an online survey tool assist users in recollecting their experiences with a product? Technology Assisted Reconstruction Can mobile sensors assist participants in reconstructing their daily experiences and whereabouts?
- 9. Soft Reliability 48% ofreturned products are not attributed to a violation of product specifications
- 10. problems rooted early in(concept) design phase
- 11. failure to trulyincorporate it in one’s life
- 12. User experience over Hme An exploratory study How do users form overall evalua/ve judgments about interac/ve products? Hassenzahl, 2004 Karapanos, E., Hassenzahl, M., & Martens, J.-B. (2008). User experience over time. CHI ’08 extended abstracts on Human factors in computing systems (pp. 3561-3566). Florence, Italy: ACM.
- 13. User experience over Hme An exploratory study Items close together are highly correlated. Lines represent clusters. Karapanos, E., Hassenzahl, M., & Martens, J.-B. (2008). User experience over time. CHI ’08 extended abstracts on Human factors in computing systems (pp. 3561-3566). Florence, Italy: ACM.
- 14. What makes forpositive experiences in the long run? Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2009). User experience over time: an initial framework. In CHI '09: Proceedings of the 27th international conference on Human factors in computing systems (pp. 729-738). New York, NY, USA: ACM.
- 15. Now think of the three experiences that were for you personally most sa.sfying or unsa.sfying experiences of today. Please, use your own feeling or a defini.on of what “sa.sfying” and “unsa.sfying experience” means. Take a couple of minutes to be sure to come up with three most crucial experiences; you may also want to write them down for yourself. We want you to be open as to which experiences to report. Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2009). User experience over time: an initial framework. In CHI '09: Proceedings of the 27th international conference on Human factors in computing systems (pp. 729-738). New York, NY, USA: ACM.
- 16. Karapanos, E., Zimmerman,J., Forlizzi, J., & Martens, J. (2009). User experience over time: an initial framework. In CHI '09: Proceedings of the 27th international conference on Human factors in computing systems (pp. 729-738). New York, NY, USA: ACM.
- 17. Cross-‐sec8onal Repeated sampling “Longitudinal” paradigms in HCI Longitudinal Retrospec8ve
- 18. Karapanos, E., Martens,J.-B., Hassenzahl, M. (2009) Reconstructing Experiences through Sketching. Arxiv preprint, arXiv:0912.5343.
- 19. Remembering is an act of reconstruc.on rather than reproduc.on Barlea (1932)
- 20. How do werecall experiences? • Validity? i.e. do memories reflect what we really experienced? • Reliability? i.e. in a second trial, will we recall the same experiences? Can we assist people in recalling - more reliably - their experiences with a product?
- 21. How do werecall emotional experiences? Two schools of thought - The Constructive approach - The Value-Account approach
- 22. The Constructive approach Theemotional experience can neither be Design principles stored nor retrieved. It is reconstructed 1. Feed-forward sketching, as each recalled event from recalled contextual details will cue more events, eventually resulting to richer episodic memories from which to infer emotions Robinson & Clore (2002) 2. Concurrency between sketching and reporting, as reporting will positively contribute Type of Knowledge Source of information Type of Self-Report towards the recall of episodic cues Experiential Online emotion, e.g. Episodic Knowledge Experience Sampling Retrospective, e.g. Episodic Episodic memory Day Reconstruction Situation-specific Semantic Exit questionnaires belief Identity-related Semantic Exit questionnaires belief
- 23. The Value-Account approach People may recall an overall evaluation of an Design principles event even when they fail to recall contextual 1. Top-down sketching (i.e., split completed line in details - “I like it but I don’t know why” parts), as participants are expected to have direct (Betsch et al., 2001) access to this value-charged information 2. Non-concurrency between sketching and reporting, as reporting (i.e., thinking about concrete This type of memory, Value-Account: episodic details) might hinder or bias the recall of value-charged information - is more accessible than episodic memory - can be used to cue the reconstruction from episodic memory - is better retained over time
- 24. Constructive Value-Account Control (no-graphing) Constructive iScale, but not the Value-Account, performed better than control condition •More experience reports •With more details (references to temporal information, discrete events) •Higher test-retest consistency of time estimation (i.e., when did an experience take place) •Higher test-retest consistency of graphed patterns (over Value- Account) 24
- 25. Technology Assisted Reconstruction Trajectoryreminders Emosnaps Footprint tracker Newcastle, 8 March 2011
- 26. Experience Sampling Method What variables do ES studies measure? We distinguish below between self-reported measures of behavior and experience (see Table 1). This distinction is sampling method. Two of these provided no justifications for their choice. Analyzing the remaining 19 papers resulted to a total of 11 reasons for choosing alternative methods to relevant as their reconstruction follows a different process ESM (see table 2). whereas behavioral information may be directly accessible through episodic memory while experiential information Table 2. Reasons for not selecting the Experience Sampling Prompts at random, or computationally has to be further inferred from recalled episodic cues [63]. Method along with frequency of occurence (No of papers). to self-report on ongoing estimated times,measures of of ES studies eliciting or both. Table 1. Number behavior, experience, self-reported Reason No behaviors and experiences. Disrupts the activity 6 Type of measures that studies elicit No Imposes high burden to participants 3 – Where areSelf-reported measures of behavior you? 5 Requires high effort from researchers 3 – What are you doing? of experience Self-reported measures 22 – How far isSelf-reported measures phone?& experience your mobile of behavior 18 Inappropriate for eliciting rich qualitative data 3 – How do you feel? Misses rare and brief events 3 Behavioral measures related most frequently to the activity a few relevant venues would still miss a substantial number method while following a user-initiated diary approach. of studies. We instead used the ACM Guide towas engaged with prior to of the remaining 49 user should be reported (n=18) that the participant Computing The analysis being The studies is being in control of when, what and 2 Literature querying for the term “experience duration (n=2) (e.g. [35]), the interrupted (e.g. [30]), its sampling” below. how often to report No of papers referring to without constraining to particular venues. This query participant’s current physical location (n=15) length, sampling frequency, and response rate returned 284 papers, published in more than fifty venues. Study (e.g. [17]) Experience Sampling and the social context (n=10), e.g. the number or nature of of the studies had a duration of several The majority (80%) Limits sample size 2 60 relationship of people that are in close proximity or days up to one month with 14 studies (34%) lasting participate with in a conversation (e.g. [34]). Other seven days (see figure 3). Only two between four and Depends on participants’ ability to articulate 2 45 measures of behavior related to mode studies had a duration of more than a month. of transit [25], ongoing experience participants’ current physical engagement [18] and mode of 15 convrersation (e.g. f2f, fixed/mobile phone etc.) [32]. 30 Poses privacy concerns 2 Experiential measures related to: 15 10 • Attitudes towards behaviors or events (n=15) such as Limits number of measured variables 1 0 being interrupted (e.g. [55]), disclosing information to 2001 2003 2005 2007 2009 2011being video recorded (e.g. relevant others (e.g. [17]), or Technology limitations 1 5 [58]). Figure 2. 243 papers referring to experience sampling over a
- 27. Day Reconstruction Method Kahneman et al. (2004) Can a retrospective method help participants in recalling more accurately their experiences? ry attempt to Type of Knowledge Source of information Type of Self-Report ntation of the an unfamiliar Episodic Experiential Knowledge Online emotion, e.g. Experience Sampling called stories tails, altering in applying Retrospective, e.g. Episodic Episodic memory riginal story. Day Reconstruction h repeated Situation-specific Semantic Exit questionnaires econstruction belief e support. At ion between Identity-related hile episodic Semantic belief Exit questionnaires om the past, lar event but Robinson & Clore (2002)
- 28. Technology Assisted ReconstrucHon Can mobile sensors assist participants in reconstructing their daily experiences and whereabouts? Trajectory reminders in location-based preferences Do trajectory reminders (locations visited before and after) increase the test-retest reliability of the reconstruction process? Emosnaps - inferring emotion from self-face pics Can self-face pictures assist in recalling momentary emotions? If so, is it through a recognition or a reconstruction process? Footprint tracker How do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls made or received) assist in reconstructing daily behaviors and experiences?
- 29. Technology Assisted ReconstrucHon Can mobile sensors assist participants in reconstructing their daily experiences and whereabouts? Trajectory reminders in location-based preferences Do trajectory reminders (locations visited before and after) increase the test-retest reliability of the reconstruction process? Emosnaps - inferring emotion from self-face pics Can self-face pictures assist in recalling momentary emotions? If so, is it through a recognition or a reconstruction process? Footprint tracker How do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls made or received) assist in reconstructing daily behaviors and experiences?
- 30. Control condition With trajectory reminders ! ! When recalling with trajectory reminders, participants were more consistent over two repeated recalls separated by 1 week
- 31. Technology Assisted ReconstrucHon Can mobile sensors assist participants in reconstructing their daily experiences and whereabouts? Trajectory reminders in location-based preferences Do trajectory reminders (locations visited before and after) increase the test-retest reliability of the reconstruction process? Emosnaps - inferring emotion from self-face pics Can self-face pictures assist in recalling momentary emotions? If so, is it through a recognition or a reconstruction process? Footprint tracker How do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls made or received) assist in reconstructing daily behaviors and experiences?
- 32. Measuring emoHons in mobile contexts Self-Cam ’06 Wearable EMG interface ’10 Teeters, Kaliouby & Picard Gruebler & Suzuki Can we develop a tool that is truly unobtrusive to daily life and can be employed in long-term field studies? Emosnaps - inferring emotion from self-face pics Can self-face pictures assist in recalling momentary emotions? If so, is it through a recognition or a reconstruction process?
- 33. Experience Sampling (Groundtruth) 78% of pictures could be used for inferring emotions Time-Day Photo-day Photo-week control condition Emotion reconstruction Emotion recognition
- 34. Technology Assisted ReconstrucHon Can mobile sensors assist participants in reconstructing their daily experiences and whereabouts? Trajectory reminders in location-based preferences Do trajectory reminders (locations visited before and after) increase the test-retest reliability of the reconstruction process? Emosnaps - inferring emotion from self-face pics Can self-face pictures assist in recalling momentary emotions? If so, is it through a recognition or a reconstruction process? Footprint tracker How do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls made or received) assist in reconstructing daily behaviors and experiences?
- 35. 1. Sensecam 2. Location logging 3. Context logging (SMS/calls made or received)
- 38. Thank you Evangelos Karapanos ekarapanos.com User experience over time What makes for positive experiences in the long run? iScale Can an online survey tool assist users in recollecting their experiences with a product? Tech. Assist. Reconstruction Can mobile sensors assist participants in reconstructing their daily experiences and whereabouts?