Affective UX: Challenges in UX involving affective computing

01WORLD IA DAY 2016 PRESENTATION TITLE HERE
HEADER OPTION
SUB HEAD OR SHORT DESCRIPTION
Some kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.
Some kind of illustration or image?
Affective UXChallenges in UX design involving affective computing
Alfredo Sánchez
Universidad de las Américas Puebla

© 2006-2014, Alfredo Sánchez
http://www.udlap.mx

Architecturing happiness
How do we know users are happy?
Can interactive systems detect happiness (or the lack thereof)?
Can interactive systems intentionally produce happinesss?
Can technology at least not get in the way?

Agenda
• Terms of affection
• Affective computing
• Detection
• Synthesis
• Mediation
• Affect in UX
• Recent development
• Open issues

Terms of affection

Emotion
A reaction experienced usually toward a specific object and typically
accompanied by physiological and behavioral changes in the body

Mood
• A conscious state of mind (typically lasting longer than an emotion)
joy depression

Sentiment
• An attitude or opinion – a state that results from attributes assigned
to objects, which are grouped into categories

The affect spectrum
emotion
mood
sentiment
may trigger
may prompt
Duration( - ) ( + )

Affective computing
(Computer sytems) related to,
arising from, or deliberately
influencing affective phenomena
- MIT Affective Computing Group
Affective Computing
Rosalind Picard
1995

Affective computing (for engineers) 
The study and development of systems and devices that can recognize,
interpret, process, simulate and mediate human affects.

Why affective computing?

I choose
not to feel
http://animala.mx/noticias/2015/02/27/adios-mr-spock/

Why Affective Computing
• “Reason is, and ought to be,
only the slave of the passions”
David Hume, 1711-1776
• Without passions we would lack
all motivation, all impulse or
drive to act, or even to reason

• A love/hate relationship with the computer and ITs in general
https://www.youtube.com/watch?v=ZcZBTB-dkqQ

Why affective computing? (for engineers) 
• Studies by many researchers show a vital role of emotion
in many processes:
• Perception
• Decision making
• Creativity
• Comprehension
• Memory
• Social interaction
http://www.bintegra.com/wp-content/uploads/2012/05/Fotolia_socila_users_S.jpg

Some application areas
• Education – intelligent tutoring
• Health – monitoring patients
• Safety – monitoring drivers / pilots
• Security – detecting lies
• E-commerce – attitudes towards
products or services
• Video games – producing engaging
environments, enriching UX
• User studies – incorporating
emotion into evaluation

Why affective computing (for our community)
Emotion is the way
we communicate our
experience

Main aspects of affective computing
perception
synthesis
mediation
(of emotions, moods and sentiments)

Affect detection

Emotion detection: From facial cues
http://npx.autismspot.com/facial_expression.jpg

Emotion detection: From postural cues
http://instridebodywork.com/wp-content/uploads/2013/05/postures-of-emotion1.jpg

Emotion detection: From voice cues
http://pandodaily.files.wordpress.com/2013/07/whisper-in-ear.jpg?w=584&h=329
http://gigaom2.files.wordpress.com/2013/07/emotion-speech.jpg?w=300&h=200&crop=1

Emotion detection: From usage patterns
http://web.deepnetsecurity.com/products2/typesense.asp
Keystroke dynamics Navigation patterns Eye tracking

Emotion detection: From brain activity
© 2006-2014, Alfredo SánchezEmotivEPOCsensor
HondaBCIsystem

Emotion detection: From sensors of
autonomic activity
• Skin conductance increases even
with a little surprise or anxiety
• Blood volume/pressure
increases with anger or stress
• Respiratory rhythm varies with
fear, sadness, happiness, anxiety,
fear
• Muscle tension increases with
surprise and anxiety
Affectivasensor
NeXussensor

Emotion detection: From sensors of
autonomic activity
“Know your inner self”
W/Me
“Hack happiness”
Feel (by Sentio)

Some general techniques to
support affective computing
For detection, synthesis and mediation

Facial Action Coding System (FACS)
http://www.theroadtosiliconvalley.com/tag/tim-roth/

FACS
• Facial Action Coding System (FACS), very popular
• Defines Action Units (AUs),
contraction or relaxation of one or more face muscles
…

FACS (for “universal” emotions)
http://www.paulekman.com/facs/

Posture, motion and gesture analysis

Posture analysis
http://people.ict.usc.edu/~gratch/CSCI534/ACII-Handbook-GestureSyn.pdf

Posture and motion analysis
• E.g. arm movements and emotion:

Gesture analysis

MPEG-4
• MPEG-4 (ISO144996) defines
• 86 Face Animation Parameters (FAPs)
• 196 Body Animation Parameters (BAPs)

Some existing tools
• FaceReader, SHORE, eMotion, RealEyes

Some challenges
• Prototypical vs. non-prototypical emotions
• Pure vs. combined emotions
• Multiple subjects
• Partly covered faces / bodies
• Tracking face or body movement
• Lighting conditions
• Integrating techniques
• Supporting application development

Emotion in UX evaluation

Evaluating user experience with novel technologies –
impact of emotion (Gestures + contextual menus - video)
© 2006-2014, Alfredo Sánchez[Magallanes, Molina, Méndez & Sánchez 2012]

Heuristic evaluation
• Problems observed in actual use
• Issues identified by experts
ID Issue Occurrences
P1 Movement of objects using two
fingers. 13
P2 Size of the keyboard. 8
P3 Finding help for the user. 5
P4 Finding the toolbar associated
with each object. 4
P5 Navigating using the fan menu. 3
P6 Navigating between nodes.
3
P7 Identifying the icon for linking to
new pages 2
P8 Incomplete or inaccessible
menus.
1
ID Task Issue
1 Insert two images, change their
position and size.
P1
2 Enter a text next to the images as
a caption.
P2
3 Obtain the toolbar for an image. P3, P4
4 Pop a menu and discover how it
works.
P5
5 Build a hierarchy of linked
workspaces.
P6, P7
6 Get the menu in different
positions. Cause the menu to go
off the screen.
P8
7 Insert two images, change their
position and size.
P1
8 Enter a text next to the images as
a caption.
P2

Additional feedback on emotions, based on Product
Emotion Measurement Tool (PrEmo)
Disgust Indignation Contempt Dissatisfaction Disappointment Unpleasant
surprise
boredom
Desire Inspiration Admiration Satisfaction Fascination Pleasant
surprise
Amusement
[Desmet 2002]
Participants were asked to select an animated representation in correspondence with their emotional reactions
[In collaboration with Universidad del Cauca, Colombia]

Positive emotions in spite of problems!
Positive emotions associated with novelty?
Negative emotions associated with known
interface elements?[Magallanes, Molina, Méndez & Sánchez 2012]

Implications
• Techniques for evaluation of usability and user experience are needed
that take affect into account
• Framework developed at UniCauca, Colombia
[Méndez, Collazos, Granollers, Sánchez 2014; Méndez 2015]

EmoVal
• A framework for emotional evaluation designed to enrich existing UX
evaluation methods
• Three major components
• EmoVal-C: Behavioral evaluation (facial expressions, gaze, posture,
movements)
• EmoVal-S: Self-report of emotions while using a system
• EmoVal-F: Physiological sensing, mainly brain activity during system usage
• Supporting website
(http://artemisa.unicauca.edu.co/~ymendal/EmoVal)
[Méndez 2015]

EmoVal: Detailed support / facial expressions
[Méndez 2015]

EmoVal: Detailed support / posture and
movement
© 2006-2016, Alfredo Sánchez [Méndez 2015]

Building capabilities for
affect detection

Current scenario: Extending an app to include more techniques
is not straightforward
Facial cue analysisMonitoring
pilot alertness
Lie detec on
Usability
tes ng
Online tutoring
Posture analysis
Voice analysis
Keystroke
dynamics
Eye tracking
Naviga on
pa erns
Application domain Technique Technologies

Goal: Integration platform
[Sánchez et al. 2015]

Architecture for implementation
Database
Controller
EmotionsController
View
Data Model
Home
- Index
- UploadUserContent
HomeController
SessionsController
Vikara
Services
Interface
Data
entities
cshtml
Session
- Index
- List
EmotionModel
UserContentController
FileController
Emotion
UserCont
entSession
MSSQL
Resources
Google Chart
Tools
WebGrease
Modernizr
Android
Emotion
Interpreter
Windows
Emotion
Interpeter
Server
Visualization
interfaces
Other
Emotion
Interpreters
Observer /
administrator
Endusers

Simple data model and messaging protocol

Prototypical apps: Kinect-based emotion
detection
14 sorpresa

Prototypical apps: Kinect-based emotion
detection

Prototype evaluation

Prototypical apps: Android-based emotion
self reporter

Prototype:
Monitoring
services

BCIs for UX
http://www.gadgetify.com/wp-content/uploads/2014/09/15/epoc.png

Experimenting with EPOC+ (video)

What about sentiment?

Sentiment analysis
• By definition,
The use of NLP and computational linguistics to identify attitudes
in speakers or writers with respect to specific subjects
http://datameer2.datameer.com/blog/wp-content/uploads/2012/10/22nd.png

Applications of sentiment analysis
Detecting / studying:
• Responses to marketing/political campaigns
• User satisfaction
• Bias in news
• Appropriateness of contents for different audiences

Some techniques
• Mostly heuristic methods
• Social media analysis
• Keyword-based
• Semantic approaches
• Annotated datasets
• Support Vector Machines (SVMs)
for classification

Expression Architecture
A flexible architecture enables Expression to listen from any data
source and have an independent component-based front-end.
Expression – Sentiment Analysis Platform at UDLAP
Architecture
[Cervantes, Gutiérrez, Gutiérrez, Castillo, Sánchez, Wan 2015]

The front end of Expression is based on sentiments cards
that represent topics of interest.
Sentiment cards
https://github.com/Innova4D/expression

A Sentiment Card graphically summarizes opinion
trends in social media
Sentiment Card

Some challenges
• Language subtleties – Irony, sarcasm, abbreviations, misspellings,
idioms
• Representativeness – Dissatisfied users tweet more actively
• Context dependent semantics
• Spam
• Classification beyond positive or negative

Affect mediation

Mediation: Conveying affect through ICTs
• Videoconferencing
• Immersive environments
• Asynchronous communication
• Email, forums, blogs…

Some issues and challenges
• Audio and video quality over
Internet
• Quality of multimedia experience
(http://i-guide.org/)
• Metrics, assessment methods
• Gaze awareness
• 3D modeling
• Issues shared with affect synthesis

Emotion over low data rates:
Sending codes instead of full video
In collaboration with University of Agder, Norway
1. Sensing
2. Encoding
3. Transmitting
4. Reconstructing
[Lamei, Li, Sánchez 2014]

Conveying mood

Instant messaging
• Emoticons work for emotions, instant reactions associated to specific
statements
• Mood is longer lasting, overarching
• Identifying mood can be a challenge, even for the person in that affective
state

Russell’s Circumplex Model can help identify
mood
• Two-dimensional, valence-
arousal model
• Consistent spatial location
in self-report experiments
across individuals and
across cultures/languages
[Russell 1983]

Extending Russell’s model - Intensity

Extended, discrete model
User testing showed
accuracy in affect identification
But using the model directly
would result in a cluttered
interface

Gradual revelation
annoyed happy
sad calm
Representative affectives states per quadrant (according to user testing)
sleepy
aroused
upset satisfied

Emoticon design

Prototype (“Russkman”)

Prototype

Evaluation
• Chat sessions among 2-3 subjects
• Hypothetical situations (upcoming exams, good grades, recent party,
lost wallet …)
• Subjects were able to specify their mood rapidly
• Subjects were able to identify mood of other users (high correlation)
• Persistent emoticons helped in keeping awareness of moods

Affect synthesis:
Modeling and displaying
emotion and mood

Synthesizing emotions
• If an emotion (or mood or sentiment) is recognized in the user, the
computer (or an adaptive environment) could (should?) respond
accordingly
• Approaches to emotion synthesis
• Reasoning about emotions—given a set of conditions, produce specific
reactions
• Characters with personality (temperament, sentiment)

Emotion synthesis
Characters with animated facial expressions
(facial cues for sensing also apply here)
[CSLU]

Responding to emotion
Affective Companion
Text from automatic speech recognizer and EmoVoice(for the user)
Natural language understanding [Cavazza et al. 2010]

FACS applied to emotion synthesis
FACSGen[Roeschetal.2010]

Affective animated avatars
• Design using
popular tools and a
reduced set of
FACS’ AUs
• Available at
http://chopis.com
• Testing in México
and China

Some challenges
• Prototypical vs. non-prototypical emotions
• Pure vs. combined emotions
• Integrating facial, posture and voice expressions
• Offering service platforms for app construction

Perspectives of affective UX
• Affect is key in UX design and evaluation
• Opportunities for research and development to improve detection,
synthesis and mediation
• Work needed in mood and sentiment analysis
• Opportunities for inter-disciplinary collaboration

Affective computing becoming mainstream?
“Pepper”, SoftBank’s affect-aware robot,
1,000 units sold out, December 2015, for about USD 1,800…
Involves facial recognition, video and audio processing, sensors, machine learning,…

But wait…
• Forward-looking statements are by their nature subject to various
risks and uncertainties, including, without limitation, a decline in
general economic conditions, general market conditions,
technological developments, changes in customer demand for
products and services, increased competition, risks associated with
international operations, and other important factors, each of which
may cause actual results and future developments to differ materially
from those expressed or implied in any forward-looking statement.
[…] Use of or reliance on the information in this presentation is at
your own risk.
http://webcast.softbankmobile.co.jp/en/conference/20140605/index.html

01WORLD IA DAY 2016 PRESENTATION TITLE HERE
HEADER OPTION
SUB HEAD OR SHORT DESCRIPTION
Some kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.
Some kind of illustration or image?
Affective UXChallenges in UX design involving affective computing
Alfredo Sánchez
Universidad de las Américas Puebla
http://ict.udlap.mx

References
• Cervantes, O., Gutiérrez, F., Gutiérrez, E., Castillo E., Sánchez, J. A., Wan, W. 2015. Expression:
Understanding affective content in social media. 7th Latin American Conference on Human-
computer Interaction (CLIHC 2015), Cordoba, Argentina, Nov. 18-23. Available from the ACM
Digital Library (http://dl.acm.org/citation.cfm?doid=2824893.2824903).
• Magallanes, Y., Molina, A., Méndez, Y., Sánchez, J. A. 2012. Combining gestures and graphical
elements for collaboration using multitouch surfaces. Proceedings of the 22nd International
Conference on Electronics, Communications and Computers (CONIELECOMP 2012, Feb.). IEEE
Computer Society Press, 173-178.
• Magallanes, Y., Molina, A., Méndez, Y., Sánchez, J. A. 2012. Towards an emotional validation of
heuristic approaches for usability evaluation. Acta Universitaria 22, 3, 119-125.
(http://www.actauniversitaria.ugto.mx/index.php/acta/article/view/351).
• Méndez, Y. A., Collazos, C. A., Sánchez, J. A., Merchán Paredes, L. 2012. Criterios para la
evaluación de superficies multitáctiles y sus aplicaciones. Lámpsakos 7 (Jan.), 64-68. (disponible
en http://www.funlam.edu.co/revistas/index.php/lampsakos/article/view/847).
• Méndez, Y. A. 2015. Marco de referencia para la captura semi-invasiva y análisis de las emociones
en el proceso de evaluación con usuarios de sistemas interactivos. Tesis doctoral. Universidad del
Cauca, Colombia. Disponible en
http://www.unicauca.edu.co/doctoradoce/publicaciones/Monografia_Mendez.pdf

References
• Sánchez, J. A., Hernández, N. P., Penagos, J. C., Ostróvskaya, Y. 2006. Conveying mood and
emotion in instant messaging by using a two-dimensional model for affective states.
Proceedings of the Symposium on Human Factors in Computer Systems (IHC 2006, Natal,
Brazil), 66-72. Available from the ACM Digital Library
(http://dl.acm.org/citation.cfm?id=1298033&CFID=582956970&CFTOKEN=61490837).
• Méndez, Y. A., Collazos, C. A., Granollers, T., Sánchez, J. A. 2014. Evaluating interactive
systems from an emotional perspective. Revista Científica Guillermo de Ockham 12(1),
43-49. (disponible en http://www.redalyc.org/pdf/1053/105332478005.pdf).
• Sánchez, J. A., Cortés, X., Starostenko, O., Cervantes, O., Wan, W. 2015. An Extensible
Platform for Seamless Integration and Management of Applications for Emotion Sensing
and Interpretation. Journal of Ambient Intelligence and Smart Environments 7(1), 5-19.
(available from http://iospress.metapress.com/content/y1g64422t78107v7).
• Starostenko, O., Cortés, X., Sánchez, J. A. 2015. Unobtrusive emotion sensing and
interpretation in smart environments. Journal of Ambient Intelligence and Smart
Environments 7, 59-83. (avalilable from
http://iospress.metapress.com/content/w2681p2827l2u22p/).

Affective UX: Challenges in UX involving affective computing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Affective UX: Challenges in UX involving affective computing

Similar to Affective UX: Challenges in UX involving affective computing (20)

More from Alfredo Sánchez

More from Alfredo Sánchez (10)

Recently uploaded

Recently uploaded (20)

Affective UX: Challenges in UX involving affective computing