WIAD 2016 Mexico City - UX afectiva: Retos del diseño de la experiencia del usuario con cómputo afectivo

2/21/16
1
01
WORLD IA DAY 2016
PRESENTATION TITLE HERE
HEADER OPTION
SUB HEAD OR SHORT DESCRIPTION!
Some kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.
Some kind of illustration or image?
Affective UX"Challenges in UX design involving affective computing

Alfredo Sánchez
Universidad de las Américas Puebla
© 2006-2014, Alfredo Sánchez
h8p://www.udlap.mx
Architecturing happiness
How do we know users are happy?

Can interacIve systems detect happiness (or the lack thereof)?

Can interacIve systems intenIonally produce happinesss?

Can technology at least not get in the way?
Agenda
•  Terms of affecIon
•  AffecIve compuIng
•  DetecIon
•  Synthesis
•  MediaIon
•  Affect in UX
•  Recent development
•  Open issues
Terms of affection
Emotion
A reacIon experienced usually toward a specific object and typically
accompanied by physiological and behavioral changes in the body

2/21/16
2
Mood
•  A conscious state of mind (typically lasIng longer than an emoIon)
joy depression
Sentiment
•  An aYtude or opinion – a state that results from a8ributes assigned
to objects, which are grouped into categories
The affect spectrum
emoIon
mood
senIment
may trigger
may prompt
Duration( - ) ( + )
Affective computing
(Computer sytems) related to,
arising from, or deliberately
influencing affecIve phenomena

-  MIT Affec)ve Compu)ng Group

AffecIve CompuIng
Rosalind Picard
1995
Affective computing (for engineers) J
The study and development of systems and devices that can
recognize, interpret, process, simulate and mediate human affects.

Why affective computing?

2/21/16
3
I choose
not to feel
h8p://animala.mx/noIcias/2015/02/27/adios-mr-spock/
Why Affective Computing
•  “Reason is, and ought to be,
only the slave of the passions”
David Hume, 1711-1776

•  Without passions we would lack
all moIvaIon, all impulse or
drive to act, or even to reason
•  A love/hate relaIonship with the computer and ITs in general
h8ps://www.youtube.com/watch?v=ZcZBTB-dkqQ
Why affective computing? (for engineers) J
•  Studies by many researchers show a vital role of emoIon
in many processes:
•  PercepIon
•  Decision making
•  CreaIvity
•  Comprehension
•  Memory
•  Social interacIon
h8p://www.bintegra.com/wp-content/uploads/2012/05/Fotolia_socila_users_S.jpg
Some application areas
•  EducaIon – intelligent tutoring
•  Health – monitoring paIents
•  Safety – monitoring drivers / pilots
•  Security – detecIng lies
•  E-commerce – aYtudes towards
products or services
•  Video games – producing engaging
environments, enriching UX
•  User studies – incorporaIng
emoIon into evaluaIon
Why affective computing (for our community)
EmoIon is the way
we communicate our
experience

2/21/16
4
Main aspects of affective computing
perception
synthesis
mediation
(of emoIons, moods and senIments)
Affect detection
Emotion detection: From facial cues
h8p://npx.auIsmspot.com/facial_expression.jpg
Emotion detection: From postural cues
h8p://instridebodywork.com/wp-content/uploads/2013/05/postures-of-emoIon1.jpg
Emotion detection: From voice cues
h8p://pandodaily.ﬁles.wordpress.com/2013/07/whisper-in-ear.jpg?w=584&h=329
h8p://gigaom2.ﬁles.wordpress.com/2013/07/emoIon-speech.jpg?w=300&h=200&crop=1
Emotion detection: From usage patterns
h8p://web.deepnetsecurity.com/products2/typesense.asp
Keystroke dynamics NavigaIon pa8erns Eye tracking

2/21/16
5
Emotion detection: From brain activity
Emo)v EPOC sensor
Honda BCI system
Emotion detection: From sensors of
autonomic activity
•  Skin conductance increases even
with a li8le surprise or anxiety
•  Blood volume/pressure
increases with anger or stress
•  Respiratory rhythm varies with
fear, sadness, happiness,
anxiety, fear

•  Muscle tension increases with
surprise and anxiety
Aﬀec)va sensor
NeXus sensor
Emotion detection: From sensors of
autonomic activity
“Know your inner self”
W/Me
“Hack happiness”
Feel (by SenIo)
Some general techniques to
support affective computing
For detecIon, synthesis and mediaIon
Facial Action Coding System (FACS)
h8p://www.theroadtosiliconvalley.com/tag/Im-roth/
FACS
•  Facial AcIon Coding System (FACS), very popular
•  Deﬁnes AcIon Units (AUs),
contracIon or relaxaIon of one or more face muscles
…

2/21/16
6
FACS (for “universal” emotions)
h8p://www.paulekman.com/facs/
Posture, motion and gesture analysis
Posture analysis
h8p://people.ict.usc.edu/~gratch/CSCI534/ACII-Handbook-GestureSyn.pdf
Posture and motion analysis
•  E.g. arm movements and emoIon:
Gesture analysis
MPEG-4
•  MPEG-4 (ISO144996) deﬁnes
•  86 Face AnimaIon Parameters (FAPs)
•  196 Body AnimaIon Parameters (BAPs)

2/21/16
7
Some existing tools
•  FaceReader, SHORE, eMoIon, RealEyes
Some challenges
•  Prototypical vs. non-prototypical emoIons
•  Pure vs. combined emoIons
•  MulIple subjects
•  Partly covered faces / bodies
•  Tracking face or body movement
•  LighIng condiIons
•  IntegraIng techniques
•  SupporIng applicaIon development
Emotion in UX evaluation
Evaluating user experience with novel
technologies – impact of emotion
© 2006-2014, Alfredo Sánchez [Magallanes, Molina, Méndez & Sánchez 2012]
Gestures + Contextual Menus
Heuristic evaluation
•  Problems observed in actual use
•  Issues idenIﬁed by experts
ID Issue Occurrences
P1 Movement of objects using two
fingers. 13
P2 Size of the keyboard. 8
P3 Finding help for the user. 5
P4 Finding the toolbar associated
with each object. 4
P5 Navigating using the fan menu. 3
P6 Navigating between nodes.
3
P7 Identifying the icon for linking to
new pages 2
P8 Incomplete or inaccessible
menus.
1
ID Task Issue
1 Insert two images, change their
position and size. P1
2 Enter a text next to the images as
a caption. P2
3 Obtain the toolbar for an image.
P3, P4
4 Pop a menu and discover how it
works. P5
5 Build a hierarchy of linked
workspaces. P6, P7
6 Get the menu in different
positions. Cause the menu to go
off the screen.
P8
7 Insert two images, change their
position and size. P1
8 Enter a text next to the images as
a caption. P2

2/21/16
8
Additional feedback on emotions, based on
Product Emotion Measurement Tool (PrEmo)
Disgust Indignation Contempt Dissatisfaction Disappointment Unpleasant
surprise
boredom

Desire Inspiration Admiration Satisfaction Fascination Pleasant
surprise
Amusement

[Desmet 2002]
ParIcipants were asked to select an animated representa:on in correspondence with their emoIonal reacIons
[In collaboraIon with Universidad del Cauca, Colombia]
Positive emotions in spite of problems!
!
! !
PosiIve emoIons associated with novelty?
NegaIve emoIons associated with known
interface elements? [Magallanes, Molina, Méndez & Sánchez 2012]
Implications
•  Techniques for evaluaIon of usability and user experience are
needed that take aﬀect into account
•  Framework developed at UniCauca, Colombia
[Méndez, Collazos, Granollers, Sánchez 2014; Méndez 2015]
EmoVal
•  A framework for emoIonal evaluaIon designed to enrich exisIng UX
evaluaIon methods
•  Three major components
•  EmoVal-C: Behavioral evalua:on (facial expressions, gaze, posture,
movements)
•  EmoVal-S: Self-report of emo:ons while using a system
•  EmoVal-F: Physiological sensing, mainly brain acIvity during system usage
•  SupporIng website (
h8p://artemisa.unicauca.edu.co/~ymendal/EmoVal)
[Méndez 2015]
EmoVal: Detailed support / facial
expressions
[Méndez 2015]
EmoVal: Detailed support / posture and
movement
© 2006-2014, Alfredo Sánchez [Méndez 2015]

2/21/16
9
Building capabilities for
affect detection
Current scenario: Extending an app to include more
techniques is not straightforward
Facial&cue&analysis&Monitoring&
pilot&alertness&
Lie&detec4on&
Usability&
tes4ng&
Online&tutoring&
Posture&analysis&
Voice&analysis&
Keystroke&
dynamics&
Eye&tracking&
Naviga4on&
pa@erns&
ApplicaIon domain Technique Technologies
Goal: Integration platform
Usability)
tes+ng)
Eye)tracking)
Vikara)
Integra+on)
Services)
Vikara)
Monitoring)
Services)
Lie)
detec+on)
Online)
tutoring)
Alertness)
monitor)
Naviga+on)
pa@erns)
Keystroke)
dynamics)
Voice)analysis)
Posture)
analysis)
Facial)cue)
analysis)
Visualiza+on)
interfaces)
[Sánchez et al. 2015]
Architecture for implementation
Database
Controller
EmotionsController
View
Data Model
Home
- Index
- UploadUserContent
HomeController
SessionsController
Vikara
Services
Interface
Data
entities
cshtml
Session
- Index
- List
EmotionModel
UserContentController
FileController
Emotion
UserCont
entSession
MS SQL
Resources
Google Chart
Tools
WebGrease
Modernizr
Android
Emotion
Interpreter
Windows
Emotion
Interpeter
Server
Visualization
interfaces
Other
Emotion
Interpreters
Observer /
administrator
Endusers
Simple data model and messaging
protocol
!
Prototypical apps: Kinect-based emotion
detection
14 sorpresa

2/21/16
10
Prototypical apps: Kinect-based emotion
detection
Prototype evaluation
!
Prototypical apps: Android-based
emotion self reporter
!
Prototype:
Monitoring
services
UX for BCI
h8p://www.gadgeIfy.com/wp-content/uploads/2014/09/15/epoc.png
Experimenting with EPOC+

2/21/16
11
What about sentiment?
Sentiment analysis
•  By definiIon,
The use of NLP and computaIonal linguisIcs to idenIfy aLtudes
in speakers or writers with respect to specific subjects
h8p://datameer2.datameer.com/blog/wp-content/uploads/2012/10/22nd.png
Applications of sentiment analysis
DetecIng / studying:

•  Responses to markeIng/poliIcal campaigns
•  User saIsfacIon
•  Bias in news
•  Appropriateness of contents for different audiences
Some techniques
•  Mostly heurisIc methods
•  Social media analysis
•  Keyword-based
•  SemanIc approaches
•  Annotated datasets
•  Support Vector Machines
(SVMs) for classificaIon
Expression Architecture
A flexible architecture enables Expression to listen from any data
source and have an independent component-based front-end.
Expression – SenIment Analysis Plazorm at UDLAP
Architecture
[Cervantes, GuIérrez, GuIérrez, CasIllo, Sánchez, Wan 2015]

2/21/16
12
The front end of Expression is based on sen:ments cards
that represent topics of interest.
SenIment cards
h8ps://github.com/Innova4D/expression
A Sen:ment Card graphically summarizes opinion
trends in social media
SenIment Card
Some challenges
•  Language subtleIes – Irony, sarcasm, abbreviaIons, misspellings,
idioms
•  RepresentaIveness – DissaIsfied users tweet more acIvely
•  Context dependent semanIcs
•  Spam
•  ClassificaIon beyond posiIve or negaIve
Affect mediation
Mediation: Conveying affect through ICTs
•  Videoconferencing
•  Immersive environments
•  Asynchronous communicaIon
•  Email, forums, blogs…
Some issues and challenges
•  Audio and video quality over
Internet
•  Quality of mulImedia experience
(h8p://i-guide.org/)
•  Metrics, assessment methods
•  Gaze awareness
•  3D modeling
•  Issues shared with affect synthesis

2/21/16
13
Emotion over low data rates:
Sending codes instead of full video
In collaboraIon with University of Agder, Norway
h8p://w
ww.paulekm
an.com/facs/
1.  Sensing
2.  Encoding
3.  TransmiYng
4.  ReconstrucIng
[Lamei, Li, Sánchez 2014]
Conveying mood
Instant messaging
•  EmoIcons work for emoIons, instant reacIons associated to specific
statements
•  Mood is longer las:ng, overarching
•  IdenIfying mood can be a challenge, even for the person in that affecIve
state
Russell’s Circumplex Model can help
identify mood
•  Two-dimensional, valence-
arousal model
•  Consistent spaIal locaIon
in self-report experiments
across individuals and
across cultures/languages
[Russell 1983]
Extending Russell’s model - Intensity
Extended, discrete model
User tesIng showed
accuracy in affect idenIficaIon

But using the model directly
would result in a clu8ered
interface

2/21/16
14
Gradual revelation
annoyed happy
sad calm
Representative affectives states per quadrant (according to user testing)
sleepy
aroused
upset satisfied
Emoticon design
Prototype (“Russkman”)
Prototype
Prototype
Evaluation
•  Chat sessions among 2-3 subjects
•  HypotheIcal situaIons (upcoming exams, good grades, recent party,
lost wallet …)
•  Subjects were able to specify their mood rapidly
•  Subjects were able to idenIfy mood of other users (high correlaIon)
•  Persistent emoIcons helped in keeping awareness of moods

2/21/16
15
Affect synthesis:
Modeling and displaying
emotion and mood
Synthesizing emotions
•  If an emoIon (or mood or senIment) is recognized in the user, the
computer (or an adapIve environment) could (should?) respond
accordingly
•  Approaches to emoIon synthesis
•  Reasoning about emoIons—given a set of condiIons, produce speciﬁc
reacIons
•  Characters with personality (temperament, senIment)
Emotion synthesis
Characters with animated facial expressions
(facial cues for sensing also apply here)
[CSLU]
Responding to emotion
AﬀecIve Companion
Text from automaIc speech recognizer and EmoVoice(for the user)
Natural language understanding [Cavazza et al. 2010]
FACS applied to emotion synthesis
FACSGen [Roesch et al. 2010]
Affective animated avatars
•  Design using
popular tools and a
reduced set of
FACS’ AUs
•  Available at
h8p://chopis.com
•  TesIng in México
and China

2/21/16
16
Some challenges
•  Prototypical vs. non-prototypical emoIons
•  Pure vs. combined emoIons
•  IntegraIng facial, posture and voice expressions
•  Offering service plazorms for app construcIon
Perspectives of affective UX
•  Affect is key in UX design and evaluaIon
•  OpportuniIes for research and development to improve detecIon,
synthesis and mediaIon
•  Work needed in mood and senIment analysis
•  OpportuniIes for inter-disciplinary collaboraIon
Affective computing becoming
mainstream?
“Pepper”, So}Bank’s affect-aware robot,
1,000 units sold out, December 2015, for about USD 1,800…
Involves facial recogniIon, video and audio processing, sensors, machine learning,…
But wait…
•  Forward-looking statements are by their nature subject to various
risks and uncertain:es, including, without limitaIon, a decline in
general economic condiIons, general market condiIons,
technological developments, changes in customer demand for
products and services, increased compeIIon, risks associated with
internaIonal operaIons, and other important factors, each of which
may cause actual results and future developments to differ materially
from those expressed or implied in any forward-looking statement.
[…] Use of or reliance on the informa:on in this presenta:on is at
your own risk.
h8p://webcast.so}bankmobile.co.jp/en/conference/20140605/index.html
01
WORLD IA DAY 2016
PRESENTATION TITLE HERE
HEADER OPTION
SUB HEAD OR SHORT DESCRIPTION!
Some kind of explanatory text, reference or footnote can go here and wrap to two lines, if needed.
Some kind of illustration or image?
Affective UX"Challenges in UX design involving affective computing

Alfredo Sánchez
Universidad de las Américas Puebla
http://ict.udlap.mx
References
•  Cervantes, O., GuIérrez, F., GuIérrez, E., CasIllo E., Sánchez, J. A., Wan, W. 2015. Expression:
Understanding affecIve content in social media. 7th LaIn American Conference on Human-
computer InteracIon (CLIHC 2015), Cordoba, ArgenIna, Nov. 18-23. Available from the ACM
Digital Library (h8p://dl.acm.org/citaIon.cfm?doid=2824893.2824903).
•  Magallanes, Y., Molina, A., Méndez, Y., Sánchez, J. A. 2012. Combining gestures and graphical
elements for collaboraIon using mulItouch surfaces. Proceedings of the 22nd InternaIonal
Conference on Electronics, CommunicaIons and Computers (CONIELECOMP 2012, Feb.). IEEE
Computer Society Press, 173-178.
•  Magallanes, Y., Molina, A., Méndez, Y., Sánchez, J. A. 2012. Towards an emoIonal validaIon of
heurisIc approaches for usability evaluaIon. Acta Universitaria 22, 3, 119-125. (
h8p://www.actauniversitaria.ugto.mx/index.php/acta/arIcle/view/351).
•  Méndez, Y. A., Collazos, C. A., Sánchez, J. A., Merchán Paredes, L. 2012. Criterios para la
evaluación de superficies mulItácIles y sus aplicaciones. Lámpsakos 7 (Jan.), 64-68. (disponible
en h8p://www.funlam.edu.co/revistas/index.php/lampsakos/arIcle/view/847).
•  Méndez, Y. A. 2015. Marco de referencia para la captura semi-invasiva y análisis de las emociones
en el proceso de evaluación con usuarios de sistemas interacIvos. Tesis doctoral. Universidad del
Cauca, Colombia. Disponible en h8p://www.unicauca.edu.co/doctoradoce/publicaciones/
Monografia_Mendez.pdf

2/21/16
17
References
•  Sánchez, J. A., Hernández, N. P., Penagos, J. C., Ostróvskaya, Y. 2006. Conveying mood
and emoIon in instant messaging by using a two-dimensional model for aﬀecIve states.
Proceedings of the Symposium on Human Factors in Computer Systems (IHC 2006, Natal,
Brazil), 66-72. Available from the ACM Digital Library (
h8p://dl.acm.org/citaIon.cfm?id=1298033&CFID=582956970&CFTOKEN=61490837).
•  Méndez, Y. A., Collazos, C. A., Granollers, T., Sánchez, J. A. 2014. EvaluaIng interacIve
systems from an emoIonal perspecIve. Revista Cien•ﬁca Guillermo de Ockham 12(1),
43-49. (disponible en h8p://www.redalyc.org/pdf/1053/105332478005.pdf).
•  Sánchez, J. A., Cortés, X., Starostenko, O., Cervantes, O., Wan, W. 2015. An Extensible
Plazorm for Seamless IntegraIon and Management of ApplicaIons for EmoIon Sensing
and InterpretaIon. Journal of Ambient Intelligence and Smart Environments 7(1), 5-19.
(available from h8p://iospress.metapress.com/content/y1g64422t78107v7).
•  Starostenko, O., Cortés, X., Sánchez, J. A. 2015. Unobtrusive emoIon sensing and
interpretaIon in smart environments. Journal of Ambient Intelligence and Smart
Environments 7, 59-83. (avalilable from
h8p://iospress.metapress.com/content/w2681p2827l2u22p/).

WIAD 2016 Mexico City - UX afectiva: Retos del diseño de la experiencia del usuario con cómputo afectivo

Recommended

Recommended

More Related Content

Similar to WIAD 2016 Mexico City - UX afectiva: Retos del diseño de la experiencia del usuario con cómputo afectivo

Similar to WIAD 2016 Mexico City - UX afectiva: Retos del diseño de la experiencia del usuario con cómputo afectivo (20)

Recently uploaded

Recently uploaded (20)

WIAD 2016 Mexico City - UX afectiva: Retos del diseño de la experiencia del usuario con cómputo afectivo