This document summarizes the work of Elisabeth André and the Human-Centered Multimedia lab on developing socially sensitive interfaces. The lab focuses on human-computer interaction, social signal processing, and building embodied conversational agents and social robots. Their work aims to enrich computer interfaces with human abilities like nonverbal communication. While social signal processing research has grown, applications have not translated well. The challenges include dealing with noisy real-world data, non-prototypical behaviors, and multimodal fusion. The lab works on analyzing social signals, generating expressive behaviors in virtual agents and robots, and applications like training presentation skills. Current work focuses on mobile social signal processing.
Affective Computing by Michal Shmueli-Scheuer, PhD, IBM Haifa Research Lab presented on Cognitive Systems Institute Group Speaker Series call 28 May 2015.
Emotion-oriented computing: Possible uses and applicationsAndré Valdestilhas
This article discusses the concepts of using digital television affective computing and computer vision.
The proposal involves the union of some techniques such as capturing facial expressions through a video
camera, use of accelerometers in ball and touch holograms to work a certain level of interactivity with the
viewer. Some uses of the proposal in question are described, such as control of the hearing, background
content, among others. This article reveals numerous benefits that can be addressed with the use of
matters presented which can be applied in a broad context, such as for the blind in video games, among
others
Project NEMO - towards empathy-enabling digital environmentsKatri Saarikivi
A presentation on the interdisciplinary research project NEMO (Natural Emotions in digital interaction) that won the science-based idea competition arranged by the University of Helsinki, the Helsinki Challenge, in 2015. Presented at the Presented at the Digital International Collaboration in Education seminar arranged by the Finnish Ministry of Education and Culture on May 26th, 2016.
Affective Computing by Michal Shmueli-Scheuer, PhD, IBM Haifa Research Lab presented on Cognitive Systems Institute Group Speaker Series call 28 May 2015.
Emotion-oriented computing: Possible uses and applicationsAndré Valdestilhas
This article discusses the concepts of using digital television affective computing and computer vision.
The proposal involves the union of some techniques such as capturing facial expressions through a video
camera, use of accelerometers in ball and touch holograms to work a certain level of interactivity with the
viewer. Some uses of the proposal in question are described, such as control of the hearing, background
content, among others. This article reveals numerous benefits that can be addressed with the use of
matters presented which can be applied in a broad context, such as for the blind in video games, among
others
Project NEMO - towards empathy-enabling digital environmentsKatri Saarikivi
A presentation on the interdisciplinary research project NEMO (Natural Emotions in digital interaction) that won the science-based idea competition arranged by the University of Helsinki, the Helsinki Challenge, in 2015. Presented at the Presented at the Digital International Collaboration in Education seminar arranged by the Finnish Ministry of Education and Culture on May 26th, 2016.
Pathways from the Laboratory to Real World ProductsMiniFAB
After 20 years of incubation, Lab-on-a-chip technologies are ready to realize their potential. Critical to achieving this goal is their ability to attract investment. Traditionally there has been a disproportionate emphasis on prototypes in the quest for investment. On too many occasions this disparate focus has created a vicious cycle of expensive and lengthy failures, leaving exciting technologies and ideas in the 'valley of death'.
Understanding user requirements, adopting a design-for-manufacture approach and having a clear idea of what your end-goal looks like can eliminate the dangers of the 'valley of death'.
Another crucial strategy is implementing a staged development strategy that segments the project into manageable, investible and technically achievable stages, concentrating on risk-based developments as opposed to agile ones.
For the investment community, processes such as these represent a mature and disciplined approach to translating lab-on-a-chip technologies from the bench top, into real-world products that address new commercial opportunities.
MiniFAB CEO Dr Erol Harvey presented this talk at Lab-on-a-Chip Asia in Singapore, November 2014.
Past, present, and future of Recommender Systems: an industry perspectiveXavier Amatriain
Keynote for the ACM Intelligent User Interface conference in 2016 in Sonoma, CA. I start with the past by talking about the Recommender Problem, and the Netflix Prize. Then I go into the Present and the Future by talking about approaches that go beyond rating prediction and ranking and by finishing with some of the most important lessons learned over the years. Throughout my talk I put special emphasis on the relation between algorithms and the User Interface.
Products are Services, how ubiquitous computing changes designMike Kuniavsky
As more products, from tablets to bathroom scales to washing machines go online, our relationship to them changes. We start to think of them as representatives of online services, and to think of services as represented by products. Ubiquitous computing changes our understanding of where the boundaries of a hardware product and a service stop, and fundamentally challenges how we design both.
Internet of Things (IoT) will enable dramatic society transformation. This seminar presents an introduction to the IoT and explains why IoT Security is important.
Then it presents security issues in wireless sensor networks that constitute a main ingredient of IoT.
Seminar given at Centre Tecnològic de Telecomunicacions de Catalunya (CTTC) on 28 January 2015.
Who talks to whom? What communication channels do they use and why? What emotions are involved? Summer School on Software Engineering. Oct 9, 2018. Oulu, Finland.
Facial Expression Recognition System: A Digital Printing Applicationijceronline
International Journal of Computational Engineering Research (IJCER) is dedicated to protecting personal information and will make every reasonable effort to handle collected information appropriately. All information collected, as well as related requests, will be handled as carefully and efficiently as possible in accordance with IJCER standards for integrity and objectivity.
Facial Expression Recognition System: A Digital Printing Applicationijceronline
International Journal of Computational Engineering Research (IJCER) is dedicated to protecting personal information and will make every reasonable effort to handle collected information appropriately. All information collected, as well as related requests, will be handled as carefully and efficiently as possible in accordance with IJCER standards for integrity and objectivity.
Hybrid Facial Expression Recognition (FER2013) Model for Real-Time Emotion Cl...BIJIAM Journal
Facial expression recognition is a vital research topic in most fields ranging from artificial intelligence and gaming to human–computer interaction (HCI) and psychology. This paper proposes a hybrid model for facial expression recognition, which comprises a deep convolutional neural network (DCNN) and a Haar Cascade deep learning architecture. The objective is to classify real-time and digital facial images into one of the seven facial emotion categories considered. The DCNN employed in this research has more convolutional layers, ReLU activation functions, and multiple kernels to enhance filtering depth and facial feature extraction. In addition, a Haar Cascade model was also mutually used to detect facial features in real-time images and video frames. Grayscale images from the Kaggle repository (FER-2013) and then exploited graphics processing unit (GPU) computation to expedite the training and validation process. Pre-processing and data augmentation techniques are applied to improve training efficiency and classification performance. The experimental results show a significantly improved classification performance compared to state-of-the-art (SoTA) experiments and research. Also, compared to other conventional models, this paper validates that the proposed architecture is superior in classification performance with an improvement of up to 6%, totaling up to 70% accuracy, and with less execution time of 2,098.8 s.
From NLP to NLU: Why we need varied, comprehensive, and stratified knowledge,...Amit Sheth
#ChatGPT #LLM #DistributionalSemantics are hot topics--both for their successes and failures/shortcomings. In Prof. Amit Sheth's keynote he delivered yesterday at #knowledgeNLP2023 -"From #NLP to #NLU: Why we need varied, comprehensive, and #StratifiedKnowledge, and how to use it for neuro-symbolic AI", he discusses several categories of deficiencies, and more importantly, how to address them using Knowledge-infused #neurosymbolicAI. #WorldModel #RealWorldSemantics Slides: http://bit.ly/kNLP2023 Abstract: https://lnkd.in/grzi5UyJ.
Photos: https://lnkd.in/gS_KRFvQ
Pathways from the Laboratory to Real World ProductsMiniFAB
After 20 years of incubation, Lab-on-a-chip technologies are ready to realize their potential. Critical to achieving this goal is their ability to attract investment. Traditionally there has been a disproportionate emphasis on prototypes in the quest for investment. On too many occasions this disparate focus has created a vicious cycle of expensive and lengthy failures, leaving exciting technologies and ideas in the 'valley of death'.
Understanding user requirements, adopting a design-for-manufacture approach and having a clear idea of what your end-goal looks like can eliminate the dangers of the 'valley of death'.
Another crucial strategy is implementing a staged development strategy that segments the project into manageable, investible and technically achievable stages, concentrating on risk-based developments as opposed to agile ones.
For the investment community, processes such as these represent a mature and disciplined approach to translating lab-on-a-chip technologies from the bench top, into real-world products that address new commercial opportunities.
MiniFAB CEO Dr Erol Harvey presented this talk at Lab-on-a-Chip Asia in Singapore, November 2014.
Past, present, and future of Recommender Systems: an industry perspectiveXavier Amatriain
Keynote for the ACM Intelligent User Interface conference in 2016 in Sonoma, CA. I start with the past by talking about the Recommender Problem, and the Netflix Prize. Then I go into the Present and the Future by talking about approaches that go beyond rating prediction and ranking and by finishing with some of the most important lessons learned over the years. Throughout my talk I put special emphasis on the relation between algorithms and the User Interface.
Products are Services, how ubiquitous computing changes designMike Kuniavsky
As more products, from tablets to bathroom scales to washing machines go online, our relationship to them changes. We start to think of them as representatives of online services, and to think of services as represented by products. Ubiquitous computing changes our understanding of where the boundaries of a hardware product and a service stop, and fundamentally challenges how we design both.
Internet of Things (IoT) will enable dramatic society transformation. This seminar presents an introduction to the IoT and explains why IoT Security is important.
Then it presents security issues in wireless sensor networks that constitute a main ingredient of IoT.
Seminar given at Centre Tecnològic de Telecomunicacions de Catalunya (CTTC) on 28 January 2015.
Who talks to whom? What communication channels do they use and why? What emotions are involved? Summer School on Software Engineering. Oct 9, 2018. Oulu, Finland.
Facial Expression Recognition System: A Digital Printing Applicationijceronline
International Journal of Computational Engineering Research (IJCER) is dedicated to protecting personal information and will make every reasonable effort to handle collected information appropriately. All information collected, as well as related requests, will be handled as carefully and efficiently as possible in accordance with IJCER standards for integrity and objectivity.
Facial Expression Recognition System: A Digital Printing Applicationijceronline
International Journal of Computational Engineering Research (IJCER) is dedicated to protecting personal information and will make every reasonable effort to handle collected information appropriately. All information collected, as well as related requests, will be handled as carefully and efficiently as possible in accordance with IJCER standards for integrity and objectivity.
Hybrid Facial Expression Recognition (FER2013) Model for Real-Time Emotion Cl...BIJIAM Journal
Facial expression recognition is a vital research topic in most fields ranging from artificial intelligence and gaming to human–computer interaction (HCI) and psychology. This paper proposes a hybrid model for facial expression recognition, which comprises a deep convolutional neural network (DCNN) and a Haar Cascade deep learning architecture. The objective is to classify real-time and digital facial images into one of the seven facial emotion categories considered. The DCNN employed in this research has more convolutional layers, ReLU activation functions, and multiple kernels to enhance filtering depth and facial feature extraction. In addition, a Haar Cascade model was also mutually used to detect facial features in real-time images and video frames. Grayscale images from the Kaggle repository (FER-2013) and then exploited graphics processing unit (GPU) computation to expedite the training and validation process. Pre-processing and data augmentation techniques are applied to improve training efficiency and classification performance. The experimental results show a significantly improved classification performance compared to state-of-the-art (SoTA) experiments and research. Also, compared to other conventional models, this paper validates that the proposed architecture is superior in classification performance with an improvement of up to 6%, totaling up to 70% accuracy, and with less execution time of 2,098.8 s.
From NLP to NLU: Why we need varied, comprehensive, and stratified knowledge,...Amit Sheth
#ChatGPT #LLM #DistributionalSemantics are hot topics--both for their successes and failures/shortcomings. In Prof. Amit Sheth's keynote he delivered yesterday at #knowledgeNLP2023 -"From #NLP to #NLU: Why we need varied, comprehensive, and #StratifiedKnowledge, and how to use it for neuro-symbolic AI", he discusses several categories of deficiencies, and more importantly, how to address them using Knowledge-infused #neurosymbolicAI. #WorldModel #RealWorldSemantics Slides: http://bit.ly/kNLP2023 Abstract: https://lnkd.in/grzi5UyJ.
Photos: https://lnkd.in/gS_KRFvQ
This presentation was on Empathic Mixed Reality, which we applied Mixed Reality technology to Empathic Computing in our studies. We shared an overview of our research and selected findings. This talk was given at ETRI and KAIST in Daejeon, South Korea, on the 24th of May 2017.
Privacy-Preserving and Scalable Affect Detection in Online Synchronous Learning
written by Felix Böttger, Ufuk Cetinkaya, Daniele Di Mitri, Sebastian Gombert, Krist Shingjergji, Deniz Iren & Roland Klemke was accepted at the Seventeenth European Conference on Technology Enhanced Learning (EC-TEL 2022) Educating for a new future: Making sense of technology-enhanced learning adoption – Toulouse, France, 12-16 September 2022
The paper reports on a research prototype which stems from the cooperation between DIPF and the Open University of the Netherlands.
Abstract
The recent pandemic has forced most educational institutions to shift to distance learning. Teachers can perceive various non-verbal cues in face-to-face classrooms and thus notice when students are distracted, confused, or tired. However, the students’ non-verbal cues are not observable in online classrooms. The lack of these cues poses a challenge for the teachers and hinders them in giving adequate, timely feedback in online educational settings. This can lead to learners not receiving proper guidance and may cause them to be demotivated. This paper proposes a pragmatic approach to detecting student affect in online synchronized learning classrooms. Our approach consists of a method and a privacy-preserving prototype that only collects data that is absolutely necessary to compute action units and is highly scalable by design to run on multiple devices without specialized hardware. We evaluated our prototype using a benchmark for the system performance. Our results confirm the feasibility and the applicability of the proposed approach.
Cosmetic shop management system project report.pdfKamal Acharya
Buying new cosmetic products is difficult. It can even be scary for those who have sensitive skin and are prone to skin trouble. The information needed to alleviate this problem is on the back of each product, but it's thought to interpret those ingredient lists unless you have a background in chemistry.
Instead of buying and hoping for the best, we can use data science to help us predict which products may be good fits for us. It includes various function programs to do the above mentioned tasks.
Data file handling has been effectively used in the program.
The automated cosmetic shop management system should deal with the automation of general workflow and administration process of the shop. The main processes of the system focus on customer's request where the system is able to search the most appropriate products and deliver it to the customers. It should help the employees to quickly identify the list of cosmetic product that have reached the minimum quantity and also keep a track of expired date for each cosmetic product. It should help the employees to find the rack number in which the product is placed.It is also Faster and more efficient way.
Forklift Classes Overview by Intella PartsIntella Parts
Discover the different forklift classes and their specific applications. Learn how to choose the right forklift for your needs to ensure safety, efficiency, and compliance in your operations.
For more technical information, visit our website https://intellaparts.com
6th International Conference on Machine Learning & Applications (CMLA 2024)ClaraZara1
6th International Conference on Machine Learning & Applications (CMLA 2024) will provide an excellent international forum for sharing knowledge and results in theory, methodology and applications of on Machine Learning & Applications.
HEAP SORT ILLUSTRATED WITH HEAPIFY, BUILD HEAP FOR DYNAMIC ARRAYS.
Heap sort is a comparison-based sorting technique based on Binary Heap data structure. It is similar to the selection sort where we first find the minimum element and place the minimum element at the beginning. Repeat the same process for the remaining elements.
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...Amil Baba Dawood bangali
Contact with Dawood Bhai Just call on +92322-6382012 and we'll help you. We'll solve all your problems within 12 to 24 hours and with 101% guarantee and with astrology systematic. If you want to take any personal or professional advice then also you can call us on +92322-6382012 , ONLINE LOVE PROBLEM & Other all types of Daily Life Problem's.Then CALL or WHATSAPP us on +92322-6382012 and Get all these problems solutions here by Amil Baba DAWOOD BANGALI
#vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore#blackmagicformarriage #aamilbaba #kalajadu #kalailam #taweez #wazifaexpert #jadumantar #vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore #blackmagicforlove #blackmagicformarriage #aamilbaba #kalajadu #kalailam #taweez #wazifaexpert #jadumantar #vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore #Amilbabainuk #amilbabainspain #amilbabaindubai #Amilbabainnorway #amilbabainkrachi #amilbabainlahore #amilbabaingujranwalan #amilbabainislamabad
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)MdTanvirMahtab2
This presentation is about the working procedure of Shahjalal Fertilizer Company Limited (SFCL). A Govt. owned Company of Bangladesh Chemical Industries Corporation under Ministry of Industries.
Saudi Arabia stands as a titan in the global energy landscape, renowned for its abundant oil and gas resources. It's the largest exporter of petroleum and holds some of the world's most significant reserves. Let's delve into the top 10 oil and gas projects shaping Saudi Arabia's energy future in 2024.
NUMERICAL SIMULATIONS OF HEAT AND MASS TRANSFER IN CONDENSING HEAT EXCHANGERS...ssuser7dcef0
Power plants release a large amount of water vapor into the
atmosphere through the stack. The flue gas can be a potential
source for obtaining much needed cooling water for a power
plant. If a power plant could recover and reuse a portion of this
moisture, it could reduce its total cooling water intake
requirement. One of the most practical way to recover water
from flue gas is to use a condensing heat exchanger. The power
plant could also recover latent heat due to condensation as well
as sensible heat due to lowering the flue gas exit temperature.
Additionally, harmful acids released from the stack can be
reduced in a condensing heat exchanger by acid condensation. reduced in a condensing heat exchanger by acid condensation.
Condensation of vapors in flue gas is a complicated
phenomenon since heat and mass transfer of water vapor and
various acids simultaneously occur in the presence of noncondensable
gases such as nitrogen and oxygen. Design of a
condenser depends on the knowledge and understanding of the
heat and mass transfer processes. A computer program for
numerical simulations of water (H2O) and sulfuric acid (H2SO4)
condensation in a flue gas condensing heat exchanger was
developed using MATLAB. Governing equations based on
mass and energy balances for the system were derived to
predict variables such as flue gas exit temperature, cooling
water outlet temperature, mole fraction and condensation rates
of water and sulfuric acid vapors. The equations were solved
using an iterative solution technique with calculations of heat
and mass transfer coefficients and physical properties.
2. 2
Human-Centered Multimedia
Founded: April 2001
Chair: Elisabeth André
Research Topics:
Human-Computer Interaction
Social Signal Processing
Affective Computing
Embodied Conversational Agents
Social Robotics
3. 3
Motivation
There is another level in human communication, which is
just as important as the spoken message:
nonverbal communication
How can we enrich the precise and useful functions of
computers with the human’s ability to shape the meaning
of a message through nonverbal messages?
4. 4
Observation
Social signal processing has developed from a
side issue to a major area of research.
Undertaken effort has not translated well into
applications. Why is this?
1998 ………………. …………………….. 2005 2006 ……. 2009 .. 2011 2012 2013 2015
Special Session on Face and Gesture Recognition
Keynote „Honest Signals“
1st HCM Workshop
1/3 of Grand Challenge Papers on Affective Computing
3 Workshops on „Social Cues“Brave New Topic: Affective Multimodal HCI
ACM MM
5. 5
Challenge: Real-Life
Applications
Total of 434 publications on SSPNet
10% include term “real(-)time” and are related to detection
Only 2 % address multi-modal detection
Social Signal Processing in the Wild
90%
3%
2%
2%
1%0%2%
face (15) gesture (9) speech (9)
interaction (8) physiological (2) multimodal (13)
Meta Analysis by J. Wagner
6. 6
Organization of the Talk
Analysis of Emotional and Social Signals
Generation of Expressive Behaviors in Virtual Agents
and Robots
Applications of Socially Signal Processing and Embodied
Agents
Socially sensitive Robots
Training of Presentation Skills in
• Job Interviews
• Public Speaking
Providing Information on Social Context to Blind
People
7. 7
Challenge: Noisy and Corrupted
Data
We only rely on previously seen data.
We have to deal with noisy and corrupted data.
?
now
time
noise missing
8. 8
Challenge: Non-Prototypical
Behaviors
Previous research focused on the analysis of
prototypical samples in preferably pure form
In daily life, we also observe subtle, blended and
suppressed emotions, i.e. non-prototypical emotional
displays.
Pictures from Ekman and Friesen’s database of emotional faces
9. 9
Accuracy Drops with
Naturalness
Systems developed under laboratory conditions
often perform poorly in real-world scenarios
100%
80% 70%
Accuracy
Naturalness
Acted Read WOZ
10. 10
Contextualized Analysis
Improvement by context-sensitive analysis
Gender-specific information (Vogt & André 2006)
Success / failure of student in tutoring applications
(Conati & McLaren 2009)
Dialogue behavior of virtual agent / robot (Baur et al.
2014)
Learning context using (B)LSTM (Metallinou et al.
2014)
11. 11
Challenge: Multimodal Fusion
Meta study by D’Mello and Kory on multimodal
affect detection shows that improvement
correlates with naturalness of corpus:
>10% for acted and only <5% for natural data
In natural interaction people draw on a mixture
of strategies to express emotion leading to a
complementary rather than consistent display of
social behaviour
S.K. D'Mello, J.M. Kory: Consistent but modest: a
meta-analysis on unimodal and multimodal affect
detection accuracies from 30 studies. ICMI 2012: 31-38
12. 12
Event-Based Fusion
In case of contradictory cues, fusion methods
trust the “right” modality just as often as “wrong”
one
single modalities
fusion
techniques
sample
correct classification
incorrect classification
J. Wagner, E. André, F. Lingenfelser, J. Kim: Exploring Fusion Methods for
Multimodal Emotion Recognition with Missing Data. T. Affective Computing 2(4): 206-
218 (2011)
13. 13
Event-Based Fusion
Amount of misclassified samples significantly
higher when annotations mismatch
Yes 71%
No 29%
62% 36%
Agreement?
15. 15
Synchronous Fusion
Synchronous fusion approaches are characterized by
the consideration of multiple modalities within the same
time frame
16. 16
Asynchronous Fusion
Asynchronous fusion algorithms refer to past time
frames with the help of some kind of memory support.
Therefore, they are able to capture the asynchronous
nature of observed modalities.
18. 18
Event-Based Fusion
Take into account temporal relationships between
channels and learn when to combine information
Move from segmentation-based processing to
asynchronous event-driven approaches
More robust in the case of missing or noisy data
+
0
Fusion
time
haha hehe
Event
F. Lingenfelser, J. Wagner, E. André, G. McKeown, W. Curran: An Event Driven Fusion Approach
for Enjoyment Recognition in Real-time. ACM Multimedia 2014: 377-386
19. 19
SSI Framework
The Social Signal Interpretation (SSI) framework
is the attempt to provide a general architecture
to tackle the challenges we have discussed:
collection of large and rich multi-modal corpora
investigation of advanced fusion techniques
simplifying the development of online systems
hehe
hehe
Johannes Wagner, Florian Lingenfelser, Tobias
Baur, Ionut Damian, Felix Kistler, Elisabeth André:
The social signal interpretation (SSI) framework:
multimodal signal processing and recognition in
real-time. ACM Multimedia 2013: 831-834
SSI is freely available under:
http://www.openssi.net
24. 24
Generation of Facial
Expressions
FACS (Facial Action Coding System) can be used to
generate and recognize facial expressions.
Action Units are used to describe emotional
expressions.
Seven Action Units were identified for the robotic face
(out of 40 Action Units for the human face)
Lower face:
lip corner puller (AU 12),
lip corner depressor (AU 15)
and lip opening (AU 25)
Upper face:
inner brows raiser (AU 1),
brown lowerer (AU 4),
upper lid raiser (AU 5)
and eye closure (AU 43).
26. 26
Realization of Social Lies for the
Hanson Robokind
Social lies constitute a great part of human conversation.
Social lies, as used for politeness reasons, are generally
accepted.
Humans often show deceptive cues in their nonverbal
behavior while lying.
Humanoid robots should show deceptive cues while
conducting social lies as well.
27. 27
Deceptive Cues
Deceptive cues in human faces, according to Ekman and
colleagues:
Micro-expressions: A false emotion is displayed but
the felt emotion is unconsciously expressed for the
fraction of a second.
Masks: The felt emotion is intentionally masked by a
not corresponding facial expression.
Timing: The longer an expression is shown the more
likely it is accompanying a lie.
Asymmetry: Voluntarily shown facial expressions
tend to be displayed in an asymmetrical way.
30. 30
Real versus Faked Smile
Smile with blended anger (in the
eye region
Real smile
31. 31
Results of a Study
It was easier to detect faked smiles by the mouth region.
Robots with an asymmetrical smile were rated as
significantly less happy than robots with a genuine smile.
Results are in line with research on virtual agents:
Rehm & André, AAMAS 2005:
• Agents that fake emotions are perceived as less trustworthy
and less convincing
• Subjects were not able to name reasons for their uneasiness
with the deceptive agent
B. Endrass, M. Häring, G. Akila, E. André: Simulating
Deceptive Cues of Joy in Humanoid Robots. IVA 2014:
174-177
33. 33
Social Feedback Loop
Improve
Social Skills
Implicit Social Response
Generate Feedback
Explicit Hint on Social Behavior
Behavior Analysis
Social Behavior
Sensors
34. 34
Behavior
Analysis
Real-time multimodal analysis and classification
of social signals
Expressivity features (Energy, Openness, Fluidity)
Facial expressions (Smiles, Lip biting)
Speech quality (Speech rate, Loudness, Pitch)
Engagement, Nervousness
35. 35
Evaluation
Location:
Parkschule School in Stadtbergen, Germany
Participants:
20 pupils (10m/10f), 13-16 years old, job seeking
Two practitioners
I. Damian, T. Baur, B.Lugrin, P. Gebhard, G.
Mehlmann, E. André: Games are Better than Books:
In-Situ Comparison of an Interactive Job Interview
Game with Conventional Training. AIED 2015: 84-94
37. 37
Day 1 Day 2 Day 3
Pre-Interviews Training (Control) Training (TARDIS) Post-Interviews
20 pupils
2 practitioners
Task: mock-
interviews
Duration: ~10 min
10 pupils
Task: reading a
job interview guide
Duration: ~10 min
10 pupils
Task: Interaction
with TARDIS +
NovA
Duration: ~10 min
20 pupils
2 practitioners
Task: mock-
interviews
Duration: ~10 min
2x performance
questionnaires
(user +
practitioner)
user experience
questionnaires
user experience
questionnaires
2x performance
questionnaires
(user +
practitioner)
Experimental Setting
38. 38
Results
The overall behavior of the pupils who had interacted
with TARDIS was rated significantly better by job trainers
than the overall behavior of the pupils who prepared
themselves for the job interview using books.
Only for the pupils who trained with TARDIS we were
able to measure statistically significant improvements:
Their use of smiles appeared more appropriate.
Their use of eye contact appeared more appropriate.
They appeared significantly less nervous.
39. [...] using the system, pupils seem
to be highly motivated and able to
learn how to improve their
behaviour […] they usually lack
such motivation during class
[...] transports the experience into
the youngster’s own world
[...] makes the feedback be much
more believable
40. 40
Augmenting
Social
Interactions
I. Damian, C.S. Tan, T. Baur, J. Schöning,
K. Luyten, E. André: Augmenting Social
Interactions: Realtime Behavioural
Feedback using Social Signal Processing
Techniques. CHI 2015: 565-574
42. 42
Social Feedback Loop
Behavior Analysis
Social Behavior
Explicit
Feedback Generation
Haptic Feedback
Sensors
Improve
Social Skills
43. 15 speakers, 2 observers
Task: Hold 5 min presentation
2 Conditions: system on, system off
- within subjects
- randomized order, 2 weeks apart
Data acquisition: social signal recordings,
questionnaires (speaker/observers)
Study 1: Quantitative study in
controlled environment
44. Objective analysis of recordings:
Amount of inappropriate behaviour
decreased when system was on
Off
On
%inappropriatebehaviour
(lowerisbetter)
44
46. 3 speakers, 13 observers
Task: Present PhD progress
Data acquisition: semi-structured
interview
Study 2: Qualitative study in a
real presentation setting
47. [...] once I saw the feedback that
I was talking too fast, I tried to
adapt
48. [...] once I saw the feedback that
I was talking too fast, I tried to
adapt
[...] most of the time I did not
perceive the system, only when I
consciously looked at the feedback
49. [...] once I saw the feedback that
I was talking too fast, I tried to
adapt
[...] most of the time I did not
perceive the system, only when I
consciously looked at the feedback
It was a good feeling seeing
everything [the icons] green ...
it’s like applause, or as if
someone looks at you and nods.
However, the green lasts longer
than a nod [laughs]
53. 53
User Study
Users:
7 blind and visually impaired participants
Criteria:
No nystagmus, unrestricted eye
movements
Age Gender Visual impairment Control method
68 male Cataract center point
49 female Cataract (early stage) eye gaze
43 female Optic atrophy eye gaze
73 male Congenital blindness center point
68 male Optic nerve damage (accident) center point
87 female Macular degeneration eye gaze
70 male Retinal degeneration eye gaze
54. 54
Experiment
Scenario:
Two videos with a speaker giving a monologue are
shown
Task:
Rate emotional state of the speaker
Results:
Videos were rated more accurately with the system on
56. 56
Overall Conclusions
Social and emotional sensitivity are key elements of
human intelligence.
Social signals are particularly difficult to interpret
requiring to understand and model the causes and
consequences of them.
Offline applications start from too optimistic recognition
rates.
More work needs to be devoted to interactive online
applications.
More information and software available under:
http://www.hcm-lab.de
57. 57
Current Work:
Mobile Social Signal Processing
SSJ: Realtime Social Signal Processing for Java/Android
SSI – Unix/Android build compatibility