Emotive Media are media environments where an emotional channel is added in addition to other transmission channels in computer mediated communication. The medium actively recognizes human emotions, and is capable of simulating emotions as well. In this presentation, very basic aspects of emotive media are presented. The talk layouts the basics through an investigation of methods emerging from affective computation, brain-interfaces, and cognitive concepts. Practical examples, such as cognitive Big Data, the LudoViCo UX Machine, Empathy Interfaces, Portable Personality (P2), Financial Texts Mining for Sentiment Analysis, and Visualization are presented. More information (in particular publications) can be found on www.artur-lugmayr.com.

1. Emotive Media
Visualisation and Analysis of Human Bio-Feedback Data
26/08/2016 @ Curtin HIVE, Perth, Australia
A/Prof. Artur Lugmayr
www.artur-lugmayr.com
artur.lugmayr@artur-lugmayr.com

2. https://www.fitbit.com/au, http://emotiv.com/

3. Trigger
Peak of
Expectations
Disillusion Enlightenment
Plateau of
Productivity
Speech
Recognition
Artificial
Intelligence
Deep Learning
Open AI
Gesture
Recognition
Automated
Narratives
Traditional
Media
IoT
Affective
Computation
Smart Home
Smart
Accessories
Brain Interfaces
Sensor
Interfaces
Biosensors
Big Data NOW
2-5 YEARS
5-10 YEARS
> 10 YEARS
Technologies –
Between Hype
and Maturity
QoE (Quality of
Experience)
Intelligent
Machines

4. Practical Applications
• Health care and wellness
• eLearning
• Robotics and digital pets
• Identification and safety
• Sentiment and text analysis
• Small gadgets and wearables
• Sports analysis
• User-experience research
• Quality of Experience (QoE)
• Analysis of archives and media
• Advertisement analytics
• Game behaviour and analytics
• Organizational management
• Intelligent machines
• …

5. Affective
Computation
Neurosciences
Media Studies and
Communication
Theories
User Experience,
and Human-
Computer-
Interaction
Psychology of ‘The
Human Mind’
Computational
Models
Involved Research Fields

6. http://muvis.cs.tut.fi/index.html
Previously…. the conveyer belt for data processing

7. In the Future… we need a ‘cognitive’ conveyer belt
for knowledge and wisdom

8. · Situation
· Object
· Collaboration
· Behaviour
· Perception
· Attention
· Arousal
· Emotion
· Experience
· Feelings
· Cognition
Mind
Communication
Cause
· Language
· Personality
· Aesthetics
· Emotional Binding
· Artificial Intelligence
· Motives and Believes
Concepts
The Key in Emotional Media are Higher Level
Concepts, as e.g. Language, Aesthetics, and
Personality
Lugmayr, Artur, Tillmann Dorsch and Pabo Roman Humanes. "Emotional Ambient Media." Handbook of Research on Synthetic Emotions and Sociable
Robotics: New Applications in Affective Computing and Artificial Intelligence. IGI Global, 2009. 443-459. Web. 27 Aug. 2016. doi:10.4018/978-1-60566-354-
8.ch022

9. Emotional
State
Signal Level
(Signals)
Concept Level
(Concepts)
Symbol Level
(Symbols)
Pattern Level
(Patterns)
ActionMind
Reaction
Level
CAPTURE
SIMULATE
CONCEPT
RECOGNIZE
LEARN
COMMUNICATE
Emotional Media are Dual – Media that Understand
Emotions, and Media that Simulate Emotions
“getting computers to ‘think’ like
humans, [and] AI’s next natural
step seems to be getting
computers to ‘feel’ like humans”
J. Krikke and B. Alfonsi, "In the News," IntelligentSystems, IEEE, vol. 21, pp.
102-104, c3, 2006

10. • Hayley Hung, Hatice Gunes, Introduction to Social Signal Processing, ACM Multimedia 2015: Emotional and Social Signals Tutorial
• New Social Signals in a New Interaction World: The Next Frontier for Social Signal Processing, A Vinciarelli, AS Pentland, Systems, Man, and
Cybernetics Magazine, IEEE 1(2), 2015
“A Social Signal is a communicative or informative signal that,
either directly or indirectly provides information concerning social
interactions, social emotions, social attitudes or social relations”
• http://www.theaustralian.com.au/news/d
eputy-opposition-leader-julie-bishop-says-
peter-slipper-cannot-remain-as-
speaker/story-e6frg6n6-1226491792937
• http://www.abc.net.au/pm/content/2015
/s4331627.htm
NOTE! Due to eventual copyright
reasons, I removed 2 images of two
Australian Politicians. You want to
know what was here? Click on the
links below…

11. Semantics & Meaning (Stock Exchange Sentiment)
Macroeconomic Indicators
Interest rate
Inflation
Domestic credit
Industrial production
Unemployment rate
Consumer sentiment
Consumer prices
Growth durable goods
GDP
….
Implicit Sentiment
Indicators
Call/Put ratio
Open interest
IPOs
IPO first day returns
Mutual fund flows
…
Explicit Sentiment Indicators
Boerse Frankfurt sentiment index
Economic sentiment index (ESI)
ZEW Indicator
Ifo Business Climate Index
…
Textual Sentiment
Sources Sources
Newsletters
RSS Feeds
News
…
Sentiment Indicator
Sentiment index
Trend and trend strength
(Resistance/Support)
(Hot news)
Textual
Sentiment
Analysis
Text Mining
Sentiment
Aggregation
Sentiment
Model
Technical Indicators
Volatility
Volume
Turnover
RSI, CCI, ARMS, …
….
Lugmayr and G. Gossen, “Evaluation of methods and techniques for language based sentiment analysis for DAX30 stock exchange - a first concept of a
"LUGO" sentiment indicator,” in Proceedings of the 5th International Workshop on Semantic Ambient Media Experience (SAME) - in Conjunction with
Pervasive 2012, A. Lugmayr, T. Risse, B. Stockleben, J. Kaario, B. Pogorelc, and E. S. Asensio, Eds. Newcastle, UK: Tampere University of Technology (TUT),

12. Techniques in Social Signal Recognition
• Verbal communication
• Non-verbal communication
• Gestures, facial expressions, postures, gaze, pupils, vocal behaviour, speech
activity, personal space, seating arrangements, …
• Media archive analytics
• Multimedia content analysis, low-level features, semantic relations, …
• Social media analytics
• Media achieve analytics
• Hayley Hung, Hatice Gunes, Introduction to Social Signal Processing, ACM Multimedia 2015:
Emotional and Social Signals Tutorial

13. Bio-Feedback Visualization
https://processing.org/
Examples: https://processing.org/examples/ https://www.youtube.com/watch?v=SWCkEKrgaXk
NOTE! Due to eventual copyright
reasons, I removed 2 images of the
example. You want to know what
was here? Click on the links below…

14. Emotional Interaction and Interfaces
Lugmayr, A., Dorsch, T. & Humanes, P.R., 2009. Emotional Ambient Media. In
Handbook of Research on Synthetic Emotions and Sociable Robotics: New
Applications in Affective Computing and Artificial Intelligence. IGI Global.

15. Empathy Glasses (CHI 2016)
• Masai, Katsutoshi et al. “Empathy Glasses.” CHI (2016)., https://www.semanticscholar.org/paper/Empathy-Glasses-Masai-
Kunze/8cc9f1a26ef16490d4688bb9928fe91e3f71d02f
NOTE! Due to eventual copyright
reasons, I removed the images of the
example. You want to know what
was here? Click on the links below…

16. Personality
Personality
Personality
Personality
Personality
Personality
Personality
Personality
Personality
Collaborative
Space
Personality Computation
Bits of personality everywhere (Portable Personality – P2),
similar to Microsoft’s Cortana and Apple’s Siri
NOTE! Due to eventual copyright
reasons, I removed 2 images of the
example. You want to know what
was here? Click on the links below…
P2 is similar to the film
idea in ‘The Golden
Compass’. Check it out:
https://en.wikipedia.or
g/wiki/The_Golden_Co
mpass_(film)
S. Reymann, V. Bruns, and A. Lugmayr, "P2 - Portable Personality a Middleware Solution for Smart
User Profile Management and Distribution," in Interactive TV: A Shared Experience, TISCP Adjunct
Proceedings of EuroITV 2007, Amsterdam, 2007
S. Uhlmann and A. Lugmayr. Media in the Ubiquitous Era: Ambient, Social and Gaming Media, chapter
Portable Personality and its Personalization Algorithms: An Overview and Directions, pages 66–93. IGI
Global, Hershey, USA, 2011. ISBN 9781609607746.

17. P2 – Portable Personality
www.portablepersonality.org
Provider
P2 Service P2 ServiceP2 Mobile Service
Consumer
Service
S. Reymann, V. Bruns, and A. Lugmayr, "P2 - Portable Personality a Middleware Solution for Smart
User Profile Management and Distribution," in Interactive TV: A Shared Experience, TISCP Adjunct
Proceedings of EuroITV 2007, Amsterdam, 2007
S. Uhlmann and A. Lugmayr. Media in the Ubiquitous Era: Ambient, Social and Gaming Media, chapter
Portable Personality and its Personalization Algorithms: An Overview and Directions, pages 66–93. IGI
Global, Hershey, USA, 2011. ISBN 9781609607746.

18. Free UX Testing Tool: The LudoVico UX Machine for Physiological Sensor Data
Recording, Analysis, and Visualization for User Experience Design Experiments,
SEACHI Workshop, SIGCHI, 2016
The Ludovico UX Machine
WWW.UX-MACHINE.COM (freely available sometimes in 2016)

19. The LudoVico UX Machine
Medical Sensor Data: Collection and interpretation of
medical sensor data, such as EMC, EMG, breath, pulse, body
temperature, body position, and galvinistic skin response.
Smart Environment Sensor Data: data coming from smart
city environments, mashup data, location based data, mobile
phone data.
Body Motion Sensor Data: Collection and interpretation of
body motion sensor data through multiple sensor attached to
a human body.
Extendible Platform: A plug-in concept and a light-weight
sensor data protocol allows high extendibility of the platform,
add new sensor types, additional features, new analysis
techniques, and new functionality.
Freely Available: The LudoViko UX Tool is freely available
for download: http://www.artur-lugmayr.com starting from
mid-2016. Subscribe to our newsletter to get more
information.

20. The Key is Visualisation
Data Analysis
Re-
presentation
Presentation
Interaction
Cognition &
Perception
Communication
Context

21. VALENCE
AROUSAL
LOW
(passive)
HIGH
(active)
NEGATIVE
POSITIVE
… and developing user-interfaces that people
understand in particular domains….
…a film-maker might not understand complex time-series data in form of diagrams, but a user-
interface that he is comfortable with as e.g. Adobe Premiere integrating an ‘emotional
monitor’…

22. Marshall McLuhan
• Electronic culture as follow up culture for 400 years of print culture
• Media extend human senses: sight, hearing, touch, and smell
• Media technologies change society and culture: extensions of our bodies change
mind & society affecting our mental functions
• generating ideas, perception of the world, emotions, experiences
• The effect of McLuhan’s thesis:
• Technology DETERMINES social outcome
• Technology enables man extended and a live model outside the mind
• Cybernetics
• Examples: light bulb, hammer, 3D cinema, ambient media ...
https://en.wikipedia.org/wiki/Marshall_McLuhan
1964 Understanding Media:
The Extensions of Man; 1st
ed. McGraw Hill, NY;
reissued by MIT Press, 1994,
with introduction by Lewis
H. Lapham; reissued by
Gingko Press, 2003. ISBN 1-
58423-073-8.