This talk is about PLEA, the virtual being and the robot. It is about the vision how PLEA is made and what is her story. She samples its environment to determine how a person feels, and then demonstrates the affection back. She analyses and interprets different sources of social signals from those who interact with to generate hypotheses. Then she produces non-verbal expressions using information visualization techniques. PLEA is a proof-of-concept, and she was presented at many festivals including British Science Festival and Art & AI Festival in Leicester, the UK. At the end of this talk if we are lucky, PLEA would visit the audience from the screen.

1. PLEA - Affective interactive virtual agents
that can occupy different environments
Tomislav Stipancic, University of Zagreb, Faculty of Mechanical Engineering and Naval
Architecture, Department for Robotics and Automation
Research Talk, Data Science Adria Conference, 17.05.2022., Zagreb, Croatia
The work has been supported in part by Croatian Science Foundation under the project ‘Affective Multimodal Interaction based on
Constructed Robot Cognition – AMICORC (UIP-2020-02-7184)’.

2. ... contents ...
02/29
o ... where are we today & our dreams for the future ...
o … the problem …
o … the solution approach …
o … what is context ? …
o ... the goal ...
o … theoretical background …
o ... Social Robotics - PLEA ...
o … at the end, plans, conclusions …
o ... questions, discussion ...

3. ... where are we today & our dreams for the future ...
o Smart robots à today mostly data-driven.
03/29

4. ... where are we today & our dreams for the future ...
o Smart robots à today mostly data-driven.
o Robots à ... able to manipulate objects, designed for uncertainty and
dexterity, adapt to dynamic reality ...
04/29

5. ... where are we today & our dreams for the future ...
o Smart robots à today mostly data-driven.
o Robots à ... able to manipulate objects, designed for uncertainty and
dexterity, adapt to dynamic reality ...
o ... the scientific dreams go further ... as partners, able to establish
interaction with humans and related environment, exhibit emergent
behavior, understand concepts ...
05/29

6. ... where are we today & our dreams for the future ...
o Smart robots à today mostly data-driven.
... are more like humans ...
o Robots à ... able to manipulate objects, designed for uncertainty and
dexterity, adapt to dynamic reality ...
o ... the scientific dreams go further ... as partners, able to establish
interaction with humans and related environment, exhibit emergent
behavior, understand concepts ...
... or not?
All men by nature desire to know (Aristotle)!
06/29

7. o Deterministic chaos.
o It is not possible to completely
determine any environment no matter
how tight tolerance ranges we are
using.
... the problem ...
o Deterministic chaos should be accepted as a natural phenomenon.
07/29

8. ... the solution approach ...
o How could we build the context-based algorithms?
o Context is any information that can be used to characterize the situation of an
entity. Context-aware applications look at the who’s, where’s, when’s, and what’s
of entities and use this information to determine why a situation is occurring
[Dey, A. K., 2010].
08/29

9. ... the solution approach ...
o How could we build the context-based algorithms?
o Context is any information that can be used to characterize the situation of an
entity. Context-aware applications look at the who’s, where’s, when’s, and what’s
of entities and use this information to determine why a situation is occurring
[Dey, A. K., 2010].
09/29
Explicit
(understandable
to computers)
Implicit
(understandable
to humans)
Computation
model
Implicit to explicit transformation.

10. 10/28 10/29
... … what is context ? …

11. 11/29
... what is context ? …

12. 12/29
o A trilled Serena Williams
after she beat her sister
Venus in the 2008 U.S.
Open tennis finals (source:
Barrett, L. F., Mesquita, B.,
& Gendron, M. (2011).
Context in Emotion
Perception. Science, 20(5),
286-290)
... … what is context ? …

13. Can we build a software agent to reason like a person?
n In the domain of strong AI - today is still impossible!
... the goal...
13/29

14. Can we build a software agent to reason like a person?
n In the domain of strong AI - today is still impossible!
... the goal...
14/29
Input data à f(x) à Output (robot actions)
Levels of adequacy (linguistic theory)
1. Observational adequacy
n account for the observations
2. Descriptive adequacy
n formally specified rules accounting for
all observed arrangements of the data,
n account for the observations
3. Explanatory adequacy
n explain the observations

15. 2. Semantic domain description (Knowledge base ).
3. Uncertainty based reasoning (AI).
Approach: computational model as an implicit to explict converter.
... theoretical background ...
1. Acquisition of information from the environment (Ubiquitous Computing).
15/29
Explicit
(understandable
to computers)
Implicit
(understandable
to humans)
Computation
model
Implicit to explicit transformation.

16. 3. Uncertainty based reasoning (AI).
Approach: computational model as an implicit to explict converter.
... theoretical background ...
1. Acquisition of information from the environment (Ubiquitous Computing).
2. Semantic domain description (Knowledge base ).
16/29

17. 17/19
... theoretical background ...
Ubiquitous computing (or "ubicomp") is a concept in software engineering
and computer science where computing is made to appear anytime and
everywhere.
17/29

18. 3. Uncertainty based reasoning (AI).
Approach: computational model as an implicit to explict converter.
... theoretical background ...
1. Acquisition of information from the environment (Ubiquitous Computing).
2. Semantic domain description (Knowledge base ).
18/29

19. 2. Semantic domain description (Knowledge base ).
3. Uncertainty based reasoning (AI).
Approach: computational model as an implicit to explict converter.
... theoretical background ...
1. Acquisition of information from the environment (Ubiquitous Computing).
o Search for all objects that are partialy
gray and loud, conditionaly nice and
weight lower then 100 kg.
n Home Theatre,
n Washing machine,
n Mother in law,
n …
19/29

20. 3. Uncertainty based reasoning (AI).
Approach: computational model as an implicit to explict converter.
... theoretical background ...
1. Acquisition of information from the environment (Ubiquitous Computing).
2. Semantic domain description (Knowledge base ).
o Supervised Learning
o Unsupervised Learning
o Reinforcement Learning
o Semi-supervised learning
20/29

21. 21/19
PLEA is an interactive biomimicking robot head.
PLEA samples its environment and your face to determine how you are feeling - it then demonstrates its affection for
you.
PLEA is analysing different sources of social signals from those who interact with it, including facial emotions, levels of
loudness in the room, intensity of body movements of those moving around the installation and also sentiment analysis
of speech that it hears.
The work on this robot has been supported in part by Croatian Science Foundation under the project ‘Affective Multimodal
Interaction based on Constructed Robot Cognition – AMICORC (UIP-2020-02-7184)’.
21/29
... Social Robotics - PLEA ...

22. 22/19
PLEA is an interactive biomimicking robot head.
PLEA samples its environment and your face to determine how you are feeling - it then demonstrates its affection for
you.
PLEA is analysing different sources of social signals from those who interact with it, including facial emotions, levels of
loudness in the room, intensity of body movements of those moving around the installation and also sentiment analysis
of speech that it hears.
The work on this robot has been supported in part by Croatian Science Foundation under the project ‘Affective Multimodal
Interaction based on Constructed Robot Cognition – AMICORC (UIP-2020-02-7184)’.
22/29
... Social Robotics - PLEA ...

23. 23/19
PLEA is using a multimodal approach in which the
hypothesis about emotion is extracted from
• facial expressions,
• intensity of body movements,
• speech detected in a surrounding environment
(using acoustic and linguistic submodalities)
23/29
... Social Robotics - PLEA ...

24. 24/19
... Social Robotics - PLEA ...
PLEA is using a multimodal approach in which the hypothesis about emotion is extracted from
• face expressions,
• intensity of body movements,
• speech detected in a surrounding,
• (WIKI Data knowledge base)
• …
Sad Happy
Angry Neutral
Surprised Disgust
Angry Fear
Granularity
24/29

25. 25/19
... Social Robotics - PLEA ...
PLEA is using a multimodal approach in which the hypothesis about emotion is extracted from
• face expressions,
• intensity of body movements,
• speech detected in a surrounding,
• (WIKI Data knowledge base)
• …
Sad Happy
Angry Neutral
Surprised Disgust
Angry Fear
Granularity
25/29

26. Sad Happy
Angry Neutral
Surprised Disgust
Angry Fear
Granularity
Autonomous
generation of
facial
expressions
26/29
... Social Robotics - PLEA ...

27. 27/29
... Social Robotics - PLEA ...

28. … at the end, plans, conclusions …
28/29
o Current applications of interest - related to HCI in different environments:
o People in public places in interaction … we are already doing that.
o Smart aging applications … we started preparing a new Horizon project. PLEA
could help an ageing person not to feel alone that much.
o People at space stations … astronauts are often feeling lonely and separated, and
we strongly believe that PLEA could help there.
o Let’s call PLEA?

29. ... questions, discussion ...
29/29
Tomislav Stipancic
Department of Robotics and Production System
Automation
Faculty of Mechanical Engineering and Naval Architecture
University of Zagreb, Croatia
tomislav.stipancic@fsb.hr