Exploring UiPath Orchestrator API: updates and limits in 2024 🚀
Towards Queryable User Profiles: Introducing Conversational Agents in a Platform for Holistic User Modeling
1. @cataldomusto
cataldo.musto@uniba.it
Towards Queryable User Profiles:
Introducing Conversational Agents in
a Platform for Holistic User Modeling
CATALDO MUSTO*, FEDELUCIO NARDUCCI^, MARCO POLIGNANO*, PASQUALE LOPS*, MARCO DE GEMMIS*, GIOVANNI SEMERARO*
*UNIVERSITÀ DEGLI STUDI DI BARI ‘ALDO MORO’ – ITALY
^POLYTECHNIC UNIVERSITY OF BARI - ITALY
ExUM 2020 – International
Workshop on Explainable User
Models and Personalised Systems
jointly held with UMAP 2020
Online from Genoa (Italy) – July 18, 2020
2. 2
User Modeling (in the ‘egosystem’ era)
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
3. Holistic User Model*
Affects
Demographics
Interests
Behaviors
Social
Relations
Knowledge
and Skills
Physical
States
Psychological
Aspects
6
(*) Musto, C., Polignano, M.,
Semeraro, G. et al. MYRROR: a
platform for holistic user
modeling. User Modeling and
User-Adapted
Interaction (2020).
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
4. 7
Myrror: a platform for holistic user modeling*
(*) Musto, C., Polignano, M., Semeraro, G. et al. MYRROR: a platform for holistic
user modeling. User Modeling and User-Adapted Interaction (2020).
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
5. 8
Myrror: a platform for holistic user modeling*
(*) Musto, C., Polignano, M., Semeraro, G. et al. MYRROR: a platform for holistic
user modeling. User Modeling and User-Adapted Interaction (2020).
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
6. 10
Myrror: a platform for holistic user modeling*
(*) Musto, C., Polignano, M., Semeraro, G. et al. MYRROR: a platform for holistic
user modeling. User Modeling and User-Adapted Interaction (2020).
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
7. 11
Myrror: a platform for holistic user modeling*
(*) Musto, C., Polignano, M., Semeraro, G. et al. MYRROR: a platform for holistic
user modeling. User Modeling and User-Adapted Interaction (2020).
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
8. 13
Myrror: a platform for holistic user modeling*
(*) Musto, C., Polignano, M., Semeraro, G. et al. MYRROR: a platform for holistic
user modeling. User Modeling and User-Adapted Interaction (2020).
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
9. 14
Myrror: a platform for holistic user modeling*
(*) Musto, C., Polignano, M., Semeraro, G. et al. MYRROR: a platform for holistic
user modeling. User Modeling and User-Adapted Interaction (2020).
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
10. 15
Myrror: a platform for holistic user modeling*
(*) Musto, C., Polignano, M., Semeraro, G. et al. MYRROR: a platform for holistic
user modeling. User Modeling and User-Adapted Interaction (2020).
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
11. Myrror: a platform for holistic user modeling*
• Current Issues
1. Overall transparency may be improved
◦ Users should be aware of the information encoded in their
profiles
◦Interaction with a web-based dashboard can be difficult,
especially for beginners
◦ «How can I check my current interests?»
2. It is difficult to get personal «insights» from the current
dashboard
◦ «Did I sleep enough?»
16
(*) Musto, C., Polignano, M., Semeraro, G. et al. MYRROR: a platform for holistic
user modeling. User Modeling and User-Adapted Interaction (2020).
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
12. Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020 17
Goals
To improve the way people access to the
information encoded in their profiles
13. 18
Intuition: Conversational Agents
What is a conversational
agent?
• A tool that understands
natural-language
requests and provides
natural-language answers
to general-purposes
services
• Popularized by Alexa, Siri,
Google Now, etc.
Alexa, play Africa
by Toto.
Here’s Africa by Toto
from Amazon Music.
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
14. 19
Intuition: Conversational Agents
Intuition
• To exploit Conversational
Agents to allow the user
to query her holistic user
model in natural
language
• Easier to access to the
information encoded
in the profile
• Easier to get insights
Alexa, play Africa
by Toto.
Here’s Africa by Toto
from Amazon Music.
Did I sleep enough?
No, your sleep is not
enough. Currently, you
sleep 6h per day.
A first step towards Queryable User Profiles
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
15. 21
Contribution: MyrrorBot
MyrrorBot is a conversational
agent built on top of Myrror
Based on HUMs
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
16. MyrrorBot is a conversational
agent built on top of Myrror
Based on HUMs
Answers to natural-language
requests concerning the
information encoded in the
HUM
Provides personalized services
Alexa-like (Music, News,
Recipes, etc.)
22
Contribution: MyrrorBot
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
17. 23
Contribution: MyrrorBot
MyrrorBot is a conversational
agent built on top of Myrror
Based on HUMs
Answers to natural-language
requests concerning the
information encoded in the
HUM
Provides personalized services
Alexa-like (Music, News,
Recipes, etc.)
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
18. 24
MyrrorBot: Intent Recognizer
MyrrorBot is a conversational
agent built on top of Myrror
Based on HUMs
Answers to natural-language
requests concerning the
information encoded in the
HUM
Provides personalized services
Alexa-like (Music, News,
Recipes, etc.)
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
19. 25
MyrrorBot: Intent Recognizer
Our natural language
understanding module is based
on Google DialogFlow.
Input:
Natural Language
Request
Output:
Intent (i.e., a
representation of the
informative need)
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
Alexa, play
Africa by Toto.
Here’s Africa by Toto
from Amazon Music.
Intent=music
20. 26
MyrrorBot: Intent Recognizer
Our natural language
understanding module is based
on Google DialogFlow.
Input:
Natural Language
Request
Output:
Intent (i.e., a
representation of the
informative need)
What kind of
questions can
we handle?
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
21. 29
Intents in MyrrorBot
Demographics
Interests
Affects
Physical States
Social Relations
Behaviors
Knowledge and Skills
Psychological Traits
Intents are directly mapped to the facets of HUMs
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
22. 30
Intents in MyrrorBot
Demographics
Interests
Affects
Physical States
Social Relations
Behaviors
Knowledge and Skills
Psychological Traits
Each intent is further split into two sub-intents
Informative
(«How much did I
sleep today?»)
Insights
(«Did I sleep
enough today?»)
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
23. 35
Training the Intent Recognizer
Intent Recognizer is
based on Machine
Learning
For each intent, a set of
training sentences
covering all the
possible utterances
should be provided
We feed the algorithm
with 20-30 sentences
for each intent
Behaviors - Insights Behaviors - Insights
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
24. 36
Intent Recognizer @Work
Affects - Informative
Psychological Traits - Informative
Behaviors - Insights
Physical States - Insights
Demographics - Informative
What is my mood right now, according
to your data?
Tell me about my personality traits
Did I burn enough calories?
Did I sleep enough today?
Where do I live?
Intent
Recognizer
(NLU)
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
25. 38
MyrrorBot: Workflow
MyrrorBot is a conversational
agent built on top of Myrror
Based on HUMs
Answers to natural-language
requests concerning the
information encoded in the
HUM
Provides personalized services
Alexa-like (Music, News,
Recipes, etc.)
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
26. 40
MyrrorBot: Generator
The Generator module produces a natural
language answer that fulfill user’s
information need.
In case of Informative intent, no computation
is carried out. The profile is queried and a
template is filled in with the values
returned by the HUM.
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
27. 41
MyrrorBot: Generator
The Generator module
produces a natural language
answer that fulfill user’s
information need.
Differently, intents that aim to
get some insights require some
computation (i.e., compare
current values with some
threshold value) before the
answer is generated.
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020
28. Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020 42
MyrrorBot: Recap
Acquires a HUM
Recognizes users’ requests
Generate the answer
29. Experimental Evaluation
Research Question 1
What is the opinion of the users, in terms of usability, ease of use and learning curve?
Research Question 2
Does a conversational interface reduces the time needed to access to the personal information
encoded in a user profile?
Experimental Design
User Study with a Web Application
76 subjects (previous experience with chatbots= 35.5%)
Metrics RQ1: Questions based on SUS (System Usability Survey) Questionnaire
5-points Likert scale
Metrics RQ2: Seconds required to complete a task (web-based vs. conversational interface)
Randomized Order of interaction, Wilcoxon Test
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020 43
30. Results – Research Question 1
Metric Question %Compl. Agree %Agree %Others
Learning Curve «I became familiar with
the system very quickly»
51.3% 38.2% 10.5%
Interaction «It was easy to formulate
my questions»
40.8% 48.7% 11.8%
Accuracy «The system provided me
with the right answers»
55.3% 35.5% 9.2%
Daily Use «I would use the system
for my daily tasks»
28.9% 40.8% 29.2%
Switching «I would prefer the system
to a web platform»
40.8% 40.8% 18.4%
Cataldo Musto, Fedelucio Narducci, Marco Polignano, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Towards Queryable User Profiles:
Introducing Conversational Agents in a Platform for Holistic User Modeling. ExUM Workshop@UMAP 2020 – Online – July 18, 2020 44
31. Results – Research Question 2
Facet Task MyrrorBot
(sec.)
Myrror
(sec.)
Gap%
Demographics Check your birthday 7.72 10.31 -25.12%
(p<0.001)
Interests Check your interests 9.62 22.11 -56.49%
(p<0.001)
Affects Check your latest mood 10.34 20.02 -48.35%
(p<0.001)
Psychological
Aspects
Check your personality
traits
9.98 12.33 -19.10%
(p<0.05)
45
Significant reduction of the task completion time
32. Results – Research Question 2 (cont.)
Facet Task MyrrorBot
(sec.)
Myrror
(sec.)
Gap%
Behaviors
Check your daily activities 13.25 24.99 -47.20%
(p<0.001)
Check if you burnt
enough calories
14.18 23.17 -19.05%
(p<0.05)
Physical States
Check if you slept enough 13.44 19.20 -30.47%
(p<0.001)
Check your heart rate 10.20 16.49 -38.14%
(p<0.001)
Significant reduction of the task completion time
46
33. Research Question 2 – Recap
Avg. Reduction40.3%
11,09
18,58
0,00 2,00 4,00 6,00 8,00 10,00 12,00 14,00 16,00 18,00 20,00
MyrrorBot
Myrror
Average Task Completion Time (sec.)
47
34. Recap
A first step towards Queryable User Profiles
Starting Point: Myrror
◦ Holistic User Modeling platform
◦ Collects data from several heterogeneous sources
◦ Processes the data and aggregates them to eight pre-defined facets
Contribution: MyrrorBot
◦ A Conversational agent built on top of MyrrorBot
◦ Acquires a HUM and recognizes users’ requests (based on an intent recognizer)
◦ Generate natural language answers fulfilling the requests
◦ Informative and Insight-based intents
◦ Good impact on the users. Significant reduction of task completion time.
48
36. Future Work
Evaluation of personalized services
Evaluation of new interaction
methods
◦ Integration with social robots (e.g.,
Pepper)
◦ Integration with voice-based interfaces
(e.g., Amazon Alexa-like)
Training of more (and more
sophisticated) intents
◦ Goal: to get more useful insights from the
platform (e.g., is my lifestyle correct?)
50