Talk straps: Interactivity between Human and Artificial Intelligence

Enabling Interactivity between Human and Artiﬁcial Intelligence December 14, 2020 @ STRAPS’20
Enabling Interactivity between
Human and Artiﬁcial Intelligence
Behrooz Omidvar-Tehrani
@BehroozOmidvar
2nd Workshop on Smart Data Integration and Processing on Service Based Environments
December 14, 2020

• Users, in their different roles, have different information seeking needs.
• Typical paradigms of information seeking are search, recommendation,
and exploration (search by experience).
Information seeking
2
Data scientist Domain expert Information consumer

For a full-fledged information seeking, users should be able to interact with the data system and
fulfill their tasks.
Main goal in information seeking
3
Data scientist
explores the banking data
to find patterns and trends
in loan records.
Medical expert
explores the effect of air
pollution on chronic respiratory
diseases in different regions.
Information consumer
explores Amazon to buy
a good point-and-shoot
camera.

State of the art in information seeking — are we done?
4
Tableau Visual Analytics tool

• In their paper called Subjective Databases, Li et el. claim that “making decisions that involve
subjective preferences is labor intensive for the end user.”
Subjective and objective tasks in information seeking
5
User
Subjective
task
Objective
task
their price is reasonable.
their price is lower than $300
per night.
I want to receive the list
of hotel such that …
Data
[VLDB’19]

• A challenge in information seeking is that most often complex and cold users carry vague tasks.
• The vagueness emanates from the lack of knowledge about the data schema and/or data
distribution.
Subjectivity in information seeking
6
Data scientist explores the banking data find
patterns and trends in loan records.
Medical expert explores the effect of air pollution on
chronic respiratory diseases in different regions.
Information consumer explores Amazon to buy a
good point-and-shoot camera.
How is an interesting effect defined?
What does distinguish between an
interesting and futile pattern/trend?
How is a good camera specified?

• No scientiﬁc method can provide a precise answer for a subjective task. However a frame of
reference is more feasible to deliver.
• The Fermi estimation is an approach to obtain this frame of reference for Fermi problems, e.g.,
the number of piano tuners in Chicago, or number of extraterrestrial civilizations in the Milky
Way galaxy (Drake equation).
• Fermi estimates work because overestimates and underestimates help cancel each other out, in
the absence of a consistent bias.
How to obviate subjectivity?
7
List hotels with reasonable price! I can afford more!
User
AI agent
Can’t do! Should be okay.
Is the price range $50-$100
per night reasonable?
What about $500-
$700 per night?
Hence, $200-$300?
[Weinstein 2012]

Data mining, databases, machine learning, and visual analytics, contribute to fulﬁlling clear tasks.
Objective tasks
8
Find the correlation between the loan
approval and the investees’ demographics.
Medical expert compares the survival rate in
a high and low air polluted areas.
Find a point-and-shoot camera with 4K video
resolution and the price lower than $100.
Statistics and visual analytics, such as
Kaplan-Meier charts.
ML techniques such as linear regression
Find groups of investees with success stories.
Pattern mining techniques such as
frequent item-set mining.
SQL querying

Typically, an AI pipeline with ML components can easily perform objective tasks.
Status quo for objective tasks
9
Data
Preparation
(ETL)
Black-box
Machine
Learning
Data
Presentation
Raw data User
Inductiv

Typical AI pipelines lack personalization, customization, and transparency, because the user is
left out of the loop.
Shortcomings of AI pipelines for subjective tasks
10
AI system
Raw data User
User context
(personalization)
User feedback
(Customization)
Explanation
(Transparency)
Focus of today’s talk!

• Inspired by Fermi estimation, incorporating interactivity between human and artiﬁcial
intelligence in AI pipelines can be solution for subjective information seeking tasks.
• Two big questions about interactivity …
How to formalize interactions?
How can the interaction between users and the AI agent happen?
How to learn interactions?
Can these interactions get automated? Can AI predict user decisions? Can AI assist users in these
interactions?
Interactivity for subjective information seeking
11

Formalizing interactions
Part 1

• Moravec’s paradox. Machines and humans frequently have opposite strengths and weaknesses.
Machines do reasoning, humans do sensory.
• An interactive pipeline becomes a mixed-initiative system.
Proposed model
13
Raw data User
Exploration
AI agent as an ML component
in the pipeline
Feedback
Guide
[Hans Moravec, Harvard University Press 1988]

• In a mixed-initiative system, there is a conversation between the user and the AI agent.
• The system guides the user in the plethora of data points, and the user provides feedback on
how this guidance should be performed.
• In such system, both the user and the agent grow together toward converging on a target.
Interactions in mixed-initiative system
14
User
AI agent
Target
Towards clarifying subjective tasks
Towards comprehending user needs
Guide
Feedback

Assume a program committee (PC) chair wants to build a PC for a conference, formed by
geographically distributed male and female researchers with different seniority and dexterity
levels.
Example: PC formation
15
Data Pipelines for User Group Analy4cs: SIGMOD 2019 Tutorial July 5, 2019 at Amsterdam
By-example explora4on
• Explora4on is expressed as similarity with an example group: explore-
around and explore-within.
IUGA
[Omidvar-Tehrani et al., CIKM’15]
proliﬁc, high publi.,
SIGMOD (29)
SIGMOD (29)
junior, high publi.
(46)
explore
around
ACM (46)
Itera4on 1
produc4ve,
temporal databases
(11)
highly senior, VLDB
(119)
SIGMOD, schema
matching, male
Itera4on 2
The group
contains L. Popa, A. Doan,
M. Benedikt, and S. Amer-
Yahia
The group contains F. Bonchi, K.
Chakrabar4, P. Fratenali and F. Naumann
Sihem Amer-Yahia
Denilson Barbosa
Michael Benedikt
Francesco Bonchi
Kaushik Chakrabar4
Lei Chen
Piero Fraternali
Felix Naumann
Paolo Papov
Lucian Popa
Mar4n Theobald
Fei Wu
Program Commiee
(not exhaus4ve)
Build a program commi"ee for WebDB’14 formed by geograph
male and female researchers with diﬀerent seniority and
explore
within
[CIKM’15] [VLDBJ’19]
User (PC chair)

Assume a tourist (who is a cold user) is walking in the area of the Pompidou center in Paris. After
30 minutes of walking, she gets tired and asks the system for “me time” recommendations.
Another example: cold-start POI recommendation
16
[SIGSPATIAL’20]
Visitors who have many friends, check in actively, and tend to visit
historical landmarks.
Visitors who post many photos tend to visit Asian restaurants.
Visitors who have many friends and tend to visit coffee shops.
I look for “me time”
Visitors who have many friends and visit restaurants on evenings.
Visitors who visit Modern Art Museums .
Visitors with many check-ins who visit shopping centers.
User clicks on
a yellow POI.
STEP 1 STEP 2
‘I’m now hungry”

• User’s feedback is provided using exploration actions.
• Given a set of data cubes as guidance, the user expresses her interest in what she wants to
receive in the next iteration using an exploration action (feedback).
• The function explore(d,k,e) takes as input a data cube d (feedback), an integer k, and an
exploration action e, and returns k other cubes as guidance, called D.
Feedback
17
proliﬁc, high publi,
SIGMOD
Feedback d
Exploration action e
(explore around)
D = explore(d,k,e) given k=3
SIGMOD, Schema
matching, male
highly senior, VLDB
productive, temporal
databases
Iteration t Iteration t+1

• Exploration actions are combination of “interestingness” measures to steer users towards a
particular cube set of interest. They capture representativeness and informativeness.
Exploration actions
18
User
Feedback d
at iteration t
Guidance D
at iteration t+1
[TKDE’19] [VLDBJ’2018]
Exploration actions
Exploration action Semantics
explore around
constrain Jaccard sim. with d and
maximize diversity in D
explore within
stay inside d and maximize
coverage in D
by distribution
constrain Earth Mover Distance with the
score distributions in d and maximize
diversity in D
by topic
constrain Cosine sim. with the LDA
topics in d and maximize diversity in D

Example of exploration actions
19
proliﬁc, high publi,
SIGMOD
SIGMOD, Schema
matching, male
highly senior, VLDB
productive, temporal
databases
Cube c1 Cube c2 Cube c3
Local constraint
Global constraint
(given the similarity threshold to be 0.7)
Jacc(d,c1) = 0.81
Jacc(d,c2) = 0.93
Jacc(d,c3) = 0.74
diversity({c1,c2,c3}) → max
explore around
Jacc(d,c4) = 0.65
d

Enabling Interactivity between Human and Artiﬁcial Intelligence December 14, 2020 @ STRAPS’20 20
Numberofiterations
0
4
8
12
16
WebDB'14 SIGMOD'14 VLDB'14 CIKM'14
Reaching 50% of the PC Reaching 80% of the PC
Effect of exploration actions

Learning interactions
Part 2

• Per Robin Hogarth (Economics Professor at UPF Barcelona), there exist two learning
environments: kind and wicked.
• Kind (or close-world) environments need tactics which are obtained
using repetition and practice.
• In Chess or Tennis, practice can make a grandmaster.
• ML systems are good to imitate kind environments.
• Wicked (or open-world) environments needs strategy, which differs
from tactics, and may even contradict it.
• Even a senior ﬁnancial expert may get surprised by a share
investment situation.
• ML systems do not imitate well due to lack of logs.
• Information seeking often occurs in a wicked environment.
Learning interactions is challenging
22
Judit Polgár

• Exploration state. The exploration action at iteration t is denoted as st = (dt, Dt), where Dt
is obtained by applying et-1 to dt-1.
• Exploration session. A sequence of exploration states and actions S = [(s1,e1)…(sn,en)]
• Policy. A state-action mapper π(st)=e1
• Objective. Find the optimal policy π*
Learning interactions
23
User with a sub-optimal policy
AI agent with an optimal policy
!
"
Target
next action?
optimal action!

• A good exploration policy collects more rewards along the way.
• Cube utility. Given a cube d and and an exploration target T, d’s utility is symbolically shown as
d ∩ T, and can be computed by any similarity measure.
• Reward. Given the current state st and current action et , the reward of transitioning to
another state st+1 denoted as R(st+1|st,et)is equal to the cube utility of dt+1.
• Hence an optimal policy is the one that maximizes the discounted cumulative reward.
What is a good exploration policy?
24

• Ofﬂine phase. An RL agent simulating a human analyst is trained to learn an optimal policy. The
policy is updated as the agent interacts with the data cubes via exploration actions. Action
selection is based on ε-greedy method (optimal under GLIE)
• Online phase. Once a policy is learned, it is provided to a user who applies it to generate an
interpretable exploration session.
General solution
25
policy
Update policyexplore
ofﬂine phase online phase
Explore with the
recommended action
recommend
action
feedback

• The optimal policy requires that the value function V and the action-value function Q are
maximized.
• Following Bellman equation:
• An optimal policy is obtained when we know X o or .
Obtaining the optimal policy
26

• Typically RL learns matrix for .
• Instead, we learn where and
• Given the function f as the state-feature function:
• Features include (but are not limited to) diversity, coverage, size, previously seen targets,
distribution, and previous action.
Improved policy learning with feature functions
27

• To learn we apply SGD by the minimization of mean squared error:
• Updates are incrementally done by SARSA:
Learning procedure
28

An example exploration session
29
by-distribu*on
features ↓
explore-
around
by-distrib. by-topic
support < 200 -0.04 0.01 0.03
no conferences 0.02 0.05 -0.04
demographic attributes 0.04 -0.02 0.03
recent papers 0.06 0.03 0.00
0-1 targets 0.14 0.10 -0.00
no seniority 0.10 0.02 0.03
reward=0 -0.12 0.04 0.02
sum 0.2 0.23 0.07
# discovered PC: 0
Beginningof
thesession
# discovered PC: 0
explore-around
[female, very productive, Europe, SIGIR]
[CEUR Workshop Proceedings, very
productive, Europe]
[female, very productive, SIGIR]
[female, Europe]
[female, EDBT, very productive]
[Asia, very active, male]
[Europe, confirmed, ICDE]
[UK/Ireland, very productive, male]
[ICDM, North America, ICDE]
[productive, female, Europe]
active
features
features ↓ explore-
around
diversity >0.8 0.06 0.04 0.03
support < 50 0.11 -0.02 0.01
conference in label -0.03 -0.04 0.02
demographic attribs. 0.04 -0.02 0.03
0-1 targets 0.14 0.10 -0.00
seniority -0.07 0.02 0.00
reward=0 -0.12 0.04 0.02
sum 0.19 0.15 0.11
active
features
by-topic
[female, highly senior, Enc. of DB]
[VLDB J., senior, IEEE, PVLDB]
[Asia, EDBT, confirmed, male]
[highly senior, very productive, Europe,
Enc. of DB]
[GRADES]
# discovered PC: 1
[female, Europe, confirmed, PVLDB]
[female, EDBT, SIGMOD]
[female, Europe, Enc. of DB]
[female, prolific, ICDE]
[Europe, confirmed, ICDE]
features ↓ explore-
around
diversity > 0.8 0.06 0.04 0.03
support < 50 0.11 -0.02 0.01
conference in label -0.03 -0.04 0.02
demographic attribs. 0.04 -0.02 0.03
2-3 targets -0.03 0.01 0.12
no seniority 0.10 0.02 0.03
reward=0 -0.12 0.04 0.02
sum 0.19 0.06 0.26
# discovered PC: 2 active
features

• We examine the target spread in 1, 2, 4, and 10 cubes.
Target spread
30

• We examine the impact of training and test datasets.
• Left is WebDB PC and right is SIGMOD.
Potentials of transfer learning
31

• We examine the impact of features.
• We consider the feature “the number of targets discovered so far”.
• Left is WebDB 2017 PC and right is SIGMOD 2017 PC.
Experiment on decision making
32

• Human-in-the loop data analytics is a must for today’s ML applications.
• Users and AI systems can cooperate to reach a common goal.
• Reinforcement learning is an appropriate model to capture the dynamics of interactions
between users and AI systems.
• Transfer learning (from general models to more specialized domains) shows great potentials for
the future interactive systems.
Conclusion
33

Collaborators
34
Sihem Amer-Yahia Eric Simon Alexandre Termier Ria Mae Borromeo Mariia Seleznova
CNRS, University of
Grenoble Alpes
SAP Paris Inria Rennes
University of the
Philippines
TU Berlin

Question Time
Enabling Interactivity between Human and Artiﬁcial Intelligence
Behrooz Omidvar-Tehrani
@BehroozOmidvar
2nd Workshop on Smart Data Integration and Processing on Service Based Environments
December 14, 2020

Talk straps: Interactivity between Human and Artificial Intelligence

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Talk straps: Interactivity between Human and Artificial Intelligence

Similar to Talk straps: Interactivity between Human and Artificial Intelligence (20)

More from Genoveva Vargas-Solar

More from Genoveva Vargas-Solar (10)

Recently uploaded

Recently uploaded (20)

Talk straps: Interactivity between Human and Artificial Intelligence