Relevance and Speed of Answers: How Can MAS Answering Systems Deal With That?

1. Relevance and Speed of Answers:
How Can MAS Answering Systems
Deal With That?
Albert Trias i Mansilla
Girona
12th july of 2011

2. Search Paradigms
• Library Paradigm:
– Search is based on Catalogues.
– Search results on published content.
– Trust is based on authority.
• Village Paradigm:
– People ask with natural language.
– Answers are generated in real-time by anyone in
the community.
– Trust is based on intimacy
– “It’s not what you know, but who you know”
[Nardi2000]
2

3. Village Paradigm
3
• C1: “The village paradigm (social search) has
some advantages in front of the library paradigm”
– Example: People can address new questions.
• C2: “Village Paradigm can be automated”
– Example: Recommender System.

4. Agents and Q&A Automation
4
• Agents are relevant for Social Search:
• Reactivity
• Sociability
• Proactivity
• Autonomy
• Agents enable the construction of information
systems from multiple heterogeneous sources
[Dignum2006]
• C3: “Agents are a natural approach for social
search automation”.

5. Automated Q&A in Agents
Social Network
5
• Approach
• P2P Social Network.
• Each user is represented by an agent.
• Each agent has a FAQ list.
• Agents Can:
• Send Questions (asker).
• Forward Questions (mediator).
• Answer Questions (answerer).

6. Automated Q&A in Agents
Social Network
6
• Related Work:
• MARS.
• P2P Multi-Agent Referral System.
• 6Search.
• P2P web search (bookmark based)
• BFS (Gnutella)
• TTL
• [Walter2008]
• Recommender System, filtering with trust using BFS.
• Trust Path (in base of trust transitivity)
• [Mychlmir2007].
• Query Routing in P2P
• Ant Optimization techniques.

7. 7
Question Waves

8. 8
Question Waves
Assumptions:
• Model does not consider context.
• Agents are homogeneous.
• Agents are benevolent and cooperative.
• Agents are always online.
• Answering time is constant.
• Static Social Network

9. 9
Question Waves
“T1: Answers relevancy (in village paradigm) is
correlated with answer time”
• A question wave is an attempt to find an answer to a
question.
• In every attempt, the same question is sent to a subset
of acquaintances.
• The expectancy of finding appropriate answers decays
after every attempt.
• In P2P, to request a question to all possible peers is not
feasible because it can overload the system.
• However, reducing the number of recipients too far can
provide the worst results.

10. 10
Question Waves
T=1
T=1T=0.7
T=0.7
Example:
Shelly
Bob
Dale
Gordon
Laura
Leo
T=0.2
0 1 5 6 11 40

11. 11
Evaluation
Simulations:
Get answers and sort by answer relevance, compare the
rankings using Spearman’s correlation
Agents use 4 waves: T={t1,t2,t3}
1st : after 1 simulation step; Trust > t1
2nd : after 5 simulation steps; t1 >Trust > t2
3rd : after 20 simulation steps; t2> Trust > t3
4th: after 40 simulation steps; t3>Trust > 0

12. 12
Results
• Correlation between answers sorted by relevance, and
sorted by the following heuristics:
• Answer Distance (D)
• Trust of the last sender (Tr).
• Receiving Order (H).
• Answer Distance and Trust (DT).
• Receiving Order and Trust (HT).
• Transitive Trust (TT).
• Trust of the Last Mediator (TLM).

13. 13
Results
• Heuristics Example
T=1
T=1T=0.7
T=0.7
Shelly
Bob
Dale
Gordon
Laura
Leo
T=0.2
0 1 5 6 11 40
Laura Gordon Bob
D 1 2 2
H 1 +1 6 +2 11 +2
Tr 1 0.7 (Dale) 0.7 (Dale)
TT 1 1* 0.7 0.7 * 0.7
TLM 1 1 0.7

14. 14
Evaluation
Ev(a) T D H DT HT Tr TT TLM 𝝑𝝑
mean .8,.7,.6 .14 .67 .17 .66 .14 .52 .9 .66
mean .85,.8,.7 .10 .49 .16 .48 .17 .56 .91 .68
mean .85,.75,.5 .11 .43 .16 .43 .19 .53 .91 .67
mean .85,.7,.5 .12 .56 .16 .55 .16 .52 .9 .67
max .8,.7,.6 .23 .7 .27 .69 .14 .53 .83 .72
max .85,.8,.7 .13 .62 .2 .61 .2 .57 .87 .73
max .85,.75,.5 .15 .6 .23 .59 .22 .58 .87 .72
max .85,.7,.5 .19 .67 .24 .65 .16 .56 .85 .72
Spearman’s Correlation

15. Evaluation
– Using Question Waves behavior and under
our assumptions, answer relevance is
correlated with answering time.
– Benefits:
• Relevant: answers come ranked
• Faster: reduce the burden of questions
• Robust: agents search answers persistently.
– Risks:
• Different point of view as answer quality.
• Trust is needed: Answering always the same is really fast.
15

16. 16
Discussion
Answer velocity is affected by:
• Answering time.
• Expertise (Algorithms with faster convergence)
• Effort (Example: numerical analysis, more iterations more
precision)
• State of answerer
• Automated or “Manual” answer?
• Implication: Most important tasks will be performed early.
• Communication time.
• Answering delay (time after receiving and before trying to
answer)
• Can MAS use answering time to consider answer relevance?
• Can MAS behavior be based on reciprocity?

17. Thank you very much for your
attention
17

18. 18
Evaluation

19. 19
Evaluation

20. Village Paradigm
• Proverbs:
– “A teacher is better than 2 books”
– “A library of books does not equal one good teacher”
• Researchers:
– Sometimes information only can be accessed asking the right
people [Yu2003].
– “It’s not what you know, but who you know” [Nardi2000]
20

21. Social Search
21
• Social search use social interactions (implicit or
explicit) to obtain results.
• (Chi, 2009) Social Search Engines can be
classified in:
– Social Feedback Systems. (Sorting results).
• Immediate Answer.
• Cannot adress new questions.
– Social Answering Systems. (People answers
questions)
• Can handle new questions.
• Answer not immediate
• Experts can get several times same question.

22. 22
Content
•Introduction
•Social Search
•Agents and Q&A Automation
•Automated Q&A in Agents Social Network
•Question Waves
•Evaluation
•Discussion and Future Work

23. 23
Introduction
•Centralized Search Engines provide generally
good results, but they go down with atypical
searches.
•Interest is Social Networking sites is growing.
•Researchers and Companies show interest in the
“village paradigm”.

24. Automated Q&A in Agents
Social Network
24
• Why?
• Reuse previous pairs of questions and answers.
• 30% of the time that a query was performed, it had been
carried out before by the same user. [Smyth2005]
• 70% of the time it was searched before by an acquaintance of
the user. [Smyth2005]

25. 25
Evaluation
set of agents A={a0 , a1, …, ai}, connected in a p2p social network
Method Step
For each Received Answers
If Own Question
Update result and Trust
Else
Forward it and update trust
If I have a new Own question
Select contacts in contact waves;
Program messages
For each received question
If I received the same question before
Ignore it
Else If I am good enough for answering,
Generate Answer Value; Send answer
Else
Select contacts in contacts waves;
Program messages
Send programmed messages

26. Ev(a) T D H DT HT Tr TT TLM 𝝑𝝑
mean .8,.7,.6 .12 .51 .12 .49 .10 .38 .72 .66
mean .85,.8,.7 .09 .37 .11 .34 .12 .41 .74 .68
mean .85,.75,.5 .1 .32 .11 .30 .14 .39 .74 .67
mean .85,.7,.5 .1 .42 .11 .39 .12 .38 .73 .67
max .8,.7,.6 .2 .54 .19 .52 .11 .39 .64 .72
max .85,.8,.7 .12 .47 .15 .44 .15 .41 .68 .73
max .85,.75,.5 .13 .46 .16 .43 .16 .42 .69 .72
max .85,.7,.5 .16 .52 .17 .48 .12 .41 .67 .72
26
Evaluation
Kendall’s Correlation