Previous studies on the practice of asking questions on social networking sites have shown that most questions remain unanswered and that most of the replies, if any, are only from members of the questioner's neighborhood. In this paper, we specifically consider the challenging task of solving a question posted on Twitter. The latter generally remains unanswered and most of the replies, if any, are only from members of the questioner's neighborhood. As outlined in previous work related to community Q\&A, we believe that question-answering is a collaborative process and that the relevant answer to a question post is an aggregation of answer nuggets posted by a group of relevant users. Thus, the problem of identifying the relevant answer turns into the problem of identifying the right group of users who would provide useful answers and would possibly be willing to collaborate together in the long-term. Accordingly, we present a novel method, called CRAQ, that is built on the collaboration paradigm and formulated as a group entropy optimization problem. To optimize the quality of the group, an information gain measure is used to select the most likely ``informative" users according to topical and collaboration likelihood predictive features. Crowd-based experiments performed on two crisis-related Twitter datasets demonstrate the effectiveness of our collaborative-based answering approach.

1. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
Answering Twitter Questions:
a Model for Recommending Answerers through Social
Collaboration
Laure Soulier
Pierre and Marie Curie University
LIP6, Paris - France
Lynda Tamine
Paul Sabatier University
IRIT, Toulouse - France
Gia-Hung Nguyen
Paul Sabatier University
IRIT, Toulouse - France
October 25, 2016
1 / 30

2. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
PLAN
1. Context and motivations
Social media-based information access
Collaboration and social media-based informationa access
2. Related Work
3. The CRAQ Model
4. Experimental Evaluation
5. Conclusion
2 / 30

3. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONTEXT AND MOTIVATIONS
SOCIAL MEDIA-BASED INFORMATION ACCESS
• Activity on social platforms
• Social networks: communication tool for the general public
3 / 30

4. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONTEXT AND MOTIVATIONS
SOCIAL MEDIA-BASED INFORMATION ACCESS
• Why choosing social platforms for asking questions?
4 / 30

5. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONTEXT AND MOTIVATIONS
SOCIAL MEDIA-BASED INFORMATION ACCESS
• Why choosing social platforms for asking questions?
Large audience and wide range of topics
[Harper et al., 2008, Jeong et al., 2013,
Tamine et al., 2016]
Specific audience, expertise → trust,
personalisation, and contextualisation
[Morris et al., 2010]
Friendsourcing through people addressing
(”@”, forward) [Liu and Jansen, 2013,
Teevan et al., 2011, Fuchs and Groh, 2015]
Communication, exchange, sensemaking
[Morris, 2013, Evans and Chi, 2010,
Tamine et al., 2016]
4 / 30

6. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONTEXT AND MOTIVATIONS
SOCIAL MEDIA-BASED INFORMATION ACCESS
• Why choosing social platforms for asking questions?
Large audience and wide range of topics
[Harper et al., 2008, Jeong et al., 2013,
Tamine et al., 2016]
Specific audience, expertise → trust,
personalisation, and contextualisation
[Morris et al., 2010]
Friendsourcing through people addressing
(”@”, forward) [Liu and Jansen, 2013,
Teevan et al., 2011, Fuchs and Groh, 2015]
Communication, exchange, sensemaking
[Morris, 2013, Evans and Chi, 2010,
Tamine et al., 2016]
• Limitations of social platforms
4 / 30

7. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONTEXT AND MOTIVATIONS
SOCIAL MEDIA-BASED INFORMATION ACCESS
• Why choosing social platforms for asking questions?
Large audience and wide range of topics
[Harper et al., 2008, Jeong et al., 2013,
Tamine et al., 2016]
Specific audience, expertise → trust,
personalisation, and contextualisation
[Morris et al., 2010]
Friendsourcing through people addressing
(”@”, forward) [Liu and Jansen, 2013,
Teevan et al., 2011, Fuchs and Groh, 2015]
Communication, exchange, sensemaking
[Morris, 2013, Evans and Chi, 2010,
Tamine et al., 2016]
• Limitations of social platforms
Majority of questions without response
[Jeong et al., 2013, Paul et al., 2011]
Answers mostly provided by members of the
immediate follower network
[Morris et al., 2010, Rzeszotarski et al., 2014]
Social and cognitive cost of friendsourcing
(e.g., spent time and deployed effort)
[Horowitz and Kamvar, 2010, Morris, 2013].
4 / 30

8. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONTEXT AND MOTIVATIONS
SOCIAL MEDIA-BASED INFORMATION ACCESS
• Why choosing social platforms for asking questions?
Large audience and wide range of topics
[Harper et al., 2008, Jeong et al., 2013,
Tamine et al., 2016]
Specific audience, expertise → trust,
personalisation, and contextualisation
[Morris et al., 2010]
Friendsourcing through people addressing
(”@”, forward) [Liu and Jansen, 2013,
Teevan et al., 2011, Fuchs and Groh, 2015]
Communication, exchange, sensemaking
[Morris, 2013, Evans and Chi, 2010,
Tamine et al., 2016]
• Limitations of social platforms
Majority of questions without response
[Jeong et al., 2013, Paul et al., 2011]
Answers mostly provided by members of the
immediate follower network
[Morris et al., 2010, Rzeszotarski et al., 2014]
Social and cognitive cost of friendsourcing
(e.g., spent time and deployed effort)
[Horowitz and Kamvar, 2010, Morris, 2013].
Design implications
• Enhancement of social awareness (creating social ties to active/relevant users)
• Recommendation of collaborators (asking questions to crowd instead of followers)
4 / 30

9. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONTEXT AND MOTIVATIONS
COLLABORATION AND SOCIAL MEDIA-BASED INFORMATION ACCESS: TWO SIDE IN THE SAME COIN?
• Social media-based information access
Seeking, answering, sharing,
bookmarking, and spreading information
Improving the search outcomes through
social interactions
• Collaboration
Identifying and solving a shared
complex problem
Creating and sharing knowledge within
a work team
5 / 30

10. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONTEXT AND MOTIVATIONS
COLLABORATION AND SOCIAL MEDIA-BASED INFORMATION ACCESS: TWO SIDE IN THE SAME COIN?
• Social media-based information access
Seeking, answering, sharing,
bookmarking, and spreading information
Improving the search outcomes through
social interactions
• Collaboration
Identifying and solving a shared
complex problem
Creating and sharing knowledge within
a work team
• Social media-based collaboration
Leveraging from the ”wisdom of the
crowd”
Implicit or explicit intents (sharing,
questioning, and/or answering)
Tasks: social question-answering, social
search, real-time search
5 / 30

11. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONTEXT AND MOTIVATIONS
COLLABORATION AND SOCIAL MEDIA-BASED INFORMATION ACCESS: TWO SIDE IN THE SAME COIN?
• Social media-based information access
Seeking, answering, sharing,
bookmarking, and spreading information
Improving the search outcomes through
social interactions
• Collaboration
Identifying and solving a shared
complex problem
Creating and sharing knowledge within
a work team
• Social media-based collaboration
Leveraging from the ”wisdom of the
crowd”
Implicit or explicit intents (sharing,
questioning, and/or answering)
Tasks: social question-answering, social
search, real-time search
Our contribution
• Identifying a group of socially authoritative users with complementary skills to overpass the local
social network
• Gathering diverse pieces of information
→ Recommending a group of collaborators
5 / 30

12. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
PLAN
1. Context and motivations
2. Related Work
Pioneering work
Comparison of previous work
3. The CRAQ Model
4. Experimental Evaluation
5. Conclusion
6 / 30

13. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
RELATED WORK
PIONEERING WORK: AARDVARK [HOROWITZ AND KAMVAR, 2010]
Aardvark [Horowitz and Kamvar, 2010]
• The village paradigm: towards a social dissemination of knowledge
Information is passed from person to person
Finding the right person rather than the right document
7 / 30

14. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
RELATED WORK
PIONEERING WORK: SEARCHBUDDIES [HECHT ET AL., 2012]
SearchBuddies [Hecht et al., 2012]
• A crowd-powered socially embedded search engine
• Leveraging users’ personal network to reach the right people/information
• Soshul Butterflie: Recommending people • Investigaetore: Recommending urls
8 / 30

15. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
RELATED WORK
COMPARISON OF PREVIOUS APPROACHES
Previous work
Expertise/interest
Responsiveness
Socialactivity
U
sers’connectedness
C
om
patibility
O
ptim
ization
ofthe
outcom
e
C
om
plem
entarity
of
users’skills
Reco.users
Expert finding [Balog et al., 2012]
Authoritative users/influencers
[Pal and Counts, 2011]
Aardvark
[Horowitz and Kamvar, 2010]
SearchBuddies [Hecht et al., 2012]
Mentionning users/spreaders
[Wang et al., 2013, Gong et al., 2015]
Reco.groupofusers
CrowdStar [Nushi et al., 2015]
Question routing for collab.
Q&A[Chang and Pal, 2013]
Recommended targeted stranger
[Mahmud et al., 2013]
Crowdworker
[Ranganath et al., 2015]
Our work
9 / 30

16. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
PLAN
1. Context and motivations
2. Related Work
3. The CRAQ Model
Overview
Learning the pairwise collaboration likelihood
Building the collaborative group of users
4. Experimental Evaluation
5. Conclusion
10 / 30

17. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
THE CRAQ MODEL: ANSWERING TWITTER QUESTIONS THROUGH A
COLLABORATIVE GROUP RECOMMENDATION ALGORITHM
OVERVIEW
• Indentifying a group
of complementary
answerers who could
provide the questioner
with a cohesive and
relevant answer.
• Gathering diverse
pieces of information
posted by users
• Maximization of the
group entropy
11 / 30

18. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
THE CRAQ MODEL: ANSWERING TWITTER QUESTIONS THROUGH A
COLLABORATIVE GROUP RECOMMENDATION ALGORITHM
STAGE A: LEARNING THE PAIRWISE COLLABORATION LIKELIHOOD
Objective
Estimating the potential of collaboration between a pair of users
• Hypotheses:
On Twitter, collaboration between users is noted by the @ symbol
[Ehrlich and Shami, 2010, Honey and Herring, 2009]
Trust and authority enable to improve the effectiveness of the collaboration
[McNally et al., 2013]
Collaboration is a structured search process in which users might or might not be
complementary [Sonnenwald et al., 2004, Soulier et al., 2014]
12 / 30

19. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
THE CRAQ MODEL: ANSWERING TWITTER QUESTIONS THROUGH A
COLLABORATIVE GROUP RECOMMENDATION ALGORITHM
STAGE A: LEARNING THE PAIRWISE COLLABORATION LIKELIHOOD
Objective
Estimating the potential of collaboration between a pair of users
• Hypotheses:
On Twitter, collaboration between users is noted by the @ symbol
[Ehrlich and Shami, 2010, Honey and Herring, 2009]
Trust and authority enable to improve the effectiveness of the collaboration
[McNally et al., 2013]
Collaboration is a structured search process in which users might or might not be
complementary [Sonnenwald et al., 2004, Soulier et al., 2014]
12 / 30

20. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
THE CRAQ MODEL: ANSWERING TWITTER QUESTIONS THROUGH A
COLLABORATIVE GROUP RECOMMENDATION ALGORITHM
STAGE A: LEARNING THE PAIRWISE COLLABORATION LIKELIHOOD
Objective
Estimating the potential of collaboration between a pair of users
• Hypotheses:
On Twitter, collaboration between users is noted by the @ symbol
[Ehrlich and Shami, 2010, Honey and Herring, 2009]
Trust and authority enable to improve the effectiveness of the collaboration
[McNally et al., 2013]
Collaboration is a structured search process in which users might or might not be
complementary [Sonnenwald et al., 2004, Soulier et al., 2014]
12 / 30

21. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
THE CRAQ MODEL: ANSWERING TWITTER QUESTIONS THROUGH A
COLLABORATIVE GROUP RECOMMENDATION ALGORITHM
STAGE B: BUILDING THE COLLABORATIVE GROUP OF USERS
Objective
Building the smallest group of collaborators maximizing the cohesiveness and relevance of the
collaborative response
• Identifying candidate collaborators through a temporal ranking model
[Berberich and Bedathur, 2013]
13 / 30

22. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
THE CRAQ MODEL: ANSWERING TWITTER QUESTIONS THROUGH A
COLLABORATIVE GROUP RECOMMENDATION ALGORITHM
STAGE B: BUILDING THE COLLABORATIVE GROUP OF USERS
Objective
Building the smallest group of collaborators maximizing the cohesiveness and relevance of the
collaborative response
• Extracting the collaborator group
Maximizing entropy equivalent to minimizing the information gain [Quinlan, 1986]
IG(g, uk) = [H(g)
↓
H(g) ∝ −
uj∈g
P(uj|q) · log(P(uj|q))
− H(g|uk)
↓
H(g|uk) = p(uk) · [−
uj∈g
uj=uk
P(uj|uk) · log(P(uj|uk))]
] (1)
Recursive decrementation of candidate collaborators through the information gain metric
t
∗
= arg max
t∈[0,...,|U|−1]
∂2IGr(gt,u)
∂u2 |u=ut (2)
Given u
t
= argminu
j
∈gt IGr(gt
, uj )
And g
t+1
= gt
ut
14 / 30

23. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
PLAN
1. Context and motivations
2. Related Work
3. The CRAQ Model
4. Experimental Evaluation
Evaluation Protocol
Results
5. Conclusion
15 / 30

24. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
EVALUATION PROTOCOL
Evaluation objectives
• RQ1: Do the tweets posted by the collaborative group members recommended by the CRAQ allow
the building of an answer?
• RQ2: Are the recommended group-based answers relevant?
• RQ3: What is the synergic effect of the CRAQ-based collaborative answering methodology?
• Datasets
1 Hurricane #Sandy
(October 2012)
2 #Ebola virus epidemic
(2013-2014)
Collection Sandy Ebola
Tweets 2,119,854 2,872,890
Microbloggers 1,258,473 750,829
Retweets 963,631 1,157,826
Mentions 1,473,498 1,826,059
Reply 63,596 69,773
URLs 596,393 1,309,919
Pictures 107,263 310,581
16 / 30

25. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
EVALUATION PROTOCOL
• Question identification [Jeong et al., 2013]
Filtering tweets ending with a question mark
Excluding mention tweets and tweets including URLs
Filtering tweets with question-oriented hashtags [Jeffrey M. Rzeszotarski, 2014] (e.g., #help,
#askquestion, ...)
Excluding rhetorical questions (Crowdflower)
Sandy 41 questions Would love to #help to clear up the mess #Sandy made. Any way
people can help? Voluntery groups?
Ebola 21 questions How do you get infected by this Ebola virus though?? #Twoogle
17 / 30

26. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
EVALUATION PROTOCOL
• Question identification [Jeong et al., 2013]
Filtering tweets ending with a question mark
Excluding mention tweets and tweets including URLs
Filtering tweets with question-oriented hashtags [Jeffrey M. Rzeszotarski, 2014] (e.g., #help,
#askquestion, ...)
Excluding rhetorical questions (Crowdflower)
Sandy 41 questions Would love to #help to clear up the mess #Sandy made. Any way
people can help? Voluntery groups?
Ebola 21 questions How do you get infected by this Ebola virus though?? #Twoogle
• Collaboration likelihood features
Name Description
Authority
Importance Number of followers
Number of followings
Number of favorites
Engagement Number of tweets
Activity Number of topically-related tweets
within the In-degree in the topic
topic Out-degree in the topic
Complementarity
Topic Jansen-Shanon distance between topical-
based representation of users’ interests
obtained through the LDA algorithm
Multimedia Number of tweets with video
Number of tweets with images
Number of tweets with links
Number of tweets with hashtags
Number of tweets with only text
Opinion Number of tweets with positive opinion
polarity Number of tweets with neutral opinion
Authority-based features (trust and
expertise of each user)
Xjj = log(
µ(Xj, Xj )
σ(Xj, Xj )
) (3)
Complementarity-based features
(complementairty of collaborators)
Xjj =
|Xj − Xj |
Xj + Xj
(4)
17 / 30

27. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
EVALUATION PROTOCOL
• Baselines
MMR: diversity-based ranking model of tweets [Carbonell and Goldstein, 1998] (tweet level)
U: best user of the temporal ranking model [Berberich and Bedathur, 2013] (individual)
CRAQ-TR: CRAQ w/o group entropy maximization
SM: community detection algorithm based on the graph structure [Cao et al., 2015]
STM: Topical Sensitive PageRank applied on users [Haveliwala, 2002]
• Evaluation workflow
Question +
Tweets of users
Evaluating tweets
and building an answer
Assessing
the relevance
of users’ tweets
of built answers
Ground truth
18 / 30

28. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
EVALUATION PROTOCOL
• Baselines
MMR: diversity-based ranking model of tweets [Carbonell and Goldstein, 1998] (tweet level)
U: best user of the temporal ranking model [Berberich and Bedathur, 2013] (individual)
CRAQ-TR: CRAQ w/o group entropy maximization
SM: community detection algorithm based on the graph structure [Cao et al., 2015]
STM: Topical Sensitive PageRank applied on users [Haveliwala, 2002]
• Evaluation workflow
Question +
Tweets of users
Evaluating tweets
and building an answer
Assessing
the relevance
of users’ tweets
of built answers
Ground truth
Question Top ranked tweets of recommended group members Answer built by the crowd
Would love to
#help to clear up
the mess #Sandy
made. Any way
people can help?
Voluntery groups?
- My prayers go out to those people out there that have been
affected by the storm. #Sandy
- Makes me want to volunteer myself and help the Red Cross and
rescue groups.#Sandy
- Rescue groups are organized and dispatched to help animals in
Sandy’s aftermath. You can help by donating. #SandyPets
- ASPCA, HSUS, American Humane are among groups on the
ground helping animals in Sandy’s aftermath. Help them with a
donation. #SandyPets #wlf
Rescue groups are organized and dis-
patched ASPCA, HSUS, American Hu-
mane, Donate to @RedCross, @Hu-
maneSociety, @ASPCA.
18 / 30

29. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
EVALUATION PROTOCOL
• Baselines
MMR: diversity-based ranking model of tweets [Carbonell and Goldstein, 1998] (tweet level)
U: best user of the temporal ranking model [Berberich and Bedathur, 2013] (individual)
CRAQ-TR: CRAQ w/o group entropy maximization
SM: community detection algorithm based on the graph structure [Cao et al., 2015]
STM: Topical Sensitive PageRank applied on users [Haveliwala, 2002]
• Evaluation workflow
Question +
Tweets of users
Evaluating tweets
and building an answer
Assessing
the relevance
of users’ tweets
of built answers
Ground truth
18 / 30

30. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ1: Do the tweets posted by the collaborative group members recommended by the CRAQ allow the
building of an answer?
• Testing whether the CRAQ is effective in providing useful tweets in terms of relatedness to the
question topic and complementarity.
0 1 2 3
10
20
30
40
50
Sandy
0 1 2 3
0
10
20
30
40
50
60
70
Ebola
MMR U CRAQ-TR SM STM CRAQ
• Lowest rate for the Not related category (0)
19 / 30

31. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ1: Do the tweets posted by the collaborative group members recommended by the CRAQ allow the
building of an answer?
• Testing whether the CRAQ is effective in providing useful tweets in terms of relatedness to the
question topic and complementarity.
0 1 2 3
10
20
30
40
50
Sandy
0 1 2 3
0
10
20
30
40
50
60
70
Ebola
MMR U CRAQ-TR SM STM CRAQ
• Lowest rate for the Not related category (0)
• Highest proportion of 2+3 Related and helpful
19 / 30

32. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ1: Do the tweets posted by the collaborative group members recommended by the CRAQ allow the
building of an answer?
• Testing whether the CRAQ is effective in providing useful tweets in terms of relatedness to the
question topic and complementarity.
0 1 2 3
10
20
30
40
50
Sandy
0 1 2 3
0
10
20
30
40
50
60
70
Ebola
MMR U CRAQ-TR SM STM CRAQ
• Lowest rate for the Not related category (0)
• Highest proportion of 2+3 Related and helpful
• Complementarity of tweets is not satisfying w.r.t. baselines
Negative regression estimate of complementarity-based features in the collaboration
likelihood model - phase A
19 / 30

33. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ1: Do the tweets posted by the collaborative group members recommended by the CRAQ allow the
building of an answer?
• Testing whether the CRAQ is effective in providing useful tweets in terms of relatedness to the
question topic and complementarity.
• Testing whether those provided tweets allow building a cohesive answer.
Sandy Ebola
10
15
20
25
30
35
Avg Percentage of selected tweets
Sandy Ebola
10
20
30
40
50
60
70
80
Number of built answers
MMR U CRAQ-TR SM STM CRAQ
• U: highest rate of selected tweets / lowest number of built answers
20 / 30

34. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ1: Do the tweets posted by the collaborative group members recommended by the CRAQ allow the
building of an answer?
• Testing whether the CRAQ is effective in providing useful tweets in terms of relatedness to the
question topic and complementarity.
• Testing whether those provided tweets allow building a cohesive answer.
Sandy Ebola
10
15
20
25
30
35
Avg Percentage of selected tweets
Sandy Ebola
10
20
30
40
50
60
70
80
Number of built answers
MMR U CRAQ-TR SM STM CRAQ
• U: highest rate of selected tweets / lowest number of built answers
• CRAQ: Lack of tweet complementarity does not impact the ability of the
recommended group to answer the query
20 / 30

35. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ2: Are the recommended group-based answers relevant?
• Testing the relevance of the built answers
MMR U CRAQ-TR SM STM CRAQ
Sandy
ba 43 29 75 74 67 77
1 11: 25.58% 9: 31.03% 23: 30.67% 24: 32.43% 21: 31.34% 17: 22.08%
2 20: 46.51% 14: 48.28% 33: 44.00% 34: 45.95% 24: 35.82% 39: 50.65%
3 12: 27.91% 6: 20.69% 19: 25.33% 16: 21.62% 22: 32.84% 21: 27.27%
2+3 32: 74.42% 20: 68.97% 52: 69.33% 50: 67.57% 46: 68.66% 60: 77.92%
Ebola
ba 22 11 39 30 37 41
1 4: 21.05% 3: 27.27% 15: 38.46% 8: 26.67% 15: 40.54% 10: 24.39%
2 6: 31.58% 4: 36.36% 18: 46.15% 15: 50% 16: 43.24% 22: 53.66%
3 9: 47.37% 4: 36.36% 6: 15.38% 7: 23.33% 6: 16.22% 9: 21.95%
2+3 15: 78.95% 8: 72.72% 24: 1.53% 22: 73.33% 22:59.46% 31: 75.61%
• CRAQ enables to build a higher number of answers, among them a higher proportion
of Partly relevant and Relevant answers
U: reinforces our intuition that a single user might have an insufficient knowledge (even if
related) to solve a tweeted question.
21 / 30

36. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ2: Are the recommended group-based answers relevant?
• Testing the relevance of the built answers
MMR U CRAQ-TR SM STM CRAQ
Sandy
ba 43 29 75 74 67 77
1 11: 25.58% 9: 31.03% 23: 30.67% 24: 32.43% 21: 31.34% 17: 22.08%
2 20: 46.51% 14: 48.28% 33: 44.00% 34: 45.95% 24: 35.82% 39: 50.65%
3 12: 27.91% 6: 20.69% 19: 25.33% 16: 21.62% 22: 32.84% 21: 27.27%
2+3 32: 74.42% 20: 68.97% 52: 69.33% 50: 67.57% 46: 68.66% 60: 77.92%
Ebola
ba 22 11 39 30 37 41
1 4: 21.05% 3: 27.27% 15: 38.46% 8: 26.67% 15: 40.54% 10: 24.39%
2 6: 31.58% 4: 36.36% 18: 46.15% 15: 50% 16: 43.24% 22: 53.66%
3 9: 47.37% 4: 36.36% 6: 15.38% 7: 23.33% 6: 16.22% 9: 21.95%
2+3 15: 78.95% 8: 72.72% 24: 1.53% 22: 73.33% 22:59.46% 31: 75.61%
• CRAQ enables to build a higher number of answers, among them a higher proportion
of Partly relevant and Relevant answers
U: reinforces our intuition that a single user might have an insufficient knowledge (even if
related) to solve a tweeted question.
MMR: gives rise to the benefit of building answers from the users perspective rather than the
tweets regardless of their context.
21 / 30

37. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ2: Are the recommended group-based answers relevant?
• Testing the relevance of the built answers
MMR U CRAQ-TR SM STM CRAQ
Sandy
ba 43 29 75 74 67 77
1 11: 25.58% 9: 31.03% 23: 30.67% 24: 32.43% 21: 31.34% 17: 22.08%
2 20: 46.51% 14: 48.28% 33: 44.00% 34: 45.95% 24: 35.82% 39: 50.65%
3 12: 27.91% 6: 20.69% 19: 25.33% 16: 21.62% 22: 32.84% 21: 27.27%
2+3 32: 74.42% 20: 68.97% 52: 69.33% 50: 67.57% 46: 68.66% 60: 77.92%
Ebola
ba 22 11 39 30 37 41
1 4: 21.05% 3: 27.27% 15: 38.46% 8: 26.67% 15: 40.54% 10: 24.39%
2 6: 31.58% 4: 36.36% 18: 46.15% 15: 50% 16: 43.24% 22: 53.66%
3 9: 47.37% 4: 36.36% 6: 15.38% 7: 23.33% 6: 16.22% 9: 21.95%
2+3 15: 78.95% 8: 72.72% 24: 1.53% 22: 73.33% 22:59.46% 31: 75.61%
• CRAQ enables to build a higher number of answers, among them a higher proportion
of Partly relevant and Relevant answers
U: reinforces our intuition that a single user might have an insufficient knowledge (even if
related) to solve a tweeted question.
MMR: gives rise to the benefit of building answers from the users perspective rather than the
tweets regardless of their context.
CRAQ-TR (best baseline): building a group by gathering individual users identified as
relevant through their skills (tweet topical similarity with the question) is not always
appropriate.
21 / 30

38. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ3: What is the synergic effect of the CRAQ-based collaborative answering methodology?
• Measuring the synergic effect of simulated collaboration within the recommended groups of
users with respect to the search effectiveness based on the tweets published by those group
members.
MMR U CRAQ-TR SM STM CRAQ
Value %Chg Value %Chg Value %Chg Value %Chg Value %Chg Value
Sandy
Precision 0.24 +92.93** 0.46 +2.32 0.33 +42.01* 0.21 +124.71***0.49 -4.1 0.47
Recall 0.09 +95.19* 0.1 +81.63* 0.15 +16.18 0.09 +105.96* 0.1 +80.09* 0.18
F-measure 0.12 +78.22* 0.15 +41.79 0.19 +10.59 0.12 +84.42* 0.15 +40.48 0.21
Ebola
Precision 0.22 +153.65***0.64 -12.12 0.5 +12.22 0.3 +89.59** 0.45 +24.5 0.57
Recall 0.07 +155.59***0.11 +69.96* 0.22 -18.08 0.12 +46.80 0.06 +216.56***0.18
F-measure 0.09 +164.17***0.21 +17.46 0.28 -11.64 0.17 +50.07 0.1 +159.07***0.25
• MMR: sustains observed analysis on the lack of user context
22 / 30

39. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ3: What is the synergic effect of the CRAQ-based collaborative answering methodology?
• Measuring the synergic effect of simulated collaboration within the recommended groups of
users with respect to the search effectiveness based on the tweets published by those group
members.
MMR U CRAQ-TR SM STM CRAQ
Value %Chg Value %Chg Value %Chg Value %Chg Value %Chg Value
Sandy
Precision 0.24 +92.93** 0.46 +2.32 0.33 +42.01* 0.21 +124.71***0.49 -4.1 0.47
Recall 0.09 +95.19* 0.1 +81.63* 0.15 +16.18 0.09 +105.96* 0.1 +80.09* 0.18
F-measure 0.12 +78.22* 0.15 +41.79 0.19 +10.59 0.12 +84.42* 0.15 +40.48 0.21
Ebola
Precision 0.22 +153.65***0.64 -12.12 0.5 +12.22 0.3 +89.59** 0.45 +24.5 0.57
Recall 0.07 +155.59***0.11 +69.96* 0.22 -18.08 0.12 +46.80 0.06 +216.56***0.18
F-measure 0.09 +164.17***0.21 +17.46 0.28 -11.64 0.17 +50.07 0.1 +159.07***0.25
• MMR: sustains observed analysis on the lack of user context
• U: consistent with previous work highlighting the synergic effect of a collaborative
group
22 / 30

40. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ3: What is the synergic effect of the CRAQ-based collaborative answering methodology?
• Measuring the synergic effect of simulated collaboration within the recommended groups of
users with respect to the search effectiveness based on the tweets published by those group
members.
MMR U CRAQ-TR SM STM CRAQ
Value %Chg Value %Chg Value %Chg Value %Chg Value %Chg Value
Sandy
Precision 0.24 +92.93** 0.46 +2.32 0.33 +42.01* 0.21 +124.71***0.49 -4.1 0.47
Recall 0.09 +95.19* 0.1 +81.63* 0.15 +16.18 0.09 +105.96* 0.1 +80.09* 0.18
F-measure 0.12 +78.22* 0.15 +41.79 0.19 +10.59 0.12 +84.42* 0.15 +40.48 0.21
Ebola
Precision 0.22 +153.65***0.64 -12.12 0.5 +12.22 0.3 +89.59** 0.45 +24.5 0.57
Recall 0.07 +155.59***0.11 +69.96* 0.22 -18.08 0.12 +46.80 0.06 +216.56***0.18
F-measure 0.09 +164.17***0.21 +17.46 0.28 -11.64 0.17 +50.07 0.1 +159.07***0.25
• MMR: sustains observed analysis on the lack of user context
• U: consistent with previous work highlighting the synergic effect of a collaborative
group
• CRAQ-TR: no significant differences in effectiveness / lower ratio of relevant answers.
Benefit of the group entropy maximization based on the collaboration likelihood.
22 / 30

41. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ3: What is the synergic effect of the CRAQ-based collaborative answering methodology?
• Measuring the synergic effect of simulated collaboration within the recommended groups of
users with respect to the search effectiveness based on the tweets published by those group
members.
MMR U CRAQ-TR SM STM CRAQ
Value %Chg Value %Chg Value %Chg Value %Chg Value %Chg Value
Sandy
Precision 0.24 +92.93** 0.46 +2.32 0.33 +42.01* 0.21 +124.71***0.49 -4.1 0.47
Recall 0.09 +95.19* 0.1 +81.63* 0.15 +16.18 0.09 +105.96* 0.1 +80.09* 0.18
F-measure 0.12 +78.22* 0.15 +41.79 0.19 +10.59 0.12 +84.42* 0.15 +40.48 0.21
Ebola
Precision 0.22 +153.65***0.64 -12.12 0.5 +12.22 0.3 +89.59** 0.45 +24.5 0.57
Recall 0.07 +155.59***0.11 +69.96* 0.22 -18.08 0.12 +46.80 0.06 +216.56***0.18
F-measure 0.09 +164.17***0.21 +17.46 0.28 -11.64 0.17 +50.07 0.1 +159.07***0.25
• MMR: sustains observed analysis on the lack of user context
• U: consistent with previous work highlighting the synergic effect of a collaborative
group
• CRAQ-TR: no significant differences in effectiveness / lower ratio of relevant answers.
Benefit of the group entropy maximization based on the collaboration likelihood.
• SM: benefit of overpassing strong ties (users’ local network) to select relevant strangers
22 / 30

42. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
EXPERIMENTAL EVALUATION
RESULTS
RQ3: What is the synergic effect of the CRAQ-based collaborative answering methodology?
• Measuring the synergic effect of simulated collaboration within the recommended groups of
users with respect to the search effectiveness based on the tweets published by those group
members.
MMR U CRAQ-TR SM STM CRAQ
Value %Chg Value %Chg Value %Chg Value %Chg Value %Chg Value
Sandy
Precision 0.24 +92.93** 0.46 +2.32 0.33 +42.01* 0.21 +124.71***0.49 -4.1 0.47
Recall 0.09 +95.19* 0.1 +81.63* 0.15 +16.18 0.09 +105.96* 0.1 +80.09* 0.18
F-measure 0.12 +78.22* 0.15 +41.79 0.19 +10.59 0.12 +84.42* 0.15 +40.48 0.21
Ebola
Precision 0.22 +153.65***0.64 -12.12 0.5 +12.22 0.3 +89.59** 0.45 +24.5 0.57
Recall 0.07 +155.59***0.11 +69.96* 0.22 -18.08 0.12 +46.80 0.06 +216.56***0.18
F-measure 0.09 +164.17***0.21 +17.46 0.28 -11.64 0.17 +50.07 0.1 +159.07***0.25
• MMR: sustains observed analysis on the lack of user context
• U: consistent with previous work highlighting the synergic effect of a collaborative
group
• CRAQ-TR: no significant differences in effectiveness / lower ratio of relevant answers.
Benefit of the group entropy maximization based on the collaboration likelihood.
• SM: benefit of overpassing strong ties (users’ local network) to select relevant strangers
• STM: topically relevant tweets issued from the most socially authoritative are not
obviously relevant to answer the tweeted question.
22 / 30

43. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
PLAN
1. Context and motivations
2. Related Work
3. The CRAQ Model
4. Experimental Evaluation
5. Conclusion
23 / 30

44. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONCLUSION AND PERSPECTIVES
Discussion
• Novel method for answering questions on social networks: recommending a group of
socially active and complementary collaborators.
• Relevant factors:
Information gain provided by a user to the group
Complementarity and topical relevance of the related tweets
Trust and authority of the group members
• Method applicable for other social platforms (Facebook, community Q&A sites, ...)
24 / 30

45. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
CONCLUSION AND PERSPECTIVES
Discussion
• Novel method for answering questions on social networks: recommending a group of
socially active and complementary collaborators.
• Relevant factors:
Information gain provided by a user to the group
Complementarity and topical relevance of the related tweets
Trust and authority of the group members
• Method applicable for other social platforms (Facebook, community Q&A sites, ...)
Future Directions
• Limitation of the predictive model of collaboration likelihood relying on basic
assumptions of collaborations (mentions, replies, retweets).
Deeper analysis of collaboration behavior on social networks to identify collaboration
patterns.
• Automatic summarization of candidate answers to build a collaborative answer.
24 / 30

46. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
THANK YOU!
@LaureSoulier @LyndaTamine @ngiahung
25 / 30

47. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
REFERENCES I
Balog, K., Fang, Y., de Rijke, M., Serdyukov, P., and Si, L. (2012).
Expertise retrieval.
Foundations and Trends in Information Retrieval, 6(2-3):127–256.
Berberich, K. and Bedathur, S. (2013).
Temporal Diversification of Search Results.
In SIGIR #TAIA workshop. ACM.
Cao, C., Caverlee, J., Lee, K., Ge, H., and Chung, J. (2015).
Organic or organized?: Exploring URL sharing behavior.
In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 513–522.
Carbonell, J. and Goldstein, J. (1998).
The use of MMR, diversity-based reranking for reordering documents and producing summaries.
In Proceedings of the Annual International SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’98, pages 335–336.
ACM.
Chang, S. and Pal, A. (2013).
Routing questions for collaborative answering in community question answering.
In ASONAM ’13, pages 494–501. ACM.
Ehrlich, K. and Shami, N. S. (2010).
Microblogging inside and outside the workplace.
In Proceedings of the Fourth International Conference on Weblogs and Social Media, ICWSM 2010.
Evans, B. M. and Chi, E. H. (2010).
An elaborated model of social search.
Information Processing & Management (IP&M), 46(6):656–678.
26 / 30

48. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
REFERENCES II
Fuchs, C. and Groh, G. (2015).
Appropriateness of search engines, social networks, and directly approaching friends to satisfy information needs.
In Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2015, pages
1248–1253.
Gong, Y., Zhang, Q., Sun, X., and Huang, X. (2015).
Who will you ”@”?
pages 533–542. ACM.
Harper, F. M., Raban, D. R., Rafaeli, S., and Konstan, J. A. (2008).
Predictors of answer quality in online q&a sites.
In Proceedings of the 2008 Conference on Human Factors in Computing Systems, CHI 2008, 2008, Florence, Italy, April 5-10, 2008, pages 865–874.
Haveliwala, T. H. (2002).
Topic-sensitive PageRank.
In Proceedings of the International Conference on World Wide Web, WWW ’02, pages 517–526. ACM.
Hecht, B., Teevan, J., Morris, M. R., and Liebling, D. J. (2012).
Searchbuddies: Bringing search engines into the conversation.
In WSDM ’14.
Honey, C. and Herring, S. (2009).
Beyond Microblogging: Conversation and Collaboration via Twitter.
In HICSS, pages 1–10.
Horowitz, D. and Kamvar, S. D. (2010).
The Anatomy of a Large-scale Social Search Engine.
In WWW ’10, pages 431–440. ACM.
27 / 30

49. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
REFERENCES III
Jeffrey M. Rzeszotarski, M. R. M. (2014).
Estimating the social costs of friendsourcing.
In Proceedings of CHI 2014. ACM.
Jeong, J.-W., Morris, M. R., Teevan, J., and Liebling, D. (2013).
A crowd-powered socially embedded search engine.
In ICWSM ’13. AAAI.
Liu, Z. and Jansen, B. J. (2013).
Factors influencing the response rate in social question and answering behavior.
In Computer Supported Cooperative Work, CSCW 2013, pages 1263–1274.
Mahmud, J., Zhou, M. X., Megiddo, N., Nichols, J., and Drews, C. (2013).
Recommending targeted strangers from whom to solicit information on social media.
In IUI ’13, pages 37–48. ACM.
McNally, K., O’Mahony, M. P., and Smyth, B. (2013).
A model of collaboration-based reputation for the social web.
In ICWSM.
Morris, M. R. (2013).
Collaborative Search Revisited.
In Proceedings of the Conference on Computer Supported Cooperative Work, CSCW ’13, pages 1181–1192. ACM.
Morris, M. R., Teevan, J., and Panovich, K. (2010).
What do people ask their social networks, and why?: a survey study of status message q&a behavior.
In Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, pages 1739–1748.
28 / 30

50. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
REFERENCES IV
Nushi, B., Alonso, O., Hentschel, M., and Kandylas, V. (2015).
Crowdstar: A social task routing framework for online communities.
In ICWE ’15, pages 219–230.
Pal, A. and Counts, S. (2011).
Identifying topical authorities in microblogs.
In WSDM ’11, pages 45–54. ACM.
Paul, S. A., Hong, L., and Chi, E. H. (2011).
Is twitter a good place for asking questions? A characterization study.
In Proceedings of the Fifth International Conference on Weblogs and Social Media.
Quinlan, J. (1986).
Induction of decision trees.
Machine Learning, 1(1):81–106.
Ranganath, S., Wang, S., Hu, X., Tang, J., and Liu, H. (2015).
Finding time-critical responses for information seeking in social media.
In 2015 IEEE International Conference on Data Mining, ICDM 2015, pages 961–966.
Rzeszotarski, J. M., Spiro, E. S., Matias, J. N., Monroy-Hern´andez, A., and Morris, M. R. (2014).
Is anyone out there?: unpacking q&a hashtags on twitter.
In CHI Conference on Human Factors in Computing Systems, CHI’14, pages 2755–2758.
Sonnenwald, D. H., Maglaughlin, K. L., and Whitton, M. C. (2004).
Designing to support situation awareness across distances: an example from a scientific collaboratory.
Information Processing & Management (IP&M), 40(6):989–1011.
29 / 30

51. 1. Context and motivations 2. Related Work 3. The CRAQ Model 4. Experimental Evaluation 5. Conclusion
REFERENCES V
Soulier, L., Shah, C., and Tamine, L. (2014).
User-driven System-mediated Collaborative Information Retrieval.
In SIGIR ’14, pages 485–494. ACM.
Tamine, L., Soulier, L., Jabeur, L. B., Amblard, F., Hanachi, C., Hubert, G., and Roth, C. (2016).
Social media-based collaborative information access: Analysis of online crisis-related twitter conversations.
In HT ’16.
Teevan, J., Ramage, D., and Morris, M. R. (2011).
#twittersearch: a comparison of microblog search and web search.
In Proceedings of the Forth International Conference on Web Search and Web Data Mining, WSDM 2011, pages 35–44.
Wang, B., Wang, C., Bu, J., Chen, C., Zhang, W. V., Cai, D., and He, X. (2013).
Whom to mention: Expand the diffusion of tweets by @ recommendation on micro-blogging systems.
In Proceedings of the 22Nd International Conference on World Wide Web, WWW ’13, pages 1331–1340. ACM.
30 / 30