Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Azureを有効活用しよう
Next
Download to read offline and view in fullscreen.

Share

Leveraging Textual Features for Best Answer Prediction in Community-based Question Answering

Download to read offline

Abstract:
This paper addresses the problem of determining the best answer in Community-based Question Answering (CQA) websites by focussing on the content. In particular, we present a system, ACQUA1, that can be installed onto the majority of browsers as a plugin. The service o↵ers a seamless and accurate prediction of the answer to be accepted. Previous research on this topic relies on the exploitation of community feedback on the answers, which involves rating of either users (e.g., reputation) or answers (e.g. scores manually assigned to answers). We propose a new technique that leverages the content/textual features of answers in a novel way. Our approach delivers better results than related linguistics-based solutions and manages to match rating-based approaches. More specifically, the gain in performance is achieved by rendering the values of these features into a discretised form. We also show how our technique manages to deliver equally good results in real-time settings, as opposed to having to rely on information not always readily available, such as user ratings and answer scores. We ran an evaluation on 21 StackExchange websites covering around 4 million questions and more than 8 million answers. We obtain 84% average precision and 70% recall, which shows that our technique is robust, effective, and widely applicable.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Leveraging Textual Features for Best Answer Prediction in Community-based Question Answering

  1. 1. GEORGE GKOTSIS 1, MARIA LIAKATA 2, CARLOS PEDRINACI 3, JOHN DOMINGUE 3 Leveraging Textual Features for Best Answer Prediction in Community-based Question Answering 1King’s College London 2Department of Computer Science, University of Warwick 3Knowledge Media Institute, The Open University
  2. 2. Outline 8-11June 2015ICCSS 2015  Motivation  Problem description  Proposed solution  Evaluation  ACQUA
  3. 3. 8-11June 2015ICCSS 2015 Motivation
  4. 4. Questions on social networking sites 8-11June 2015ICCSS 2015 Recommendations & opinions Authoritative responses Expert & Empirical knowledge
  5. 5. Queries on CQA 8-11June 2015ICCSS 2015
  6. 6. 8-11June 2015ICCSS 2015 Problem description
  7. 7. 8-11June 2015ICCSS 2015
  8. 8. Reputation based Answer Rating based 8-11June 2015ICCSS 2015 “…we observe significant assortativity in the reputations of co-answerers, relationships between reputation and answer speed, and that the probability of an answer being chosen as the best one strongly depends on temporal characteristics of answer arrivals.” Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, Jure Leskovec Discovering Value from Community Activity on Focused Question Answering Sites: A Case Study of Stack Overflow. KDD 2012 “When available, scoring (or rating) features improve prediction results significantly, which demonstrates the value of community feedback and reputation for identifying valuable answers.” Grégoire Burel, Yulan He, Harith Alani. Automatic Identification of Best Answers in Online Enquiry Communities ESWC 2012 State of the art solutions
  9. 9. Best answer prediction in Social Q&A 8-11June 2015ICCSS 2015  Binary classification problem  Is it solved?  Yes, partially  Current solutions depend on: Answer Ratings • Score, #comments Knowledge is Future & Unknown User Ratings • User Reputation • UpVotes etc • Preferential attachment Knowledge is Past & Not always available
  10. 10. State of the art solutions Summary 8-11June 2015ICCSS 2015 Our solution 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% Linguistic User Ratings Answer ratings Average Precision
  11. 11. StackExchange network 8-11June 2015ICCSS 2015 SE “is all about getting answers, it’s not a discussion forum, there’s no chit-chat”  123 Q&A sites  5,622,330 users  9.5 million questions  16.3 million answers  9.3 million visits per day 20 June 2014:
  12. 12. 8-11June 2015ICCSS 2015 StackOverflow 91% The Rest 9% 3,375,817 3,795,276 0 1,000,000 2,000,000 3,000,000 4,000,000 5,000,000 6,000,000 7,000,000 8,000,000 stackoverflow Non Accepted Answers Accepted Answers September 2013 dump Questions with Accepted Answers
  13. 13. Shallow Linguistic features 8-11June 2015ICCSS 2015  Long history, coming from studies on readability 1. Average number of characters per word 2. Average number of words per sentence 3. Number of words in the longest sentence 4. Answer length 5. Log Likehood: Pitler & Nenkova, 2008
  14. 14. StackOverflow Overview of shallow features’ evolution 8-11June 2015ICCSS 2015
  15. 15. Shallow features: Observations 8-11June 2015ICCSS 2015  Accepted answers tend to be:  Longer  Differ more from the community vocabulary  Contain shorter words  Have longer longest sentences  Have more words per sentence But how good are shallow features?
  16. 16. But how good are shallow features? 8-11June 2015ICCSS 2015  58% macro precision (our baseline)  Possible reasons 1. Evolution of language characteristics  Language becomes more eloquent 2. Variance is huge 3. Universal classifier looks unreachable, e.g.:  SuperUser average length is 577  Skeptics average length is 2,154 Bad Good
  17. 17. StackOverflow vrs. SuperUser 8-11June 2015ICCSS 2015
  18. 18. 8-11June 2015ICCSS 2015 Proposed solution
  19. 19. Objectives 8-11June 2015ICCSS 2015  Build a classifier which is: 1. Based on linguistic features solely 2. Robust  Performs equally well to other classifiers that use user ratings (past knowledge) or answer ratings (future knowledge) 3. Universal  Same classifier applicable to as many SE websites possible (domain agnostic)
  20. 20. Feature discretisation Example for Length 8-11June 2015ICCSS 2015 Group by question Question Id 1 5 Answer Id 6 7 Length 2 200 3 150 4 250 150 100 Sort by Length in descending order Rank LengthD 1 2 3 1 2
  21. 21. Feature discretisation 8-11June 2015ICCSS 2015 Category Name Information Gain Linguistic Length 0.0226 LongestSentence 0.0121 LL 0.0053 WordsPerSentence 0.0048 CharactersPerWord 0.0052 Linguistic Discretisation LengthD 0.2168 LongestSentenceD 0.1750 LLD 0.1180 WordsPerSentenceD 0.1404 CharactersPerWordD 0.1162 20x increase
  22. 22. User and answer rating features 8-11June 2015ICCSS 2015 Category Name Other Age CreationDateD AnswerCount User Rating UserReputation UserUpVotes UserDownVotes UserViews UserUpDownVotes Answer rating Score CommentCount ScoreRatio
  23. 23. 8-11June 2015ICCSS 2015 Evaluation
  24. 24. Evaluation Comparison 8-11June 2015ICCSS 2015 Case Features Used P R FM AUC 1 Linguistic 0.58 0.60 0.56 0.60 2 Linguistic & Discretisation 0.81 0.70 0.74 0.84 3 Linguistic & Discretisation & Other 0.84 0.7 0.76 0.87 4 Linguistic & Other & User Rating (no discretisation) 0.82 0.69 0.75 0.86 5 Linguistic & Other & User Rating (with discretisation) 0.82 0.72 0.77 0.88 6 All features (Answer and User Rating with discretisation) 0.88 0.85 0.86 0.94
  25. 25. 8-11June 2015ICCSS 2015 ACQUA Automatic Community-based Question Answering https://acqua.kmi.open.ac.uk/
  26. 26. 8-11June 2015ICCSS 2015 ACQUA - Architecture
  27. 27. ACQUA - Screenshot 8-11June 2015ICCSS 2015
  28. 28. Read more about our work 8-11June 2015ICCSS 2015  It’s All in the Content: State of the Art Best Answer Prediction based on Discretisation of Shallow Linguistic Features. WebSci ’14  ACQUA: Automated Community-based Question Answering through the Discretisation of Shallow Linguistic Features. The Journal of Web Science, 1(1) (preprint available)
  29. 29. Thank you 8-11June 2015ICCSS 2015 http://xkcd.com/386/

Abstract: This paper addresses the problem of determining the best answer in Community-based Question Answering (CQA) websites by focussing on the content. In particular, we present a system, ACQUA1, that can be installed onto the majority of browsers as a plugin. The service o↵ers a seamless and accurate prediction of the answer to be accepted. Previous research on this topic relies on the exploitation of community feedback on the answers, which involves rating of either users (e.g., reputation) or answers (e.g. scores manually assigned to answers). We propose a new technique that leverages the content/textual features of answers in a novel way. Our approach delivers better results than related linguistics-based solutions and manages to match rating-based approaches. More specifically, the gain in performance is achieved by rendering the values of these features into a discretised form. We also show how our technique manages to deliver equally good results in real-time settings, as opposed to having to rely on information not always readily available, such as user ratings and answer scores. We ran an evaluation on 21 StackExchange websites covering around 4 million questions and more than 8 million answers. We obtain 84% average precision and 70% recall, which shows that our technique is robust, effective, and widely applicable.

Views

Total views

943

On Slideshare

0

From embeds

0

Number of embeds

161

Actions

Downloads

13

Shares

0

Comments

0

Likes

0

×