Download to read offline
Uploaded byAlexander Panchenko
IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for Community Question Answering and Implicit Dialogue Identification
The document describes a system for identifying answers and implicit dialogues in community question answering sites. It examines multiple feature types, including string similarity, word embeddings, topic modeling and keyword features. An SVM classifier is used to rank answers for three subtasks: comment classification, question-comment similarity and question-external comment similarity. Evaluation on the SemEval 2017 Task 3 dataset shows the full feature set performs best, with string and embedding features also providing significant contributions to performance.
More Related Content
Similar to IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for Community Question Answering and Implicit Dialogue Identification
IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for Community Question Answering and Implicit Dialogue Identification
- 1. 1/34 IIT-UHH at SemEval-2017Task 3: Exploring Multiple Features for Community Question Answering and Implicit Dialogue Identification Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 1Indian Institute of Technology Patna, India 2Universit¨at Hamburg, Germany {titas.ee13,deepak.pcs16,asif,pb}@iitp.ac.in {biemann,yimam,kohail}@informatik.uni-hamburg.de Presented by Alexander Panchenko2 August 3, 2017 Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 2. 2/34 Outline 1 Task Description Structureof the Task Related Work 2 System Description Basic Features Implicit Dialogue Identification Statistical Model 3 Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017 4 Conclusions Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 3. 3/34 Outline 1 Task Description Structureof the Task Related Work 2 System Description Basic Features Implicit Dialogue Identification Statistical Model 3 Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017 4 Conclusions Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 4. 4/34 SemEval 2017 Task3: the Three Sub-Tasks Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 5. 5/34 Outline 1 Task Description Structureof the Task Related Work 2 System Description Basic Features Implicit Dialogue Identification Statistical Model 3 Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017 4 Conclusions Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 6. 6/34 Related Work Useful ideasfrom the best systems of 2015 and 2016 tasks: Belinkov (2015): word vectors and meta-data features Nicosia (2015): derived features from a comment in the context of the entire thread Filice (2016): stacking classifiers across subtasks Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 7. 7/34 Outline of theMethod Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 8. 8/34 Outline 1 Task Description Structureof the Task Related Work 2 System Description Basic Features Implicit Dialogue Identification Statistical Model 3 Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017 4 Conclusions Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 9. 9/34 String Similarity Features Stringsimilarity between a question-comment/question pair: Jaro-Winkler Levenshtein Jaccard Sorensen-Dice n-gram LCS Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 10. 10/34 Domain (Task) SpecificFeatures If a comment by asker of the question is an acknowledgement Position of comment in the thread Coverage (the ratio of the number of tokens) of question by the comment and comment by the question Presence of URLs, emails or HTML tags Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 11. 11/34 Word Embedding Features Trainedword embedding model using Word2Vec on unannotated data Sentence vectors averaging word vectors wscore = wquestion − wcomment Distance scores Based on the computed sentence vectors Cosine Distance (1 − cos) Manhattan Distance Euclidean Distance Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 12. 12/34 Topic Modeling Features TrainedLDA Topic model using Mallet tool on training data Extracted the 20 most relevant topics for the data Topic Vector of a Question/Comment wscore = wquestion − wcomment Topic Vocabulary of a Question/Comment Vocabulary(T) = 10 i=1 topic words(ti ) where ti is one of the top topics for comment/question T. Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 13. 13/34 Keyword and NamedEntity Features Extracted keywords or focus words from question and comment using the RAKE algorithm (Rose et al., 2010) Keyword match between question and comment Extracted Named Entities from question and comment Entity tags consisted of LOCATION, PERSON, ORGANIZATION, DATE, MONEY, PERCENT and TIME Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 14. 14/34 Outline 1 Task Description Structureof the Task Related Work 2 System Description Basic Features Implicit Dialogue Identification Statistical Model 3 Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017 4 Conclusions Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 15. 15/34 Implicit Dialogue Identification Identifiedimplicit dialogues among users User Interaction Graph Each user is in dialogue with some other user who came before him/her Asker - desirable Other users - not desirable Vertices - Users in a comment thread Edges - Directed edges showing interaction Edge weight - the level of interaction Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 16. 16/34 Implicit Dialogue Identification:an Example Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 17. 17/34 Implicit Dialogue Identification:an Example Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 18. 18/34 Implicit Dialogue Identification:an Example Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 19. 19/34 Implicit Dialogue Identification:an Example Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 20. 20/34 Implicit Dialogue Identification:an Example Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 21. 21/34 Implicit Dialogue Identification:an Example Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 22. 22/34 Implicit Dialogue Identification:an Example Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 23. 23/34 Computing Edge Weights Theedge weight is computed (or revised) on the basis of: Explicit dialogue score. If one user refers the other explicitly, then add 1.0 to the edge score. Embedding score. For each word in a comment, find the word in the other comment that has maximum cosine similarity with it. Then finally average all those max cosine scores to get a value. Topic score. The cosine of topic vectors of the two comments. Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 24. 24/34 Titas Nandi1 , ChrisBiemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 25. 25/34 Outline 1 Task Description Structureof the Task Related Work 2 System Description Basic Features Implicit Dialogue Identification Statistical Model 3 Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017 4 Conclusions Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 26. 26/34 Classification Model Normalized allfeature values with Z-scores Feature Selection using wrapper methods to maximize accuracy on the development set Used SVM confidence probabilities for ranking (RBF kernel) Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 27. 27/34 Subtask C: Similarityof Questions and External Comments Oversample the data using the SMOTE (Chawla, 2002) technique and run classifier on original question - external comment pair Stacking across tasks: the SVM scores of all three subtasks are combined: Score C = log(SVM Score) + log(Score A) + log(Score B) Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 28. 28/34 Outline 1 Task Description Structureof the Task Related Work 2 System Description Basic Features Implicit Dialogue Identification Statistical Model 3 Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017 4 Conclusions Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 29. 29/34 Feature Ablation Results:Impact of Different Feature Sets Features Development Set 2017 Subtask A MAP P R F1 Acc All Features 65.50 58.43 62.71 60.50 72.54 All — string 65.53 57.84 62.71 60.18 72.17 All — embedding 62.11 53.03 53.42 53.23 68.52 All — domain 61.85 54.46 54.52 54.49 69.47 All — topic 65.15 59.02 61.98 60.47 72.83 All — keyword 65.73 57.98 62.59 60.20 72.25 IR Baseline 53.84 - - - - Runs Test Set 2017 Subtask A MAP P R F1 Acc Primary 86.88 73.37 74.52 73.94 72.70 Contrastive 1 86.35 79.42 51.94 62.80 68.02 Contrastive 2 85.24 81.22 57.65 67.43 71.06 Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 30. 30/34 Outline 1 Task Description Structureof the Task Related Work 2 System Description Basic Features Implicit Dialogue Identification Statistical Model 3 Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017 4 Conclusions Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 31. 31/34 Comparison of Resultson Subtask A at SemEval 2017 Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 32. 32/34 Comparison of Resultson Subtask C at SemEval 2017 Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 33. 33/34 Observations and Conclusions Smallin-domain texts are better for training, compared to large out-of-domain pre-trained GoogleNews embeddings Most instrumental are features based on: User dialogues Word embeddings Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34
- 34. 34/34 Thank you! Any questionsfrom the community? Titas Nandi1 , Chris Biemann2 , Seid Muhie Yimam2 , Deepak Gupta1 , Sarah Kohail2 , Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT PatAnswer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34