A support framework for community managers to help them in managing discussions based on NLP and argumentation theory

1. A Support Framework for Argumentative
Discussions Management in the Web
Elena Cabrio, Serena Villata, Fabien Gandon
Wimmics Team
INRIA, I3S - Sophia Antipolis, France

2. Supporting community managers using
NLP and argumentation
COMMUNITY
MANAGER
GOAL
Efficient management of
wiki pages by community
managers and animations
of communities
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 2

3. Supporting community managers using
NLP and argumentation
TEXTUAL
ENTAILMENT
TEXTUAL
ENTAILMENT
How to detect the arguments,
And the relationships
among them?
1
COMMUNITY
MANAGER
GOAL
Efficient management of
wiki pages by community
managers and animations
of communities
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 3

4. Supporting community managers using
NLP and argumentation
TEXTUAL
ENTAILMENT
TEXTUAL
ENTAILMENT
ARGUMENTATION
THEORY
ARGUMENTATION
THEORY
How to detect the arguments,
And the relationships
among them?
1
How to build the overall
graph of the changes and
discover the winning
arguments?
2
COMMUNITY
MANAGER
GOAL
Efficient management of
wiki pages by community
managers and animations
of communities
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 4

5. Supporting community managers using
NLP and argumentation
TEXTUAL
ENTAILMENT
TEXTUAL
ENTAILMENT
ARGUMENTATION
THEORY
ARGUMENTATION
THEORY
How to detect the arguments,
And the relationships
among them?
1
How to build the overall
graph of the changes and
discover the winning
arguments?
2
RDF/
SPARQL
RDF/
SPARQL
3
COMMUNITY
MANAGER
How to extract
further insightful
information?
GOAL
Efficient management of
wiki pages by community
managers and animations
of communities
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 5

6. Outline
1 Related literature
2 Textual Entailment and Argumentation
3 Combined Framework
4 Experimental setting on Wikipedia revisions
5 Conclusions
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 6

7. Related literature
• Wikipedia revisions in NLP tasks
Zanzotto and Pennacchiotti (2010)
Expanding textual entailment corpora from Wikipedia using co-training
Cabrio et al. (2012)
Extracting context-rich entailment rules from wikipedia revision history
Nelken and Yamangil (2008)
Mining wikipedia revision histories for improving sentence compression
Max and Wisniewski (2010)
Mining naturally-occurring corrections and paraphrases from wikipedia’s revision
history
Dutrey et al. (2011)
Local modifications and paraphrases in wikipedia’s revision history
• Argumentation and NLP
Moens et al. (2007)
Automatic detection of arguments in legal texts
Carenini and Moore (2006)
Generating and evaluating evaluative arguments
Wyner and van Engers (2010)
A framework for enriched, controlled online discussion forums for e-government
policy-making
Heras et al. (2010)
How argumentation can enhance dialogues in social networks
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 7

8. Related literature
• Wikipedia revisions in NLP tasks
Zanzotto and Pennacchiotti (2010)
Expanding textual entailment corpora from Wikipedia using co-training
Cabrio et al. (2012)
Extracting context-rich entailment rules from wikipedia revision history
Nelken and Yamangil (2008)
Mining wikipedia revision histories for improving sentence compression
Max and Wisniewski (2010)
Mining naturally-occurring corrections and paraphrases from wikipedia’s revision
history
Dutrey et al. (2011)
Local modifications and paraphrases in wikipedia’s revision history
• Argumentation and NLP
Moens et al. (2007)
Automatic detection of arguments in legal texts
Carenini and Moore (2006)
Generating and evaluating evaluative arguments
Wyner and van Engers (2010)
A framework for enriched, controlled online discussion forums for e-government
policy-making
Heras et al. (2010)
How argumentation can enhance dialogues in social networks
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 8

9. Textual Entailment
• Generic framework for capturing major semantic inference
needs in NLP applications (Dagan and Glickman, 2004).
• Relation between two textual fragments T and H:
T ⇒ H: meaning of H can be inferred from meaning of T, as
interpreted by a typical language user.
T (Wiki11): The land area of the contiguous United States is approximately
1,800 million acres (7,300,000 km2)
H (Wiki10): The land area of the contiguous United States is approximately
1.9 billion acres (770 million hectares)
T (Wiki10): The land area of the contiguous United States is approximately
1.9 billion acres (770 million hectares)
H (Wiki09): The total land area of the contiguous United States is approxima-
tely 1.9 billion acres.
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 9

10. Textual Entailment
• Generic framework for capturing major semantic inference
needs in NLP applications (Dagan and Glickman, 2004).
• Relation between two textual fragments T and H:
T ⇒ H: meaning of H can be inferred from meaning of T, as
interpreted by a typical language user.
T (Wiki11): The land area of the contiguous United States is approximately
1,800 million acres (7,300,000 km2)
H (Wiki10): The land area of the contiguous United States is approximately
1.9 billion acres (770 million hectares)
T (Wiki10): The land area of the contiguous United States is approximately
1.9 billion acres (770 million hectares)
H (Wiki09): The total land area of the contiguous United States is approxima-
tely 1.9 billion acres.
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 10

11. Abstract Argumentation Theory
• Directed graph (Dung, 1995)
Nodes: abstract arguments
Edges: attack relation
argument
A
argument
B
argument
C
argument
A
argument
B
IN OUT IN OUT IN
ATTACK ATTACK ATTACK
• Bipolar argumentation
(Cayrol & Lagasquie-Schiex, 2005),(Boella et al., 2010)
b ca a bc b ca
Supported attack Secondary attack Mediated attack
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 11

12. Combined Framework
Wikipedia revisions for the article “United States”
T (Wiki12): The land area of the contiguous United States is 2,959,064 square miles (7,663,941 km2).
H (Wiki11): The land area of the contiguous United States is approximately 1,800 million acres
(7,300,000 km2)
T (Wiki11): The land area of the contiguous United States is approximately 1,800 million acres
(7,300,000 km2)
H (Wiki10): The land area of the contiguous United States is approximately 1.9 billion acres )
(770 million hectares)
T (Wiki10): The land area of the contiguous United States is approximately 1.9 billion acres
(770 million hectares)
H (Wiki09): The total land area of the contiguous United States is approximately 1.9 billion acres.
A2
Wiki10
A3
Wiki11
A1
Wiki09
A4
Wiki12
(a)
A2
Wiki10
A3
Wiki11
A1
Wiki09
A4
Wiki12
(b)
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 12

13. Revisions in RDF using
SIOC-Argumentation extended vocabulary
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 13

14. Revisions in RDF using
SIOC-Argumentation extended vocabulary
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 14

15. Extracting further information
from revisions in RDF
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 15

16. Experimental setting:
preprocessing Wikipedia dumps
• 4 dumps of English Wikipedia (2009, 2010, 2011, 2012)
• 5 most revised pages: United States, World War II, George
Bush, Michael Jackson, Britney Spears
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 16

17. Experimental setting:
extraction of entailment pairs
• Documents are sentence splitted, and sentences are aligned
• To measure the similarity between the sentences: Position
Independent Word Error Rate (PER) [Tillman et al., 1997]
• Different thresholds are set to cluster pairs into different sets
• Sentences with major editing are selected (0.2<PER<0.6)
• TE pair: revised sentence as T, original sentence as H
Entailment No Entailment
Training Set 114 pairs 114 pairs
Test Set 101 pairs 123 pairs
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 17

18. Experimental setting: evaluation
• EDITS system (Edit Distance Textual Entailment Suite)
(Kouylekov and Negri, 2010), off-the-shelf system
Basic configuration: word overlap and cosine similarity
algorithms; distance calculated on lemmas; stopword list
• FIRST STEP: TEXTUAL ENTAILMENT
Train Test
EDITS configurations rel Precision Recall Accuracy Precision Recall Accuracy
WordOverlap
yes 0.83 0.82
0.83
0.83 0.82
0.78
no 0.76 0.73 0.79 0.82
CosineSimilarity
yes 0.58 0.89
0.63
0.52 0.87
0.58
no 0.77 0.37 0.76 0.34
• SECOND STEP: TE+ARGUMENTATION THEORY
Test
Configuration Precision Recall F-measure
WordOverlap + AT 0.90 0.92 0.91
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 18

19. Experimental setting: evaluation
• EDITS system (Edit Distance Textual Entailment Suite)
(Kouylekov and Negri, 2010), off-the-shelf system
Basic configuration: word overlap and cosine similarity
algorithms; distance calculated on lemmas; stopword list
• FIRST STEP: TEXTUAL ENTAILMENT
Train Test
EDITS configurations rel Precision Recall Accuracy Precision Recall Accuracy
WordOverlap
yes 0.83 0.82
0.83
0.83 0.82
0.78
no 0.76 0.73 0.79 0.82
CosineSimilarity
yes 0.58 0.89
0.63
0.52 0.87
0.58
no 0.77 0.37 0.76 0.34
• SECOND STEP: TE+ARGUMENTATION THEORY
Test
Configuration Precision Recall F-measure
WordOverlap + AT 0.90 0.92 0.91
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 19

20. Experimental setting: evaluation
• EDITS system (Edit Distance Textual Entailment Suite)
(Kouylekov and Negri, 2010), off-the-shelf system
Basic configuration: word overlap and cosine similarity
algorithms; distance calculated on lemmas; stopword list
• FIRST STEP: TEXTUAL ENTAILMENT
Train Test
EDITS configurations rel Precision Recall Accuracy Precision Recall Accuracy
WordOverlap
yes 0.83 0.82
0.83
0.83 0.82
0.78
no 0.76 0.73 0.79 0.82
CosineSimilarity
yes 0.58 0.89
0.63
0.52 0.87
0.58
no 0.77 0.37 0.76 0.34
• SECOND STEP: TE+ARGUMENTATION THEORY
Test
Configuration Precision Recall F-measure
WordOverlap + AT 0.90 0.92 0.91
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 20

21. 1 Connect users to their arguments in online communities
2 Arguments’ evaluation depending on sources’ expertise
3 TE three-way judgement task:
entailment, contradiction, unknown
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 21

22. Thanks for your attention!
http://bit.ly/WikipediaDatasetXML
http://bit.ly/WikipediaDatasetRDF
http://bit.ly/SIOC_Argumentation
E. Cabrio, S. Villata, F. Gandon, Argumentative Discussions Management in the Web. 22