Paper for the First Workshop on Argumentation Mining at the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, Maryland, June 26 2014 Abstract: Argumentation mining, a relatively new area of discourse analysis, involves automatically identifying and structuring arguments. Following a basic introduction to argumentation, we describe a new possible domain for argumentation mining: debates in open online collaboration communities. Based on our experience with manual annotation of arguments in debates, we envision argumentation mining as the basis for three kinds of support tools, for authoring more persuasive arguments, finding weaknesses in others’ arguments, and summarizing a debate’s overall conclusions. Full paper: http://jodischneider.com/pubs/aclargmining2014.pdf Proceedings with links: http://acl2014.org/acl2014/W14-21/index.html Workshop homepage: http://www.uncg.edu/cmp/ArgMining2014/

1. Jodi Schneider
First Workshop on Argumentation Mining at ACL 2014
Baltimore, Maryland
2014-06-26

2. Argumentation mining today
• No unified vision of the field. Multiple:
– Interrelated problems
– Application domains
– Tools handling one aspect of annotation
• Few corpora
• Need for
– Common definition(s) of argumentation
– "Challenge problems"
– Shared corpora
– Applications

3. Argumentation mining today
• No unified vision of the field. Multiple:
– Interrelated problems
– Application domains
– Tools handling one aspect of annotation
• Few corpora
• Need for
– Common definition(s) of argumentation
– "Challenge problems"
– Shared corpora
– Our Application: debates in online collaboration

4. Application: Debates in Open Online
Collaboration
• Wikipedia
• HTML5
• OpenStreetMap
• Project Gutenberg
• Apache projects
• Mozilla Firefox
• …

5. Debates argue for & against
a single conclusion

6. Which content belongs in Wikipedia?

7. 500 discussions per week!!!

8. Argument-based support
• How can I win an argument?
Which arguments sway the community?
• Why were previous decisions made?
• Which ongoing debates need more comments?

9. Argument-based support
• How can I win an argument?
Which arguments sway the community?
• Why were previous decisions made?
• Which ongoing debates need more comments?

10. Corpus: 72 discussions started on 1 day
• Each discussion has:
3-33 messages
2-15 participants
• 741 messages contributed by 244 users.
Each message has 3-350+ words.
• 98 printed A4 sheets

11. Approach
• Compare two argumentation theories
• Iterative annotation with multiple annotators
– Refine to get good inter-annotator agreement
• 4 rounds of annotation
– Rounds 1-2 by me
– Rounds 3-4 by 2 assistants

12. We used two argumentation theories
• Walton’s Argumentation Schemes
(Walton, Reed, and Macagno 2008)
– Informal argumentation
(philosophical & computational argumentation)
– Identify & prevent errors in reasoning (fallacies)
– 60 patterns
• Factors/Dimensions Analysis
(Ashley 1991; Bench-Capon and Rissland, 2001)
– Case-based reasoning
– E.g. factors for deciding cases in trade secret law,
favoring either party (the plaintiff or the defendant).

13. Walton’s Argumentation Schemes
Example Argumentation Scheme:
Argument from Rules – “we apply rule X”
Critical Questions
1. Does the rule require carrying out this type of action?
2. Are there other established rules that might conflict
with or override this one?
3. Are there extenuating circumstances or an excuse for
noncompliance?
Walton, Reed, and Macagno 2008

14. Example: "Stop at a red light"
1. Does the rule require carrying out this type of action?
Were you driving a vehicle?
2. Are there other established rules that might conflict
with or override this one?
Did a police officer direct you to continue without
stopping?
3. Are there extenuating circumstances or an excuse for
noncompliance?
Were you driving an ambulance with its siren on?
Critical Questions from Argument from Rules based on Walton, Reed, and Macagno 2008

15. Walton’s Argumentation Schemes
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker.
“Arguments about Deletion: How Experience Improves the Acceptability of Arguments in
Ad-hoc Online Task Groups”. In CSCW 2013.
! "#$%&' ()*+((&"' *"&, +-&' . &
! "#$%&' ()*"+%), - ./&' 0&)(+)123+(4&5.5 67897:
! "#$%&' ()*"+%); $<&5 6=87>:
? +(& 6@8=7:
! "#$%&' (A(.+' )*"+%)BA<$&5 C89>:
! "#$%&' ()*"+%)? &&/)*+")1&<3 C869:
! "#$%&' ()*"+%)D.A5 @8EF:
? +)"&A5+' )#.- &' @899:
! "#$%&' ()*"+%)G+5.(.+' )(+)H' +I @8>J :
! "#$%&' ()*"+%)G"&0&/&' ( @8>J :
! "#$%&' ()*"+%)K#' +"A' 0& 987F:
! "#$%&' ()*"+%)L+%3+5.(.+' 98J =:
! "#$%&' ()*"+%)LA$5&)(+), **&0( 98@6:
! "#$%&' ()*"+%)! ' A<+#2 989@:
! "#$%&' ()*"+%)MA5(& 989@:
G"A0(.0A<); &A5+' .' # 989@:
! "#8)*"+%)B&"NA<)L<A55.*.0A(.+' 98>=:

16. How to win an argument with a
Wikipedian?
• Argument from Evidence to Hypothesis (19%)
• Argument from Rules (17%)

17. How to win an argument (Arucaria)?
Classifying Arguments by Scheme. Vanessa Wei Feng. Master's thesis, Toronto, 2010.

18. Experts vs. Novices
• Experts were more likely to use
– Argument from Precedent
• Novices were more likely to use
– Argumentation from Values
– Argumentation from Cause to Effect
– Argument from Analogy

19. Unsuccessful arguments from novices
• Emsworth Cricket Club is one of the oldest
cricket clubs in the world, and this really is
worth a mention. Especially on a website,
where pointless people … gets a mention.
• Why just because it is a small team and not
major does it not deserve it’s (sic) own page
on here?

20. Newcomers don't understand
how to counterargue

21. Factors/Dimensions Analysis
• Factors (case-based reasoning)
– All or nothing
• Either present ("applicable") or absent
• When present, a factor always favors the same side
• Dimensions
– More complex/subtle
• Can be applicable to a varying degree ("sliding scale")
• Favor plantiff on one extreme; defendant on the other
Ashley 1991; Bench-Capon and Rissland, 2001

22. Example factors analysis
(Aleven 1997)

23. Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussions
“Other” as 5th catchall

24. Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussions
“Other” as 5th catchall
Factor Example (used to justify `keep')
Notability Anyone covered by another
encyclopedic reference is
considered notable enough for
inclusion in Wikipedia.
Sources Basic information about this
album at a minimum is certainly
verifiable, it's a major label
release, and a highly notable
band.
Maintenance …this article is savable but at its
current state, needs a lot of
improvement.
Bias It is by no means spam (it does
not promote the products).
**Other I'm advocating a blanket
"hangon" for all articles on
newly-drafted players

25. Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussions
“Other” as 5th catchall
Factor Example (used to justify `keep')
Notability Anyone covered by another
encyclopedic reference is
considered notable enough for
inclusion in Wikipedia.
Sources Basic information about this
album at a minimum is certainly
verifiable, it's a major label
release, and a highly notable
band.
Maintenance …this article is savable but at its
current state, needs a lot of
improvement.
Bias It is by no means spam (it does
not promote the products).
**Other I'm advocating a blanket
"hangon" for all articles on
newly-drafted players

26. Factor-based Summarization

27. Comparison of Annotation
• Cohen’s kappa (Cohen, 1960)
.48 for Walton’s argumentation schemes
.64-.82 for factors, depending on the factor
• Potential for task support
– Argumentation schemes
• Write effective arguments
• Ask critical questions to check others' arguments
– Factors
• Summarize debates

28. Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide personalized feedback on drafts
– Requires: knowing which arguments are accepted;
identifying argumentation in a drafts
• Find weaknesses in others’ arguments
– How: suggest & instantiate relevant critical questions
– Requires: identifying argumentation schemes
• Summarize the overall conclusions of the debate
– How: identify the winning and losing rationales
– Requires: identifying rationales and contradictions

29. Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide personalized feedback on drafts
– Requires: knowing which arguments are accepted;
identifying argumentation in a drafts
• Find weaknesses in others’ arguments
– How: suggest & instantiate relevant critical questions
– Requires: identifying argumentation schemes
• Summarize the overall conclusions of the debate
– How: identify the winning and losing rationales
– Requires: identifying rationales and contradictions

30. Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide personalized feedback on drafts
– Requires: knowing which arguments are accepted;
identifying argumentation in a drafts
• Find weaknesses in others’ arguments
– How: suggest & instantiate relevant critical questions
– Requires: identifying argumentation schemes
• Summarize the overall conclusions of the debate
– How: identify the winning and losing rationales
– Requires: identifying rationales and contradictions

31. Argumentation Mining papers
Arguing on Wikipedia
• “Arguments about Deletion: How Experience Improves the Acceptability of Arguments
in Ad-hoc Online Task Groups” CSCW 2013.
• “Deletion Discussions in Wikipedia: Decision Factors and Outcomes” WikiSym2012.
Arguing in Social Media
• “Dimensions of Argumentation in Social Media" EKAW 2012
• “Why did they post that argument? Communicative intentions of Web 2.0 arguments.”
Arguing on the Web 2.0 at ISSA 2014
Arguing in Reviews
• “Identifying Consumers' Arguments in Text” SWAIE 2012
• “Semi-Automated Argumentative Analysis of Online Product Reviews" COMMA 2012
• “Arguing from a Point of View” Agreement Technologies 2012
Structuring Arguments on the Social Semantic Web
• “A Review of Argumentation for the Social Semantic Web” Semantic Web –
Interoperability, Usability, Applicability, 2013.
• “Identifying, Annotating, and Filtering Arguments and Opinions in Open Collaboration
Systems" 2013 Thesis: purl.org/jsphd
• “Modeling Arguments in Scientific Papers” at ArgDiaP 2014
http://jodischneider.com/jodi.html

33. http://purl.org/jsphd

34. An argument gives a rationale

35. What do we mean by an argument?
• The simplest possible argument connects two
Statements by means of an Inference Rule

36. Cases
• Cases support a single conclusion.

37. Attacks & Debates
• Debates argue for and against a single
conclusion.

38. Open collaboration based on
• Technological infrastructure
• People
• Social structures: rules, policies, procedures,…

39. Open collaboration based on
• Technological infrastructure
• People
• Social structures:
– joint decision-making
– Importance of rationales: reasons for opinions

40. HTML5

41. OpenStreetMap

42. Example from Corpus

43. Results: Important tasks
for consensus discussions
1. Determine one’s personal position
2. Express one’s personal position in accordance
with community norms
3. Determine the consensus

44. Related work
• Dissent and rhetorical devices in bug reporting
(Ko and Chilana, 2011)
• how Python listservs select enhancement
proposals (Barcellini et al., 2005).
– role of a participant is related to the kinds of
message they quote (Syntheses, Disagreements,
Proposals, or Agreements)
– Syntheses and Disagreements are the most
quoted