Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Jodi Schneider
First Workshop on Argumentation Mining at ACL 2014
Baltimore, Maryland
2014-06-26
Argumentation mining today
• No unified vision of the field. Multiple:
– Interrelated problems
– Application domains
– Too...
Argumentation mining today
• No unified vision of the field. Multiple:
– Interrelated problems
– Application domains
– Too...
Application: Debates in Open Online
Collaboration
• Wikipedia
• HTML5
• OpenStreetMap
• Project Gutenberg
• Apache project...
Debates argue for & against
a single conclusion
Which content belongs in Wikipedia?
500 discussions per week!!!
Argument-based support
• How can I win an argument?
Which arguments sway the community?
• Why were previous decisions made...
Argument-based support
• How can I win an argument?
Which arguments sway the community?
• Why were previous decisions made...
Corpus: 72 discussions started on 1 day
• Each discussion has:
3-33 messages
2-15 participants
• 741 messages contributed ...
Approach
• Compare two argumentation theories
• Iterative annotation with multiple annotators
– Refine to get good inter-a...
We used two argumentation theories
• Walton’s Argumentation Schemes
(Walton, Reed, and Macagno 2008)
– Informal argumentat...
Walton’s Argumentation Schemes
Example Argumentation Scheme:
Argument from Rules – “we apply rule X”
Critical Questions
1....
Example: "Stop at a red light"
1. Does the rule require carrying out this type of action?
Were you driving a vehicle?
2. A...
Walton’s Argumentation Schemes
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker.
“Arguments about Deletion:...
How to win an argument with a
Wikipedian?
• Argument from Evidence to Hypothesis (19%)
• Argument from Rules (17%)
How to win an argument (Arucaria)?
Classifying Arguments by Scheme. Vanessa Wei Feng. Master's thesis, Toronto, 2010.
Experts vs. Novices
• Experts were more likely to use
– Argument from Precedent
• Novices were more likely to use
– Argume...
Unsuccessful arguments from novices
• Emsworth Cricket Club is one of the oldest
cricket clubs in the world, and this real...
Newcomers don't understand
how to counterargue
Factors/Dimensions Analysis
• Factors (case-based reasoning)
– All or nothing
• Either present ("applicable") or absent
• ...
Example factors analysis
(Aleven 1997)
Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussio...
Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussio...
Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussio...
Factor-based Summarization
Comparison of Annotation
• Cohen’s kappa (Cohen, 1960)
.48 for Walton’s argumentation schemes
.64-.82 for factors, dependi...
Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide pe...
Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide pe...
Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide pe...
Argumentation Mining papers
Arguing on Wikipedia
• “Arguments about Deletion: How Experience Improves the Acceptability of...
http://purl.org/jsphd
An argument gives a rationale
What do we mean by an argument?
• The simplest possible argument connects two
Statements by means of an Inference Rule
Cases
• Cases support a single conclusion.
Attacks & Debates
• Debates argue for and against a single
conclusion.
Open collaboration based on
• Technological infrastructure
• People
• Social structures: rules, policies, procedures,…
Open collaboration based on
• Technological infrastructure
• People
• Social structures:
– joint decision-making
– Importa...
HTML5
OpenStreetMap
Example from Corpus
Results: Important tasks
for consensus discussions
1. Determine one’s personal position
2. Express one’s personal position...
Related work
• Dissent and rhetorical devices in bug reporting
(Ko and Chilana, 2011)
• how Python listservs select enhanc...
Envisioning argumentation and decision making support for debates in open online collaboration communities--acl arg mining...
Upcoming SlideShare
Loading in …5
×

Envisioning argumentation and decision making support for debates in open online collaboration communities--acl arg mining workshop 2014-06-26

1,143 views

Published on

Paper for the First Workshop on Argumentation Mining at the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, Maryland, June 26 2014

Abstract:
Argumentation mining, a relatively new area of discourse analysis, involves automatically identifying and structuring arguments. Following a basic introduction to argumentation, we describe a new possible domain for argumentation mining: debates in open online collaboration communities. Based on our experience with manual annotation of arguments in debates, we envision argumentation mining as the basis for three kinds of support tools, for authoring more persuasive arguments, finding weaknesses in others’ arguments, and summarizing a debate’s overall conclusions.

Full paper:
http://jodischneider.com/pubs/aclargmining2014.pdf

Proceedings with links:
http://acl2014.org/acl2014/W14-21/index.html

Workshop homepage:
http://www.uncg.edu/cmp/ArgMining2014/

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Envisioning argumentation and decision making support for debates in open online collaboration communities--acl arg mining workshop 2014-06-26

  1. 1. Jodi Schneider First Workshop on Argumentation Mining at ACL 2014 Baltimore, Maryland 2014-06-26
  2. 2. Argumentation mining today • No unified vision of the field. Multiple: – Interrelated problems – Application domains – Tools handling one aspect of annotation • Few corpora • Need for – Common definition(s) of argumentation – "Challenge problems" – Shared corpora – Applications
  3. 3. Argumentation mining today • No unified vision of the field. Multiple: – Interrelated problems – Application domains – Tools handling one aspect of annotation • Few corpora • Need for – Common definition(s) of argumentation – "Challenge problems" – Shared corpora – Our Application: debates in online collaboration
  4. 4. Application: Debates in Open Online Collaboration • Wikipedia • HTML5 • OpenStreetMap • Project Gutenberg • Apache projects • Mozilla Firefox • …
  5. 5. Debates argue for & against a single conclusion
  6. 6. Which content belongs in Wikipedia?
  7. 7. 500 discussions per week!!!
  8. 8. Argument-based support • How can I win an argument? Which arguments sway the community? • Why were previous decisions made? • Which ongoing debates need more comments?
  9. 9. Argument-based support • How can I win an argument? Which arguments sway the community? • Why were previous decisions made? • Which ongoing debates need more comments?
  10. 10. Corpus: 72 discussions started on 1 day • Each discussion has: 3-33 messages 2-15 participants • 741 messages contributed by 244 users. Each message has 3-350+ words. • 98 printed A4 sheets
  11. 11. Approach • Compare two argumentation theories • Iterative annotation with multiple annotators – Refine to get good inter-annotator agreement • 4 rounds of annotation – Rounds 1-2 by me – Rounds 3-4 by 2 assistants
  12. 12. We used two argumentation theories • Walton’s Argumentation Schemes (Walton, Reed, and Macagno 2008) – Informal argumentation (philosophical & computational argumentation) – Identify & prevent errors in reasoning (fallacies) – 60 patterns • Factors/Dimensions Analysis (Ashley 1991; Bench-Capon and Rissland, 2001) – Case-based reasoning – E.g. factors for deciding cases in trade secret law, favoring either party (the plaintiff or the defendant).
  13. 13. Walton’s Argumentation Schemes Example Argumentation Scheme: Argument from Rules – “we apply rule X” Critical Questions 1. Does the rule require carrying out this type of action? 2. Are there other established rules that might conflict with or override this one? 3. Are there extenuating circumstances or an excuse for noncompliance? Walton, Reed, and Macagno 2008
  14. 14. Example: "Stop at a red light" 1. Does the rule require carrying out this type of action? Were you driving a vehicle? 2. Are there other established rules that might conflict with or override this one? Did a police officer direct you to continue without stopping? 3. Are there extenuating circumstances or an excuse for noncompliance? Were you driving an ambulance with its siren on? Critical Questions from Argument from Rules based on Walton, Reed, and Macagno 2008
  15. 15. Walton’s Argumentation Schemes Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups”. In CSCW 2013. ! "#$%&' ()*+((&"' *"&, +-&' . & ! "#$%&' ()*"+%), - ./&' 0&)(+)123+(4&5.5 67897: ! "#$%&' ()*"+%); $<&5 6=87>: ? +(& 6@8=7: ! "#$%&' (A(.+' )*"+%)BA<$&5 C89>: ! "#$%&' ()*"+%)? &&/)*+")1&<3 C869: ! "#$%&' ()*"+%)D.A5 @8EF: ? +)"&A5+' )#.- &' @899: ! "#$%&' ()*"+%)G+5.(.+' )(+)H' +I @8>J : ! "#$%&' ()*"+%)G"&0&/&' ( @8>J : ! "#$%&' ()*"+%)K#' +"A' 0& 987F: ! "#$%&' ()*"+%)L+%3+5.(.+' 98J =: ! "#$%&' ()*"+%)LA$5&)(+), **&0( 98@6: ! "#$%&' ()*"+%)! ' A<+#2 989@: ! "#$%&' ()*"+%)MA5(& 989@: G"A0(.0A<); &A5+' .' # 989@: ! "#8)*"+%)B&"NA<)L<A55.*.0A(.+' 98>=:
  16. 16. How to win an argument with a Wikipedian? • Argument from Evidence to Hypothesis (19%) • Argument from Rules (17%)
  17. 17. How to win an argument (Arucaria)? Classifying Arguments by Scheme. Vanessa Wei Feng. Master's thesis, Toronto, 2010.
  18. 18. Experts vs. Novices • Experts were more likely to use – Argument from Precedent • Novices were more likely to use – Argumentation from Values – Argumentation from Cause to Effect – Argument from Analogy
  19. 19. Unsuccessful arguments from novices • Emsworth Cricket Club is one of the oldest cricket clubs in the world, and this really is worth a mention. Especially on a website, where pointless people … gets a mention. • Why just because it is a small team and not major does it not deserve it’s (sic) own page on here?
  20. 20. Newcomers don't understand how to counterargue
  21. 21. Factors/Dimensions Analysis • Factors (case-based reasoning) – All or nothing • Either present ("applicable") or absent • When present, a factor always favors the same side • Dimensions – More complex/subtle • Can be applicable to a varying degree ("sliding scale") • Favor plantiff on one extreme; defendant on the other Ashley 1991; Bench-Capon and Rissland, 2001
  22. 22. Example factors analysis (Aleven 1997)
  23. 23. Wikipedia Factors Analysis Factors determined by iterative annotation 4 Factors cover – 91% of comments – 70% of discussions “Other” as 5th catchall
  24. 24. Wikipedia Factors Analysis Factors determined by iterative annotation 4 Factors cover – 91% of comments – 70% of discussions “Other” as 5th catchall Factor Example (used to justify `keep') Notability Anyone covered by another encyclopedic reference is considered notable enough for inclusion in Wikipedia. Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band. Maintenance …this article is savable but at its current state, needs a lot of improvement. Bias It is by no means spam (it does not promote the products). **Other I'm advocating a blanket "hangon" for all articles on newly-drafted players
  25. 25. Wikipedia Factors Analysis Factors determined by iterative annotation 4 Factors cover – 91% of comments – 70% of discussions “Other” as 5th catchall Factor Example (used to justify `keep') Notability Anyone covered by another encyclopedic reference is considered notable enough for inclusion in Wikipedia. Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band. Maintenance …this article is savable but at its current state, needs a lot of improvement. Bias It is by no means spam (it does not promote the products). **Other I'm advocating a blanket "hangon" for all articles on newly-drafted players
  26. 26. Factor-based Summarization
  27. 27. Comparison of Annotation • Cohen’s kappa (Cohen, 1960) .48 for Walton’s argumentation schemes .64-.82 for factors, depending on the factor • Potential for task support – Argumentation schemes • Write effective arguments • Ask critical questions to check others' arguments – Factors • Summarize debates
  28. 28. Argumentation mining could be the basis for support tools • Help participants write persuasive arguments – How: provide personalized feedback on drafts – Requires: knowing which arguments are accepted; identifying argumentation in a drafts • Find weaknesses in others’ arguments – How: suggest & instantiate relevant critical questions – Requires: identifying argumentation schemes • Summarize the overall conclusions of the debate – How: identify the winning and losing rationales – Requires: identifying rationales and contradictions
  29. 29. Argumentation mining could be the basis for support tools • Help participants write persuasive arguments – How: provide personalized feedback on drafts – Requires: knowing which arguments are accepted; identifying argumentation in a drafts • Find weaknesses in others’ arguments – How: suggest & instantiate relevant critical questions – Requires: identifying argumentation schemes • Summarize the overall conclusions of the debate – How: identify the winning and losing rationales – Requires: identifying rationales and contradictions
  30. 30. Argumentation mining could be the basis for support tools • Help participants write persuasive arguments – How: provide personalized feedback on drafts – Requires: knowing which arguments are accepted; identifying argumentation in a drafts • Find weaknesses in others’ arguments – How: suggest & instantiate relevant critical questions – Requires: identifying argumentation schemes • Summarize the overall conclusions of the debate – How: identify the winning and losing rationales – Requires: identifying rationales and contradictions
  31. 31. Argumentation Mining papers Arguing on Wikipedia • “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups” CSCW 2013. • “Deletion Discussions in Wikipedia: Decision Factors and Outcomes” WikiSym2012. Arguing in Social Media • “Dimensions of Argumentation in Social Media" EKAW 2012 • “Why did they post that argument? Communicative intentions of Web 2.0 arguments.” Arguing on the Web 2.0 at ISSA 2014 Arguing in Reviews • “Identifying Consumers' Arguments in Text” SWAIE 2012 • “Semi-Automated Argumentative Analysis of Online Product Reviews" COMMA 2012 • “Arguing from a Point of View” Agreement Technologies 2012 Structuring Arguments on the Social Semantic Web • “A Review of Argumentation for the Social Semantic Web” Semantic Web – Interoperability, Usability, Applicability, 2013. • “Identifying, Annotating, and Filtering Arguments and Opinions in Open Collaboration Systems" 2013 Thesis: purl.org/jsphd • “Modeling Arguments in Scientific Papers” at ArgDiaP 2014 http://jodischneider.com/jodi.html
  32. 32. http://purl.org/jsphd
  33. 33. An argument gives a rationale
  34. 34. What do we mean by an argument? • The simplest possible argument connects two Statements by means of an Inference Rule
  35. 35. Cases • Cases support a single conclusion.
  36. 36. Attacks & Debates • Debates argue for and against a single conclusion.
  37. 37. Open collaboration based on • Technological infrastructure • People • Social structures: rules, policies, procedures,…
  38. 38. Open collaboration based on • Technological infrastructure • People • Social structures: – joint decision-making – Importance of rationales: reasons for opinions
  39. 39. HTML5
  40. 40. OpenStreetMap
  41. 41. Example from Corpus
  42. 42. Results: Important tasks for consensus discussions 1. Determine one’s personal position 2. Express one’s personal position in accordance with community norms 3. Determine the consensus
  43. 43. Related work • Dissent and rhetorical devices in bug reporting (Ko and Chilana, 2011) • how Python listservs select enhancement proposals (Barcellini et al., 2005). – role of a participant is related to the kinds of message they quote (Syntheses, Disagreements, Proposals, or Agreements) – Syntheses and Disagreements are the most quoted

×