Jodi Schneider
First Workshop on Argumentation Mining at ACL 2014
Baltimore, Maryland
2014-06-26
Argumentation mining today
• No unified vision of the field. Multiple:
– Interrelated problems
– Application domains
– Too...
Argumentation mining today
• No unified vision of the field. Multiple:
– Interrelated problems
– Application domains
– Too...
Application: Debates in Open Online
Collaboration
• Wikipedia
• HTML5
• OpenStreetMap
• Project Gutenberg
• Apache project...
Debates argue for & against
a single conclusion
Which content belongs in Wikipedia?
500 discussions per week!!!
Argument-based support
• How can I win an argument?
Which arguments sway the community?
• Why were previous decisions made...
Argument-based support
• How can I win an argument?
Which arguments sway the community?
• Why were previous decisions made...
Corpus: 72 discussions started on 1 day
• Each discussion has:
3-33 messages
2-15 participants
• 741 messages contributed ...
Approach
• Compare two argumentation theories
• Iterative annotation with multiple annotators
– Refine to get good inter-a...
We used two argumentation theories
• Walton’s Argumentation Schemes
(Walton, Reed, and Macagno 2008)
– Informal argumentat...
Walton’s Argumentation Schemes
Example Argumentation Scheme:
Argument from Rules – “we apply rule X”
Critical Questions
1....
Example: "Stop at a red light"
1. Does the rule require carrying out this type of action?
Were you driving a vehicle?
2. A...
Walton’s Argumentation Schemes
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker.
“Arguments about Deletion:...
How to win an argument with a
Wikipedian?
• Argument from Evidence to Hypothesis (19%)
• Argument from Rules (17%)
How to win an argument (Arucaria)?
Classifying Arguments by Scheme. Vanessa Wei Feng. Master's thesis, Toronto, 2010.
Experts vs. Novices
• Experts were more likely to use
– Argument from Precedent
• Novices were more likely to use
– Argume...
Unsuccessful arguments from novices
• Emsworth Cricket Club is one of the oldest
cricket clubs in the world, and this real...
Newcomers don't understand
how to counterargue
Factors/Dimensions Analysis
• Factors (case-based reasoning)
– All or nothing
• Either present ("applicable") or absent
• ...
Example factors analysis
(Aleven 1997)
Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussio...
Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussio...
Wikipedia Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
– 91% of comments
– 70% of discussio...
Factor-based Summarization
Comparison of Annotation
• Cohen’s kappa (Cohen, 1960)
.48 for Walton’s argumentation schemes
.64-.82 for factors, dependi...
Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide pe...
Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide pe...
Argumentation mining could be the
basis for support tools
• Help participants write persuasive arguments
– How: provide pe...
Argumentation Mining papers
Arguing on Wikipedia
• “Arguments about Deletion: How Experience Improves the Acceptability of...
http://purl.org/jsphd
An argument gives a rationale
What do we mean by an argument?
• The simplest possible argument connects two
Statements by means of an Inference Rule
Cases
• Cases support a single conclusion.
Attacks & Debates
• Debates argue for and against a single
conclusion.
Open collaboration based on
• Technological infrastructure
• People
• Social structures: rules, policies, procedures,…
Open collaboration based on
• Technological infrastructure
• People
• Social structures:
– joint decision-making
– Importa...
HTML5
OpenStreetMap
Example from Corpus
Results: Important tasks
for consensus discussions
1. Determine one’s personal position
2. Express one’s personal position...
Related work
• Dissent and rhetorical devices in bug reporting
(Ko and Chilana, 2011)
• how Python listservs select enhanc...
Envisioning argumentation and decision making support for debates in open online collaboration communities--acl arg mining...
Upcoming SlideShare
Loading in …5
×

Envisioning argumentation and decision making support for debates in open online collaboration communities--acl arg mining workshop 2014-06-26

854 views
687 views

Published on

Paper for the First Workshop on Argumentation Mining at the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, Maryland, June 26 2014

Abstract:
Argumentation mining, a relatively new area of discourse analysis, involves automatically identifying and structuring arguments. Following a basic introduction to argumentation, we describe a new possible domain for argumentation mining: debates in open online collaboration communities. Based on our experience with manual annotation of arguments in debates, we envision argumentation mining as the basis for three kinds of support tools, for authoring more persuasive arguments, finding weaknesses in others’ arguments, and summarizing a debate’s overall conclusions.

Full paper:
http://jodischneider.com/pubs/aclargmining2014.pdf

Proceedings with links:
http://acl2014.org/acl2014/W14-21/index.html

Workshop homepage:
http://www.uncg.edu/cmp/ArgMining2014/

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
854
On SlideShare
0
From Embeds
0
Number of Embeds
34
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 12-12:30
    20 minutes + questions

    4th paper in 90 minute session 11-12:30

    Paper:
    http://jodischneider.com/pubs/aclargmining2014.pdf


    Workshop homepage:
    http://www.uncg.edu/cmp/ArgMining2014/

    Proceedings with links:
    http://acl2014.org/acl2014/W14-21/index.html


    Abstract:
    Argumentation mining, a relatively new area of discourse analysis, involves automatically identifying and structuring arguments. Following a basic introduction to argumentation, we describe a new possible domain for argumentation mining: debates in open online collaboration communities. Based on our experience with manual annotation of arguments in debates, we envision argumentation mining as the basis for three kinds of support tools, for authoring more persuasive arguments, finding weaknesses in others’ arguments, and summarizing a debate’s overall conclusions.
  • “people form ties with others &
    create things together”
    (Forte and Lampe 2013)
  • Factors provide a good way to organize the debate;

    Filtering discussions based on each factor can show the rationale topic by topic, which supported decision making in a pilot user-based evaluation

    16 of 19 participants (84%) preferred
  • See (Schneider et al., WikiSym 2012) and (Schneider et al., CSCW 2013)
  • When the argumentation scheme used in a draft message is not generally accepted, the author could be warned that their message might not be persuasive, and given personalized suggestions

    Listing these questions in concrete and contextualized form (drawing on the premises, inference rules, and conclusions to instantiate and contextualize them) would encourage participants to consider the pos- sible flaws in reasoning and might prompt partici- pants to request answers within the debate.

    Macro- argumentation, such as the factors analysis de- scribed above, would be a natural choice for sum- marization, as it has already proven useful for fil- tering discussions. A more reasoning-intensive approach would be to calculate consistent out- comes (Wyner and van Engers, 2010), if debates can be easily formalized.

  • When the argumentation scheme used in a draft message is not generally accepted, the author could be warned that their message might not be persuasive, and given personalized suggestions

    Listing these questions in concrete and contextualized form (drawing on the premises, inference rules, and conclusions to instantiate and contextualize them) would encourage participants to consider the pos- sible flaws in reasoning and might prompt partici- pants to request answers within the debate.

    Macro- argumentation, such as the factors analysis de- scribed above, would be a natural choice for sum- marization, as it has already proven useful for fil- tering discussions. A more reasoning-intensive approach would be to calculate consistent out- comes (Wyner and van Engers, 2010), if debates can be easily formalized.

  • When the argumentation scheme used in a draft message is not generally accepted, the author could be warned that their message might not be persuasive, and given personalized suggestions

    Listing these questions in concrete and contextualized form (drawing on the premises, inference rules, and conclusions to instantiate and contextualize them) would encourage participants to consider the pos- sible flaws in reasoning and might prompt partici- pants to request answers within the debate.

    Macro- argumentation, such as the factors analysis de- scribed above, would be a natural choice for sum- marization, as it has already proven useful for fil- tering discussions. A more reasoning-intensive approach would be to calculate consistent out- comes (Wyner and van Engers, 2010), if debates can be easily formalized.

  • Envisioning argumentation and decision-making support for debates in open online collaboration communities
  • “people form ties with others &
    create things together”
    (Forte and Lampe 2013)

    The prototypical open collaboration system is an online environment that
    supports the collective production of an artifact
    through a technologically mediated collaboration platform
    (3) that presents a low barrier to entry and exit, and
    (4) supports the emergence of persistent but malleable social structures.



    "open collaboration system" = Artifact + Technology + Ad hoc community + Social Structures

    Image from http://breakingenergy.com/2013/04/08/energy-legal-work-takes-the-number-two-spot-growth-forecast/

    ***Picture conveying social norms/policies – e.g. growth over time from Butler (if there’s some image of that)



  • “people form ties with others &
    create things together”
    (Forte and Lampe 2013)

    The prototypical open collaboration system is an online environment that
    supports the collective production of an artifact
    through a technologically mediated collaboration platform
    (3) that presents a low barrier to entry and exit, and
    (4) supports the emergence of persistent but malleable social structures.



    "open collaboration system" = Artifact + Technology + Ad hoc community + Social Structures

    Image from http://breakingenergy.com/2013/04/08/energy-legal-work-takes-the-number-two-spot-growth-forecast/

    ***Picture conveying social norms/policies – e.g. growth over time from Butler (if there’s some image of that)



  • https://github.com/gravitystorm/openstreetmap-carto/issues/435
  • The organizational relevance of these open decision making discussions in collaborative communities makes them a promising target for support, and argumentation mining technology is an appropriate tool to deploy towards that end.

  • Envisioning argumentation and decision making support for debates in open online collaboration communities--acl arg mining workshop 2014-06-26

    1. 1. Jodi Schneider First Workshop on Argumentation Mining at ACL 2014 Baltimore, Maryland 2014-06-26
    2. 2. Argumentation mining today • No unified vision of the field. Multiple: – Interrelated problems – Application domains – Tools handling one aspect of annotation • Few corpora • Need for – Common definition(s) of argumentation – "Challenge problems" – Shared corpora – Applications
    3. 3. Argumentation mining today • No unified vision of the field. Multiple: – Interrelated problems – Application domains – Tools handling one aspect of annotation • Few corpora • Need for – Common definition(s) of argumentation – "Challenge problems" – Shared corpora – Our Application: debates in online collaboration
    4. 4. Application: Debates in Open Online Collaboration • Wikipedia • HTML5 • OpenStreetMap • Project Gutenberg • Apache projects • Mozilla Firefox • …
    5. 5. Debates argue for & against a single conclusion
    6. 6. Which content belongs in Wikipedia?
    7. 7. 500 discussions per week!!!
    8. 8. Argument-based support • How can I win an argument? Which arguments sway the community? • Why were previous decisions made? • Which ongoing debates need more comments?
    9. 9. Argument-based support • How can I win an argument? Which arguments sway the community? • Why were previous decisions made? • Which ongoing debates need more comments?
    10. 10. Corpus: 72 discussions started on 1 day • Each discussion has: 3-33 messages 2-15 participants • 741 messages contributed by 244 users. Each message has 3-350+ words. • 98 printed A4 sheets
    11. 11. Approach • Compare two argumentation theories • Iterative annotation with multiple annotators – Refine to get good inter-annotator agreement • 4 rounds of annotation – Rounds 1-2 by me – Rounds 3-4 by 2 assistants
    12. 12. We used two argumentation theories • Walton’s Argumentation Schemes (Walton, Reed, and Macagno 2008) – Informal argumentation (philosophical & computational argumentation) – Identify & prevent errors in reasoning (fallacies) – 60 patterns • Factors/Dimensions Analysis (Ashley 1991; Bench-Capon and Rissland, 2001) – Case-based reasoning – E.g. factors for deciding cases in trade secret law, favoring either party (the plaintiff or the defendant).
    13. 13. Walton’s Argumentation Schemes Example Argumentation Scheme: Argument from Rules – “we apply rule X” Critical Questions 1. Does the rule require carrying out this type of action? 2. Are there other established rules that might conflict with or override this one? 3. Are there extenuating circumstances or an excuse for noncompliance? Walton, Reed, and Macagno 2008
    14. 14. Example: "Stop at a red light" 1. Does the rule require carrying out this type of action? Were you driving a vehicle? 2. Are there other established rules that might conflict with or override this one? Did a police officer direct you to continue without stopping? 3. Are there extenuating circumstances or an excuse for noncompliance? Were you driving an ambulance with its siren on? Critical Questions from Argument from Rules based on Walton, Reed, and Macagno 2008
    15. 15. Walton’s Argumentation Schemes Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups”. In CSCW 2013. ! "#$%&' ()*+((&"' *"&, +-&' . & ! "#$%&' ()*"+%), - ./&' 0&)(+)123+(4&5.5 67897: ! "#$%&' ()*"+%); $<&5 6=87>: ? +(& 6@8=7: ! "#$%&' (A(.+' )*"+%)BA<$&5 C89>: ! "#$%&' ()*"+%)? &&/)*+")1&<3 C869: ! "#$%&' ()*"+%)D.A5 @8EF: ? +)"&A5+' )#.- &' @899: ! "#$%&' ()*"+%)G+5.(.+' )(+)H' +I @8>J : ! "#$%&' ()*"+%)G"&0&/&' ( @8>J : ! "#$%&' ()*"+%)K#' +"A' 0& 987F: ! "#$%&' ()*"+%)L+%3+5.(.+' 98J =: ! "#$%&' ()*"+%)LA$5&)(+), **&0( 98@6: ! "#$%&' ()*"+%)! ' A<+#2 989@: ! "#$%&' ()*"+%)MA5(& 989@: G"A0(.0A<); &A5+' .' # 989@: ! "#8)*"+%)B&"NA<)L<A55.*.0A(.+' 98>=:
    16. 16. How to win an argument with a Wikipedian? • Argument from Evidence to Hypothesis (19%) • Argument from Rules (17%)
    17. 17. How to win an argument (Arucaria)? Classifying Arguments by Scheme. Vanessa Wei Feng. Master's thesis, Toronto, 2010.
    18. 18. Experts vs. Novices • Experts were more likely to use – Argument from Precedent • Novices were more likely to use – Argumentation from Values – Argumentation from Cause to Effect – Argument from Analogy
    19. 19. Unsuccessful arguments from novices • Emsworth Cricket Club is one of the oldest cricket clubs in the world, and this really is worth a mention. Especially on a website, where pointless people … gets a mention. • Why just because it is a small team and not major does it not deserve it’s (sic) own page on here?
    20. 20. Newcomers don't understand how to counterargue
    21. 21. Factors/Dimensions Analysis • Factors (case-based reasoning) – All or nothing • Either present ("applicable") or absent • When present, a factor always favors the same side • Dimensions – More complex/subtle • Can be applicable to a varying degree ("sliding scale") • Favor plantiff on one extreme; defendant on the other Ashley 1991; Bench-Capon and Rissland, 2001
    22. 22. Example factors analysis (Aleven 1997)
    23. 23. Wikipedia Factors Analysis Factors determined by iterative annotation 4 Factors cover – 91% of comments – 70% of discussions “Other” as 5th catchall
    24. 24. Wikipedia Factors Analysis Factors determined by iterative annotation 4 Factors cover – 91% of comments – 70% of discussions “Other” as 5th catchall Factor Example (used to justify `keep') Notability Anyone covered by another encyclopedic reference is considered notable enough for inclusion in Wikipedia. Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band. Maintenance …this article is savable but at its current state, needs a lot of improvement. Bias It is by no means spam (it does not promote the products). **Other I'm advocating a blanket "hangon" for all articles on newly-drafted players
    25. 25. Wikipedia Factors Analysis Factors determined by iterative annotation 4 Factors cover – 91% of comments – 70% of discussions “Other” as 5th catchall Factor Example (used to justify `keep') Notability Anyone covered by another encyclopedic reference is considered notable enough for inclusion in Wikipedia. Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band. Maintenance …this article is savable but at its current state, needs a lot of improvement. Bias It is by no means spam (it does not promote the products). **Other I'm advocating a blanket "hangon" for all articles on newly-drafted players
    26. 26. Factor-based Summarization
    27. 27. Comparison of Annotation • Cohen’s kappa (Cohen, 1960) .48 for Walton’s argumentation schemes .64-.82 for factors, depending on the factor • Potential for task support – Argumentation schemes • Write effective arguments • Ask critical questions to check others' arguments – Factors • Summarize debates
    28. 28. Argumentation mining could be the basis for support tools • Help participants write persuasive arguments – How: provide personalized feedback on drafts – Requires: knowing which arguments are accepted; identifying argumentation in a drafts • Find weaknesses in others’ arguments – How: suggest & instantiate relevant critical questions – Requires: identifying argumentation schemes • Summarize the overall conclusions of the debate – How: identify the winning and losing rationales – Requires: identifying rationales and contradictions
    29. 29. Argumentation mining could be the basis for support tools • Help participants write persuasive arguments – How: provide personalized feedback on drafts – Requires: knowing which arguments are accepted; identifying argumentation in a drafts • Find weaknesses in others’ arguments – How: suggest & instantiate relevant critical questions – Requires: identifying argumentation schemes • Summarize the overall conclusions of the debate – How: identify the winning and losing rationales – Requires: identifying rationales and contradictions
    30. 30. Argumentation mining could be the basis for support tools • Help participants write persuasive arguments – How: provide personalized feedback on drafts – Requires: knowing which arguments are accepted; identifying argumentation in a drafts • Find weaknesses in others’ arguments – How: suggest & instantiate relevant critical questions – Requires: identifying argumentation schemes • Summarize the overall conclusions of the debate – How: identify the winning and losing rationales – Requires: identifying rationales and contradictions
    31. 31. Argumentation Mining papers Arguing on Wikipedia • “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups” CSCW 2013. • “Deletion Discussions in Wikipedia: Decision Factors and Outcomes” WikiSym2012. Arguing in Social Media • “Dimensions of Argumentation in Social Media" EKAW 2012 • “Why did they post that argument? Communicative intentions of Web 2.0 arguments.” Arguing on the Web 2.0 at ISSA 2014 Arguing in Reviews • “Identifying Consumers' Arguments in Text” SWAIE 2012 • “Semi-Automated Argumentative Analysis of Online Product Reviews" COMMA 2012 • “Arguing from a Point of View” Agreement Technologies 2012 Structuring Arguments on the Social Semantic Web • “A Review of Argumentation for the Social Semantic Web” Semantic Web – Interoperability, Usability, Applicability, 2013. • “Identifying, Annotating, and Filtering Arguments and Opinions in Open Collaboration Systems" 2013 Thesis: purl.org/jsphd • “Modeling Arguments in Scientific Papers” at ArgDiaP 2014 http://jodischneider.com/jodi.html
    32. 32. http://purl.org/jsphd
    33. 33. An argument gives a rationale
    34. 34. What do we mean by an argument? • The simplest possible argument connects two Statements by means of an Inference Rule
    35. 35. Cases • Cases support a single conclusion.
    36. 36. Attacks & Debates • Debates argue for and against a single conclusion.
    37. 37. Open collaboration based on • Technological infrastructure • People • Social structures: rules, policies, procedures,…
    38. 38. Open collaboration based on • Technological infrastructure • People • Social structures: – joint decision-making – Importance of rationales: reasons for opinions
    39. 39. HTML5
    40. 40. OpenStreetMap
    41. 41. Example from Corpus
    42. 42. Results: Important tasks for consensus discussions 1. Determine one’s personal position 2. Express one’s personal position in accordance with community norms 3. Determine the consensus
    43. 43. Related work • Dissent and rhetorical devices in bug reporting (Ko and Chilana, 2011) • how Python listservs select enhancement proposals (Barcellini et al., 2005). – role of a participant is related to the kinds of message they quote (Syntheses, Disagreements, Proposals, or Agreements) – Syntheses and Disagreements are the most quoted

    ×