Digital Enterprise Research Institute                                                                 www.deri.ie         ...
Ad-hoc online task groups• Open source software development• Collaborative writing groups• Standardization bodies
Let’s do something together at the banquet!
How’s the talk length?
Making an argument
Arguments give a POSITION & RATIONALEPosition: Add more question timeRationale: the flow of people between sessions can be...
Arguments are used in decision-makingPosition: Add more question time   Attack positionRationale: the flow of people betwe...
John Danaher
Case Study of Argumentation in    Wikipedia Deletion Discussions1. What arguments are given in content deletion   discussi...
Should we delete this Wikipedia article?                  [Delete the article]...hasnt                    played since 200...
[Delete the article]...hasnt  played since 2008. His 66-73  record is far from stellar and,  in my opinion, does not merit...
“Rule” Argumentation Scheme
“Evidence” Argumentation Scheme
Evidence + Rule -> Conclusion
Corpus Analysis• English Wikipedia deletion discussions• Representative “typical” day: 72 discussions• Iterative Manual An...
Which arguments are given?Compared to general arguing,Wikipedia arguing involves more:  • Rules  • Values  • Bias  • Prece...
Novices often misunderstand policy“if you folks had been around actively working onthe Web in 2000, you would know when th...
Problems with novices’ arguments• Confusion about what “reliable source” (RS) means• Confusion about establishing notabili...
Problems with experts’ arguments• Vagueness or lack of justification• Boilerplate messages lacking in detail• Sourcing (ve...
TypicalPotentially problematic                                                                               Wikipedia    ...
Novices are more likely to use some      problematic arguments • Values    o“obviously of interest to the public in genera...
Emotion• Emotional involvement vs. action  o“I know Wikipedia has a dislike for all things   [article topic]”  o“I and oth...
Future Work• Templates for effective arguments• Semi-automatic argument identification• Reusing argument analysis methods•...
Future Work• Templates for effective arguments• Semi-automatic argument identification• Reusing argument analysis methods•...
Acknowledgements• Science Foundation Ireland Grant No.  SFI/09/CE/I1380 (Líon2)• Annotators: Laura O’Connor and Lyndia Pet...
Research Questions• [RQ1] What arguments are given?• [RQ2] Do people with different levels of experience  with Wikipedia e...
Previous ResearchShallow analysis of large datasets• Redacted content      • West & Lee, “What Wikipedia deletes” WikiSym ...
Argument from Rules -               From Established RuleMajor Premise: If carrying out types of actions includingA is the...
Find counterarguments with “critical               questions”1. Does the rule require carrying out this type of   action?2...
InstantiatingIf stopping at a red light is the established rule fordriving a vehicle, then (unless the case is anexception...
1. Were you driving a vehicle?2. Did a police officer direct you to continue without   stopping?3. Were you driving an ambu...
Differences in Novices vs. Experts• Experts may read all debates
Case Study of Argumentation in   Wikipedia Deletion DiscussionsRQ 1: What arguments are given in contentdeletion discussio...
Arguments about deletion cscw2013 how experience improves the acceptability of arguments in ad hoc online task groups
Arguments about deletion cscw2013 how experience improves the acceptability of arguments in ad hoc online task groups
Arguments about deletion cscw2013 how experience improves the acceptability of arguments in ad hoc online task groups
Arguments about deletion cscw2013 how experience improves the acceptability of arguments in ad hoc online task groups
Arguments about deletion cscw2013 how experience improves the acceptability of arguments in ad hoc online task groups
Arguments about deletion cscw2013 how experience improves the acceptability of arguments in ad hoc online task groups
Upcoming SlideShare
Loading in …5
×

Arguments about deletion cscw2013 how experience improves the acceptability of arguments in ad hoc online task groups

813
-1

Published on

CSCW2013 talk about our analysis of Wikipedia article discussions (Articles for Deletion) using Walton's argumentation schemes.

Paper:
http://jodischneider.com/pubs/cscw2013.pdf

Abstract: Increasingly, ad-hoc online task groups must make decisions about jointly created artifacts such as open source software and Wikipedia articles. Time-consuming and laborious attention to textual discussions is needed to make such decisions, for which computer support would be beneficial. Yet there has been little study of the argumentation patterns that distributed ad-hoc online task groups use in evaluation and decision-making. In a corpus of English Wikipedia deletion discussions, we investigate the argumentation schemes used, the role of the arguer’s experience, and which arguments are acceptable to the audience. We report three main results: First, the most prevalent patterns are the Rules and Evidence schemes from Walton’s catalog of argumentation schemes [34], which comprise 36% of arguments. Second, we find that familiarity with community norms correlates with the novices’ ability to craft persuasive arguments. Third, acceptable arguments use community-appropriate rhetoric that demonstrate knowledge of policies and community values while problematic arguments are based on personal preference and inappropriate analogy to other cases.

Citation:
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups”. In Computer Supported Cooperative Work and Social Computing. San Antonio, TX, February 23-27, 2013.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
813
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Characterised by: communication, shared goals
  • HTML5 decision policyhttp://dev.w3.org/html5/decision-policy/decision-policy-v3.html
  • Twitter #wedo2 searchDecision-making conversations
  • http://philosophicaldisquisitions.blogspot.com/2010/03/argumentation-schemes-part-1.htmlCC-BY-NC-ND Image URL: http://3.bp.blogspot.com/_wuSqJG5bIKE/S7JfZUsNhII/AAAAAAAAAWY/YYG9avlB-Ko/s1600/Argumentation+Schemes.006.png
  • Isn’t it funny that people tweet about this
  • Corpus: a typical day’s deletion discussions500 discussions per week~12% of deletions Lam & Riedl. “Is Wikipedia growing a longer tail?” GROUP ’09Most contentiousArticulated decision making72 deletion discussions - January 29, 2011 - English Wikipedia only
  • Corpus: a typical day’s deletion discussions500 discussions per week~12% of deletions Lam & Riedl. “Is Wikipedia growing a longer tail?” GROUP ’09Most contentiousArticulated decision making72 deletion discussions - January 29, 2011 - English Wikipedia only
  • We would count this as an Argument from Rules
  • five of Wikipedia’s top third mostcommon deletion arguments–Arguments from Rules, Values,Bias, Precedent, and Waste–are not in the top two-thirds ofthe most commonly used arguments (pers. Communication, Snaith) in the only widely-available informal argument corpus, the Araucaria corpus [21]
  • five of Wikipedia’s top third mostcommon deletion arguments–Arguments from Rules, Values,Bias, Precedent, and Waste–are not in the top two-thirds ofthe most commonly used arguments (pers. Communication, Snaith) in the only widely-available informal argument corpus, the Araucaria corpus [21]
  • Experience with deletion has some effect on the skill and rhetoric with which people argue a case. In Wikipedia, policy knowledge is a particular stumbling block for novices. The difference between real-world importance and Wikipedia-importance (notability) is quite confusing to newcomers; this newcomer asks Why an article on Juvenile Justice System Rules prevelant (sic) in the largest province of Punjab having population of more than 90 million people including juveniles is not Notable? Among novices, confusion about reliable sources (RS) is particularly prominent–for instance claiming that a selfpublished biographical website counts as a source for a biography, or providing insufficient detail (e.g. I see notability and RS, without specifying the reliable sources). Verifiability, which is of particular importance for avoiding bias, can also be misunderstood; this newcomer wants to rely on plausibility instead: if you folks had been around actively working on the Web in 2000, you would know when the Dot Com Bubble burst, many, many companies went out of business. Servers with information about me... were taken off line.
  • Delete – notability not demonstrated in a reliable secondary source”five of Wikipedia’s top third mostcommon deletion arguments–Arguments from Rules, Values,Bias, Precedent, and Waste–are not in the top two-thirds ofthe most commonly used arguments (pers. Communication, Snaith) in the only widely-available informal argument corpus, the Araucaria corpus [21]
  • Delete – notability not demonstrated in a reliable secondary source”five of Wikipedia’s top third mostcommon deletion arguments–Arguments from Rules, Values,Bias, Precedent, and Waste–are not in the top two-thirds ofthe most commonly used arguments (pers. Communication, Snaith) in the only widely-available informal argument corpus, the Araucaria corpus [21]
  • Arguing from personal vs. community values
  • Also causeto effectArgumentation from Values is the second most common argument given by novices; this is unfortunate since it is problematic when it is used to argue from individual (rather than accepted community values). Hence novice arguers sometimes provide uncompelling arguments, based on ignoranceof community standards, e.g. to keep an article because it isobviously of interest to the public in general.)Emsworth Cricket Club is one of the longest debates in our sample, yet could have been quickly decided. All four novices–two IP users and two with newly created accounts– advocate keeping, while experienced participants argue unanimously for deletion. While one novice argues well, pointing to the British Newspaper Library (though not specifically to any individual references), the other novices’ comments are largely uncompelling, arguing from personal, but not shared Wikipedian values (e.g. Why just because it is a small team and not major does it not deserve it’s own page on here?). Delete comments, however, cite specific notability policies and the need for reliable secondary sources. The novices have not sufficiently understood the criteria to be applied, or the possible counterarguments. Such discussions can escalate into heated discussions which increase effort and may alienate participants, unless diffused by skillful arguing backed by strong policy. In this case there is strong policy, but perhaps a lack of tact. The final word goes to an experienced participant who urges reading the policy, saying This speicalness (sic) argument is getting lame. Socializing newcomers involves not only informing them about policy, but also taking an encouraging tone; in this case, warning away inexperienced (and hence presumably non-valuable) contributors takes priority over socializing them.
  • five of Wikipedia’s top third mostcommon deletion arguments–Arguments from Rules, Values,Bias, Precedent, and Waste–are not in the top two-thirds ofthe most commonly used arguments (pers. Communication, Snaith) in the only widely-available informal argument corpus, the Araucaria corpus [21]
  • We also suggest increased sensitivity and attention to supportinvolved parties–page creators and those who edited a page22http://en.wikipedia.org/wiki/WP:Notabilitybefore it was nominated for deletion. The emotional attachment of this group adds complication, especially when theyare new to the deletion discussion process. Based on our corpus, newcomers are more likely to take the process personally, making statements such as I know wikipedia (sic) has adislike for all things [article topic]. In contrast, experiencedWikipedians’ contributions do not typically show high levels of emotional involvement. Rather, experienced Wikipedians appear to use personal pronouns to report on actions theyhave taken (e.g. changes made to the article), to suggest newpolicies, to mark uncertainty (e.g. I would say), or to showa personal opinion that may difference from the consensusview (e.g. I don’t expect my saying that to alter the outcomeof this AfD). Statistical analysis of the use of sentiment andpersonal pronouns could provide further evidence for the differences in personal engagement we see. Yet we think thatthese differences are sufficiently clear to warrant immediatecommunity response in attending skillfully to newcomers
  • We also suggest increased sensitivity and attention to supportinvolved parties–page creators and those who edited a page22http://en.wikipedia.org/wiki/WP:Notabilitybefore it was nominated for deletion. The emotional attachment of this group adds complication, especially when theyare new to the deletion discussion process. Based on our corpus, newcomers are more likely to take the process personally, making statements such as I know wikipedia (sic) has adislike for all things [article topic]. In contrast, experiencedWikipedians’ contributions do not typically show high levels of emotional involvement. Rather, experienced Wikipedians appear to use personal pronouns to report on actions theyhave taken (e.g. changes made to the article), to suggest newpolicies, to mark uncertainty (e.g. I would say), or to showa personal opinion that may difference from the consensusview (e.g. I don’t expect my saying that to alter the outcomeof this AfD). Statistical analysis of the use of sentiment andpersonal pronouns could provide further evidence for the differences in personal engagement we see. Yet we think thatthese differences are sufficiently clear to warrant immediatecommunity response in attending skillfully to newcomers
  • Isn’t it funny that people tweet about this
  • 22% of all deletions are speedy deleted for A7: No indication of importance (Geiger & Ford WikiSym 2011)======R. S. Geiger and H. Ford.. In WikiSym ’11, pages 201–202.http://www.wikisym.org/ws2011/_media/proceedings:p201-geiger.pdfS. K. Lam, J. Karim, and J. Riedl. The effects of groupcomposition on decision quality in a social productioncommunity. In GROUP ’10, pages 55–64.Reid Priedhorsky, Jilin Chen, Shyong (Tony) K. Lam, Katherine Panciera, Loren Terveen, and John Riedl. Creating, destroying, and restoring value in wikipedia. In GROUP '07: Proceedings of the 2007 International ACM Conference on Supporting Group Work, pages 259-268.D. Taraborelli and G. L. Ciampaglia. Beyond notability. Collective deliberation on content inclusion in Wikipedia. In Fourth IEEE International Conference on Self-Adaptive and Self-Organizing Systems Workshops, 2010) G. West and I. Lee. What Wikipedia deletes: Characterizing dangerous collaborative content. In WikiSym ’11, pages 25–28.
  • http://philosophicaldisquisitions.blogspot.com/2010/03/argumentation-schemes-part-1.html
  • five of Wikipedia’s top third mostcommon deletion arguments–Arguments from Rules, Values,Bias, Precedent, and Waste–are not in the top two-thirds ofthe most commonly used arguments (pers. Communication, Snaith) in the only widely-available informal argument corpus, the Araucaria corpus [21]
  • The volume of participation is markedly different betweennovices and experts. In our corpus, only 6 novices made morethan one comment, and only 3 novices participated in morethan one debate (2,6,8 debates)(debates avg 2.2, stddev 1.9).By comparison, only 2 experts commented in just one of the72 debates in our sample; overall, experts averaged 5.4 comments in 3.6 articles (stddev 10.1 for comments and 6.6 for articles). While the two largest number of comments (99 comments in 62 articles and 32 comments in 24 articles) were dueto administrative notes and debate closing decisions, three experts participated in 11 to 20 debates and seven experts made10 to 20 comments.While regulars may read all debates, or all debates on a topic,novices are attracted to a debate by their knowledge of a topic,or by their desire to become more active in the community(for instance to support a nomination for adminship). Novicesmay be interested parties – creators or subjects – who may argue to keep an article without sufficiently understanding thecriteria to be applied. Cases which attract many novices canbecome contentious, causing challenges unless there is strongpolicy to be applied [24]. In our sample, ‘no consensus’ discussions either lacked significant discussion (2) or involveddiscussions with a novice creator (2). Discussions with article creators showed strong emotion in five out of six cases, asindicated by the use of high sentiment words.Novices arguing for deletion of an article are more likely tobe participating in multiple arguments, and are more likely tocite policy correctly. Our sample showed a clear differencein the rhetoric of keep and delete comments, for example, inthe ten comments left by IP users. Keep (6) comments exhibited emotional involvement and indicated confusion aboutpolices, especially on notability, original research, and veri-fiability. Delete comments (4) used standard formatting andsupporting resources. While further research is needed, thisprovides deeper insight on Lam et al.’s earlier findings thatKeep outcomes involving novices are more likely to be overturned [24].Experience with
  • Isn’t it funny that people tweet about this
  • Arguments about deletion cscw2013 how experience improves the acceptability of arguments in ad hoc online task groups

    1. 1. Digital Enterprise Research Institute www.deri.ie Arguments about Deletion How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups Jodi Schneider, Krystian Samp, Alexandre Passant, and Stefan Decker @jschneider CSCW 2013: Computer Supported Tuesday 26th February 2013 Cooperative Work and Social Computing San Antonio, Texas Copyright 2011 Digital Enterprise Research Institute. All rights reserved. Enabling Networked Knowledge 1
    2. 2. Ad-hoc online task groups• Open source software development• Collaborative writing groups• Standardization bodies
    3. 3. Let’s do something together at the banquet!
    4. 4. How’s the talk length?
    5. 5. Making an argument
    6. 6. Arguments give a POSITION & RATIONALEPosition: Add more question timeRationale: the flow of people between sessions can be distracting
    7. 7. Arguments are used in decision-makingPosition: Add more question time Attack positionRationale: the flow of people between sessions can be distracting Attack rationale Attack inference
    8. 8. John Danaher
    9. 9. Case Study of Argumentation in Wikipedia Deletion Discussions1. What arguments are given in content deletion discussions?2. Differences in novices’ and experts’ arguments?3. Which argumentation schemes are accepted?
    10. 10. Should we delete this Wikipedia article? [Delete the article]...hasnt played since 2008. His 66-73 record is far from stellar and, in my opinion, does not merit an article. >>He pitched last month and plays for the Venezuelan League. This meets our article criteria.
    11. 11. [Delete the article]...hasnt played since 2008. His 66-73 record is far from stellar and, in my opinion, does not merit an article.>>He pitched last month and plays for the Venezuelan League. This meets our article criteria.
    12. 12. “Rule” Argumentation Scheme
    13. 13. “Evidence” Argumentation Scheme
    14. 14. Evidence + Rule -> Conclusion
    15. 15. Corpus Analysis• English Wikipedia deletion discussions• Representative “typical” day: 72 discussions• Iterative Manual Annotation w/ multiple coders
    16. 16. Which arguments are given?Compared to general arguing,Wikipedia arguing involves more: • Rules • Values • Bias • Precedent • Avoiding Waste
    17. 17. Novices often misunderstand policy“if you folks had been around actively working onthe Web in 2000, you would know when the DotCom Bubble burst, many, many companies wentout of business. Servers with information aboutme... were taken off line.”
    18. 18. Problems with novices’ arguments• Confusion about what “reliable source” (RS) means• Confusion about establishing notability with RS’s• Confusion about verification and need for RS’s
    19. 19. Problems with experts’ arguments• Vagueness or lack of justification• Boilerplate messages lacking in detail• Sourcing (verification vs. importance of topic)• Mixing case-debate with policy development
    20. 20. TypicalPotentially problematic Wikipedia Argumentationargument terminology SchemesPersonal preference Values ILIKEITFew search engine hits Ignorance Google testMany search engine hits Cause to Effect Google test Values or Waste orRequesting a favor PLEASEDONT Practical ReasoningAnalogy to other cases Analogy OTHERNo harm in keeping an article Copyright 2011 Digital Enterprise Research Institute. All rights reserved. Values or Waste NOHARMTopic will be notable in the Practical Reasoning Networked Knowledge Enabling CRYSTALBALLfuture
    21. 21. Novices are more likely to use some problematic arguments • Values o“obviously of interest to the public in general” • Analogy o“just as special as an article on a breed of dog or something similar”
    22. 22. Emotion• Emotional involvement vs. action o“I know Wikipedia has a dislike for all things [article topic]” o“I and others have added several third-party reliable sources to the article.”• Understanding the process helps o “I believe that (much as it would break my heart based on the no of hours I have put in over the years working on the article) it is perhaps sensible that the piece is deleted.”
    23. 23. Future Work• Templates for effective arguments• Semi-automatic argument identification• Reusing argument analysis methods• Incentivizing social sensitivity• Classifying emotional needs and triggers• Understanding impact on newcomers & article creators
    24. 24. Future Work• Templates for effective arguments• Semi-automatic argument identification• Reusing argument analysis methods• Incentivizing social sensitivity• Classifying emotional needs and triggers• Understanding impact on newcomers & article creators Thanks! @jschneider jodi.schneider@deri.org
    25. 25. Acknowledgements• Science Foundation Ireland Grant No. SFI/09/CE/I1380 (Líon2)• Annotators: Laura O’Connor and Lyndia Peters• Trevor Bench-Capon, Luigina Ciolfi, Bernie Hogan, David Randall, Mark Snaith, Adam Wyner• Thanks to reviewers and CSCW revisions process!
    26. 26. Research Questions• [RQ1] What arguments are given?• [RQ2] Do people with different levels of experience with Wikipedia editing or the Wikipedia deletion process provide different types of arguments?• [RQ3] Which argumentation schemes are accepted?
    27. 27. Previous ResearchShallow analysis of large datasets• Redacted content • West & Lee, “What Wikipedia deletes” WikiSym 2011• Vote sequencing • Taraborelli & Ciampaglia “Beyond notability” SASOW 2011• Decision quality • Lam, Karim & Riedl “The effects of group composition on decision quality in a social production community”, GROUP 2010• Who participates, what & how much gets deleted • Priedhorsky, Chen, Lam, Panciera, Terveen, & Riedl. “Creating, destroying, and restoring value in Wikipedia”, GROUP 2007 • Geiger & Ford “Participation in Wikipedia’s article deletion processes”, WikiSym 2011 30
    28. 28. Argument from Rules - From Established RuleMajor Premise: If carrying out types of actions includingA is the established rule for x, then (unless the case is anexception), a must carry out A.Minor Premise: Carrying out types of actions including Aisthe established rule for a.Conclusion: Therefore, a must carry out A.
    29. 29. Find counterarguments with “critical questions”1. Does the rule require carrying out this type of action?2. Are there other established rules that might conflict with or override this one?3. Are there extenuating circumstances or an excuse for noncompliance?
    30. 30. InstantiatingIf stopping at a red light is the established rule fordriving a vehicle, then (unless the case is anexception), drivers must stop at a red light.Stopping at a red light is the established rule fordrivers.Therefore, drivers must stop at a red light.
    31. 31. 1. Were you driving a vehicle?2. Did a police officer direct you to continue without stopping?3. Were you driving an ambulance with its siren on?
    32. 32. Differences in Novices vs. Experts• Experts may read all debates
    33. 33. Case Study of Argumentation in Wikipedia Deletion DiscussionsRQ 1: What arguments are given in contentdeletion discussions?RQ 2: Differences in novices’ and experts’arguments?RQ 3: Which argumentation schemes areaccepted?

    ×