Identifying, annotating, and filtering arguments and opinions on the social web  - PhD viva slides reprised for WIMMICS seminar 2014-02-21
Upcoming SlideShare
Loading in...5
×
 

Identifying, annotating, and filtering arguments and opinions on the social web - PhD viva slides reprised for WIMMICS seminar 2014-02-21

on

  • 114 views

How do we support people in using/reusing arguments and opinions on the World Wide Web? WIMMICS, INRIA seminar based on my PhD viva slides.

How do we support people in using/reusing arguments and opinions on the World Wide Web? WIMMICS, INRIA seminar based on my PhD viva slides.

Statistics

Views

Total Views
114
Views on SlideShare
114
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Arguments are everywhere – we would like to reuse argumentsincluding on the Web.From http://www.sendareview.com/The Web is full of opinions & commentary.A lot of it disagrees.How do we learn from other people, when they disagree?
  • 1“only 0.6 percent of those whose articles are met with deletion stayed editing, compared to 4.4 percent of the users whose articles remained”, http://enwp.org/Wikipedia:Wikipedia_ Signpost/2011-04-04/Editor_retentionOriginal stats from Mr.Z-man: http://en.wikipedia.org/wiki/User:Mr.Z-man/newusers
  • ***Get just blue figures***
  • Image from http://breakingenergy.com/2013/04/08/energy-legal-work-takes-the-number-two-spot-growth-forecast/***Picture conveying social norms/policies – e.g. growth over time from Butler (if there’s some image of that)
  • Image from http://kozinets.net/archives/509
  • Technically started or relistedCorpus is https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Log/2011_January_29
  • Categories (Walton’s argumentation schemes) vs. process (factors analysis)
  • Major Premise: If carrying out types of actions including A is the established rule for x, then (unless the case is an exception), a must carry out A.Minor Premise: Carrying out types of actions including A is the established rule for a.Conclusion: Therefore, a must carry out A.
  • Earlier in CSCW: Jodi Schneider, KrystianSamp, Alexandre Passant, Stefan Decker. “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups”. In Computer Supported Cooperative Work and Social Computing (CSCW). San Antonio, TX, February 23-27, 2013.Used as categoriesInitial annotation60 categories (each Walton argumentation scheme)all arguments in each messageRound 415 most common argumentation schemesmain argument in each messageGood inter-annotator agreement for hard task:54% agreement (compared to 12% chance) among 2 annotators
  • 20 novice participants used both systems“The ability to navigate the comments made it a bit easier to filter my mind set and to come to a conclusion.”“summarise and, at the same time, evaluate which factor should be considered determinant for the final decision”
  • **go out with a bang**

Identifying, annotating, and filtering arguments and opinions on the social web  - PhD viva slides reprised for WIMMICS seminar 2014-02-21 Identifying, annotating, and filtering arguments and opinions on the social web - PhD viva slides reprised for WIMMICS seminar 2014-02-21 Presentation Transcript

  • Jodi Schneider WIMMICS seminar INRIA Sophia Antipolis – Méditerranée & I3S vendredi 21 février 2014
  • How can we make sense of disagreement?
  • Arguments & opinions give a rationale
  • Reuse arguments & rationales o How do we make arguments more clear to BOTH humans and machines? o Explicit arguments are not available • Important in bug reports, political commentary, product reviews, etc. o Machine-readable arguments could help • Gather information – e.g. finding issues, claims, and opinion clusters • Connect opinions to explicit evidence • Navigate claims networks
  • Arguments in collective decision-making
  • Arguments about content deletion
  • Arguments in open collaboration systems
  • Open collaboration systems ―people form ties with others & create things together‖ (Forte and Lampe 2013)
  • Open collaboration systems ―people form ties with others & create things together‖ (Forte and Lampe 2013) Examples: o Wikipedia o HTML5 working group o OpenStreetMap o Project Gutenberg – Distributed Proofreaders o Apache projects, Mozilla Firefox, …
  • How do we enable the reuse of arguments and opinions in open collaboration systems?
  • Use case: deletion in Wikipedia o 1 in 4 new Wikipedia articles is deleted – within minutes or hours o Demotivating! • 1 in 3 newcomers start by writing a new article • 7X less likely to stay if their article is deleted! o Can we support editor retention? Source: http://enwp.org/Wikipedia:Wikipedia_Signpost/2011-04-04/Editor_retention
  • Thousands of new editors each month Source: http://reportcard.wmflabs.org/ English All languages
  • Policies grew 10-15X (2002-2008) See e.g. Butler, Joyce, and Pike. CHI 2008 "Don't look now, but we've created a bureaucracy: the nature and roles of policies and rules in Wikipedia."
  • Supporting open collaboration systems o Can we support editor retention? o Make criteria explicit to: • Explain community expectations (how to be convincing) • Support making & auditing decisions
  • Research Questions
  • How do we enable the reuse of arguments and opinions on the World Wide Web? o RQ1: What are the opportunities and requirements for providing argumentation support? o RQ2: Which arguments are used in open collaboration systems? o RQ3: How can we structure and display opinions and arguments to support filtering?
  • How do we enable the reuse of arguments and opinions on the World Wide Web? o RQ1: What are the opportunities and requirements for providing argumentation support? o RQ2: Which arguments are used in open collaboration systems? o RQ3: How can we structure and display opinions and arguments to support filtering?
  • How do we enable the reuse of arguments and opinions on the World Wide Web? o RQ1: What are the opportunities and requirements for providing argumentation support? o RQ2: Which arguments are used in open collaboration systems? o RQ3: How can we structure and display opinions and arguments to support filtering?
  • How do we enable the reuse of arguments and opinions on the World Wide Web? o RQ1: What are the opportunities and requirements for providing argumentation support? Netnography o RQ2: Which arguments are used in open collaboration systems? Iterative Annotation o RQ3: How can we structure and display opinions and arguments to support filtering? Semantic Web Systems Development
  • RQ1: What are the opportunities and requirements for providing argumentation support?
  • Methodology: Netnography Kozinets, Robert V. Netnography: Doing ethnographic research online. Sage Publications, 2010.
  • Methodology: Netnography 1. Planning and community selection 2. Participant observation and data collection 3. Data analysis and iterative interpretation 4. Presenting results
  • Results: Sample corpus 72 discussions started on 1 day o Each discussion has: • 3—33 messages • 2—15 participants o 741 messages contributed by 244 users. Each message has 3—350+ words. o 98 printed A4 sheets
  • Example from Corpus
  • Example from Corpus
  • Results: Terminology and policy knowledge becomes an obstacle
  • Results: Important tasks for consensus discussions 1. Determine one’s personal position 2. Express one’s personal position in accordance with community norms 3. Determine the consensus
  • RQ2: Which arguments are used in open collaboration systems?
  • Methods o Use corpus of 72 discussions o Two types of annotation: 2 argumentation theories o Iterative annotation for each theory • Multiple annotators • Refine to get good inter-annotator agreement • 4 rounds of annotation
  • Two argumentation theories o Walton’s Argumentation Schemes (Walton, Reed, and Macagno 2008) • Informal argumentation (philosophical & computational argumentation) • Identify & prevent errors in reasoning (fallacies) • 60 patterns o Factors Analysis (Ashley 1991) • Case-based reasoning • E.g. factors for deciding cases in trade secret law, favoring either party (the plaintiff or the defendant).
  • Walton’s Argumentation Schemes Example Argumentation Scheme: Argument from Rules – “we apply rule X” Critical Questions 1. Does the rule require carrying out this type of action? 2. Are there other established rules that might conflict with or override this one? 3. Are there extenuating circumstances or an excuse for noncompliance? Walton, Reed, and Macagno 2008
  • Walton’s Argumentation Schemes Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. ―Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups‖. In CSCW 2013. ! "#$%&' ()*+((&"' *"&, +-&' . & ! "#$%&' ()*"+%), -./&' 0&)(+)123+(4&5.5 67897: ! "#$%&' ()*"+%); $<&5 6=87>: ? +(& 6@8=7: ! "#$%&' (A(.+' )*"+%)BA<$&5 C89>: ! "#$%&' ()*"+%)? &&/)*+")1&<3 C869: ! "#$%&' ()*"+%)D.A5 @8EF: ? +)"&A5+' )#.- &' @899: ! "#$%&' ()*"+%)G+5.(.+' )(+)H' +I @8>J : ! "#$%&' ()*"+%)G"&0&/&' ( @8>J : ! "#$%&' ()*"+%)K#' +"A' 0& 987F: ! "#$%&' ()*"+%)L+%3+5.(.+' 98J =: ! "#$%&' ()*"+%)LA$5&)(+), **&0( 98@6: ! "#$%&' ()*"+%)! ' A<+#2 989@: ! "#$%&' ()*"+%)MA5(& 989@: G"A0(.0A<); &A5+' .' # 989@: ! "#8)*"+%)B&"NA<)L<A55.*.0A(.+' 98>=:
  • Factors Analysis Factors determined by iterative annotation
  • Factors Analysis Factors determined by iterative annotation 4 Factors cover • 91% of comments • 70% of discussions
  • Factors Analysis Factors determined by iterative annotation 4 Factors cover • 91% of comments • 70% of discussions ―Other‖ as 5th catchall
  • Factors Analysis Factors determined by iterative annotation 4 Factors cover • 91% of comments • 70% of discussions ―Other‖ as 5th catchall Factor Example (used to justify `keep') Notability Anyone covered by another encyclopedic reference is considered notable enough for inclusion in Wikipedia. Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band. Maintenance …this article is savable but at its current state, needs a lot of improvement. Bias It is by no means spam (it does not promote the products). **Other I'm advocating a blanket "hangon" for all articles on newly- drafted players
  • RQ3: How can we structure and display opinions and arguments to support filtering?
  • Methodology o Linked Data Application Development o User testing – 20 users Original Discussion Ontology Semantic Enrichment Semantically Enriched RDFa Querying Queryable User Interface With Barchart
  • Add a discussion summary
  • Add a discussion summary
  • Semantically enrich messages
  • Use semantic structure Implementation based on Jodi Schneider and Krystian Samp “Alternative Interfaces for Deletion Discussions in Wikipedia: Some Proposals Using Decision Factors. [Demo]” In WikiSym2012.
  • Experimental design
  • System A (Control)
  • System B (Experimental)
  • Experimental design
  • Experimental design
  • PU* - Perceived usefulness PE* - Perceived ease of use DC -Decision completeness PF - Perceived effort IC* - Information completeness Statistical Significance PU* p < .001 PE* p .001 IC* p .039
  • Final survey
  • Results: 84% prefer our system ―Information is structured and I can quickly get an overview of the key arguments.‖ ―The ability to navigate the comments made it a bit easier to filter my mind set and to come to a conclusion.‖ ―It offers the structure needed to consider each factor separately, thus making the decision easier. Also, the number of comments per factor offers a quick indication of the relevance and the deepness of the decision.‖ Based on a 20 participant user test. 1 participant did not take the final survey
  • Overall contributions o A procedure for providing argumentation support o A demonstration of this procedure, including • A requirements analysis • A categorization of the most common arguments used according to two theories o Walton’s argumentation schemes o Factors-dimensions theory • An ontology for argumentation in Wikipedia deletion discussions. • An argumentation filtering system that visually summarizes arguments with bar charts of the decision factors.
  • Main papers used in the thesis o Jodi Schneider, Krystian Samp, Alexandre Passant, and Stefan Decker. ―Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups‖. In CSCW 2013. o Jodi Schneider, Tudor Groza, Alexandre Passant, ―A Review of Argumentation for the Social Semantic Web.‖ Semantic Web – Interoperability, Usability, Applicability, 2013, 4(2), 159-218. o Jodi Schneider and Krystian Samp. ―Alternative Interfaces for Deletion Discussions in Wikipedia: Some Proposals Using Decision Factors. [Demo]‖ In WikiSym2012. o Jodi Schneider, Alexandre Passant, and Stefan Decker. ―Deletion Discussions in Wikipedia: Decision Factors and Outcomes.‖ In WikiSym2012.
  • Summary o We need better ways of structuring arguments on the Web. o Arguments vary across Social Media. o Different theories of argumentation stress different aspects. o Factors analysis is useful for providing a brief summary of discussions. This can help find consensus.
  • Copyright 2011 Digital Enterprise Research Institute. All rights reserved. Digital Enterprise Research Institute www.deri.ie Enabling Networked Knowledge Identifying, Annotating, and Filtering Arguments and Opinions in Open Collaboration Systems Jodi Schneider vendredi 21 fev 2014 60 WIMMICS seminar INRIA Sophia Antipolis - Méditerranée and I3S