Jodi Schneider
WIMMICS seminar
INRIA Sophia Antipolis – Méditerranée & I3S
vendredi 21 février 2014
How can we make sense of disagreement?
Arguments & opinions give a rationale
Reuse arguments & rationales
o How do we make arguments more clear to BOTH
humans and machines?
o Explicit arguments are n...
Arguments in collective decision-making
Arguments about content deletion
Arguments in open collaboration systems
Open collaboration systems
―people form ties with others & create things together‖
(Forte and Lampe 2013)
Open collaboration systems
―people form ties with others & create things together‖
(Forte and Lampe 2013)
Examples:
o Wiki...
How do we enable the reuse of
arguments and opinions
in open collaboration systems?
Use case: deletion in Wikipedia
o 1 in 4 new Wikipedia articles is deleted – within
minutes or hours
o Demotivating!
• 1 i...
Thousands of new editors each month
Source: http://reportcard.wmflabs.org/
English
All languages
Policies grew 10-15X (2002-2008)
See e.g. Butler, Joyce, and Pike. CHI 2008
"Don't look now, but we've created a bureaucra...
Supporting open collaboration systems
o Can we support editor retention?
o Make criteria explicit to:
• Explain community ...
Research Questions
How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and
requirem...
How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and requirem...
How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and requirem...
How do we enable
the reuse of arguments and opinions
on the World Wide Web?
o RQ1: What are the opportunities and requirem...
RQ1: What are the opportunities and
requirements for providing
argumentation support?
Methodology: Netnography
Kozinets, Robert V.
Netnography: Doing ethnographic research online.
Sage Publications, 2010.
Methodology: Netnography
1. Planning and community selection
2. Participant observation and data collection
3. Data analys...
Results: Sample corpus
72 discussions started on 1 day
o Each discussion has:
• 3—33 messages
• 2—15 participants
o 741 me...
Example from Corpus
Example from Corpus
Results: Terminology and policy
knowledge becomes an obstacle
Results: Important tasks
for consensus discussions
1. Determine one’s personal position
2. Express one’s personal position...
RQ2: Which arguments are used in
open collaboration systems?
Methods
o Use corpus of 72 discussions
o Two types of annotation: 2 argumentation theories
o Iterative annotation for each...
Two argumentation theories
o Walton’s Argumentation Schemes
(Walton, Reed, and Macagno 2008)
• Informal argumentation
(phi...
Walton’s Argumentation Schemes
Example Argumentation Scheme:
Argument from Rules – “we apply rule X”
Critical Questions
1....
Walton’s Argumentation Schemes
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker.
―Arguments about Deletion:...
Factors Analysis
Factors determined
by iterative annotation
Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
• 91% of comments
• 70% of discussions
Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
• 91% of comments
• 70% of discussions
―Other‖...
Factors Analysis
Factors determined
by iterative annotation
4 Factors cover
• 91% of comments
• 70% of discussions
―Other‖...
RQ3: How can we structure and
display opinions and arguments to
support filtering?
Methodology
o Linked Data Application Development
o User testing – 20 users
Original
Discussion
Ontology
Semantic
Enrichme...
Add a discussion summary
Add a discussion summary
Semantically enrich messages
Use semantic structure
Implementation based on Jodi Schneider and Krystian Samp
“Alternative Interfaces for Deletion Discu...
Experimental design
System A (Control)
System B (Experimental)
Experimental design
Experimental design
PU* - Perceived usefulness
PE* - Perceived ease of use
DC -Decision completeness
PF - Perceived effort
IC* - Information
c...
Final survey
Results: 84% prefer our system
―Information is structured and I can quickly get an
overview of the key arguments.‖
―The ab...
Overall contributions
o A procedure for providing argumentation support
o A demonstration of this procedure, including
• A...
Main papers used in the thesis
o Jodi Schneider, Krystian Samp, Alexandre Passant, and
Stefan Decker. ―Arguments about Del...
Summary
o We need better ways of structuring arguments
on the Web.
o Arguments vary across Social Media.
o Different theor...
Copyright 2011 Digital Enterprise Research Institute. All rights reserved.
Digital Enterprise Research Institute www.deri....
Identifying, annotating, and filtering arguments and opinions on the social web  - PhD viva slides reprised for WIMMICS se...
Identifying, annotating, and filtering arguments and opinions on the social web  - PhD viva slides reprised for WIMMICS se...
Identifying, annotating, and filtering arguments and opinions on the social web  - PhD viva slides reprised for WIMMICS se...
Identifying, annotating, and filtering arguments and opinions on the social web  - PhD viva slides reprised for WIMMICS se...
Identifying, annotating, and filtering arguments and opinions on the social web  - PhD viva slides reprised for WIMMICS se...
Identifying, annotating, and filtering arguments and opinions on the social web  - PhD viva slides reprised for WIMMICS se...
Upcoming SlideShare
Loading in …5
×

Identifying, annotating, and filtering arguments and opinions on the social web - PhD viva slides reprised for WIMMICS seminar 2014-02-21

448 views

Published on

How do we support people in using/reusing arguments and opinions on the World Wide Web? WIMMICS, INRIA seminar based on my PhD viva slides.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
448
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Arguments are everywhere – we would like to reuse argumentsincluding on the Web.From http://www.sendareview.com/The Web is full of opinions & commentary.A lot of it disagrees.How do we learn from other people, when they disagree?
  • 1“only 0.6 percent of those whose articles are met with deletion stayed editing, compared to 4.4 percent of the users whose articles remained”, http://enwp.org/Wikipedia:Wikipedia_ Signpost/2011-04-04/Editor_retentionOriginal stats from Mr.Z-man: http://en.wikipedia.org/wiki/User:Mr.Z-man/newusers
  • ***Get just blue figures***
  • Image from http://breakingenergy.com/2013/04/08/energy-legal-work-takes-the-number-two-spot-growth-forecast/***Picture conveying social norms/policies – e.g. growth over time from Butler (if there’s some image of that)
  • Image from http://kozinets.net/archives/509
  • Technically started or relistedCorpus is https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Log/2011_January_29
  • Categories (Walton’s argumentation schemes) vs. process (factors analysis)
  • Major Premise: If carrying out types of actions including A is the established rule for x, then (unless the case is an exception), a must carry out A.Minor Premise: Carrying out types of actions including A is the established rule for a.Conclusion: Therefore, a must carry out A.
  • Earlier in CSCW: Jodi Schneider, KrystianSamp, Alexandre Passant, Stefan Decker. “Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups”. In Computer Supported Cooperative Work and Social Computing (CSCW). San Antonio, TX, February 23-27, 2013.Used as categoriesInitial annotation60 categories (each Walton argumentation scheme)all arguments in each messageRound 415 most common argumentation schemesmain argument in each messageGood inter-annotator agreement for hard task:54% agreement (compared to 12% chance) among 2 annotators
  • 20 novice participants used both systems“The ability to navigate the comments made it a bit easier to filter my mind set and to come to a conclusion.”“summarise and, at the same time, evaluate which factor should be considered determinant for the final decision”
  • **go out with a bang**
  • Identifying, annotating, and filtering arguments and opinions on the social web - PhD viva slides reprised for WIMMICS seminar 2014-02-21

    1. 1. Jodi Schneider WIMMICS seminar INRIA Sophia Antipolis – Méditerranée & I3S vendredi 21 février 2014
    2. 2. How can we make sense of disagreement?
    3. 3. Arguments & opinions give a rationale
    4. 4. Reuse arguments & rationales o How do we make arguments more clear to BOTH humans and machines? o Explicit arguments are not available • Important in bug reports, political commentary, product reviews, etc. o Machine-readable arguments could help • Gather information – e.g. finding issues, claims, and opinion clusters • Connect opinions to explicit evidence • Navigate claims networks
    5. 5. Arguments in collective decision-making
    6. 6. Arguments about content deletion
    7. 7. Arguments in open collaboration systems
    8. 8. Open collaboration systems ―people form ties with others & create things together‖ (Forte and Lampe 2013)
    9. 9. Open collaboration systems ―people form ties with others & create things together‖ (Forte and Lampe 2013) Examples: o Wikipedia o HTML5 working group o OpenStreetMap o Project Gutenberg – Distributed Proofreaders o Apache projects, Mozilla Firefox, …
    10. 10. How do we enable the reuse of arguments and opinions in open collaboration systems?
    11. 11. Use case: deletion in Wikipedia o 1 in 4 new Wikipedia articles is deleted – within minutes or hours o Demotivating! • 1 in 3 newcomers start by writing a new article • 7X less likely to stay if their article is deleted! o Can we support editor retention? Source: http://enwp.org/Wikipedia:Wikipedia_Signpost/2011-04-04/Editor_retention
    12. 12. Thousands of new editors each month Source: http://reportcard.wmflabs.org/ English All languages
    13. 13. Policies grew 10-15X (2002-2008) See e.g. Butler, Joyce, and Pike. CHI 2008 "Don't look now, but we've created a bureaucracy: the nature and roles of policies and rules in Wikipedia."
    14. 14. Supporting open collaboration systems o Can we support editor retention? o Make criteria explicit to: • Explain community expectations (how to be convincing) • Support making & auditing decisions
    15. 15. Research Questions
    16. 16. How do we enable the reuse of arguments and opinions on the World Wide Web? o RQ1: What are the opportunities and requirements for providing argumentation support? o RQ2: Which arguments are used in open collaboration systems? o RQ3: How can we structure and display opinions and arguments to support filtering?
    17. 17. How do we enable the reuse of arguments and opinions on the World Wide Web? o RQ1: What are the opportunities and requirements for providing argumentation support? o RQ2: Which arguments are used in open collaboration systems? o RQ3: How can we structure and display opinions and arguments to support filtering?
    18. 18. How do we enable the reuse of arguments and opinions on the World Wide Web? o RQ1: What are the opportunities and requirements for providing argumentation support? o RQ2: Which arguments are used in open collaboration systems? o RQ3: How can we structure and display opinions and arguments to support filtering?
    19. 19. How do we enable the reuse of arguments and opinions on the World Wide Web? o RQ1: What are the opportunities and requirements for providing argumentation support? Netnography o RQ2: Which arguments are used in open collaboration systems? Iterative Annotation o RQ3: How can we structure and display opinions and arguments to support filtering? Semantic Web Systems Development
    20. 20. RQ1: What are the opportunities and requirements for providing argumentation support?
    21. 21. Methodology: Netnography Kozinets, Robert V. Netnography: Doing ethnographic research online. Sage Publications, 2010.
    22. 22. Methodology: Netnography 1. Planning and community selection 2. Participant observation and data collection 3. Data analysis and iterative interpretation 4. Presenting results
    23. 23. Results: Sample corpus 72 discussions started on 1 day o Each discussion has: • 3—33 messages • 2—15 participants o 741 messages contributed by 244 users. Each message has 3—350+ words. o 98 printed A4 sheets
    24. 24. Example from Corpus
    25. 25. Example from Corpus
    26. 26. Results: Terminology and policy knowledge becomes an obstacle
    27. 27. Results: Important tasks for consensus discussions 1. Determine one’s personal position 2. Express one’s personal position in accordance with community norms 3. Determine the consensus
    28. 28. RQ2: Which arguments are used in open collaboration systems?
    29. 29. Methods o Use corpus of 72 discussions o Two types of annotation: 2 argumentation theories o Iterative annotation for each theory • Multiple annotators • Refine to get good inter-annotator agreement • 4 rounds of annotation
    30. 30. Two argumentation theories o Walton’s Argumentation Schemes (Walton, Reed, and Macagno 2008) • Informal argumentation (philosophical & computational argumentation) • Identify & prevent errors in reasoning (fallacies) • 60 patterns o Factors Analysis (Ashley 1991) • Case-based reasoning • E.g. factors for deciding cases in trade secret law, favoring either party (the plaintiff or the defendant).
    31. 31. Walton’s Argumentation Schemes Example Argumentation Scheme: Argument from Rules – “we apply rule X” Critical Questions 1. Does the rule require carrying out this type of action? 2. Are there other established rules that might conflict with or override this one? 3. Are there extenuating circumstances or an excuse for noncompliance? Walton, Reed, and Macagno 2008
    32. 32. Walton’s Argumentation Schemes Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. ―Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups‖. In CSCW 2013. ! "#$%&' ()*+((&"' *"&, +-&' . & ! "#$%&' ()*"+%), -./&' 0&)(+)123+(4&5.5 67897: ! "#$%&' ()*"+%); $<&5 6=87>: ? +(& 6@8=7: ! "#$%&' (A(.+' )*"+%)BA<$&5 C89>: ! "#$%&' ()*"+%)? &&/)*+")1&<3 C869: ! "#$%&' ()*"+%)D.A5 @8EF: ? +)"&A5+' )#.- &' @899: ! "#$%&' ()*"+%)G+5.(.+' )(+)H' +I @8>J : ! "#$%&' ()*"+%)G"&0&/&' ( @8>J : ! "#$%&' ()*"+%)K#' +"A' 0& 987F: ! "#$%&' ()*"+%)L+%3+5.(.+' 98J =: ! "#$%&' ()*"+%)LA$5&)(+), **&0( 98@6: ! "#$%&' ()*"+%)! ' A<+#2 989@: ! "#$%&' ()*"+%)MA5(& 989@: G"A0(.0A<); &A5+' .' # 989@: ! "#8)*"+%)B&"NA<)L<A55.*.0A(.+' 98>=:
    33. 33. Factors Analysis Factors determined by iterative annotation
    34. 34. Factors Analysis Factors determined by iterative annotation 4 Factors cover • 91% of comments • 70% of discussions
    35. 35. Factors Analysis Factors determined by iterative annotation 4 Factors cover • 91% of comments • 70% of discussions ―Other‖ as 5th catchall
    36. 36. Factors Analysis Factors determined by iterative annotation 4 Factors cover • 91% of comments • 70% of discussions ―Other‖ as 5th catchall Factor Example (used to justify `keep') Notability Anyone covered by another encyclopedic reference is considered notable enough for inclusion in Wikipedia. Sources Basic information about this album at a minimum is certainly verifiable, it's a major label release, and a highly notable band. Maintenance …this article is savable but at its current state, needs a lot of improvement. Bias It is by no means spam (it does not promote the products). **Other I'm advocating a blanket "hangon" for all articles on newly- drafted players
    37. 37. RQ3: How can we structure and display opinions and arguments to support filtering?
    38. 38. Methodology o Linked Data Application Development o User testing – 20 users Original Discussion Ontology Semantic Enrichment Semantically Enriched RDFa Querying Queryable User Interface With Barchart
    39. 39. Add a discussion summary
    40. 40. Add a discussion summary
    41. 41. Semantically enrich messages
    42. 42. Use semantic structure Implementation based on Jodi Schneider and Krystian Samp “Alternative Interfaces for Deletion Discussions in Wikipedia: Some Proposals Using Decision Factors. [Demo]” In WikiSym2012.
    43. 43. Experimental design
    44. 44. System A (Control)
    45. 45. System B (Experimental)
    46. 46. Experimental design
    47. 47. Experimental design
    48. 48. PU* - Perceived usefulness PE* - Perceived ease of use DC -Decision completeness PF - Perceived effort IC* - Information completeness Statistical Significance PU* p < .001 PE* p .001 IC* p .039
    49. 49. Final survey
    50. 50. Results: 84% prefer our system ―Information is structured and I can quickly get an overview of the key arguments.‖ ―The ability to navigate the comments made it a bit easier to filter my mind set and to come to a conclusion.‖ ―It offers the structure needed to consider each factor separately, thus making the decision easier. Also, the number of comments per factor offers a quick indication of the relevance and the deepness of the decision.‖ Based on a 20 participant user test. 1 participant did not take the final survey
    51. 51. Overall contributions o A procedure for providing argumentation support o A demonstration of this procedure, including • A requirements analysis • A categorization of the most common arguments used according to two theories o Walton’s argumentation schemes o Factors-dimensions theory • An ontology for argumentation in Wikipedia deletion discussions. • An argumentation filtering system that visually summarizes arguments with bar charts of the decision factors.
    52. 52. Main papers used in the thesis o Jodi Schneider, Krystian Samp, Alexandre Passant, and Stefan Decker. ―Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups‖. In CSCW 2013. o Jodi Schneider, Tudor Groza, Alexandre Passant, ―A Review of Argumentation for the Social Semantic Web.‖ Semantic Web – Interoperability, Usability, Applicability, 2013, 4(2), 159-218. o Jodi Schneider and Krystian Samp. ―Alternative Interfaces for Deletion Discussions in Wikipedia: Some Proposals Using Decision Factors. [Demo]‖ In WikiSym2012. o Jodi Schneider, Alexandre Passant, and Stefan Decker. ―Deletion Discussions in Wikipedia: Decision Factors and Outcomes.‖ In WikiSym2012.
    53. 53. Summary o We need better ways of structuring arguments on the Web. o Arguments vary across Social Media. o Different theories of argumentation stress different aspects. o Factors analysis is useful for providing a brief summary of discussions. This can help find consensus.
    54. 54. Copyright 2011 Digital Enterprise Research Institute. All rights reserved. Digital Enterprise Research Institute www.deri.ie Enabling Networked Knowledge Identifying, Annotating, and Filtering Arguments and Opinions in Open Collaboration Systems Jodi Schneider vendredi 21 fev 2014 60 WIMMICS seminar INRIA Sophia Antipolis - Méditerranée and I3S

    ×