0
Formalization, User Strategy and Interaction Design: Users’ Behaviour with Discourse Tagging Semantics Bertrand Sereno*, S...
Overview <ul><li>The problem:  collaboration semantics in contested domains  —  no consensus assumed; possibly not even po...
Scholarly discourse as CKS… Philosophical Transactions of the Royal Society of London March 1665 Le Journal des S ça v ans...
Scholarly discourse as CKS…  Beyond document citations… ? applies These annotations are freeform summaries of an idea, as ...
Scholarly discourse as CKS…   Connecting freeform tags with  naturalistic connections (“dialects”)  grounded in a  formal ...
source type comparativeness polarity Mancini, C. and Buckingham Shum, S.J. (2006).  Modelling Discourse in Contested Domai...
Scholarly discourse as CKS…   Visualising claims and arguments The link-tracking service allows the user to specify struct...
Scholarly discourse as CKS…   Querying on argumentation structures
Scholarly discourse as CKS…   “What papers contrast with this paper?” <ul><li>Extract concepts for this document </li></ul...
The point is… we think these kinds of structures are worth having  But can users create them?
How to help scholars engage in CKS? Pilot study: paper-based literature modelling S. Buckingham Shum, V. Uren, G. Li, B. S...
How to help scholars engage in CKS?   From paper prototype to semiformal mapping tool <ul><li>The  ClaiMapper  tool </li><...
How to help scholars engage in CKS? Pilot study: paper-based annotation Pilot study reported in: B. Sereno, S. Buckingham ...
How to help scholars engage in CKS? <ul><li>The  ClaimSpotter  annotation tool: Web 2.0-style tagging with optional commun...
A user-centred approach to the formative evaluation of a CKS tool <ul><li>Research question: </li></ul><ul><ul><li>how do ...
Example  claims  (tag triples) from users <ul><li>Domain ontology  is about  A hierarchy of URIs on multiple levels </li><...
User study: selected results <ul><li>See paper for details </li></ul><ul><li>and the PhD for complete account B. Sereno. A...
Tag length similar for novices and experts (64% 1-3 words) B. Sereno. A Document-Centric Semantic Annotation Environment t...
Tag reuse most of them used only twice in this study (1 hour) B. Sereno. A Document-Centric Semantic Annotation Environmen...
Transcript analysis  to derive themes, sub-categories and codes
Themes from the user study <ul><li>Formalization </li></ul><ul><ul><li>the cognitive task of structured tagging </li></ul>...
Formalization the cognitive task of structured tagging <ul><li>Looking for the right tag type… </li></ul>
Formalization the cognitive task of structured tagging <ul><li>Looking for the right link type… </li></ul>
User Strategy how users approach the semantic annotation task <ul><li>What granularity and type of claims? When to stop? <...
User Strategy how users approach the semantic annotation task <ul><li>No initial use of tagging aids — focus is on own tag...
Interaction Design how behaviour is shaped by the tool’s affordances <ul><li>‘ Flip’ left/right tags to match the link typ...
Interaction Design how behaviour is shaped by the tool’s affordances <ul><li>Skimming highlighted text </li></ul>
Lessons Learnt & Design Principles <ul><li>Untrained users can do it : in their first hour they created coherent claims. U...
ClaimSpotter: papers and demos http://kmi.open.ac.uk/projects/hyperdiscourse/tools/claimspotter  Hypermedia Discourse proj...
Upcoming SlideShare
Loading in...5
×

Social and Collaborative Construction of Structured Knowledge WWW2007

1,847

Published on

Sereno, B., Buckingham Shum, S. and Motta, E. (2007). Formalization, User Strategy and Interaction Design: Users’ Behaviour with Discourse Tagging Semantics. Workshop on Social and Collaborative Construction of Structured Knowledge, 16th International World Wide Web Conference (WWW 2007), Banff, AB, Canada; 8-12 May 2007. [http://www2007.org/workshops/paper_30.pdf]

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,847
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
25
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Social and Collaborative Construction of Structured Knowledge WWW2007"

  1. 1. Formalization, User Strategy and Interaction Design: Users’ Behaviour with Discourse Tagging Semantics Bertrand Sereno*, Simon Buckingham Shum & Enrico Motta Knowledge Media Institute, The Open University, Milton Keynes, UK * Now at: Centre for Advanced Learning Technologies, INSEAD, Fontainebleau, France Workshop on Social and Collaborative Construction of Structured Knowledge, 16th International World Wide Web Conference, Banff, Canada, May 8, 2007 Licensed under Creative Commons Attribution-ShareAlike 2.0 License Acknowledgements:
  2. 2. Overview <ul><li>The problem: collaboration semantics in contested domains — no consensus assumed; possibly not even possible </li></ul><ul><li>Previous work: Scholarly discourse as Collaborative Knowledge Structuring (CKS) </li></ul><ul><ul><li>Modelling and querying Web argument structures </li></ul></ul><ul><li>How to help users engage in CKS ? </li></ul><ul><ul><li>Evaluating the ClaimSpotter tool </li></ul></ul><ul><li>Summary of evaluation results and design principles </li></ul><ul><ul><li>Formalization / User Strategy / Interaction Design </li></ul></ul>
  3. 3. Scholarly discourse as CKS… Philosophical Transactions of the Royal Society of London March 1665 Le Journal des S ça v ans January 1665 From: To: Chaomei Chen, 2006: Citation analysis Buckingham Shum et al, 2003: lineage analysis
  4. 4. Scholarly discourse as CKS… Beyond document citations… ? applies These annotations are freeform summaries of an idea, as one would also find in researchers’ journals, fieldnotes, lit. review notes or blog entries Making formal connections between ideas creates a semantic citation network —> novel literature navigation, querying and visualization Method Theory Paper: “The Scent of a Site: A System for Analyzing and Predicting Information Scent, Usage, and Usability of a Web Site” “ Web User Flow by Information Scent (WUFIS)” Paper: “Information foraging” “ Information foraging theory” “ Information scent models” “ People try to maximise their rate of gaining information” Claim
  5. 5. Scholarly discourse as CKS… Connecting freeform tags with naturalistic connections (“dialects”) grounded in a formal set of relations (from semiotics and coherence relations)
  6. 6. source type comparativeness polarity Mancini, C. and Buckingham Shum, S.J. (2006). Modelling Discourse in Contested Domains: A Semiotic and Cognitive Framework . International Journal of Human Computer Studies, 64, (11), pp.1154-1171 Cognitive Coherence Relations —towards a relational ontology for discourse
  7. 7. Scholarly discourse as CKS… Visualising claims and arguments The link-tracking service allows the user to specify structures e.g. show tags one link out from any tag on the left hand of a claim containing the string “CiteSeer” Evaluated in: V. Uren, S. Buckingham Shum, G. Li, and M. Bachler. Sensemaking Tools for Understanding Research Literatures: Design, Implementation and User Evaluation . International Journal of Human Computer Studies , 64(5):420–445, 2006
  8. 8. Scholarly discourse as CKS… Querying on argumentation structures
  9. 9. Scholarly discourse as CKS… “What papers contrast with this paper?” <ul><li>Extract concepts for this document </li></ul><ul><li>Trace concepts on which they build </li></ul><ul><li>Trace concepts challenging this set </li></ul><ul><li>Show root documents </li></ul>Evaluated in: V. Uren, S. Buckingham Shum, G. Li, and M. Bachler. Sensemaking Tools for Understanding Research Literatures: Design, Implementation and User Evaluation . International Journal of Human Computer Studies , 64(5):420–445, 2006
  10. 10. The point is… we think these kinds of structures are worth having But can users create them?
  11. 11. How to help scholars engage in CKS? Pilot study: paper-based literature modelling S. Buckingham Shum, V. Uren, G. Li, B. Sereno, and C. Mancini. Computational Modelling of Naturalistic Argumentation in Research Literatures: Representation and Interaction Design Issues . International Journal of Intelligent Systems , 22(1):17–47, 2006
  12. 12. How to help scholars engage in CKS? From paper prototype to semiformal mapping tool <ul><li>The ClaiMapper tool </li></ul>Evaluated in: V. Uren, S. Buckingham Shum, G. Li, and M. Bachler. Sensemaking Tools for Understanding Research Literatures: Design, Implementation and User Evaluation . International Journal of Human Computer Studies , 64(5):420–445, 2006 Starting from paper-based modelling, move from literature sketches… … to formal argument maps
  13. 13. How to help scholars engage in CKS? Pilot study: paper-based annotation Pilot study reported in: B. Sereno, S. Buckingham Shum, and E. Motta. (2005). ClaimSpotter: an Environment to Support Sensemaking with Knowledge Triples . Proc. Int. Conf. Intelligent User Interfaces , pages 199–206, ACM
  14. 14. How to help scholars engage in CKS? <ul><li>The ClaimSpotter annotation tool: Web 2.0-style tagging with optional community/system tag recommendations </li></ul>
  15. 15. A user-centred approach to the formative evaluation of a CKS tool <ul><li>Research question: </li></ul><ul><ul><li>how do annotators approach the task of using a new Web tool to semantically annotate a document with its key contributions? </li></ul></ul><ul><li>Focus </li></ul><ul><ul><li>..is on how the tool functionality and UI affordances shape tagging behaviour </li></ul></ul><ul><li>Quantitative and qualitative analysis </li></ul>
  16. 16. Example claims (tag triples) from users <ul><li>Domain ontology is about A hierarchy of URIs on multiple levels </li></ul><ul><li>Universal physical access is unlikely to affect Digital divide </li></ul><ul><li>Hypertext node juxtaposition is analogous to Cinematic shot juxtaposition </li></ul><ul><li>[ Evidence ] In the Bristol trial, the awareness of the presence of other players was correlated with how much our participants enjoyed the game as well as with how engaged they felt is consistent with Presence awareness of many other people is capable of causing, feel good factor </li></ul><ul><li>Magpie moves away from hypermedia towards open service-based architectures is evidence for [Magpie improves on COHSE] </li></ul>
  17. 17. User study: selected results <ul><li>See paper for details </li></ul><ul><li>and the PhD for complete account B. Sereno. A Document-Centric Semantic Annotation Environment to Support Sense-Making. PhD Thesis, Technical Report KMI-06-13, Knowledge Media Institute, The Open University, UK, September 2005. [http://kmi.open.ac.uk/publications/pdf/KMI-TR-06-13.pdf] </li></ul>
  18. 18. Tag length similar for novices and experts (64% 1-3 words) B. Sereno. A Document-Centric Semantic Annotation Environment to Support Sense-Making. PhD Thesis, Technical Report KMI-06-13 ), Knowledge Media Institute, The Open University, UK, September 2005. [http://kmi.open.ac.uk/publications/pdf/KMI-TR-06-13.pdf] — p.161
  19. 19. Tag reuse most of them used only twice in this study (1 hour) B. Sereno. A Document-Centric Semantic Annotation Environment to Support Sense-Making. PhD Thesis, Technical Report KMI-06-13 ), Knowledge Media Institute, The Open University, UK, September 2005. [http://kmi.open.ac.uk/publications/pdf/KMI-TR-06-13.pdf] — p.259
  20. 20. Transcript analysis to derive themes, sub-categories and codes
  21. 21. Themes from the user study <ul><li>Formalization </li></ul><ul><ul><li>the cognitive task of structured tagging </li></ul></ul><ul><li>User Strategy </li></ul><ul><ul><li>how users approach the semantic annotation task </li></ul></ul><ul><li>Interaction Design </li></ul><ul><ul><li>how behaviour is shaped by the tool’s affordances </li></ul></ul>
  22. 22. Formalization the cognitive task of structured tagging <ul><li>Looking for the right tag type… </li></ul>
  23. 23. Formalization the cognitive task of structured tagging <ul><li>Looking for the right link type… </li></ul>
  24. 24. User Strategy how users approach the semantic annotation task <ul><li>What granularity and type of claims? When to stop? </li></ul>
  25. 25. User Strategy how users approach the semantic annotation task <ul><li>No initial use of tagging aids — focus is on own tags </li></ul>
  26. 26. Interaction Design how behaviour is shaped by the tool’s affordances <ul><li>‘ Flip’ left/right tags to match the link type </li></ul>
  27. 27. Interaction Design how behaviour is shaped by the tool’s affordances <ul><li>Skimming highlighted text </li></ul>
  28. 28. Lessons Learnt & Design Principles <ul><li>Untrained users can do it : in their first hour they created coherent claims. UI design validated to this degree. —future work: longitudinal evaluation at scale </li></ul><ul><li>New users attend to what is highlighted for them (matching tags; primary doct.), and generally don’t click down a level — next version combines visualizations and document-centric features </li></ul><ul><li>Support incremental formalization — cf. use of is-about as a placeholder link; provide an Other… category and try to map automatically to the ontology </li></ul><ul><li>Users’ strategies vary — don’t assume a strong workflow a paper-based pilot study can provide insights into this </li></ul><ul><li>Web 2.0 UI simplicity : good design needed to provide high functionality, walk-up-and-use CKS tools — we overwhelmed some users with overlaid suggestions for tags </li></ul>
  29. 29. ClaimSpotter: papers and demos http://kmi.open.ac.uk/projects/hyperdiscourse/tools/claimspotter Hypermedia Discourse project: theories / tools / case studies / user studies: face-face and asynch. interaction http://kmi.open.ac.uk/projects/hyperdiscourse collaboration / semantics / usability / community informatics / argumentation http://www.PragmaticWeb.info Short/full paper submission deadline: 14 May 2 nd International Conference on The Pragmatic Web 22-23 Oct., Tilburg, NL
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×