Hypermedia Discourse: Theory & Technology for the Pragmatic Web?

SRI International, AI Centre, 5 May 2007 Hypermedia Discourse: Theory & Technology for the Pragmatic Web? Simon Buckingham Shum Knowledge Media Institute & Computing Research Centre The Open University, Milton Keynes, UK kmi.open.ac.uk/people/sbs sbs@acm.org Web Pragmatics Hypermedia Discourse 1 © Simon Buckingham Shum

Acknowledgements Compendium Project: Scholarly Ontologies Project: Al Selvin (Verizon/Open U.) Victoria Uren Maarten Sierhuis (NASA) Gangmin Li Jeff Conklin (CogNexus Inst.) Clara Mancini Michelle Bachler (Open U.) Neil Benn Bertrand Sereno John Domingue Enrico Motta Funding gratefully acknowledged: 2 © Simon Buckingham Shum

web pragmatics? the Pragmatic Web? pragmatic webs? “THE PRAGMATIC WEB CONFERENCE is a unique forum to envision and debate how the emerging social, semantic, multimedia Web mediates the ways in which we construct shared meaning. While there is much research and development into topics relevant to this challenge such as collaboration, usability, knowledge representation, and social informatics, the Pragmatic Web conference provides common ground for dialogue at the nexus of these topics.” 3 © Simon Buckingham Shum

CONTEXT CONVERSATION When contexts change, This is how trust is built, and meanings change in things get done. How do we conversations, documents, and do this fluently over the web? models of the world. How does this shape our use for formal semantics on the Web? What is the interest in the “Pragmatic Web”? COMMITMENT / ACTION How do we support the expression, and coordination, of commitments over the Web? 4 © Simon Buckingham Shum

Pragmatic Web research challenges —from the ICPW’07 CfP: www.PragmaticWeb.info  How can we better understand the usefulness, and limitations, of a concept such as \"Web Pragmatics”  What pragmatic design principles improve websites where trust and commitment to action are central?  What are the tradeoffs for users of more structured Web collaboration media? (e.g. in learnability, scaleability, intelligibility)  How can participatory work practices and collaboration tools be orchestrated in the design of the standards, data models and ontologies that underpin data-driven Web applications? 5 © Simon Buckingham Shum

Pragmatic Web research challenges  What role does pragmatics play in the design of personalised information and personalised actions channelled through the Web?  What impact (intended or unintended, productive or disruptive) do different levels of computational infrastructure have on Web pragmatics?  How can we clarify our understandings of increasingly important concepts on the Web such as \"social ties\", \"metadata\", \"knowledge representation\", and \"transaction\"?  If \"context\" is pivotal in making human interaction meaningful, how can we take context into account to improve Web applications? 6 © Simon Buckingham Shum

The essence of this talk: Pragmatics Web Hypermedia Discourse 7 © Simon Buckingham Shum

The essence of this talk: Web Pragmatics :-) Hypermedia Discourse 8 © Simon Buckingham Shum

Modelling ~ Discourse The discourse of modelling: How can we support the discourse that drives the development and contextualisation of the models underpinning interactive systems? Modelling discourse: Can we usefully model (structure) sensemaking discourse, without in the process obstructing it? 9 © Simon Buckingham Shum

Discourse Verbal and written workplace communication Discourse communities: “making and taking perspectives” Dialogue Argumentation Claim making Analytical narrative Meetings 10 © Simon Buckingham Shum

Hypermedia Modelling discourse relations Expressing different perspectives on a conceptual space Supporting the incremental formalization of ideas Rendering structural visualizations Connecting heterogeneous content 11 © Simon Buckingham Shum

Characteristics of Hypermedia Discourse Discourse Ontology Literacy/ Notation(s) Fluency Intuitive Computational User Interface Services 12 © Simon Buckingham Shum

2 examples of Hypermedia Discourse approach and tool support • personal or group concept mapping • real time meeting Compendium capture • participatory modelling • discourse as semantic hypertext • Web publishing of Scholarly scholarly claims and Ontologies argumentation • discourse as semantic Project hypertext 13 © Simon Buckingham Shum

• personal or group concept mapping • real time meeting Compendium capture • participatory modelling • discourse as semantic hypertext 14 © Simon Buckingham Shum

Key elements of Compendium • Shared visual display • Simple notation • Template patterns • Node transclusions Knowledge • Tagging • Hypermedia Media • Interoperability with other data, services and user interfaces Modelling Practitioner skills Frameworks e.g. • Cognitive skills to chunk and link ideas e.g. (Buckingham Shum) • IBIS • Dialogue Mapping (Conklin) • CommonKADS • Conversational Modelling (Sierhuis & Selvin) • World Modelling • Participatory Hypermedia Construction • Critical Systems Heuristics (Selvin) 15 © Simon Buckingham Shum

Discourse grounded in Horst Rittel’s IBIS: Issue-Based Information System 16 © Simon Buckingham Shum

Compendium: hypertext discourse mapping/conceptual modelling 17 © Simon Buckingham Shum

Compendium: Descendent of gIBIS 18 © Simon Buckingham Shum

Structure management in Compendium Associative linking  nodes in a shared context connected by graphical Map links Categorical membership  nodes in different contexts connected by common attributes via metadata Tags Hypertextual Transclusion  reuse of the same node in different views Templates  reuse of the same structure in different views HTML, XML and RDF data exports for interoperability  Java and SQL interfaces to add services  19 © Simon Buckingham Shum

Compendium as the technical and intellectual ‘glue’ Java/RDF Advanced Knowledge Technologies project: www.aktors.org 20 © Simon Buckingham Shum

Compendium as sensemaking hub for emergency response semantic web tools Advanced Knowledge Technologies project: www.aktors.org 21 © Simon Buckingham Shum

Modelling using Issue-templates 22 © Simon Buckingham Shum

Modelling organisational processes in Compendium using a Template 23 © Simon Buckingham Shum

Completing a Compendium template 24 © Simon Buckingham Shum

Generating Custom Documents and Diagrams from Compendium Templates Field Integrated/ Deviations/ Specific Installation Assignable Approvals Revised Changes Assignments Details/ Inventory Requirements (Engr Sched) /Assignment Specs/NDO Notice (E1) List Build Assignable Inventory Assignable Inventory 25 © Simon Buckingham Shum

Collaborative Ontology Design and Merging with Compendium Buckingham Shum, S., Motta, E. and Domingue, J. (2002). Augmenting Design Deliberation with Compendium: Motta, Domingue, The Case of Collaborative Ontology Design. in Workshop on Facilitating Hypertext-Augmented Collaborative Modelling, ACM Hypertext Conference. [PrePrint: http://cognexus.org/ht02]. Modelling, Conference. [PrePrint: http://cognexus.org/ht02]. 26 © Simon Buckingham Shum

Setup for collaborative ontology design and rationale capture 27 © Simon Buckingham Shum

Result of initial requirements discussion 28 © Simon Buckingham Shum

Dialogue Mapping informal discussion 29 © Simon Buckingham Shum

‘Home window’ of the emerging ontology, linking to design discussions 30 © Simon Buckingham Shum

Template-driven Conversational Modelling of ontology class/slot structure 31 © Simon Buckingham Shum

Using Compendium for personnel recovery planning Co-OPR Project (with Austin Tate): http://www.aiai.ed.ac.uk/project/co-opr 32 © Simon Buckingham Shum

Co-OPR – Sample Screens Left screen: Compendium Middle screen: I-X Process Right screen: I-X Process intelligence database and Panels showing current Panels communicating discussion/rationale state of plan execution and with the external world, capture. situation map e.g. Isolated Personnel 33 © Simon Buckingham Shum

Co-OPR: Operational Sensemaking Situation analysis COA refinement COA generation Operational Sensemaking support for JPRC COA Other comparison/evaluation Tools + Detailed Planning 34 © Simon Buckingham Shum

Co-OPR Scenario Information 35 © Simon Buckingham Shum

PR Doctrine for Situation Analysis extracted as an Issue Template Link to the source doctrine document Issues that require attention (as specified in the doctrine document) Relevant extract from doctrine publication inside the node for reference 36 © Simon Buckingham Shum

Completed Issue Template (1/2) 37 © Simon Buckingham Shum

Completed Issue Template (2/2) Compendium knowledge elements (nodes) can be overlaid on any image, and can point to any media file 38 © Simon Buckingham Shum

Example ONA database maps Maps of  multimedia data from the ONA db Response Mechanism Force data converted to Issue template by drag+drop from Excel spreadsheet 39 © Simon Buckingham Shum

Imported database on Blue Forces 40 © Simon Buckingham Shum

JPRC Compendium Homepage  Sensemaking workflow: Links start from JTFC Briefing, to use of ONA database, to initial COA exploration, to worksheet analysis, to a COA Comparison worksheet for final briefing back to JTFC 41 © Simon Buckingham Shum

Issue Templates for JTFC Briefing The JTFC’s Briefing is  captured in a set of issue templates For each category (menu  item on left) there are a number of issues awaiting answers 42 © Simon Buckingham Shum

JTFC Briefing: Intent template Answers to template issues provided in the JTFC Briefing. Answers may be constrained by predefined options, as specified in the XML schema 43 © Simon Buckingham Shum

Crisis Action Planning template built in an hour Following distribution of the Crisis Action Planning process which was to be followed, a CAP template was created at short notice to support the process 44 © Simon Buckingham Shum

COA Wargame Analysis Worksheet Links to the Situation Analysis and COA Key Issues from PR Comparison Doctrine are listed here worksheet Constraints raised by the JTFC are ‘docked’ here as visual reminders Actors Rows for recording critical events which provoke discussion 45 © Simon Buckingham Shum

PMESII and Relative Superiority Issue Palettes The high level Issues can be expanded if required to see the sub-Issues raised by PR doctrine 46 © Simon Buckingham Shum

Compendium Wargame Analysis Worksheet An argument map is dragged from the template when some aspect of the COA provokes discussion Input from PMESII analysts The discussion map integrated into the discussion map ‘behind’ the cell is then built on the fly 47 © Simon Buckingham Shum

COA-2.1 Wargaming Worksheet (2/2) Dialogue Map capturing the planners’ discussion of this option 48 © Simon Buckingham Shum

COA Comparison Worksheet Summary of how COAs trade off against each other Constraints Doctrinal Issues can also be displayed if required 49 © Simon Buckingham Shum

Grey Matter and Silicon I-X Inputs to Compendium • Issues and Responses • Activity Options • Constraints/Maps • Annotations/Notes 50 © Simon Buckingham Shum

Independent evaluation of Co-OPR Evaluation ratings from six members of the planning cell who were supported by Co-OPR tools in the personnel recovery simulation. (Numbers indicate the number of planners assigning the rating.) 51 © Simon Buckingham Shum

Independent evaluation of Co-OPR In the subsequent vignette of the scenario (in which Co-OPR was not due to participate) Compendium was requested to replace (the usual) PowerPoint as the information management tool. 52 © Simon Buckingham Shum

Large scale NASA e-science field trials: Interoperability with other databases, software agents and collaboration tools Clancey, W.J., Sierhuis, M., Alena, R., Berrios, D., Dowding, J., Graham, J.S., Tyree, K.S., Hirsh, R.L., Garry, W.B., Semple, A., Buckingham Shum, S.J., Shadbolt, N. and Rupert, S. (2005). “Automating CapCom Using Mobile Agents and Robotic Assistants.” 1st Space Exploration Conference, American Institute of Aeronautics and Astronautics, 31 Jan- 1 Feb, 2005, Orlando, FL. Available from: AIAA Meeting Papers on Disc [CD-ROM]: Reston, VA, and as Advanced Knowledge Technologies ePrint 375: http://eprints.aktors.org/375 53 © Simon Buckingham Shum

54 © Simon Buckingham Shum Image Credits--- Mars: NASA/JPL/MSSS; Earth: NASA/JSC; Composite: MSSS

NASA e-science field trials (2004 and 2005) Distributed Mars-Earth planning and data analysis tools for Mars Habitat field trial in Utah desert, supported from US+UK 55 © Simon Buckingham Shum

NASA Mobile Agents Architecture 56 © Simon Buckingham Shum

Collaboration Configuration Compendium used as a collaboration medium at all intersections: humans+agents, reading+writing maps Software Agent Scientist Architecture (Earth) (Mars) Scientist Scientist Scientist Scientist (Earth) (Mars) (Mars) (Earth) 57 © Simon Buckingham Shum

NASA testbed: Compendium activity plans for surface exploration, constructed by scientists on ‘Earth’, interpreted by software agents on ‘Mars’ Copyright, 2004, RIACS/NASA Ames, Open University, Southampton University Not to be used without permission The Compendium nodes and relationships in this plan were interpreted by Brahms software agents for monitoring and coordinating astronaut and robot activity during surface explorations. 58 © Simon Buckingham Shum

CoAKTinG NASA testbed: Compendium science data map, generated by software agents, for interpretation by Mars+Earth scientists Copyright, 2004, RIACS/NASA Ames, Open University, Southampton University Not to be used without permission The Compendium maps were autonomously created and populated with science data by Brahms software agents that use models of the mission plan, work process, data flow and science data relationships to create the maps. 59 © Simon Buckingham Shum

CoAKTinG NASA testbed: Compendium-based photo analysis by geologists on ‘Mars’ Copyright, 2004, RIACS/NASA Ames, Open University, Southampton University Not to be used 60 without permission © Simon Buckingham Shum

NASA testbed: Compendium scientific feedback map from Earth scientists to Mars colleagues Copyright, 2004, RIACS/NASA Ames, Open University, Southampton University Not to be used 61 without permission © Simon Buckingham Shum

Importing an Argumentation Scheme as an IBIS template compendium.open.ac.uk/openlearn 62 © Simon Buckingham Shum

Importing an Argumentation Scheme as an IBIS template compendium.open.ac.uk/openlearn 63 © Simon Buckingham Shum

Using Compendium to map and automatically index replayable video conferences CoAKTinG Project: www.aktors.org/coakting Memetic Project: www.memetic-vre.net 64 © Simon Buckingham Shum

Collaborative sensemaking in e-Science: Meeting Replay tool for Earth scientists, synchronising video of Mars crew’s discussion as they annotate their mission plans Copyright, 2004, RIACS/NASA Ames, Open University, Southampton University Not to be used without permission © Simon MR Clip: 00:50 NASA Buckingham Shum 65

Memetic Meeting Replay The CoAKTinG project’s results are now mainstreamed in the project’ Access Grid by the JISC Memetic VRE project 66 © Simon Buckingham Shum

Compendium ‘literacy’? …understanding how to write, read, talk and think in hypermedia IBIS …approaches from consultancy in the field, and video analysis in the lab… 67 © Simon Buckingham Shum

Literacy: significant user community www.CompendiumInstitute.org www.CompendiumInstitute.org 68 © Simon Buckingham Shum

Literacy: Cognitive task analysis  Cognitive tasks involved in using a graphical argumentation scheme (Buckingham Shum 1996)  Affordances of graphical DR for coordinating group design (Buckingham Shum et al 1997) 69 © Simon Buckingham Shum

Literacy: the craft skill of IBIS mapping in meetings: “Dialogue Mapping” Jeff Conklin: CogNexus Institute: www.CogNexus.org 70 © Simon Buckingham Shum

Literacy: expertise analysis (Albert Selvin) What is the nature of expert human performance in  creating and modifying real time conceptual structures for groups?  The NASA knowledge mapper role: Conventional  Listening and interpreting facilitation  Intervening in ‘normal’ conversation flow skills  Getting validation for captured material  Building hypertext representations on Knowledge the fly media  Interrelating data and objects facilitation  Adding metadata skills  Software-specific skills Aesthetic and Ethical Implications of Participatory Hypermedia Practice: First Year Report Selvin, A. (2005), Technical Report KMI-05-17, Knowledge Media Institute, Open University, UK 71 © Simon Buckingham Shum

Compendium literacy: expertise analysis Selvin 2005 Practitioner stances  The position of the practitioner with regard to the current activity:  Knowledge Navigator  Facilitator  Participant  Technical Expert  Editor 72 © Simon Buckingham Shum

• Web publishing of Scholarly scholarly claims and Ontologies argumentation • discourse as semantic Project hypertext Will scientific publishing in 2020 still depend solely on the reading, writing, and discovery of written texts? What might a more network-centric complement look like? 73 © Simon Buckingham Shum

In Gutenberg’s shadow (or standing on his shoulders) Newspapers + Invisible Colleges = Scholarly Journals Le Journal des Philosophical Transactions of Sçavans the Royal Society of London January 1665 March 1665 74 © Simon Buckingham Shum

What if we could get search results like this?… “What is the Turing Debate?” One of seven maps in the Mapping Great Debates: Can Computers Think? Series. MacroVU Press. www.macrovu.com (Horn, 2003; Yoshimi, 2006) 75 © Simon Buckingham Shum

Going beyond citations… People try to maximise their rate of gaining Information scent information models extends Web User Flow by Information Information Scent foraging (WUFIS) theory ? “The Scent of a Site: A System for Analyzing and Predicting Information Scent, Usage, and Usability of a Web Site” “Information foraging” 77 © Simon Buckingham Shum

Combining formal relations with the expressive freedom of ‘folksonomies’ Relational classes and dialects (KMi Scholarly Ontologies project) 78 © Simon Buckingham Shum

If we model concepts in a literature as concept maps… (KMi’s ClaiMapper, built on Compendium) KMi’ ClaiMapper, 79 © Simon Buckingham Shum

“Semantic del.icio.us”: KMi’s ClaimSpotter assigning and linking freeform tags Sereno, B., Buckingham Shum, S. and Motta, E. (2007). Formalization, User Strategy and Interaction Design: Users’ Behaviour with Discourse Tagging Semantics. Workshop on Social and Collaborative Construction of Structured Knowledge, 16th Int. World Wide Web Conference (WWW 2007), Banff, 8-12 May 2007. http://www2007.org/workshops/paper_30.pdf 80 © Simon Buckingham Shum

Visualising claims and arguments When multiple analysts annotate web documents via a server, they can generate a shared view of how they see the field, and where they agree/disagree ClaimFinder prototype: claimfinder.open.ac.uk 82 © Simon Buckingham Shum

“What papers contrast with this paper?” 1. Extract concepts for this document 2. Trace concepts on which they build 3. Trace concepts challenging this set 4. Show root documents 83 © Simon Buckingham Shum

Indicators of ClaiMaker literacy? expert user makes more extensive use of Claimaker’s semantic structures in interrogating the network than novices Victoria Uren, Simon Buckingham Shum, Michelle Bachler, Gary Li, (2006) Sensemaking Tools for Understanding Research Literatures: Design, Implementation and User Evaluation. International Journal of Human Computer Studies, Vol.64, 5, (420-445). 87 © Simon Buckingham Shum

Some answers to our questions… Web Pragmatics The discourse of modelling: Hypermedia Discourse Compendium+IBIS can support many forms of discourse and rationale capture throughout the design lifecycle, from early requirements through to maintenance Modelling discourse: It is possible to co-evolve discourse schemes, tools and literacy to mediate sensemaking discourse. Strong evidence for Compendium, emerging evidence for ClaiMaker 88 © Simon Buckingham Shum

Hypermedia Discourse tools: Discourse Ontology Literacy/ Notation(s) Fluency Intuitive Computational User Interface Services 89 © Simon Buckingham Shum

Common Ground between Pragmatic Web and Hypermedia Discourse perspectives Discourse Ontology negotiating Notation(s) meaning Literacy/ Fluency supporting co-evolving new human-human skillsets with & human-agent new tools communication Intuitive User Interface Computational empowering Services augmenting stakeholders collective to negotiate sensemaking meaning 90 © Simon Buckingham Shum

Social bookmarking as semiosis Mancini, C. and Buckingham Shum, S.J. (2006). Modelling Discourse in Contested Domains: A Semiotic and Cognitive Framework. International Journal of Human Computer Studies, 64, (11), pp.1154-1171 92 © Simon Buckingham Shum

Annotation as semiosis in the Scholarly Ontologies project Mancini, C. and Buckingham Shum, S.J. (2006). Modelling Discourse in Contested Domains: A Semiotic and Cognitive Framework. International Journal of Human Computer Studies, 64, (11), pp.1154-1171 93 © Simon Buckingham Shum

Making primary and secondary claims as semiotic and discourse moves Secondary claim: a discourse connective Primary claim: a semiotic move Mancini, C. and Buckingham Shum, S.J. (2006). Modelling Discourse in Contested Domains: A Semiotic and Cognitive Framework. International Journal of Human Computer Studies, 64, (11), pp.1154-1171 94 © Simon Buckingham Shum

Combining formal relations with the expressive freedom of ‘folksonomies’ Relational classes and dialects (KMi Scholarly Ontologies project) 95 © Simon Buckingham Shum

polarity source type comparativeness Cognitive Coherence Relations Mancini, C. and Buckingham Shum, S.J. (2006). Modelling Discourse in Contested Domains: A Semiotic and Cognitive Framework. International Journal of Human Computer Studies, 64, (11), pp.1154-1171 96 © Simon Buckingham Shum

Using CCR-based Coherence Patterns to detect candidate “schools of thought” Doctoral work by Neil Benn, KMi 97 © Simon Buckingham Shum

Hypermedia Discourse: Theory & Technology for the Pragmatic Web?

0 comments

Notes on slide 1

1 Favorite