XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse

9,018
-1

Published on

XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse

ABSTRACT
A key competency that we seek to build in learners is a critical mind, i.e. ability to engage with the ideas in the literature, and to identify when significant claims are being made in articles. The ability to decode such moves in texts is essential, as is the ability to make such moves in one’s own writing. Computational techniques for extracting them are becoming available, using Natural Language Processing (NLP) tuned to recognize the rhetorical signals that authors use when making a significant scholarly move. After reviewing related NLP work, we introduce the Xerox Incremental Parser (XIP), note previous work to render its output, and then motivate the design of the XIP Dashboard, a set of visual analytics modules built on XIP output, using the LAK/EDM open dataset as a test corpus. We report preliminary user reactions to a paper prototype of such a novel dashboard, describe the visualizations implemented to date, and present user scenarios for learners, educators and researchers. We conclude with a summary of ongoing design refinements, potential platform integrations, and questions that need to be investigated through end-user evaluations.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
9,018
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse

  1. 1. 1st International Workshop on Discourse-Centric Learning AnalyticsApril 8, 2013, LAK13 Conference, Leuven, BelgiumXIP Dashboard: Visual Analyticsfrom Automated Rhetorical Parsingof Scientific MetadiscourseDuygu Simsek, Simon Buckingham Shum, Anna De Liddo,Rebecca Ferguson — The Open University, UKÁgnes Sándor — Xerox Research Centre Europe, FR
  2. 2. Metadiscourse Xerox Incremental ParserVisual analytics v0.1: XIP Dashboard User Scenarios & Evaluation 2
  3. 3. Metadiscourse signals important movesin educated/scholarly narrative (When scholarly culture works well) this is what gets your papers accepted by reviewers, and quoted by others Clear statements regarding the problem, the claim, the argument, the evidence, the implications… This is what we teach students from school upwards 3
  4. 4. Rhetorical functions of metadiscourse identifiedby the Xerox Incremental Parser (XIP)BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:Recent studies indicate … ... new insights provide direct evidence ... … little is known …… the previously proposed … ... we suggest a new ... approach ... … role … has been elusive Current data is insufficient …… is universally accepted ... ... results define a novel role ...SUMMARIZING: SIGNIFICANCE: CONTRASTING IDEAS:The goal of this study ... studies ... have provided important … unorthodox view resolves … advances paradoxes …Here, we show ... Knowledge ... is crucial for ... In contrast with previousAltogether, our results ... indicate understanding hypotheses ... valuable information ... from studies ... inconsistent with past findings ...GENERALIZING: SURPRISE:... emerging as a promising approach We have recently observed ... surprisinglyOur understanding ... has grownexponentially ... We have identified ... unusual... growing recognition of the The recent discovery ... suggests intriguing rolesimportance ...
  5. 5. Xerox Incremental Parser (XIP)Sándor, Á. and Vorndran, A. (2010). The detection of salient messages from social science research papers and its application in document search. 5Workshop on Natural Language Processing Tools Applied to Discourse Analysis in Psychology, Buenos Aires, Argentina, May 10-14. 2010.
  6. 6. Xerox Incremental Parser (XIP)Sándor, Á. and Vorndran, A. (2010). The detection of salient messages from social science research papers and its application in document search. 6Workshop on Natural Language Processing Tools Applied to Discourse Analysis in Psychology, Buenos Aires, Argentina, May 10-14. 2010.
  7. 7. Xerox Incremental Parser (XIP)Sándor, Á. and Vorndran, A. (2010). The detection of salient messages from social science research papers and its application in document search. 7Workshop on Natural Language Processing Tools Applied to Discourse Analysis in Psychology, Buenos Aires, Argentina, May 10-14. 2010.
  8. 8. Xerox Incremental Parser (XIP)Sándor, Á. and Vorndran, A. (2010). The detection of salient messages from social science research papers and its application in document search. 8Workshop on Natural Language Processing Tools Applied to Discourse Analysis in Psychology, Buenos Aires, Argentina, May 10-14. 2010.
  9. 9. Initial evaluation of XIP is promising,but methodologically complexA striking example – but not all were like this (De Liddo et al, 2012)Human analyst XIPExtract from annotation comparison: Document 1 19 sentences annotated 22 sentences annotated 11 sentences same as human annotation Document 2 71 sentences annotated 59 sentences annotated 42 sentences same as human annotation
  10. 10. Xerox Incremental Parser (XIP) XIP’s raw output is fine for NLP machines/researchers, but not learner/educator friendly
  11. 11. Xerox Incremental Parser (XIP) XIP’s raw output is fine for NLP machines/researchers, but not learner/educator friendly
  12. 12. Xerox Incremental Parser (XIP) 5000 (or even 30) plain text files… we need overviews of XIP analyses from a corpus
  13. 13. Making XIP analytics visible:1. annotations on the full text using the OU’sCohere social sensemaking app (Firefox add-on)
  14. 14. Making XIP analytics visible:2. XIP annotations visualized in Cohere as a networkaround the document
  15. 15. Making XIP analytics visible (2)2nd phase analysisof document-concept clouds…Connecting? ?Merging?Re-tagging? ?Summarising? ? ?
  16. 16. XIP Dashboard: towards an earlier phasedashboard for navigating XIP outputDraw attention to patterns of potential significance tostudents, educators and experienced researchers alike:§  the occurrence of domain concepts in different metadiscourse contexts – e.g. effective tutoring dialogue in sentences classified contrast§  trends of the above over time, e.g. to show the development of an idea§  trends within and differences between research communities as reflected in their publications§  eventually, the above for one’s own writing 16
  17. 17. Paper prototype to elicit initial reactions 17
  18. 18. Paper prototype to elicit initial reactions ‘Intro movie’ from researcher Participants point + click with finger Basic navigation seems fine Enthusiasm for a tool that could help with literature analysis Also for a tool to improve one’s own writing by showing trends, or inconsistencies 18
  19. 19. XIP DashboardTemporal trends per corpus Similar patterns for LAK & EDM literatures Summary & Contrast categories relatively higher, and rising (Not controlled for different corpus sizes in these graphs) 19
  20. 20. XIP DashboardComparing corpora filtered by concept 20
  21. 21. XIP DashboardAll papers by year and concept, withcolour = concept density (v2 mockup) 21
  22. 22. XIP DashboardRhetorical function of the sentencesbehind each bubble 22
  23. 23. XIP DashboardHeatmap of all concepts byrhetorical classification (v2 mockup) 23
  24. 24. XIP Dashboard User scenarios…Student / Educator / Researcher Familiarization with the background material in a literature… Comparing different writing patterns between communities, or students… Focusing on specific concepts of interest in combination with rhetorical context 24
  25. 25. XIP Dashboard User EvaluationsSignal-noise ratio? Deeper or shallower reading? New insights, or just faster insights? Better writing, or just gaming the system? 25
  26. 26. SummaryEarly phases of work: a promising language technologynow has visual analytics we can deploy with stakeholders Beyond number / size / frequency http://www.glennsasscer.com/wordpress/wp-content/uploads/2011/10/iceberg.jpg of posts; ‘hottest thread’ An important feature of educated writing is knowing how to signal substantive rhetorical moves. NLP can detect this, and we can now generate rudimentary visual analytics. To be continued…

×