Your SlideShare is downloading. ×

LAK14 Doctoral Consortium

446
views

Published on

Published in: Education

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
446
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Learning Analytics for Scaffolding Academic Writing through Automatic Identification of Meta-discourse Duygu Simsek Doctoral Consortium, 4th Learning Analytics and Knowledge Conference, Indianapolis, USA 25th March, 2014 people.kmi.open.ac.uk/simsek duygu.simsek@open.ac.uk simsekduygu_ Supervisors: Prof. Simon Buckingham Shum, Dr. Rebecca Ferguson, & Dr. Anna De Liddo Dr. Ágnes Sándor, Xerox Research Centre Europe
  • 2. Research Aim To investigate  whether computational techniques can automatically identify the attributes of good academic writing in as correlated with grades of the essay and as identified in the literature  if this proves possible, how best to feed back actionable analytics to support students and educators  whether this feedback has any demonstrable benefits 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 2
  • 3. Where this research sits? ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers Discourse Centric Learning Analytics Meta- discourse in Student writing 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 3
  • 4. Where this research sits?- Academic Writing ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers Discourse Centric Learning Analytics Meta- discourse in Student writing Key aim of academic writing is to convince readers about the validity of the claims and arguments put forward through an effective narrative. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 4
  • 5. Where this research sits?- Meta-discourse ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers Discourse Centric Learning Analytics Meta- discourse in Student writing This effective narrative is signalled through meta-discourse! 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 5
  • 6. Meta-discourse Meta-discourse refers to the features of text that convey the author’s intended meaning and intention. It provides cues to the reader which explicitly express a viewpoint, argument and claim, and signals the writer's stance. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium Fig. 1 Meta-discourse that convey summary statements CuestoSummary statements 6
  • 7. Examples of meta-discourse cues that signal academic/analytical rhetorical moves BACKGROUND KNOWLEDGE: Recent studies indicate … the previously proposed … … is universally accepted NOVELTY: New insights provide direct evidence… …suggest a new approach… Results define a novel role ... OPEN QUESTION: Little is known … … role … has been elusive Current data is insufficient… TENDENCY: ... emerging as a promising approach Our understanding ... has grown exponentially ... Growing recognition of the importance ... CONTRASTING IDEAS: In contrast with previous hypotheses ... ... inconsistent with past findings ... SIGNIFICANCE: studies ... have provided important advances ... is crucial for ... understanding valuable information ... from SURPRISE: We have recently observed ... surprisingly We have identified ... unusual The recent discovery ... suggests intriguing roles SUMMARISING: The goal of this study ... Here, we show ... Our results ... indicate 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 7
  • 8. Where this research sits?- Meta-discourse ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers Discourse Centric Learning Analytics Meta- discourse in Student writing  In order to assess students’ writing therefore, educators will be examining students’ use of meta-discourse which make their students’ thinking visible.  However, students find it challenging to learn to write in an academically sound way.  They need to learn how to make their thinking visible by recognising and deploying meta-discourse. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 8
  • 9. Where this research sits?- Computational Text Analysis ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers (XIP) Discourse Centric Learning Analytics Meta- discourse in Student writing  Meta-discourse cues are automatically identifiable.  This PhD investigates whether it is possible to provide automatic meta-discourse analysis of student writing through the use of a particular rhetorical parser, XIP. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 9
  • 10. Example of a rhetorical parser: Incremental Parser (XIP)  Natural Language Processing (NLP) product which includes a rhetorical parser detecting meta-discourse in academic texts.  XIP extracts salient sentences based on their rhetorical functions:  Background Knowledge  Summarising  Tendency  Novelty  Significance  Surprise  Open Question  Contrasting Ideas 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 10
  • 11. Student Writing Analysed by XIP 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium CONTRAST SUMMARY 11
  • 12. Rhetorical functions classified by XIP BACKGROUND KNOWLEDGE: Recent studies indicate … the previously proposed … … is universally accepted NOVELTY: New insights provide direct evidence… …suggest a new approach… Results define a novel role ... OPEN QUESTION: Little is known … … role … has been elusive Current data is insufficient… TENDENCY: ... emerging as a promising approach Our understanding ... has grown exponentially ... Growing recognition of the importance ... CONTRASTING IDEAS: In contrast with previous hypotheses ... ... inconsistent with past findings ... SIGNIFICANCE: studies ... have provided important advances ... is crucial for ... understanding valuable information ... from SURPRISE: We have recently observed ... surprisingly We have identified ... unusual The recent discovery ... suggests intriguing roles SUMMARISING: The goal of this study ... Here, we show ... Our results ... indicate 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 12
  • 13. Fine for researchers or machines but it is not learner/educator friendly XIP’s Output 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 13
  • 14. Why XIP? – Key Features of Academic Writing? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium Relevance Understanding & Knowledge Structure & Organisation Linguistic Accuracy Illustrations Referencing Argumentation 14
  • 15. There is a mapping between good and strong features of academic writing and the XIP’s rhetorical functions. Why XIP? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 15
  • 16. Where this research sits?- Learning Analytics ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers (XIP) Discourse Centric Learning Analytics Meta- discourse in Student writing XIP is a parser with potential, if it can be embedded in a more complete learning analytics (LA) approach. It has potential for formative feedback to writing through LA. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 16
  • 17. Where this research sits?- Discourse-centric Learning Analytics ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers (XIP) Discourse Centric Learning Analytics (DCLA) Meta- discourse in Student writing How should a DCLA approach be validated? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 17
  • 18. Main Research Question To what degree can computational text analysis and visual analytics be used to support the academic writing of students in higher education? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 18
  • 19. To what extent is the rhetorical parser XIP accurate and sufficient for identifying the attributes of good academic writing within student writing, as judged by the grade, and by educators? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium XIP Evaluates Accuracy & Sufficiency Any correlation between Grades & XIP output? XIP’s Highlights vs. Marker’s RQ1 19
  • 20. To what extent is the rhetorical parser XIP accurate and sufficient for identifying the attributes of good academic writing within student writing, as judged by the grade, and by educators? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium RQ1 XIP Highlighted Student Writing Any correlation between the final grade of writing & XIP findings? Pearson for Total number of salient sentences vs. Grade Generalised Multiple Regression How strongly each rhetorical sentence type influences the final grade Grades 20
  • 21. To what extent is the rhetorical parser XIP accurate and sufficient for identifying the attributes of good academic writing within student writing, as judged by the grade, and by educators? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium RQ1 What is the overlap between XIP’s output and how tutors judge quality? Tutor Highlighted Student WritingXIP Highlighted Student Writing 21
  • 22. In what ways should XIP output be delivered to end users (students and educators)? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium XIP Evaluates Accuracy & Sufficiency Any correlation between Grades & XIP output? XIP’s Highlights vs. Marker’s Output RQ2 22
  • 23. 1st Year Pilot study In what ways should XIP output be delivered to end users (students and educators)?RQ2 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 23
  • 24. To what extent do educators value the results of XIP’s analysis of an individual student or cohort’s work when the primary focus is on assessment? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium XIP Evaluates Accuracy & Sufficiency Any correlation between Grades & XIP output? XIP’s Highlights vs. Marker’s Output What educators think RQ3 24
  • 25. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium XIP Evaluates Accuracy & Sufficiency Any correlation between Grades & XIP output? XIP’s Highlights vs. Marker’s Output What educators think To what extent do educators value the results of XIP’s analysis of an individual student or cohort’s work when the primary focus is on assessment? RQ3 25
  • 26. To what extent do students value the results of XIP’s analysis as formative feedback on their writing? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium XIP Evaluates Accuracy & Sufficiency Any correlation between Grades & XIP output? XIP’s Highlights vs. Marker’s Output What educators think What students think RQ4 26
  • 27. 1. For my quantitative study, do I have the right approach? Are there any alternative approaches? How could I make my study stronger? 2. What qualitative & quantitative methods could I use to evaluate the quality of the comparison between XIP & marker highlights? 3. Are there any available well-developed methodologies on assessing visualisations to elicit user reactions? Feedback? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 27