Successfully reported this slideshow.
Learning Analytics
for Scaffolding Academic Writing
through Automatic Identification of
Meta-discourse
Duygu Simsek
Doctor...
Research Aim
To investigate
 whether computational techniques can automatically
identify the attributes of good academic ...
Where this research sits?
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Cen...
Where this research sits?-
Academic Writing
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Par...
Where this research sits?-
Meta-discourse
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parse...
Meta-discourse
Meta-discourse refers to the features of text that convey the author’s intended
meaning and intention. It p...
Examples of meta-discourse cues that
signal academic/analytical rhetorical moves
BACKGROUND KNOWLEDGE:
Recent studies indi...
Where this research sits?-
Meta-discourse
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parse...
Where this research sits?-
Computational Text Analysis
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhe...
Example of a rhetorical parser:
Incremental Parser (XIP)
 Natural Language Processing (NLP) product which includes a
rhet...
Student Writing Analysed by XIP
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
CONTRAST
SUMMARY
11
Rhetorical functions classified by XIP
BACKGROUND KNOWLEDGE:
Recent studies indicate …
the previously proposed …
… is univ...
Fine for
researchers or
machines but it is
not
learner/educator
friendly
XIP’s Output
25/03/2014, Indianapolis, USALAK’14 ...
Why XIP? – Key Features of
Academic Writing?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
Relevance
Understandi...
There is a mapping between good and strong features
of academic writing and the XIP’s rhetorical functions.
Why XIP?
25/03...
Where this research sits?-
Learning Analytics
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
P...
Where this research sits?-
Discourse-centric Learning Analytics
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANA...
Main Research Question
To what degree can computational text analysis
and visual analytics be used to
support the academic...
To what extent is the rhetorical parser XIP accurate and
sufficient for identifying the attributes of good academic
writin...
To what extent is the rhetorical parser XIP accurate and
sufficient for identifying the attributes of good academic
writin...
To what extent is the rhetorical parser XIP accurate and
sufficient for identifying the attributes of good academic
writin...
In what ways should XIP output be delivered to end users
(students and educators)?
25/03/2014, Indianapolis, USALAK’14 Doc...
1st Year
Pilot study
In what ways should XIP output be delivered to end users
(students and educators)?RQ2
25/03/2014, Ind...
To what extent do educators value the results of XIP’s
analysis of an individual student or cohort’s work when
the primary...
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grade...
To what extent do students value the results of XIP’s
analysis as formative feedback on their writing?
25/03/2014, Indiana...
1. For my quantitative study, do I have the right approach? Are
there any alternative approaches? How could I make my stud...
Upcoming SlideShare
Loading in …5
×

LAK14 Doctoral Consortium

971 views

Published on

Published in: Education
  • Be the first to comment

  • Be the first to like this

LAK14 Doctoral Consortium

  1. 1. Learning Analytics for Scaffolding Academic Writing through Automatic Identification of Meta-discourse Duygu Simsek Doctoral Consortium, 4th Learning Analytics and Knowledge Conference, Indianapolis, USA 25th March, 2014 people.kmi.open.ac.uk/simsek duygu.simsek@open.ac.uk simsekduygu_ Supervisors: Prof. Simon Buckingham Shum, Dr. Rebecca Ferguson, & Dr. Anna De Liddo Dr. Ágnes Sándor, Xerox Research Centre Europe
  2. 2. Research Aim To investigate  whether computational techniques can automatically identify the attributes of good academic writing in as correlated with grades of the essay and as identified in the literature  if this proves possible, how best to feed back actionable analytics to support students and educators  whether this feedback has any demonstrable benefits 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 2
  3. 3. Where this research sits? ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers Discourse Centric Learning Analytics Meta- discourse in Student writing 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 3
  4. 4. Where this research sits?- Academic Writing ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers Discourse Centric Learning Analytics Meta- discourse in Student writing Key aim of academic writing is to convince readers about the validity of the claims and arguments put forward through an effective narrative. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 4
  5. 5. Where this research sits?- Meta-discourse ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers Discourse Centric Learning Analytics Meta- discourse in Student writing This effective narrative is signalled through meta-discourse! 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 5
  6. 6. Meta-discourse Meta-discourse refers to the features of text that convey the author’s intended meaning and intention. It provides cues to the reader which explicitly express a viewpoint, argument and claim, and signals the writer's stance. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium Fig. 1 Meta-discourse that convey summary statements CuestoSummary statements 6
  7. 7. Examples of meta-discourse cues that signal academic/analytical rhetorical moves BACKGROUND KNOWLEDGE: Recent studies indicate … the previously proposed … … is universally accepted NOVELTY: New insights provide direct evidence… …suggest a new approach… Results define a novel role ... OPEN QUESTION: Little is known … … role … has been elusive Current data is insufficient… TENDENCY: ... emerging as a promising approach Our understanding ... has grown exponentially ... Growing recognition of the importance ... CONTRASTING IDEAS: In contrast with previous hypotheses ... ... inconsistent with past findings ... SIGNIFICANCE: studies ... have provided important advances ... is crucial for ... understanding valuable information ... from SURPRISE: We have recently observed ... surprisingly We have identified ... unusual The recent discovery ... suggests intriguing roles SUMMARISING: The goal of this study ... Here, we show ... Our results ... indicate 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 7
  8. 8. Where this research sits?- Meta-discourse ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers Discourse Centric Learning Analytics Meta- discourse in Student writing  In order to assess students’ writing therefore, educators will be examining students’ use of meta-discourse which make their students’ thinking visible.  However, students find it challenging to learn to write in an academically sound way.  They need to learn how to make their thinking visible by recognising and deploying meta-discourse. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 8
  9. 9. Where this research sits?- Computational Text Analysis ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers (XIP) Discourse Centric Learning Analytics Meta- discourse in Student writing  Meta-discourse cues are automatically identifiable.  This PhD investigates whether it is possible to provide automatic meta-discourse analysis of student writing through the use of a particular rhetorical parser, XIP. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 9
  10. 10. Example of a rhetorical parser: Incremental Parser (XIP)  Natural Language Processing (NLP) product which includes a rhetorical parser detecting meta-discourse in academic texts.  XIP extracts salient sentences based on their rhetorical functions:  Background Knowledge  Summarising  Tendency  Novelty  Significance  Surprise  Open Question  Contrasting Ideas 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 10
  11. 11. Student Writing Analysed by XIP 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium CONTRAST SUMMARY 11
  12. 12. Rhetorical functions classified by XIP BACKGROUND KNOWLEDGE: Recent studies indicate … the previously proposed … … is universally accepted NOVELTY: New insights provide direct evidence… …suggest a new approach… Results define a novel role ... OPEN QUESTION: Little is known … … role … has been elusive Current data is insufficient… TENDENCY: ... emerging as a promising approach Our understanding ... has grown exponentially ... Growing recognition of the importance ... CONTRASTING IDEAS: In contrast with previous hypotheses ... ... inconsistent with past findings ... SIGNIFICANCE: studies ... have provided important advances ... is crucial for ... understanding valuable information ... from SURPRISE: We have recently observed ... surprisingly We have identified ... unusual The recent discovery ... suggests intriguing roles SUMMARISING: The goal of this study ... Here, we show ... Our results ... indicate 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 12
  13. 13. Fine for researchers or machines but it is not learner/educator friendly XIP’s Output 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 13
  14. 14. Why XIP? – Key Features of Academic Writing? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium Relevance Understanding & Knowledge Structure & Organisation Linguistic Accuracy Illustrations Referencing Argumentation 14
  15. 15. There is a mapping between good and strong features of academic writing and the XIP’s rhetorical functions. Why XIP? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 15
  16. 16. Where this research sits?- Learning Analytics ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers (XIP) Discourse Centric Learning Analytics Meta- discourse in Student writing XIP is a parser with potential, if it can be embedded in a more complete learning analytics (LA) approach. It has potential for formative feedback to writing through LA. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 16
  17. 17. Where this research sits?- Discourse-centric Learning Analytics ACADEMIC WRITING LEARNING ANALYTICS COMPUTATIONAL TEXT ANALYSIS Rhetorical Parsers (XIP) Discourse Centric Learning Analytics (DCLA) Meta- discourse in Student writing How should a DCLA approach be validated? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 17
  18. 18. Main Research Question To what degree can computational text analysis and visual analytics be used to support the academic writing of students in higher education? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 18
  19. 19. To what extent is the rhetorical parser XIP accurate and sufficient for identifying the attributes of good academic writing within student writing, as judged by the grade, and by educators? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium XIP Evaluates Accuracy & Sufficiency Any correlation between Grades & XIP output? XIP’s Highlights vs. Marker’s RQ1 19
  20. 20. To what extent is the rhetorical parser XIP accurate and sufficient for identifying the attributes of good academic writing within student writing, as judged by the grade, and by educators? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium RQ1 XIP Highlighted Student Writing Any correlation between the final grade of writing & XIP findings? Pearson for Total number of salient sentences vs. Grade Generalised Multiple Regression How strongly each rhetorical sentence type influences the final grade Grades 20
  21. 21. To what extent is the rhetorical parser XIP accurate and sufficient for identifying the attributes of good academic writing within student writing, as judged by the grade, and by educators? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium RQ1 What is the overlap between XIP’s output and how tutors judge quality? Tutor Highlighted Student WritingXIP Highlighted Student Writing 21
  22. 22. In what ways should XIP output be delivered to end users (students and educators)? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium XIP Evaluates Accuracy & Sufficiency Any correlation between Grades & XIP output? XIP’s Highlights vs. Marker’s Output RQ2 22
  23. 23. 1st Year Pilot study In what ways should XIP output be delivered to end users (students and educators)?RQ2 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 23
  24. 24. To what extent do educators value the results of XIP’s analysis of an individual student or cohort’s work when the primary focus is on assessment? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium XIP Evaluates Accuracy & Sufficiency Any correlation between Grades & XIP output? XIP’s Highlights vs. Marker’s Output What educators think RQ3 24
  25. 25. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium XIP Evaluates Accuracy & Sufficiency Any correlation between Grades & XIP output? XIP’s Highlights vs. Marker’s Output What educators think To what extent do educators value the results of XIP’s analysis of an individual student or cohort’s work when the primary focus is on assessment? RQ3 25
  26. 26. To what extent do students value the results of XIP’s analysis as formative feedback on their writing? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium XIP Evaluates Accuracy & Sufficiency Any correlation between Grades & XIP output? XIP’s Highlights vs. Marker’s Output What educators think What students think RQ4 26
  27. 27. 1. For my quantitative study, do I have the right approach? Are there any alternative approaches? How could I make my study stronger? 2. What qualitative & quantitative methods could I use to evaluate the quality of the comparison between XIP & marker highlights? 3. Are there any available well-developed methodologies on assessing visualisations to elicit user reactions? Feedback? 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium 27

×