This research aims to investigate whether computational techniques can automatically identify attributes of good academic writing in student essays. Specifically, it examines whether a rhetorical parser called XIP can identify meta-discourse cues that signal academic writing skills. The research also explores how to provide feedback on writing skills to students and educators to support learning. A key research question is to what extent XIP can accurately identify writing attributes in student essays, as judged by grades and educators, and how best to deliver its analysis to end users.

1. Learning Analytics
for Scaffolding Academic Writing
through Automatic Identification of
Meta-discourse
Duygu Simsek
Doctoral Consortium, 4th Learning Analytics and Knowledge Conference, Indianapolis, USA 25th March, 2014
people.kmi.open.ac.uk/simsek
duygu.simsek@open.ac.uk
simsekduygu_
Supervisors: Prof. Simon Buckingham Shum, Dr. Rebecca Ferguson, & Dr. Anna De Liddo
Dr. Ágnes Sándor, Xerox Research Centre Europe

2. Research Aim
To investigate
 whether computational techniques can automatically
identify the attributes of good academic writing in as
correlated with grades of the essay and as identified in
the literature
 if this proves possible, how best to feed back actionable
analytics to support students and educators
 whether this feedback has any demonstrable benefits
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
2

3. Where this research sits?
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
3

4. Where this research sits?-
Academic Writing
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
Key aim of academic
writing is to convince
readers about the
validity of the claims
and arguments put
forward through an
effective narrative.
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
4

5. Where this research sits?-
Meta-discourse
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
This effective narrative
is signalled through
meta-discourse!
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
5

6. Meta-discourse
Meta-discourse refers to the features of text that convey the author’s intended
meaning and intention. It provides cues to the reader which explicitly express a
viewpoint, argument and claim, and signals the writer's stance.
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
Fig. 1 Meta-discourse that convey summary statements
CuestoSummary
statements
6

7. Examples of meta-discourse cues that
signal academic/analytical rhetorical moves
BACKGROUND KNOWLEDGE:
Recent studies indicate …
the previously proposed …
… is universally accepted
NOVELTY:
New insights provide direct
evidence…
…suggest a new approach…
Results define a novel role ...
OPEN QUESTION:
Little is known …
… role … has been elusive
Current data is insufficient…
TENDENCY:
... emerging as a promising
approach
Our understanding ... has
grown exponentially ...
Growing recognition of the
importance ...
CONTRASTING IDEAS:
In contrast with previous
hypotheses ...
... inconsistent with past
findings ...
SIGNIFICANCE:
studies ... have provided
important advances
... is crucial for ... understanding
valuable information ... from
SURPRISE:
We have recently observed ...
surprisingly
We have identified ... unusual
The recent discovery ... suggests
intriguing roles
SUMMARISING:
The goal of this study ...
Here, we show ...
Our results ... indicate
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
7

8. Where this research sits?-
Meta-discourse
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
 In order to assess students’
writing therefore, educators
will be examining students’
use of meta-discourse which
make their students’ thinking
visible.
 However, students find it
challenging to learn to write
in an academically sound
way.
 They need to learn how to
make their thinking visible by
recognising and deploying
meta-discourse.
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
8

9. Where this research sits?-
Computational Text Analysis
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
(XIP)
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
 Meta-discourse cues
are automatically
identifiable.
 This PhD investigates
whether it is possible
to provide automatic
meta-discourse
analysis of student
writing through the
use of a particular
rhetorical parser, XIP.
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
9

10. Example of a rhetorical parser:
Incremental Parser (XIP)
 Natural Language Processing (NLP) product which includes a
rhetorical parser detecting meta-discourse in academic texts.
 XIP extracts salient sentences based on their rhetorical functions:
 Background Knowledge
 Summarising
 Tendency
 Novelty
 Significance
 Surprise
 Open Question
 Contrasting Ideas
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
10

11. Student Writing Analysed by XIP
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
CONTRAST
SUMMARY
11

12. Rhetorical functions classified by XIP
BACKGROUND KNOWLEDGE:
Recent studies indicate …
the previously proposed …
… is universally accepted
NOVELTY:
New insights provide direct
evidence…
…suggest a new approach…
Results define a novel role ...
OPEN QUESTION:
Little is known …
… role … has been elusive
Current data is insufficient…
TENDENCY:
... emerging as a promising
approach
Our understanding ... has
grown exponentially ...
Growing recognition of the
importance ...
CONTRASTING IDEAS:
In contrast with previous
hypotheses ...
... inconsistent with past
findings ...
SIGNIFICANCE:
studies ... have provided
important advances
... is crucial for ... understanding
valuable information ... from
SURPRISE:
We have recently observed ...
surprisingly
We have identified ... unusual
The recent discovery ... suggests
intriguing roles
SUMMARISING:
The goal of this study ...
Here, we show ...
Our results ... indicate
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
12

13. Fine for
researchers or
machines but it is
not
learner/educator
friendly
XIP’s Output
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
13

14. Why XIP? – Key Features of
Academic Writing?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
Relevance
Understanding & Knowledge
Structure & Organisation
Linguistic Accuracy
Illustrations
Referencing
Argumentation
14

15. There is a mapping between good and strong features
of academic writing and the XIP’s rhetorical functions.
Why XIP?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
15

16. Where this research sits?-
Learning Analytics
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
(XIP)
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing
XIP is a parser with
potential, if it can be
embedded in a more
complete learning
analytics (LA)
approach. It has
potential for formative
feedback to writing
through LA.
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
16

17. Where this research sits?-
Discourse-centric Learning Analytics
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
(XIP)
Discourse
Centric
Learning
Analytics
(DCLA)
Meta-
discourse
in Student
writing
How should a DCLA
approach be
validated?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
17

18. Main Research Question
To what degree can computational text analysis
and visual analytics be used to
support the academic writing of
students in higher education?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
18

19. To what extent is the rhetorical parser XIP accurate and
sufficient for identifying the attributes of good academic
writing within student writing, as judged by the grade,
and by educators?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
RQ1
19

20. To what extent is the rhetorical parser XIP accurate and
sufficient for identifying the attributes of good academic
writing within student writing, as judged by the grade,
and by educators?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
RQ1
XIP Highlighted Student Writing
Any correlation between
the final grade of writing & XIP findings?
Pearson for
Total number of salient sentences vs. Grade
Generalised Multiple Regression
How strongly each rhetorical sentence type
influences the final grade
Grades
20

21. To what extent is the rhetorical parser XIP accurate and
sufficient for identifying the attributes of good academic
writing within student writing, as judged by the grade,
and by educators?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
RQ1
What is the overlap
between XIP’s output and
how tutors judge quality?
Tutor Highlighted Student WritingXIP Highlighted Student Writing
21

22. In what ways should XIP output be delivered to end users
(students and educators)?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
RQ2
22

23. 1st Year
Pilot study
In what ways should XIP output be delivered to end users
(students and educators)?RQ2
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
23

24. To what extent do educators value the results of XIP’s
analysis of an individual student or cohort’s work when
the primary focus is on assessment?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
What educators
think
RQ3
24

25. 25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
What educators
think
To what extent do educators value the results of XIP’s
analysis of an individual student or cohort’s work when
the primary focus is on assessment?
RQ3
25

26. To what extent do students value the results of XIP’s
analysis as formative feedback on their writing?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
What educators
think
What students think
RQ4
26

27. 1. For my quantitative study, do I have the right approach? Are
there any alternative approaches? How could I make my study
stronger?
2. What qualitative & quantitative methods could I use to evaluate
the quality of the comparison between XIP & marker highlights?
3. Are there any available well-developed methodologies on
assessing visualisations to elicit user reactions?
Feedback?
25/03/2014, Indianapolis, USALAK’14 Doctoral Consortium
27