Paper presented (by Denise Whitelock) at the Computer Assisted Assessment Conference, The Netherlands, 1 July

1. Functional, frustrating and full of potential:
Learners’ experiences of a prototype for
automated essay feedback
Bethany Alden Rivers1
, Denise Whitelock1
, John T. E. Richardson1
,
Debora Field2
, and Stephen Pulman2
1
The Institute of Educational Technology, The Open University
2
Department of Computer Science, University of Oxford

2. The SAFeSEA Project...
Supportive Automated Feedback
for Short Essay Answers
OpenEssayist + EssayAnalyser

3. OpenEssayist...
• analyses drafts of students essays.
• offers instantaneous, automatic, individualised
feedback.
• scaffolds learning through skill development.
• promotes self-regulation.

4. ‘Views’
Draft Overview Structured version of the essay, highlighting key words,
phrases and sentences
Key words and
key phrases
Frequency of most used words and phrases
Key sentences Most important sentences in the essay
Key word
dispersion
Distribution of key word and phrases across the essay
Word cloud Cluster of key words in different colours and sizes,
according to their frequency
Word limit Number of words in each section compared to number of
words in the essay
Word count Pie chart showing number of words in each section
Organise the
key words
User can group key words and phrases in this page, which
shows these in different colours in the Draft Overview.

5. e.g. Draft Overview

6. Evaluation Phase 1:
Interviews with Users
• Postgraduate students at
the UK Open University
• Used OpenEssayist
between September 2013
to February 2014
• Completed evaluation
survey (online)
• Contacted for follow-up
interviews

7. The Case Study
Three storylines:
Two students who
took part in the
evaluation phase
+
One student who
opted out
Three questions:
Usefulness?
Potential?
Target user?

8. Maria’s story: “It encourages
you to think but it’s too
bewildering for a novice
learner.”
• Female, mid-50s, professional background in linguistics
• Planning is the most important part of writing.
• She has clear strategies for using feedback.
• Potential for the system to helps students with their
writing style, word choice and essay structure
• Inappropriate tool for a beginner or for someone who
was ‘not so familiar with ICT’
• OpenEssayist could be a catalyst for peer support

9. Robert’s story: “It could be
useful but mainly for students
who are less confident.”
• Lifelong student
• Opted out of evaluation study
• Different approaches to planning and writing
depending on the length of the essay.
• Has a clear process for assimilating feedback
• Beneficial for students with learning disabilities
• Helps a user focus on ‘what bits might be
important’, like ‘structure or synthesis’.

10. Karina’s story: “Worrisome, confusing and
fascinating: this system is for the
younger generation, not for mature
learners.”
• On early retirement after career in technology
• ‘I always found that writing the essay was the hardest thing.’
• Wishes there was more dialogue around essay-writing
• First thoughts were of fascination and intrigue
• Needs an in-built narrative
• Should offer word choices, like a thesaurus
• Could provide exemplars to aide reflection

11. Findings: Usefulness?
Maria was the most positive about
what the system could already do,
in relation to the usefulness of the
‘views’.
Karina brought certain
expectations of what the system
was going to do. When these
hopes were not met, this caused
disappointment.

12. Findings: Target User?
‘It’s nice—but it’s not for me.’
Based on these three ‘stories’ it
would seem that this system is
most suitable for a traditional-
aged university student in Year 2
or 3 of undergraduate study, who
does not feel sure about his or her
skills at essay-writing

13. Other current testing sites
• University of
Hertfordshire
• British University in
Dubai
• The Open
University

14. Sun unavailable but what about
the moon?
• Hints before
writing?
• R.C.T.
• 2 essays
• F(1,41) = 3.23,
p = 0.04 for hints

15. Towards the sun
• Give automatic
feedback
• Use margin
comments as a
tutor

16. Margin comments
16

17. Category systems
• Previous categorisation
schemes (Hyland, 2001;
Perpignan, 2003; Whitelock et al.,
2004; Brown & Glover, 2006;
Nelson & Schunn, 2009)
• Concerned with:
– Feed forward
– Socio-emotive feedback
• SAFeSEA concerned with:
– Decoding a margin
comment
– Expression of opinion
through NL

18. Moving forwards towards a
comment generator
• 3 layer coding
designed by
Debora Field
• Opinion, marker
attitude, linguistic
act
• In trials

19. Next steps
• Further evaluation testing
• RCT on different types of
feedback
• Motivated strategies for
learning questionnaire
(MSLQ) adapted by
Richardson (2006)
• Moving towards ‘advice
for action’

20. Contact
Professor Denise Whitelock
Denise.Whitelock@open.ac.uk
www.open.ac.uk/researchprojects/safesea