This document discusses challenges in learning analytics and using visualizations to provide feedback to students. It describes an automated tool called SAFeSEA that provides targeted feedback on student essays. The document emphasizes that feedback should prompt self-reflection and discussion between students and tutors. It also notes that visualizations of feedback can help students understand key points in their writing and identify areas for improvement, but developing effective visual analytics methods remains a challenge.
Introduction to ArtificiaI Intelligence in Higher Education
DCLA14_Denise_Whitelock_slides
1. Challenges for Learning Analytics:
Visualisation for Feedback
Denise Whitelock
Institute of Educational Technology
The Open University
denise.whitelock@open.ac.uk
2. Learning Analytics and Student Feedback
• Assessment drives
learning (Rowntree,
1987)
• Communicating
assessment in a
meaningful way
• What, when and
How?
DMW LAK Workshop March 2014
3. SAFeSEA!
Professor Denise Whitelock!
Professor John Richardson!
!
Professor Stephen Pulman!
!
An automated
tool supporting!
online writing
and
assessment
of essays
providing!
accurate
targeted
feedback!
SAFeSEA: Supportive Automated Feedback for Short Essay
Answers http://www.open.ac.uk/researchprojects/safesea/
DMW LAK Workshop March 2014
4. DMW LAK Workshop March 2014
Grand Challenge representing analysis that
can be readily understood
5. Feedback to prompt Self Reflection
• Analysis must prompt
“Advice for Action”
• Self reflective discourse
with computer feedback
• Visual representation of
feedback can open a
discourse between tutor
and student
• Prompt peer to peer
discourse
DMW LAK Workshop March 2014
9. Computer Feedback
• Non-judgemental/
objective
• Is not the tutor who
marks the final essay
• Feedback can be
accessed many times
• Can lead to contact with
tutor with more
questions
DMW LAK Workshop March 2014
10. Talk Back
• Checking
understanding by ‘talk
back’
• Summaries in
OpenEssayist
• Key words = key ideas
• http://www.open.ac.uk/
researchprojects/
safesea/
DMW LAK Workshop March 2014
11. OpenEssayist: What it tells you
• The system’s focus is to present summaries of
students’ own work in different ways, to encourage
them to reflect constructively on what they have written.
• In other words Open Essayist tells them from its
analysis what are the most important or key points in
their essay. They can then think about whether that
was what they intended to emphasise in their essay.
If not then they can make the appropriate changes.
• A very important aspect of the OpenEssayist system is
that it will not tell students what to write, or how to
rewrite sections of their essay, or even what is correct
or incorrect in their essay.
DMW LAK Workshop March 2014
12. OpenEssayist: A tool for reflection
• Different views with distinct roles presented
• The purpose of this feedback is to encourage you
• to reflect on the draft text you submitted,
• to help you consider how your essay is
organised,
• how the key terms are being used across the
essay
• how they combine to form a cohesive discussion.
DMW LAK Workshop March 2014
13. OpenEssayist: How it gives feedback
• Three aspects of the students’ essays are analysed by
the system:
• the structure of the essay (which paragraphs
constitute the introduction, the conclusion, the
discussion sections, etc.),
• the key words and key phrases of their essay
(which are the most important words and phrases,
the ones that are most representative of the
essay's overall meaning)
• the key sentences of their essay (which are
whole sentences that are most representative of
the essay's overall meaning).
DMW LAK Workshop March 2014
16. Key Sentences
• Do you think OpenEssayist
identified the parts of your
assignment correctly?
• Did it find the introduction
and conclusion and main
text?
• If not, why do you think that
happened?
• Is anything missing from your
essay?
DMW BUiD January 2014
17.
18. Key Words
• Are you surprised
about any of the
words in the top
row?
• Do you think some
important words are
missing?
DMW BUiD January 2014
19.
20.
21. Overview
• All the analyses i.e. keywords and summaries are
shown on the main text
• The results of individual analysis are shown together
• Note where key words and key sentences appear in
your essay
DMW BUiD January 2014
22.
23. Think about the structure of the essay
• Think about whether the key words identified by
OpenEssayist are the ones you need to answer the
question in your assignment. If they appear in the
introduction, are they helping the reader to understand
that this is what you are going to be writing about in the
main text?
• Are the key words in the conclusion similar to those in
the rest of the text but provide a summary of your
answer?
• Can you pick out the three biggest words or phrases?
DMW BUiD January 2014
24.
25.
26.
27. Hints when using the system
• Do you agree with OpenEssayist that these sentences are
a good summary of your assignment?
• Are any important ideas that you want to convey missing
from this summary?
• Are there lots of sentences in the summary that are about
something that you think is unimportant or irrelevant to
your assignment?
• Do you think there are important themes in your
assignment? Are they mentioned in your draft?
• Do you think the introduction section as recognised by
OpenEssayist is about the right size, or has OpenEssayist
go it wrong?
• Do you think you should try to lengthen the introduction?
Or the conclusion?
• Is your draft long enough? Too short?
DMW BUiD January 2014
31. Middle Space
• Develop new
analytic methods
• Computational
• Representational
• Statistical
• Visualisation
DMW LAK Workshop March 2014
32. Visualisations as a Thinking Tool
• Visualising text (Bertin,
1981; Johnson et al,
1993)
• Free text visualisation
still problematic
• Do users need training
with visualisations?
• Convention vs.
Instruction?
DMW LAK Workshop March 2014
33. Short text for illustration of Rainbow
Diagrams
DMW LAK Workshop March 2014
40. Participants for Empirical Study
Occupation
University
Professor of Teaching & Learning
OUC
Senior Lecturer Computer Science
OUC
Administrator x 2
OUC
Professor
OUC
Ph.D student x 6
OUC
Professor of Computer Science
OUC
Vice Rector
OUC
Senior Lecturer x 2
OU UK
Learning & Teaching Officer x 3
OU UK
Project Manager
OU UK
OU student x 2
OU UK
Professor
OU UK
Senior Administrator
OU UK
Course Administrator
OU UK
DMW LAK Workshop March 2014
41. Findings
DMW LAK Workshop March 2014
“One of the clues was talking about the way the
color nodes, that the red ones are at the end. If
they are a good connection with a good
explanation for each paragraph. They should be
connected together but the darker ones should
be in the middle. But you see the color groups
together so for me it automatically pulls my eye
to this page because all the colors are closer
together and more in the middle. So that would
be the student course assignment essay with the
highest mark.”
“You don’t show me
anybody’s text. You
are not revealing
anyone else’s essay.
So students cannot
plagiarize. But you are
saying ‘Look hang on,
this is the way this
essay connects
together.’ That’s what
telling a good story is
about this linking.”
42. Creating teaching and learning dialogues:
towards guided learning supported by
technology
• Learning to judge
• Providing
reassurance
• Providing a variety of
signposted routes to
achieve learning
goals
DMW LAK Workshop March 2014
43. Challenges
• Visualisation
• Social network analysis
• Communication and
collaboration,
• Discourse analytics
• Interpretation training
• What are the necessary
and sufficient conditions
for successful
implementation of
learning analytics?
DMW LAK Workshop March 2014
44. References
• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J. (2013) ‘OpenEssayist:
Extractive Summarisation & Formative Assessment of Free-Text Essays’. Workshop on
Discourse-Centric Learning Analytics, 3rd Conference on Learning Analytics and Knowledge
(LAK 2013), Leuven, Belgium
• Whitelock, D. (2011) Activating Assessment for Learning: are we on the way with Web 2.0? In
M.J.W. Lee & C. McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for
Tertiary Teaching. IGI Global. pp. 319–342.
• Field, D., Pulman, S., Van Labeke, N., Whitelock, D. and Richardson, J.T.E. 2013. Did I really
mean that? Applying automatic summarisation techniques to formative feedback. Proceedings
of the 9th International Conference Recent Advances in Natural Language Processing (Hissar,
Bulgaria, Sep. 2013).
• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. and Richardson, J.T.E. 2013. What is my
essay really saying? Using extractive summarization to motivate reflection and redrafting.
Proceedings of the AIED Workshop on Formative Feedback in Interactive Learning
Environments (Memphis, TN, Jul. 2013).
• Alden, B., Van Labeke, N., Field, D., Pulman, S., Richardson, J.T.E. and Whitelock, D. 2013.
Using student experience to inform the design of an automated feedback system for essay
answers. Proceedings of the 2013 International Computer Assisted Assessment Conference
(Southampton, UK, Jul. 2013).
• Field, D., Richardson, J.T.E., Pulman, S., Van Labeke, N. and Whitelock, D. 2013. Reflections
on characteristics of university students’ essays through experimentation with domain-
independent natural language processing techniques. Proceedings of the 2013 International
Computer Assisted Assessment Conference (Southampton, UK, Jul. 2013).
• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. and Richardson, J.T.E. 2013.
OpenEssayist: extractive summarisation and formative assessment of free-text essays.
Proceedings of the 1st International Workshop on Discourse-Centric Learning Analytics
(Leuven, Belgium, Apr. 2013).
DMW LAK Workshop March 2014