Buckingham Shum, S., Ágnes Sándor, Rosalie Goldsmith, Xiaolong Wang, Randall Bass and Mindy McWilliams (2016, In Press). Reflecting on Reflective Writing Analytics: Assessment Challenges and Iterative Evaluation of a Prototype Tool. 6th International Learning Analytics & Knowledge Conference (LAK16). Edinburgh, UK. ACM Press. http://dx.doi.org/10.1145/2883851.2883955 Preprint: http://bit.ly/LAK16paper

1. Reflec%ng
on
Reflec%ve
Wri%ng
Analy%cs
Assessment
Challenges
and
Itera0ve
Evalua0on
of
a
Prototype
Tool
Simon
Buckingham
Shum,
Ágnes
Sándor,
Rosalie
Goldsmith,
Xiaolong
Wang,
Randall
Bass,
Mindy
McWilliams
University
of
Technology
Sydney,
Xerox
Research
Centre
Europe,
Georgetown
University

2. Who
we
are
and
what
we
contribute…
Ágnes
Sándor
Xerox
Research
Centre
Europe
Grenoble
(linguist:
modelling
academic
wri0ng)
Natural
Language
Processing
service
Simon
Buckingham
Shum
&
Shawn
Wang
Connected
Intelligence
Centre
(learning
analy0cs
researcher
and
web
developer)
learning
analy0cs,
user
requirements,
web
applica0on,
user
interface
design
and
evalua0on
Rosalie
Goldsmith
Ins0tute
for
Mul0media
Learning
(researcher
in
reflec0ve
wri0ng
in
engineering)
Reflec0ve
wri0ng
scholarship
and
coding
Randall
Bass,
Mindy
McWilliams
Georgetown
University
(Forma0on
by
Design
Project
–
student
reflec0ve
wri0ng)

3. …IF
you
know
how
to
write
to
be1er
reflect
…AND
IF
you
know
how
to
make
your
thinking
visible
using
language
WHY
FOCUS
ON
REFLECTION?
SOME
OF
THE
GRADUATE
ATTRIBUTES
WE
WANT
TO
BUILD
Authen%c
Performance
Professional
Disposi%on
Emo%onally
Integrated
Reflec%on
Theory
&
Prac%ce
reflec=ve
wri=ng
is
a
window
onto
the
mind
Sense
of
Calling
Epistemological
ShiRs

4. REFLECTIVE
WRITING
EXAMPLE:
NURSING
PRACTICE
Price,
B
and
Harrington,
A
(2013)
Cri=cal
Thinking
and
Wri=ng
for
Nursing
Students.
London:
Sage/Learning
Ma_ers.
(Example
Reflec0ve
Essay
for
Cri0cal
Thinking
and
Wri0ng
for
Nursing
Students)
h_ps://au.sagepub.com/sites/default/files/upm-­‐binaries/54814_Example_reflec0ve_essay.pdf
“To
help
structure
this
essay
I
use
the
framework
described
by
Gibbs
(1988).
Whilst
the
episode
concerned
relates
a
stage
in
Mrs
Drew's
illness
when
she
challenged
her
treatment
protocol,
it
also
includes
some
of
the
memories
and
thoughts
that
this
pa0ent
refers
to
regarding
her
earlier
illness
and
past
ways
of
coping
with
pain.
In
par0cular,
it
prompted
me
to
ques0on
to
what
extent
I
as
a
nurse
should
recommend
analgesia,
drawing
on
what
I
had
been
taught
about
the
effec0ve
control
of
pain.
I
had
learned
that
it
was
be_er
to
control
rather
than
to
chase
pain
(e.g.
Mann
and
Carr,
2006
;
Forbes,
2007).”

5. REFLECTIVE
WRITING
EXAMPLE:
NURSING
PRACTICE
Price,
B
and
Harrington,
A
(2013)
Cri=cal
Thinking
and
Wri=ng
for
Nursing
Students.
London:
Sage/Learning
Ma_ers.
(Example
Reflec0ve
Essay
for
Cri0cal
Thinking
and
Wri0ng
for
Nursing
Students)
h_ps://au.sagepub.com/sites/default/files/upm-­‐binaries/54814_Example_reflec0ve_essay.pdf
Reflec0on
Analy0cs
from
AWA:
Academic
Wri0ng
Analy0cs
web
app
h_ps://utscic.edu.au/tools/awa

6. REFLECTIVE
WRITING
EXAMPLE:
TEACHING
PRACTICE
N.
Ha_on
and
D.
Smith,
(1995)
Reflec0on
in
Teacher
Educa0on:
Towards
Defini0on
and
Implementa0on.
Teacher
and
Teacher
Educa=on,
Vol.
11,
No
1
pp.
33-­‐49
“While
I
had
planned
to
use
mainly
wri_en
text
materials
I
became
aware
very
quickly
that
a
number
of
students
did
not
respond
to
these.
Thinking
about
this
now
there
may
have
been
several
reasons
for
this.
A
number
of
students,
while
reasonably
proficient
in
English,
even
though
they
had
been
NESB
learners,
may
s0ll
have
lacked
some
confidence
in
handling
the
level
of
language
in
the
text.
Alterna0vely,
a
number
of
students
may
have
been
visual
and
tac0le
learners.
In
any
case
I
found
that
I
had
to
employ
more
concrete
ac0vi0es
in
my
teaching.”

7. WHAT
MAKES
REFLECTING
WRITING
DIFFERENT
TO
DEMONSTRATING
YOUR
ACADEMIC
MASTERY
OF
A
TOPIC?
Students
are
encouraged
to
say
what
they
don’t
know
Expressions
of
internal
conflict
and
uncertainty
are
valued
Insights
into
one’s
changing
understanding
of
oneself,
studies
and
purpose
Experience
and
one’s
personal
responses
(including
emo0onal)
to
it
is
central
Oqen
looking
to
demonstrate
growth
of
one’s
professional
disposi%on
and
interpersonal
quali%es

8. STARTING
POINT
—
RUBRICS:
LANGUAGE
+
DISCOURSE
FEATURES
+
TYPE
OF
INFORMATION
Context
of
the
event
that
triggers
the
reflec0on
(why,
when,
where,
who,
how
much,
what)
Specific
informa0on
about
the
event;
ac0ons
,
ideas,
learning
experiences
Changes
in
assump0ons
about
learning,
or
prior
knowledge
and
actual
experience/performance
References
to
the
past:
0me
markers
and
use
of
past
tense
(when
I
started;
before
my
internship);
shiq
between
habitual
past
tense
(I
used
to)
and
the
present
or
the
recent
past
(since
then
I
have
…)

9. REFLECTIVE
WRITING
IS
CHALLENGING
FOR
EDUCATORS
AS
WELL
AS
STUDENTS
Challenging
to
teach
Different
interpreta0ons
of
what
reflec0ve
wri0ng
is/
what
it
looks
like
A
novel
genre
for
many
educators
and
students
Challenging
to
assess
How
do
you
iden0fy
a
deep
or
a
superficial
reflec0on?
How
important
is
content
mastery
vs
depth
of
reflec0on?
Can
some
students
be
advantaged/disadvantaged?

10. FROM
INFORMAL
RUBRICS
TO
RHETORICAL
PATTERNS
IN
AN
ANALYTICS
TOOL
10

11. "I am grateful for the practical component that this internship has offered as I
feel I have achieved a greater knowledge of just simply 'how things work'."
Academic’s Rubric: “Verbs that show awareness or shifts in
perception (I began to understand, I could see, I could visualise, I could
perceive, I became aware, I became, I grew, I realised, I recognised”
FROM
INFORMAL
RUBRIC
TO
RHETORICAL
PATTERN
Reflection label: shift in perception

12. "I am grateful for the practical component that this internship has offered as I
feel I have achieved a greater knowledge of just simply 'how things work'."
FROM
INFORMAL
RUBRIC
TO
RHETORICAL
PATTERN
Academic’s Rubric: “Verbs that show awareness or shifts in
perception (I began to understand, I could see, I could visualise, I could
perceive, I became aware, I became, I grew, I realised, I recognised”
Reflection label: shift in perception
"Understanding how the entire company works grants a holistic overview of
business operations and often allows me to understand the office procedures
and processers."

13. "I am grateful for the practical component that this internship has offered as I
feel I have achieved a greater knowledge of just simply 'how things work'."
FROM
INFORMAL
RUBRIC
TO
RHETORICAL
PATTERN
Academic’s Rubric: “Verbs that show awareness or shifts in
perception (I began to understand, I could see, I could visualise, I could
perceive, I became aware, I became, I grew, I realised, I recognised”
Reflection label: shift in perception
"Understanding how the entire company works grants a holistic overview of
business operations and often allows me to understand the office procedures
and processers."
"I can already see that my attitude towards University has changed."

14. AUTHOR MENTAL
FROM
INFORMAL
RUBRIC
TO
RHETORICAL
PATTERN
Rhetorical Pattern: CHANGE
"I am grateful for the practical component that this internship has offered as I
feel I have achieved a greater knowledge of just simply 'how things work'."
Academic’s Rubric: “Verbs that show awareness or shifts in
perception (I began to understand, I could see, I could visualise, I could
perceive, I became aware, I became, I grew, I realised, I recognised”
Reflection label: shift in perception
"I can already see that my attitude towards University has changed."
"Understanding how the entire company works grants a holistic overview of
business operations and often allows me to understand the office procedures
and processers."

15. "I can already see that my attitude towards University has changed."
IMPLEMENTATION:
XEROX
INCREMENTAL
PARSER
syntactic dependency parsing

16. IMPLEMENTATION:
XEROX
INCREMENTAL
PARSER
lexical database
+
"I can already see that my attitude towards University has changed."
syntactic dependency parsing

17. IMPLEMENTATION:
XEROX
INCREMENTAL
PARSER
lexical database
co-occurrence
rules
+
+
"I can already see that my attitude towards University has changed."
syntactic dependency parsing

18. STUDENT
CONTEXTS
18

19. UTS
ENGINEERING
PRACTICE
STUDENTS
§ 4
year
degree;
two
6-­‐month
internships
§ Graded
reflec0ve
report
on
changes
in
their
professional,
personal
and
technical
awareness.
§ Approx
200
students
per
semester,
40-­‐50
page
reports
—
very
0me-­‐consuming
to
mark.
§ So
very
difficult
for
tutors
to
provide
forma0ve
feedback
on
draqs
§ New
requirement
for
finer-­‐grained
assessment
and
grading
of
reflec0ve
wri0ng
—
wri0ng
analy0cs
could
make
a
significant
impact
19
Course
details
h_p://handbook.uts.edu.au/subjects/48121.html

20. GEORGETOWN
UNIVERSITY
WELL-­‐BEING
PROJECT
20
Reflective Writing Prompt:
Georgetown has a commitment
to educating the whole person.
Some students find that this
course has affected them as a
person, other students find that
it has not. Tell us about your
experience in this course.
GU
Forma0on
by
Design
Project
h_ps://futures.georgetown.edu/forma0on

21. GEORGETOWN
UNIVERSITY
WELL-­‐BEING
PROJECT
21
Example response:
“This class is the first course I have taken at
Georgetown that has made me really feel
like I can make a difference in the world and
that has helped me to grow as a whole
person. It has caused me to reflect on
several harms our world is facing and I left
class most days feeling very passionate and
motivated to make a difference in the world.
This course often helped me to put life in
perspective and helped me realize what
really matters to me as an individual in
society.”

22. EVALUATION
22

23. COMPARISON
OF
HUMAN
AND
MACHINE
ANNOTATION
23
human
machine
This
counts
as
a
“True
Posi0ve”

24. COMPARISON
OF
HUMAN
AND
MACHINE
ANNOTATION
24
human
highligh0ng
automated
highligh0ng

25. 25
METRICS
TO
QUANTIFY
HUMAN-­‐MACHINE
MATCH
v0.1:
30
pieces
of
wri%ng,
with
382
sentences
from
GU
students
Coded
without
knowledge
of
the
parser
Promising
given
the
parser
was
developed
for
engineering
reflec0on
v0.2:
expanded
lexicon,
disambiguated
words,
new
sentence
categories
(See
Sec.
6.1
for
details)

26. 26
METRICS
TO
QUANTIFY
HUMAN-­‐MACHINE
MATCH
v0.3:
312
extracts
and
2366
coded
sentences
accuracy
did
not
decrease
significantly:
promising
since
the
new
evalua0on
corpus
had
almost
ten
0mes
as
many
sentences
as
the
first,
increasing
the
number
of
poten0al
new
words
that
might
not
have
been
recognised.
A
closer
look
at
False
Nega=ves
and
False
Posi=ves
was
needed…

27. A
CLOSER
LOOK
AT
FALSE
NEGATIVES
Missing
words
from
reflec0ve
lexicon
• “Over
the
past
year
I
have
come
to
realize
that
many
of
my
close
friends
seek
support
and
counseling
through
campus
support
and
outside
healthcare
providers.”
Human
coding
for
ac0on
inspired
by
reflec0on
• “When
I
walk
into
a
lecture
hall,
I
look
for
a
familiar
face,
perhaps
one
that
I
met
during
[course
name].”
27

28. A
CLOSER
LOOK
AT
FALSE
NEGATIVES
Human
coding
for
ac0on
inspired
by
reflec0on
“Looking
back
on
the
semester,
I
don’t
think
I
could
have
felt
as
comfortable
and
at
ease
as
I
do
now
without
this
class.
When
I
walk
into
a
lecture
hall,
I
look
for
a
familiar
face,
perhaps
one
that
I
met
during
[course
name].”
28
Poten=al
future
work:
looking
at
what
comes
aQer
reflec=ve
markers.
Classifying
inten=ons
to
act
in
the
future.

29. A
CLOSER
LOOK
AT
FALSE
POSITIVES
29
0
=
surface
self-­‐reflec0on
Human
considera0ons:
• text
length
counts
–
this
was
all
the
student
wrote
• lack
of
detail
or
explana0on
=
lack
of
reflec0on
depth
Machine
annota0on:
• sentence
by
sentence
• surface/deep
reflec0on
dis0nc0on
not
implemented
Future
work:
paragraph-­‐level
analysis

30. FUTURE
WORK:
FROM
VALIDATING
AWA
WITH
EDUCATORS,
TO
SUPPORTING
STUDENT
REFLECTION
From
highligh0ng
to
ac0onable
reports
• How
to
bridge
the
gap
between
the
current
ability
to
highlight
sentences,
and
capability
to
generate
a
meaningful,
ac0onable
report
“Does
this
highligh0ng
mean
it’s
good?”
• Without
gevng
into
automated
grading,
students
(and
educators)
are
keen
to
know
if
there
are
signature
pa_erns
that
are
proxies
for
quality
From
“Algorithmic
Accountability”
to
“Analy0cs
System
Integrity”
• How
is
trust
forged
between
all
the
stakeholders?
Educators,
Analysts,
Students
30